In 1814 Thomas Metcalfe, later the 10th governor of Kentucky, built his home on what is now scenic Highway 68 (Maysville Road). He also built the tavern across the highway, and the stone barn where stagecoach horses were stabled for the night. Metcalfe, a stone mason, soldier and statesman, also built other notable buildings in Kentucky and nearby areas, providing stone work for the old state capital in Frankfort. He lived from 1780 to 1855 and is buried on the property.
Dr. Eslie Asbury of Cincinnati bought Forest Retreat in 1933 when the house and grounds had fallen into disrepair. He carefully restored the property, and he and later his son, Dr. Taylor Asbury, used it as a working farm. One of their horses, Determine, won the Kentucky Derby in 1954.
In 2001 the current owners, Dr. and Mrs. Philip Tibbs of Lexington, bought the property and further restored it.