The Hermitage, a National Historic Landmark, is located on 4.9 acres of lawn shaded by centuries-old trees. The site is listed on the State and National Registers of Historic Places. The fourteen-room Gothic Revival home was built in 1847-48 from designs by William H. Ranlett for Elijah Rosencrantz Jr. The house incorporates portions of a historic 18th-century house.

The house was visited during the Revolutionary War by Washington.