The house was built on top of the remains of an earlier summer palace of the Bishops of Caithness. It was later used as the occasional home of the Lords of Reay.