The ruins of Edlingham Castle stand next to Edlingham Burn in a rugged but attractive valley. John de Edlingham built a large two-storey Hall House in a moated enclosure in the mid 13th century. In 1296 the property was taken over by Sir William de Felton who added a palisade inside the moat and a gatehouse on the north side. In about 1340 his son further improved the castle and added the solar tower, a gate tower and stone curtain walls.

After 1420 ownership passed to the Hastings family until 1519 and then to the Swinburnes. These two families did not need the defensive capabilities of a castle, and until its final abandonment in 1650 the property was used as a farm with several of the buildings converted to house livestock.