In Domesday records, the village of Costesela appears, with mention of a mill, and of a manor with over 80 square miles (210 km2) of estate across Norfolk, including the only listed hunting park in Norfolk. This formerly belonged to one Earl Guert but was awarded by William the Conqueror to the Count of Brittany, a commander at the Battle of Hastings. Here began a 500-year period in which ownership of the manor passed through a variety of families, regularly being reverted to the Crown and reallocated.

In 1546, Henry VIII granted the manor to Anne of Cleves, although evidence suggests that she never actually occupied Costessey Hall. A surviving early Tudor building sited in what remains of Costessey Park is thought by some to be the hall granted by Henry. In 1555, Queen Mary granted Costessey Manor to Sir Henry Jernegan, heralding a long period of occupancy by the Jernegans, Jerninghams and Stafford Jerninghams. Sir Henry commissioned the building of a new Tudor Hall on Costessey Park, beginning his residency there in 1565.

In 1827, Sir George William Jerningham, 8th Baron Stafford, commissioned large-scale grand and elaborate expansions of Sir Henry's Hall, with many towers and mock-Tudor windows. The project was ongoing over several decades, continued by the 9th Baron Stafford from 1851, and although many features of the new design were realised, completion was ultimately prevented by dwindling funds. The 10th Baron Stafford, who inherited the title in 1884, was certified as a lunatic; during his ownership, the estate was held by the Lunacy Commission. The generous and reclusive Sir FitzOsbert Stafford Jerningham, 11th Baron Stafford, resided at Costessey Hall until his death in 1913, upon which the Hall's contents were auctioned at a high-profile sale.