There are no precise dates for the foundation of the castle. What is obvious, however, is that it was built to control the north-south passage between Upper and Lower Lotharingia. Popular belief has it that Turpin, Lord of Ardenne, had a fortification constructed on the hill above the valley of the Semois. A true fortified castle was only constructed around 1050 - 1067 under Godfry III. The Dukes of the House of Ardenne owned the castle until 1096, when Godfry (the fifth) sold it. For almost six centuries the castle remained property of the Prince-bishops of Liege until they lost it to the Tour d'Avergnes - La Marck family in 1678. The castle had been taken after a 20 day siege by the troops of Louis XIV of France. Parts of the castle where changed during that time by Vauban, the great French military architect.