In 1763, Julien-Ghislain de Pestre (or Depestre), merchant enriched on the trade with the Indies , orders the construction of the castle to the renowned architect Laurent-Benoît Dewez (1731-1812). Upon completion in 1768, Dewez becomes the official architect of the Austrian Netherlands, while Depestre is made count of Seneffe and Turnhout. De Pestre dies in 1774, leaving the interior decoration and layout of the park to the care of his widow and first son.

The castle change hands many times in the 19th century, eventually to be aquired by Franz Philippson, banker and industrialist, in 1909. The Germans used it as a local headquarters during WWII. It was subsequently occupied by the US Army, then turned into a school between 1952 and 1963. The castle of Seneffe was abandonned for 7 years, until the Belgian State decided to purchase it in 1970. One of Europe's most beautiful collection of silverware was given to the French Community of Belgium in 1978, on condition that it was exhibited in Seneffe.