There has been a castle on this site since 1172 when the Norman Knight, Richard de Clare, called Strongbow, built a wooden tower on this rocky height overlooking the River Nore. The first stone castle was built here twenty years later by Strongbow's son-in-law William Marshall, Earl of Pembroke. Three of this castle's original four towers survive today.

A busy and successful commercial centre grew around the original Norman settlement, and the present Hightown and Irishtown areas of Kilkenny today date from that period.