The tidal island called L'Islet lying in St. Aubin's Bay became the site of the Abbey (later Priory) of Saint Helier. The monastic buildings were finally taken over by the Crown at the Reformation. Surviving buildings were used for military purposes.
Construction of the castle began in 1594, and continued in the first years of the 17th century under the then governor of Jersey, Sir Walter Raleigh, who named it "Fort Isabella Bellissima" (the most beautiful Elizabeth) after Queen Elizabeth the First.
Governors of Jersey moved their official residence from Mont Orgueil to Elizabeth Castle.
Elizabeth Castle was first used in a military context during the English Civil War in the 17th century. Charles II visited the castle in 1646 and 1649, staying in the Governor's House, and was proclaimed King by governor Sir George Carteret despite the abolition of the monarchy in England. In 1651, Parliamentary forces landed in Jersey and bombarded the castle with mortars. The destruction of the mediaeval Abbey church in the heart of the castle complex which had been used as the storehouse for ammunition and provisions forced Carteret to surrender, and Jersey was held by Parliamentarians for nine years.
The current parade ground and surrounding buildings were constructed on the site of the destroyed Abbey church.
The castle was next involved in conflict in the late 18th century, this time it was with the French. French troops under Baron Phillipe de Rullecourt landed in St Helier on 6 January 1781, and the castle garrison was marooned. The governor Moise Corbet was tricked into surrendering to the French, but the castle garrison under Captain Mulcaster refused to surrender. The French were eventually defeated by troops under Major Francis Peirson at the Battle of Jersey. Both Peirson and de Rullecourt were killed during the battle.
The perceived vulnerability of the Island led to the construction of Fort Regent on Le Mont de la Ville, purchased by the British government from the Vingtaine de la Ville overlooking the Town. Fort Regent became the site of the main British garrison.
A project to link the castle to the mainland as part of an ambitious harbour project in the 19th century was abandoned. A breakwater linking L'Islet to the Hermitage Rock on which the Hermitage of Saint Helier is built remains, and is used by anglers.
The British government withdrew the garrison and relinquished the castle to the States of Jersey in 1923 which opened it to the public as a museum.


Pilgrims approaching Elizabeth Castle at low tide on Saint Helier's Day Pilgrimage on 17 July 2005.
Each year, on the Sunday closest to St. Helier's Day, 16 July, a municipal and ecumenical pilgrimage is held to visit the Hermitage. As part of the pilgrimage an open air service is held within the castle. Other cultural events, such as concerts and historical re-enactments are also held from time to time.
During the Second World War the Germans, who occupied the Channel Islands, modernised the castle with guns, bunkers and battlements. After the Liberation, the castle was repaired and was eventually re-opened to the public