The Compound was built in 1841 and the Prison itself completed on 9 August 1841. It was originally known as the Victoria Gaol and is said to be the first western building constructed of durable material in Hong Kong. The Prison still retains the facade of Victorian architecture, having been built mostly of granite and brick. It is also the first prison in Hong Kong and an important part of Hong Kong's history.

During World War II, the prison was occupied by the Japanese and most of the buildings were damaged by bombing. The prison was re-opened for use in 1946 after restoration. When Hong Kong was later declared a port of first asylum for Vietnamese refugees, Victoria Prison became a transit and repatriation centre. It was subsequently developed into an institution with modern management facilities for accommodating discharged inmates of both sexes prior to repatriation or deportation.

Declared monuments on 8 September 1995, the prison together with the adjacent former Central Police Station and the former Central Magistracy forms an impressive group of historical architecture.

Before the establishment of Stanley Prison in 1937, Victoria Prison was the centre of justice and law enforcement of the colony in the early days.

On 12 March 2006, it was officially decommissioned. The Hong Kong Government has not announced a full plan for the use of this very historical landmark.