This is a view of lower Manhattan from Castle Williams. You can see how important it was as a strong hold to protect the city during the War of 1812 (The British never entered the port during that conflict, btw). Sadly, war technology in the 1840's rendered the defense useless--weapons could fire over the castle.

Hence it's role as a prison during the Civil War.