At the conclusion of the war in 1865, a Union organization called the Navy Club of Key West erected this memorial obelisk with the inscription “To the memory of the officers, sailors & soldiers of the Army, Navy & Marine Corps of the United States who lost their lives in their country’s service upon this station from 1861-1865.”

Upon returning from the war, R. Vining Harris, a Confederate doctor and the father of the builder of the Southernmost House on Key West, erected the metal fence around the obelisk to commemorate the soldiers of the Confederacy. It is believed that this is the only such Union and Confederate memorial in the country.