The county seat was established in Emory in 1871. This is the fourth structure built here, with many renovations over the years. It is built from "ginger" brick which was manufactured nearby in the former Ginger, Texas.

The adjacent square has seen speeches by many notable Texas lawmakers including former president Lyndon B Johnson.