In 1334, a great fire ravaged half of the city and destroyed the home of the Tavel clan, a wealthy family belonging to the local nobility.
The Tavels rebuilt it soon after, giving it the character of both a fort and a small palace.
In the 17th century, the wealthy Calandrini merchant family gave the structure a complete renovation, including the addition of new windows and the enlargement of the original ones.