The bottom of the fence needs to be relieved for the drum. The easiest way to do this is to simply hold the fence in place, turn on the V-Drum Sander and let it do its job :-)

Here you can see the amount of relief needed - not that much.

The square isn't really as cruddy as it looks - that's wax with sawdust on the blade and handle - not corrosion...may be I'll clean it up and retake the photo...may be not... ;-)