Sunday's better> for music. The performers do mini-concerts at different places throughout the town. You can park your corpus in a place you like and not move.
Saturday usually is better for educational stuff. Last year, I went to an incredible presentation on African Americans in the Age of Sail.

It sure would be nice if the map showed all the festival locations - including the Press Room!