I think I covered your argument in my second paragraph re: NFL. A team from a lesser conference might win a few games...even might win it all, but would not win a rematch (were one avail.) with a team from the top of a power conference. For pete's sake there have been a couple of teams in the tournament with overall losing records. Do you think they should be in the tourney just because they got lucky in their conference tournament? I don't.
Call me a purist--even a snob--but the only reason for a field so big is $$$$$$$$$.
If VCU wins it all, it will be a Cinderella story. You and they will have a right to be happy, and deep down I might have a warm feeling for a moment. But I will not be happy about it in a rational way.