I've seen that explanation elsewhere, but it doesn't satisfy me, personally.

From the context, I think Gerswhin is saying that the singer doesn't care about the opinion of people who haven't been anywhere or done anything, insinuating that Missourians are plain and boring, simple people.

At least, that's how I took it.