There is probably a downside to downsizing, but I don't know from experience. Bets to get advice from people you know who did it, and find out if there were any drawbacks? It would be interesting to know.
As others have said, I'd like to, but the prospect of actually moving, clearing out all the junk, etc. is what holds me back.
I have the vague idea of gradually reducing the amoutn of junk I have, but it's a question of finding the time to sort through it. It always seems to be bottom of the list of things to do.
Sadly, an elderly relative (much older than me) is in that position and may shortly have it forced on him through poor health :(