It depends on what kind of music career you want to get going. If you merely want to work in and around Toronto, then yes you would have a problem moving about, but if you want to tour, I'm afraid the only thing you will be restricted from doing is being at home. I have been on the road for fourteen years now and I have lived loads of places, and found that it really doesn't matter where you live as you rarely see it!
As a session musician you would also have to travel, and the best thing to do would be to build up a CV of sessions you have done, get a website, people will get in touch if you're good. I don't see any barrier to moving about there either.
If you wanted to leave music for a couple of years your profile would go down a little, but if you have a catalogue that is out there and available it will never quite go away. You can keep chasing and chasing the "momentum", wear yourself out: what's the point in that?
Good luck anyway. Canada and America are huge places to work in, so hard to get your head above the crowd. Plenty of tasty road though!