I always figured a rounder was an itinerant worker or migrant worker - someone who "made the rounds" looking for seasonal work.

People who are "settled" tend to be suspicious of them, justly or not, hence the bad reputation.