As for lycra-wearing cyclists, any one who doesn't avail themselves of all the advantages of comfort and aerodynamics that high tech duds offer is going to have a hard time keeping up with a pack of determined cyclists on lightweight road bikes who are outfitted in that manner, to say nothing of pulling off a 300+ mile ride unless you're just poking along and camping when you're tired. Done that.
Nowadays I wear wicking gym shorts and lightweight long sleeve uv-blocking fishing shirts and ride a comfort bike with shocks at the front fork and seat which allow me to ride on sidewalks when necessary without rattling out my fillings. It's an XL frame to accommodate my tall stature and with it's upright handle bars my CoG is so high that the handling is relatively poor and probably was the main reason for my recent crash.