I don't know if I ever heard that exact phrase, but like Charmion, I would know exactly what was meant. I suppose an editor might suggest changing it for a better form, but in informal conversation it would not be an issue.
(I'd never use it because although I can almost do anything without my glasses, they are always with me when I go out... and an older emergency pair in the car in case of breakage.)