I can't imagaine why you would be "too shy" to invite him in - in the circles I travel in - and I don't think those circles are terribly unusual, and I'm heading into the same age range you're talking about - there's nothing untoward about a woman inviting a man into her place; it certainly isn't perceived as an automatic invitation to ... Surely this man is not so Victorian that he would lose all respect for you if you said, Why don't you come up? (or come in for a bit? or come in for a drink/coffee?, whatever makes sense in your part of the world). And if he IS that Victorian, and if runs around telling everyone about your frightful behaviour, he's going to be taken as an idiot and you're going to be taken as a normal person. Invite him in - he'll either come in or he won't. Take it from there.