There isn't any crime in spouses each having a hobby the other doesn't care for. Do you wear a wedding ring? If so, being there alone on a regular basis shouldn't mean a damn thing to anyone, as it isn't any of their business.