The only time I grew a beard & moustache was in my late teens when I got chickenpox and couldn't shave. The resulting strange mix of colours decided me to remain clean-shaven. However I did have rather large sideboards (sideburns) for years (think Noddy Holder UK 'Catters!). However she-who-must-be-obeyed, who is also my home barber, decided the "superannuated teddy boy" (her description) look had to go, and it went.
I don't think our love life suffered under either regime!