'mistress' always has a ring of 'regularity' and some sort of formal arrangement about it, even if kept mostly quiet....it also kinda implies a single woman, though not necessarily.

I knew of a guy who had done VERY well in business who, very briefly, (for a few months) had himself a younger 'trophy girlfriend' to stroke his ego (among other things) as contrast to a fairly plain wife. It soon became obvious that the GF just liked the game and the side benefits, and the guy decided that it wasn't worth messing up his life totally.....too bad more don't see it that way, and sooner.