"Going to the broom" as "making love" makes sense. As in going to lie down in the broom together - sounds a perfectly reasonable euphemism. Much akin to "going for a roll in the hay"!
As for the "living over the broom" - maybe that's a reference to the informal marriage ceremony of "jumping over the broom" (which I always thought of as a broomstick - but not sure) - ie being in effect married, but not according to the legal local definition. Maybe.
KRis