Yellow ribbons have for a long time been used to signify that someone absent was being waited for. A yellow handkerchief worn by an individual would seem to be a suitable substitute, and "in remembrance of me" could well mean nothing more than this. There's a recent film called "The Yellow Handkerchief" that uses it in this way, although I accept that this doesn't prove it's a long-standing custom.

On the other hand I can find nothing to prove or even suggest that it signified that the wearer had venereal disease, and it seems, as I indicated in my flippant comment above, unlikely that anyone thus afflicted would want to advertise their condition to the world in this way. But I'm keeping an open mind on this, and interested to learn more if anyone has evidence.

Marje