My understanding of the song has always been this:
- During that time period prostitutes (or whatever you want to call them) for the most part wore some form of green on their arms, to denote their trade perhaps.
- The man who wrote this song had gone to see this prostitute and over the course of time had fallen in love with her.

I hope that this doesn't cloud up the discussion any more.