I have given thought to this question and related ones as well. It is definitely off-putting to me to go into a music store and find a category called "Women's Music" (as distinct from, for example, "Female Vocalists" which seems a useful distinction if one is looking for a particular type of recording) as it seems to me to imply there is this small group of CDs on "women's issues" presumably by, for, and about women, and the vast majority are by and for "everyone else." I beg to differ on that; "everyone" is roughly half female and the unspoken implication that everything else is male is annoying. A similar situation exists in bookstores which have "women's books" sections. Also discouraging is the idea implicit in "girls' rules" for many activities. If it involves fit of clothing or products for which only a woman has use, fine, label that part of the store accordingly. But in most endeavors, the separation can suggest that the women should be separate because they can't hold their own with the (male) mainstream. And in the university bookstore, what in the world are "women's studies"? It makes me think of Woody Allen's comment that he shouldn't have dropped out of college, because "I was in the black studies program; I could have been black by now."
Agreed, there are some songs (mostly contemporary as we seem to be more inclined to write first-person specific songs) that sound downright silly with a gender change, but most of the really traditional ones do not. I have heard Martin Carthy, whose opinion I certainly respect, observe that he gives no weight to the idea of men's or women's songs, only songs worth singing and singers who will sing them.
On the other hand, there are a few very interesting songs that are different songs when people of different genders sing them. Linda Allen's "October Roses" comes to mind. As she wrote it, for some mature women in her life, it is (in her own description) a "woman-power song" but when Danny O'Flaherty recorded it, it is unmistakably a love song, for an older woman. (Grandmother? Teacher? Longtime sweetheart from lover of similar age? Could be any.)
Martin Carthy, Liam Clancy, Gordon Bok, many wonderful male singers sing songs which contain a female viewpoint; I know some great female "shantymen" ("shantywomen" sounds pretty silly; as a Boy Scout leader I also declined to answer to "Scoutmistress") and lullabyes sung by men are wonderful.