For it to be what you say, "big girl's blouse" would have to refer to two people at once if you think about it (don't think too hard, please😜). I think it might have referred to said garment flapping in the breeze on the washing line, just as a weak-willed and slightly fussy, effeminate chap might be said to be flapping.