Randolph-Legman, in "Roll Me in Your Arms," song 62, "The House of the Rising Sun," p. 253, discuss the meaning of "Rising Sun," and speculate that it "probably refers to some gilt or yellow-painted version of the large sculptured or gilt wooden chrysanthemum-style circular decoration often placed as a sun-ray mirror indoors, or over the outside door of fine houses in France, and derivatorily in French New Orleans...." He could have added derivatorily on occasional houses in many cities. It could refer to the semi-circular fan-like window or imitation, also found over doors.

All of this is speculation. I don't know of any literature indicating that it was a sign for brothels.

Brothels of the better type in American cities usually were just well-kept houses. Poor ones were in any kind of low-rent building or flophouse, or a series of cribs.

Another type I have seen in the Gulf Coast is a building so festooned with lights that it does look like a "rising sun." Houston, in the dock area, had two excellent Greek nightclub restaurants some 50 yards apart. One had superb food and little entertainment, the other had great dancing and music. We would eat at the one and then go to the other for drinks. The latter had an area for working ladies, where they could meet the sailors. Halfway between the two was the brothel, as I said, standing out like the sun.
Although illegal in Houston, I know of no instance of the house being troubled by police.