And also, talking about problematic Billy Joel songs, what about "Only The Good Die Young?"

Come out Virginia, don't let me wait,
You Catholic girls start much too late,
Sooner or later it comes down to fate,
I might as well be the one.

Well, they showed you a statue and told you to pray,
They built you a temple and locked you away,
But they never told you the price you could pay,
For things you might have done.
Only the good die young,
That's what I said,
Only the good die young,
Only the good die young...

The song is basically a guy trying to get this girl to come out and let loose, have fun, because she's repressed. He's trying to get her to acknowledge her sexuality as well, but it can be interpreted as him pressuring her or as an anti-Catholic song.
Billy Joel himself stated that an old crush of his was the inspiration for it and that it isn't anti-Catholic.