Joe Williams did indeed record a song called IT'S NOT EASY BEING WHITE. It's on the album "I Just Wanna Sing," 1985. Authorship is credited to "Bromley/Deering, T[ommy]." I transcribed this from a sound sample:

Well, talkin' all this weirdness makes me grin.
There's just so many categories that I don't fit in.
One day on my tombstone they will scrawl….