It's a kind of prejudice to confuse the works of people with the personal histories of people. But the nice way of saying it is that it's a judgment call. It's the kind of thing that bigots do when they find out that a nice piece of music was written by someone they don't like (or even worse, someone from an ethnic group they're not supposed to like, and they're committing mind control on themselves for cryin' out loud.
But it is human nature like everything else. There is undoubtedly a big split in listeners to those old Bill Cosby comedy albums. I can understand that.
Listening (or not) to Von Karajan recordings or the works of Wagner or Orff, I can understand it being a factor that people may judge in the artist's history versus the art; I personally do listen to 'em all, although I personally feel that Porgy and Bess, Oklahoma, South Pacific and West Side Story are greater works of art than the entire Ring Cycle. That's my judgment. Is it prejudice? Possible.