I'm not sure what's so straightforward about the question, or what's so wrong with the reply. I think his reply makes more basic sense than the question.
Do you really think if Dylan were not a popular and gifted entertainer that he would be a well-regarded "poet"? Do you really think he's widely revered particularly for his mellifluous singing voice? I doubt it. The question seems designed to set him up for ridicule on either hand, and you seem frustrated that he wasn't willing to bite. Song and dance man is very much closer to the truth than either singer or poet, and much more than Lennon calling himself an opera singer--though that sounds like something Lennon might've said to a silly question. Turn left at Greenland.