It's in the nature of GPT - Generative Pre-Training - that it will produce language based on the texts it been trained on. So it can produce 'natural sounding' language - similar to what it has analysed. It's been used for computer code because well-written code tends to re-use constructions that have withstood testing and use. You don't normally want to write novel or unusual computer code - there are no prizes for originality!

Good poetry OTOH should surely avoid common constructions - cliché if you like - and aim for unusual and striking means of expression. So I doubt if ChatGPT will produce interesting poetry. I don't think that necessarily applies to songs, though. Great songwriters often manage to fit natural phrases into their rhyme schemes. I suspect a SongBot could produce passable songs if it had 'learned' some musical rules. It might even be quite original in musical and lexical structure if it was 'told' to break those rules.

Songwiting is not all inspiration. I recently watched an interview with Stephen Sondheim about how he came up with rhymes - he used a thesaurus and a rhyming dictionary. (The interviewer was shocked!)