The original question is about rhythm, but the body speaks about tempo. Something is amiss.

Regarding rhythm, it is often an integral part of the piece as we understand it. A tune like "Happy Birthday", normally sung as a march, would be quite confusing if done as a reggae or bossa. "British Grenadiers" is another one that looses meaning if the rhythm is altered, though I would be delighted to hear it as a tango.

Tempo is another story. Generally I find it quite tedious if it wanders for any reason whatsoever. It's just uncomfortable and unpleasant because it conflicts with my internal count. The effect is the same as when people sing songs and leave out the rests. That said, something like Henry Grimes' "Fish Story", which accelerates drastically as part of the form, is most delightful.