Well, I've always cut my Buddleia right down in the autumn, and it doesn't look scruffy. I've got three different ones, one is quite pink in colour, another gorgeously dark and the third just ordinary. Are you even against roses? Because I don't think they're native, I couldn't bear not to have some roses in my garden! I love ivy too, it provides a good shelter for overwintering creatures, and a very nice dark backdrop for more colourful things. I'm thinking of Oxford Ragwort and Sycamore, both non-native and rather badly behaved, so I concede you do have a point!