I have heard of a study, though I've not actually seen it, that claims glue is an inherently poor sound conductor. So, I guess one could argue that the glue used to secure a set neck degrades the instrument's sound a bit. But, I guess one could say the same for all the other glue joints in a guitar and I don't see anyone proposing that bridges should be bolted on, braces should be attached with screws, etc.