Sounds like an emerging consensus (as long as we leave Alex's ex-wife out of this). I play both, too, and am happy to do so. In addition to Guy van Duser and Willie Nelson, I would also direct you to Duck Baker and Ralph Towner. There's a lot you can do with a classical guitar that isn't strictly classical.

However, there are some things that only a steel-string will work on, practically speaking. Bluegrass is one; it would be tough to play those Lester Flatt rhythms or Tony Rice leads on nylon strings. I'm nying you can't play bluegrass on nylon strings, 'cause every time someone says "can't" in a musical context, someone else comes along to prove them wrong. But you would be in uncharted waters, and would have to work out your own adaptations of technique to make it work.

So my short answer is the same as everyone else's -- do what you want, when you want to do it. And I agree with Spaw, if you can work out a way to have both, it's worth it. Good luck.