I've done this trip (or at least from Vienna to Istanbul) in both directions in 2008. Essesntially, the further east you get the more "interesting" the trains become.
Things can get grim after Bucharest as the sleeping cars are pretty old (don't even think of the passenger cars or even the couchettes) & the trains NEVER run on time!
This isn't a problem going to Istanbul as long as you don't have anything planned for the day of your arrival.
It can be on the way back as there are only two night trains from Bucharest to Budapest. The earlier connection (for the much better train - the Ister) at about 7:00 you won't make and sometimes the later one (about 10:00) you may not make either or it may be booked out. My advice is to book a hotel for that night and next day explore a bit before catching the Ister that night.
You will need to stock up on food & drink for the Istanbul - Bucharest leg. There isn't a restaurant or buffet car and the catering at stations is a bit hit & miss. You do NOT want to drink any tap water on the train. Also, the toilet situation can be diabolical. The conductor I talked to explained that they usually run out of flushing water after about 6 hours!!!
And just to complete the aggravation, the Border stop is non trivial. Expect to be off the train for nearly an hour which can be even longer if the staff aren't there when the train arrives.
But it is an adventure as long as you:
1/   Get a sleeping compartment to yourself
2/   Bring enough food, water & booze AND reading material
3/   Approach it realistically rather than expecting West Europe comfort
4/   Don't get fussed at delays & language difficulties

And finally, you should go to Seat61.com and read the section on Turkey. This is the best train travel website I know and has most of what you'll need including prices, how to book etc etc

Good luck

Dave