Don't build the house outright and sell it on spec. Offer the land with a "will build to suit" contract. That'll involve hours of negotiating and getting to know the people before the first nail's driven. If it turns out they suck, you can just be an asshole and scare them off.

And whatever you do, make sure that when you subdivide there's direct access to the new lot from the county road. Do not under any circumstances wind up with a situation where they have to use even part of your private driveway to access their property. That's askin' for trouble.