I do them in both directions depending on the song and the note characteristics. For instance if I was doing a double (or a triple) pull off, I'd pull away and upward. (Requires a well calloused fingertip.) For other things, I might use a downward and away motion. Executing good pull offs is a real finesse thing to get them clean sounding, and requires a fair amount of practise. Not something you're going to get the hang of in a day.