Rita et al.--It's "To the tables down at Mory's" [not Maury's]
Mory's Association is a private club in the vicinity of Yale, serving food and alcohol. Around 1909, when the Whiffenpoof Song was written, it was on Temple Street in New Haven, though it moved many years ago to York Street, where it remains today, between Toad's Place and the Hall of Graduate Studies.
--Charlie Baum