A book on New York mayors at google books suggest it may have been in 1854 that he bought a 4-storey house there.

There's also an interesting song here: lee's songbook called:

DEAD RABBITS' FIGHT WITH THE BOWERY BOYS
WORDS BY SAUGERTIES BARD
To the tune of JORDAN IS A HARD ROAD TO TRAVEL
1857.

It has the last verse:

  If you want all the good songs, 38 Chatham Street,
  There a printer lives, you may rely on;
  Lay in a large stock and supply all your friends,
  And they'll sing them on the other side of Jordan.


Mick