The Victoria Inner Harbour area is much changed. It used to extend inward, into the area now filled in and occupied by the Empress Hotel and grounds. The old buildings along Wharf Street are gone, some cast iron columns from San Francisco in the 1100 block of Wharf Street about al that is left.
http://www.victoria.ca/visitors/pdfs/wlkngt_flsrsh3.pdf

The Shanty Camp Workshop Sept. 19-20, I guess you will miss- Maritime Museum of B. C., 28 Bastion Square. I don't know anything about their holdings. They have a room devoted to whaling.

Someone at the Museum may be able to help you find the house(s) where Fox Smith lived- I don't know.