On the 1871 census Harry's listed as being married to Fanny (Edwards) though I don't think this ever happened. (I can't find it.)

If they had children together I suppose they could've been left in boarding school or with relations while the couple toured.

After Harry died, Fanny emigrated to the U.S. in 1889 and died 1908 at her home in Yonkers. Her obit in the New York Times says that she left three sisters and two brothers. It doesn't mention any children.