O,
a passage in Sterling Stuckey's book "Slave Cultures":
New York, Oxford University Press, 1987, p.34 suggest that a common practice was for baptism to occur early in the morning.

The River Jordan of course is represented by any body of water, and presumbably early in the morning it would be chilly.

Stuckey makes the case that there is a strong connection between water immersion and African religious practices.

"As a rule, the 'converts was made to ready to be baptize' the next-comin' Firs'Sunday, soon in de mornin' down by the Mill Pond.' The new converts and friends as well as members of their families would make a pilgrimage, Simon Brown tells us, down by the Mill Pond. Herskovits writes, "in ceremony after ceremony witnessed among the Yoruba, the Ashanti, and in Dahomey, one invariable element was a visit to the river or some other body of 'living' water... for the purpose of obtaining the liquid indispensable for the rites.' Thus, Yorubas, Ashantis, and Dahomeans would easily have identified with the pilgrimage to the Mill Pond for a ceremony that was heavily influenced by Bakongo religious ritual".

end of quote.

In addition to this practical explanation, it's also possible that the imagery of "chilly waters" could be used along with references to the "cold" hard hearted world to contrast with the warthm and caring of God's love.