I always thought that the reference was simply to the herb thyme.

thyme: any of the several herbs or small shrubs of the mint family having aromatic leaves, especially a garden herb, whose dried leaves and flowering tops are used as seasoning.

And I beleive it is, I will build my love a bower, as in a shelter of leafy branches, a summerhouse or arbor.