The version I remember from the Oxford University Morris Men (Cat Corlett in particular) singing went:

My (pronounced me) one man, my two men shall mow my meadow
My three men, my four men shall carry my hay away
My four, my three, my two my one, and then I'll have no more
To mow my hay, and to carry it away
On a beautiful midsummer's morn.

The tune and rhythm are quite different from

One man went to mow, went to mow a meadow
One man and his dog Spot, bottle of pop, Old Mother Riley and her cow etc went to mow a meadow.

The latter version only increases by one man per verse.