A little more to add to the pot.
This from Vance Randolph, Ozark Folksongs, vol. 3, # 547.
"This piece is derived from a popular song of the Civil War period. G. M. Miller (Dramatic Element in the Popular Ballad, 1905, p. 31) observes that the tune is of comparatively recent origin."
Lingard was English. Spaeth (Read 'Em and Weep, 1927, p. 63) remarks that the song was made popular by William Horace Lingard, an English music-hall singer who came to America in 1871.

I believe that Spaeth was wrong and that Lingard first came to America before 1868. Much of Spaeth's material was compiled by "helpers," and there are a number of errors in his work.

Several play party texts have developed from the song.
Description of the game: "The players 'ring up' to form one large circle, made up of alternate boys and girls so that each girl stands at her partner's left. All girls step to the right as the first line is sung, and at the second line each stands still while her partner dances around her. When the third line is sung everybody swings and at the fourth they all promenade to their original positions in the circle."
Randolph, # 547A, from Mrs Marie Wilbur, Pineville, MO, 1930. Her verses:

I'm Captain Jinks of the Horse Marines,
I feed my horse on corn and beans [some sing chile and beans],
Although it's quite beyond the means
Of a captain in the army.

Well Captain Jinks got drunk last night,
So pass your lady to the right,
An' swing her round with all your might,
For that's the style of the army.

Other verses in Randolph are the same as those previously posted.

"Sound Off" states that Lingard introduced the song to the United States. Not an army song, but frequently sung by soldiers. Nothing different in the lyrics in this book.

Not clear when the piece was introduced, although it was copyrighted in the United States in 1868, one of seven character songs sung by Lingard. "On the Beach at Long Branch" was also quite popular.
The 1868 sheet music clearly states "Written, Composed and Sung by William Lingard."
Claims by Pratt and others are also 1868 or later, but apparently Lingard was singing "Captain Jinks" in England before then.

Theatre ephemera and newspapers may contain more information.