Here's some history on the magazine but haven't found the particular song:

From Paper.html:

Captain Billy's Whiz Bang, published by W. H. Fawcett. September 1922 Issue (Vol. IV No 37). The great grandfather of the National Lampoon and other humor magazines...remember the line from "Ya Got Trouble" in The Music Man - "...is he starting to memorize jokes from Capt. Billy's Whiz Bang?" Once considered risqué... this is a mirror of contemporary society in the Roaring Twenties. Showing signs of age (this was not printed on great paper!), but intact and in relatively good condition. REDUCED TO $35.00

And from Language of the Land Project: Literary Journals:

Wilfort H. Fawcett's readers had done their travelling with the military (in the Spanish-American War and World War I) and then returned to a farm or small town afterwards. What Fawcett (1885-1940) thought they would purchase was a different sort of magazine: small, pocket sized, crammed with jokes, short poems, stories, and later, cartoons. The motto for his Capt. Billy's Whiz Bang (1919-c.1942) was "to make it snappy" and that it most certainly was.

Fawcett mimeographed the first issues of his monthly magazine, but after a few months the printing run had risen to 350,000 copies. It was the first magazine to be launched by what later became a very successful publishing company.

The heyday for Capt. Billy's Whiz Bang, named for the sound made by an artillery shell in flight, was in the 1920's. Its novelty, the racy quality of its jokes, and its "explosion of pedigreed bunk," as its cover caption read, found a wide audience. Fawcett and his brothers soon began other magazines which were sold, like Capt Billy's Whiz Bang, on the newsstand rather than by subscription. Among the sixty-three titles originated by Fawcett Publications before World War II were Romantic Stories, True Confessions, Mechanix Illustrated and Amateur Golfer and Spokesman. The latter magazine eventually was published by Virginia Safford.

As Gary Fine pointed out in an unpublished paper (Fine 1983), Capt. Billy's Whiz Bang was the most prominent comic magazine in America during the twenties, occupying a niche similar to that held by Playboy several decades later. Meredith Willson referred to the magazine in his song "Ya Got Trouble" from The Music Man, a Broadway musical comedy. Citing dangers that will harm young boys, the comedy character (Professor Harold Hill) tells the mothers of River City that if they see their sons hide dime novels in the corncrib or memorize jokes from Capt. Billy's Whiz Bang, these are "tell-tale signs of corruption."

By the mid-twenties Fawcett had become a millionaire, able to indulge his interests in travel, big game hunting, and rifle shooting (he served as captain of the United States Olympic shooting team in 1924). To entertain the many celebrities he had met through the years, Fawcett built a resort on Big Pelican Lake east of Pequot Lakes, calling it Breezy Point Lodge.

When Fawcett died in 1940 Cedric Adams devoted his "In This Corner" column to the man who given him his first job as a writer. As Adams noted, Fawcett's successful career as editor, publisher and resort owner had been a colorful one and it had all begun with Capt. Billy's Whiz Bang (Minneapolis Star Journal, February 8, 1940).

And here's a sample of Captain Billy's Gossip from a 1921 edition:

December 1921 CAPT. BILLY'S WHIZ BANG
H. H. Waters, scenario writer, was found clad only in a suit of pajamas, the other morning just outside the Hollywood Hotel. He was unconscious and bleeding profusely. The names of the other picture folk who attended the party have been kept under cover.

There are still a few rumbling in San Francisco regarding Arbuckle and his now famous party. The stories they tell are wonderful to listen to by way of teaching us farmers what strange means certain persons have devised to get a kick out of life. For instance, as my friend Barney Google would say, take this little "roomer": Two of the numerous members of the party decided to entertain their guests--the party was "dragging" as it were. The form of entertainment provided so I am told, was the kind few of us number among our accomplishments. Somehow or other, we have never gotten over that old- fashioned idea that certain ceremonies listed in the regular catalog or otherwise, are not for an audience. Rather, they are for occasions dedicated solely to the gods and ourselves. http://www.public.asu.edu/~bruce/Taylor74.txt
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
The magazine did publish songs such as The Raindrop Song, found here: Doughboy Verse

THE RAINDROPS ON YOUR OLD TIN HAT

The Mist hangs low and quiet on a ragged line of hills,
There's a whispering of wind across the flat,
You'd be feeling kind of lonesome if it wasn't for one thing
The patter of the raindrops on your old tin hat.

An' you just can't help a-figuring--sitting there alone--
About this war and hero stuff and that,
And you wonder if they haven't sort of got things twisted up,
While the rain keeps up its patter on your old tin hat.

When you step off with the outfit to do your little bit,
You're simply doing what you're s'posed to do--
And you don't take time to figure what you gain or lose--
It's the spirit of the game that brings you through.

But back at home she's waiting, writing cheerful little notes,
And every night she offers up a prayer,
And just keeps on a-hoping that her soldier boy is safe--
The Mother of the boy who's over there.

And, fellows, she's the hero of this great, big ugly war,
And her prayer is on the wind across the flat,
And don't you reckon it's her tears, and not the rain,
That's keeping up the patter on your old tin hat?

One of the most famous poems composed by a World War I Doughboy, Raindrops... was written by Lt. Wickersham the night before the St. Mihiel Offensive began. The next day, after being severely wounded by artillery fire, he continued leading his platoon despite a great loss of blood. He eventually died on the battlefield, receiving the Medal of Honor for his leadership, posthumously. His poem first appeared in Captain Billy's Whiz Bang.