Don't be dismissive of "The Panama Limited", it was one of America's premier luxury trains, which is why AMTRAK kept the name (though, regretably, not the level of service) after it took over the operation of passenger trains in 1971. The Panama was all first class, which meant it was an all-sleeper Pullman train, beginning 1911. It was one of the last three all-sleeper trains running in the 60s, tho I am not sure if it was still an all-sleeper at the time of the takeover.
"The City of New Orleans", by contrast, was an all-coach day train, and was, It was a very popular train, in no small part because it made the trip from Chi to NOLA in just a bit shy of 15hrs. If there were sons of Pullman porters on it, they were likely deadheading, because the City of New Orleans never ran sleeping cars(because it ran is full course during daylight hours).
As a point of information, the Pullman Porters would never have been working in Freight Yards, first, because they were on-board services, second, because they worked in passenger service, not freight.
As for the old, black, men: immediately after the Civil War, rail beds were generally built using convict leasing programs(guess what color the convicts tended to be), and for many years, track maintenance crews and and rail shop workers tended to be black. This changed as the much less labor intensive diesel locomotives replaced steam. The City of New Orleans began running in 1947, and was always diesel train--so by the time of the song, the freight yards wouldn't have been filled with any old men, black or not--
And that, to coin a phrase, is the name of that tune...