Yes, definitely Rin Tin Tin, my favorite show when I was five. I tried to remember the song and sing it at the time, but it's gone now. It was pretty hokey, with a chorus of cavalrymen and Rusty just as you described.

By the way, I've always wondered why a ten-year-old kid and his dog were being raised by a cavalry unit.