I have seen that phrase "the rest of us here" in several places.
It carries the weight of exclusivity at times and other times it has the quality of inclusion.

As a muse:
I wonder what kind of moral authority the rest of us have.

Are "we" the folk song army? Do all the rest of us care?
Are we against poverty war and injustice, unlike the rest of those squares?

When it comes to "the rest of us" try as we may, some of us will forever be more equal than others in various tasks, talents and tendencies.

The rest of us, is a comforting and almost romantic notion to me. I have seen individual evidence that some individuals have great moral intentions and actions, but 'the rest of us here' , well I am not aware if "we" have any moral authority whatsoever.

What is good about many of the 'rest of us here' are those who were already life long friends or those who made fast and lasting friendships through performances, pubs or gatherings.
Are we all part of the rest of us?
Just ask anyone here who has hung out with me ;-)