You don't have to be rude or loud. Just state firmly that you will no longer listen to his politics. If he persists, walk away.