Permission should ALWAYS be sought ! PERIOD!
I strongly agree - IF the location is a private residence, or a paid concert, etc., or a by-invitation function of any sort. If it's in a completely public place, i. e. a busker on the street, a street fair or concert, I see no problem in filming at will.
Peter