The more typical use of the term banyan referred to a man's 'housecoat'; it was a long, lightweight coat worn in a relaxed setting at home. If you didn't wear a waistcoat and frock, you were in a state of 'undress' and would typically be lounging around in your house. My guess would be that a social group calling itself Banyan is extremely informal, hence no acting like your on the job (on the quarter deck).