I had always heard that the castors referred to hats , and we have here a definition from the free dictionary
https://www.thefreedictionary.com/castor

cas·tor 1 (kas't?r)
n.
1. An oily, brown, odorous substance obtained from glands in the groin of the beaver and used as a perfume fixative. Also called castoreum.
2. A hat made of beaver fur or an imitation.
3. A heavy wool fabric used especially for overcoats.