GHS and D'Addario have guides on their websites discussing which string compositions have brighter or more mellow tones. I'd suggest looking at them, and trying what they suggest. I'm generally a D'Addario user but haven't played 12-string in more than 30 years, so can't really comment in any detail.

I wouldn't be inclined to try anything heavier than 12s on 12-string, and would consider down-tuning to D.