CAT5 won't help at all on the phone line side, unless the wires are actually faulty e.g. leakage caused by damp, or corroded connections. If it's that kind of fault, any replacement wire will work, and standard indoor telephone wiring cable would be the obvious choice.

If your modem is a router and is part of an ethernet LAN, and you're talking about the LAN cabling, then the LAN should certainly be CAT5, but I don't think that's what you meant.

It could be your router/modem again. I assume you've tried switching it off and on - forcing it to retrain can sometimes make a bad connection work better.

Anahata