It's a vexing problem for gardeners, and there is no simple solution. I sympathize--I am a gardener AND I have 4 cats.
My solution? I have put up angled netting at the top of my 5 and a half foot fenced perimeter. It discourages neighbor cats from entering, though it doesn't stop the most persistent ones--until they get in and realize that getting out is problematic until I open a gate for them. They usually avoid my yard after that. And it keeps my cats from disturbing neighbors.

The ultimate solution is for cat owners to control their pets. But don't hold your breath...

Rich McCarthy