Castor oil beans aren't going to kill you if you swallow them whole. The ricin stays inside right through your gut. They were a common ornamental plant where I grew up in NZ and we had one in the garden that produced hundreds of beans every year. I never heard of anyone getting poisoned by it. I never liked it, smelt nasty.

Something like kudzu, Japanese knotweed or giant hogweed would be the way to spread a bit of havoc.