In addition to the concerns above, is the concern that the modified gene may be passed to a wild relative of the modified crop.
Here's a scenario.
Modify a strain of oats so that it is resistant to a particular pesticide.
We now have "Crazy-Oats" (sold to you by CrazyEddie Seeds Inc.) & "Crazy-Zap" (sold by CrazyEddie Chemicals Inc.)

You plant the seed, spray the pesticide, Abracadabra, all the weeds die, and the oats thrive! Whoopee!

Happy Farmer, Happy CrazyEddie!

Unfortunately, some of the pollen fertilises a weed (Called "wild oat") in a neighbouring meadow.
After a few years, the resistant "wild oat" is rampant, because you are killing all the other weeds.

A similar situation can occur with any "advantage" you give your crop; related wild plants obtain the advantage by hybridization, and natural selection does the rest.

This is the reasoning behind the UK decision to allow GM Maize (but not other crops) to be grown. Maize is a new-world species, and has no close relatives in the wild in the UK.