Duckboots, I should have mentioned this before... when the insect appears on your plant, cut off some of the plant tip that shows the bud and leaves as well as the worm. Put it in a plastic zip bag and take it to the County Extension Office (should be in your local courthouse, or thereabouts). The entymologist for your county can then identify the problem. Having a bit of the plant helps show what your are dealing with. It is your tax dollars at work, so you should take advantage of their expertise.
alice