Jim,
I think the 'foolish/silly' reference is much more apt looking at the action that goes with it.

In the 19th century on broadsides ice-cream is 'Hokey pokey'.

There's a children's rhyme that starts 'Hokey, pokey, penny a lump....