Here's a recipe for it from 1894:

From:
Recipes Tried and True
Compiled by the Ladies' Aid Society of the First Presbyterian Church, Marion, Ohio, 1894

OYSTER STEW. MRS. J. ED. THOMAS.

Wash one quart oysters and place on the fire. When they boil, add one quart of boiling milk, and season with salt, pepper, and plenty of butter. Serve with crackers or toast.

Maybe it all started with the Feast of the Seven Fishes from Italy?:-)