I was reading a book about Ireland once, and they said that many cottages were made of rammed earth. It's still dirt, but somehow 'rammed earth' sounds rather strong. I wonder how they did it.
I think they had to coat it with whitewash or similar to keep the rain off.