The maritime museum in Stromness, the Mid Howe broch and BIG stalled tomb on Rousay, and the Broch at Gurness. The old Man of Hoy and the Dwarfie Stane. The round Church at Orphir. The Distilery!
That should do.
Orkney festival is at the end of May. If you are there then go to one of the concert/celidh out of Stromness for a GREAT experience. Music Food Dancing And great hospitality.