Leenia, you can remove books. This is how you do it on the iPad's Kindle app; I assume it's the same on the Kindle itself: Select a book with a finger-tap, but hold the tip of your finger on the selected icon. A menu then comes up with "Remove from device".
I have the same problem with occasionally mutinous fingers. Give the finger a wee lick and you may find it works better.