I don't have any specific recommendations for small hotels (I live here- never had to stay in one!) but there are absolutely loads along Minto Street which is in the Newington area, on the edge of town centre - most of these would probably fit your bill...

As for musical recommendations I strongly suggest visiting the Central Bar (which is at the bottom of Leith Walk) on a Sunday 3-7ishpm, it's an instrumental session but well worth a visit for a listen even if singing is more your thing, though you will be very welcome to join in too. It's also a really nice traditional pub (inside walls are completely tiled which might seem bad from an acoustic point of view but it's not at all) with several good beers on tap and a friendly publican.