Joe, I visited inmates in 8 UK prisons in total. It wasn't scary or difficult from the point of view of conversation, as the inmate I was assigned to was usually only too glad to chat to someone other than 'cons', with whom he had to be extremely careful. It mainly consisted of listening and also laughing, as each one needed cheering up a bit.
I had several Chapel Visits, where one was allowed into the Prison Chapel (under the supervision of the Chaplain) and the atmosphere was more relaxed. The travelling (I drove all over England in my trusty little Fiesta) was quite arduous. And of course, one had to be searched and go past sniffer dogs etc. But once the inmates came out into the Visit Room, the two hours passed like lightning.
I knew nothing at all about the criminal world, drugs, street violence and so on. So these intense conversations really educated me I can tell you! I was also occasionally asked for advice and did my best to help.
But mostly it was listening and being receptive.