whom is the object of ask. You ask whom. I still think there is a hidden, assumed, object there. Whom you ask is a phrase that goes with the hidden object. Depending on the cat whom you ask. Depending on the sailor whom you ask. It is like a contraction where the word is implied but not spoken.

But you could have one of those intransitive verbs and then who would be ok. He is the person who wrote the book. mg