"When something goes wrong, the guy at the top is ultimately responsible!.."

That is a vaguely true technical point...it is NOT an excuse to blame every screw-up directly ON the "guy at the top". YOU would make the same point if someone YOU approved of were castigated for some stupid interpretation of plan by a zealous underling!

When, like Nixon & Watergate, it is proven that he knew and approved the basic events, it is then time to remove him.
You may 'think' Holder should have followed every action of every field office on a daily basis, but it just don't work that way.