My understanding was simply that “little pitchers” refers to a slang term for children who throw or pitch when playing baseball. All children have “big ears” in that they regularly overhear adult conversations.
Simple!