In what way were they working outside their remit?

You quote their remit in general, but include that it is to only consider matters referred to it by the House of Commons as a whole.
The committee was due to decide on the historical actions and speech of Boris. On 21 April 2022, the House of Commons, without division, referred to the Committee of Privileges a matter concerning the conduct of the Rt Hon Boris Johnson MP, the Member for Uxbridge and South Ruislip, that is, whether he had misled the House and whether that conduct amounted to a contempt. From Here. Their latest actions appear to be outside this remit.