Not fired the RPG but I imagine it is similar to the 66mm rocket launcher and the old 84mm Carl Gustav.
These do not produce a recoil for the reason you give, but there is a concussion. The 84 had a telescopic sight, which I never knew to leave a mark on anyone.