Infrared light has a different range of wavelengths than visible light, so it is bent a different amountby the lens - thus it will be focused by the lens at a different distance - this means that it will be out of focus compared with normal light.

This effect is more pronounced when the lens is wide open - closing down the aperture to the f32 setting gives you an opening closer to a pinhole - where these effects disappear.