It's faulty. Get it swapped!

Then again, it really depends on your 'noise threshold' - go back to the shop and get them to show you another, then compare.

The 'personal' ones are not as quiet in operation as their 'big brothers', but they shouldn't be noticeably noisy. Like CD players, there is an inherent 'zizz' with the mechanism, but it would only be a problem with very low ambient noise.

Plugging a tieclip mic into it ought not to be a problem - unlike the old 'shoebox' cassette recorders with their built in mic that only recorded the motor noise!