It sounds like you are using a tape recorder that has automatic volume level adjustment... probably a bad thing for what you are trying to accomplish. When there is silence (such as at the start and end of your recording)the auto feature thinks it has to crank up the recording level on this "quiet music", and all this does is amplify the white noise. Once the music starts the auto feature is happy and adjusts the recording level down, along with the background noise.

If this is what is happening you will be better off finding a tape deck that has a manual setting for the recording level. Then you can adjust that with the VU meters and leave it alone thereafter.