Most importantly, will it record in a non-lossy format such as WAV? Recording in MP3 may result in 'twittering' problems with music.

As Matt said, do you really need it to be portable? If you do, make sure it has a removable memory card (usually SD or CF) rather than internal memory or mini hard drive. Not only does it mean you can swap it when it fills up, but you can probably use an ordinary card reader to take the files from it.

Also wise to remember that equipment using memory cards tend to use FAT, limiting the file size to 4Gb. So an 8Gb card won't give you 8Gb of continuous recording!