One effect of oil on the fingerboard is that it (or at least some oils) can work down around the frets, loosening them, and making reseating or replacement of frets difficult or impossible.

Dave Oesterreich