Have you set the boot priority in your BIOS?

Most machines go to the hard drive for boot up. It is possible (in most systems) to go into the BIOS settings (often by pressing F2 as the machine starts - you may have to have a quick trigger finger for this). Once in the BIOS settings you look for the boot details. If there is a USB option you want to move that above the hard drive in the list. Save changes and then exit and reboot.

Apologies if you've already been there.

An Ubuntu live installation on USB should boot straight up when you start the machine with the stick plugged in.

IMHO it is worth persevering. I'm on my second Ubuntu machine and I would never dream of going back to Windows. My old laptop, an Acer, was Ubuntu-only - I nuked Windows completely. My new Toshiba is a dual-boot Ubuntu/Win7 job. I only (grudgingly) kept Windows 7 at all because I want to use my laptop in the office and some of our bespoke software is Win only.