According to the Online Etymology Dictionary, the sexual meaning of "bugger" was its original meaning and has been around since about 1550. The nonsexual meanings ("bugger up" and "bugger off") have only been around since the 1920s.

bugger (n.)
"sodomite," 1550s, earlier "heretic" (mid-14c.), from Medieval Latin Bulgarus "a Bulgarian" (see Bulgaria), so called from bigoted notions of the sex lives of Eastern Orthodox Christians or of the sect of heretics that was prominent there 11c. Compare Old French bougre "Bulgarian," also "heretic; sodomite."

Softened secondary sense of "fellow, chap," is in British English "low language" [OED] from mid-19c. Meaning "something unpleasant, a nuisance" is from 1936. Related: Buggerly.

bugger (v.)
"to commit buggery with," 1590s, from bugger (n.). Meaning "ruin, spoil" is from 1923. Related: Buggered; buggering. Bugger off "go away" is from 1922, but the connection is obscure.