I interpret it as a metaphor for manual work in general - taken with the previous line about the office it suggests that his father was unsuited to any type of work, which was why he couldn't hold a job.