I got made redundant and hoped never to need a day job again I would have loved to have managed on the pension but the mortgage had to be paid and I really wanted to be able to eat. In the end I went back to work and reduced the hours slowly as the mortgage, kids etc dropped out of the equation. I have now emigrated and am, again, hoping never to need a day job again.
If you have children they really have to come first and they are your responsibility. Once they leave home though, and the house is paid for, you can afford the luxury of earning less and then the world is your oyster.
It sounds like you are stuck in the day job for a while yet but look forward to leaving it when the time is right later on.
Good luck