The nerves which control the muscles are the problem and the solution. Simply put the nerve is made up of individual neurons (nerve cells) which dont quite touch: when your brain sends a signal to the muscles it has to jump millions of tiny gaps - slow.
However, every time you use a nerve it starts to grow 'tentacles' at the end of the neuron (dendrites). This reduces the gap, and control becomes easier.
Growth continues following the initial stimulation for some time. The nerve pathway becomes more rapid even when you aren't using it.
Young people grow quicker, so learn new physical skills more quickly. Older people more slowly.
This explains why you find that following a 'layoff' you have developed the skills you were seeing: you're nerves have grown. The good news is, they never 'ungrow' so physical skills are learned for ever.
This is a crude explanation: there are more detailed ones available