Joy, Elizabeth died before James (in 1871) but I guess you're right that they wouldn't have wanted to acknowledge the 'other' family if they did know about them.

However, I assume that there must have been some knowledge among Elizabeth's children after the death of their mother at least. Their father remarries instantly, after all, and Mary Ann's oldest children, who he clearly acknowledges as his own, were in their late teens by the, so they must have realised at this point that their father had been living this 'double life', if they hadn't been aware of it before.

My instinct is that they did know though - after all, he's a well known figure, and children from both 'families' seem to have been involved in the theatre in some capacity. But as you say, there's no evidence.