Jeri,
It seems you have some misconception of what the expression 'first of all' really means. You are assuming that 'first of all' is an ordinal statement which it isn't. It is a 'cardinal' statement, it makes the prime position special and exclusive.
If you wan't to talk ordinally there are adequate words for doing so, 'firstly', 'secondly' and so on.

Bert.