Not so sure about US law but in Britain Assault is technically raising a hand or fist with intent or provocation. Battery has to involve physical contact. viz the distinction between assault and battery. If either are unwanted there is a case in law. As I understand it most US States have slightly differing laws anyway but owe a lot to British Law. Except Louisiana which owes more to French thinking.

I think any personal confrontation has warned the guy that he is in immediate danger if he does anything he knows will offend. The loss of the phone is not as high on his priorities.