There are other things to base a friendship on than humor. Common interests and mutual help, for example. Only you can guess whether you want to be this persons friend. If you don't, I don't think apologies are necessary. Just be silent and absent.