Time isn't physical, it's mathematical. So it is asymptotic. We can approach the end of time but we will never reach it. Only an infinite being could experience the end of time and we are finite. So, you are right, if the universe ended, how would we know? It could've happened already.