Time and physics are just things you experience. They're parts of the universe, they're within the universe. The mathematical universe itself can quite easily be infinite in both space and time, you just can't experience all of it.