AI could be used to choose which of the valid transformations to apply and in which order.

Current compilers do not guarantee that re-running the optimization passes a second time is idempotent. Every pass just does some "useful" transformations, it's best-effort.