One point to recall on this as well is that if we do develop an intelligent AI, it may have taken us millions of years, but we did it. An intelligent AI would potentially be able to iterate and self develop much faster than we have been able to.
It probably won’t take long to go from ‘human equivalent’ to being beyond that.
That's true, but I long ago got convinced that this particular worry isn't the most important one (the recursive-self-improvement one).
I'm fairly sure that Yudkowsky and others are also pushing that less, because there is a reasonable chance that it's just not a necessary part of the argument; even without self-improving, it's possible that just something like GPT-8 or something is harmful enough on its own that there are significant risks.
It probably won’t take long to go from ‘human equivalent’ to being beyond that.