I'd like to know how many tokens in total went into solving this problem. Have they talked about this? It matters whether they got this result in 10 million tokens or 10 billion. Whether it's closer to 1 human working on this for 1 year or 1000 humans for 1 year. The news feels different when the probability of one AI run solving this is 1 in a thousand vs 1 a million. Approximately I'm asking about the amount of money it cost to solve it, which has to include the failed parallel runs.
Good q. You can see that tantalizing graph where compute is displayed on a log axis and pass@1 goes up to like 50/60%. I’d love to know just how much compute is encoded on that axis.
I guess you can get some estimate from the excerpted CoT, but that CoT might be backed by quite a lot of parallel compute.
