What is the y-axis in this claim about logarithmic improvements? Any exponential curve can be trivially turned into a logarithmic and vice-versa, and the y-axis redefined as "progress", with no loss of accuracy.
One example that always bugs me is when people point to "exponential" or "sigmoidal" progress on benchmarks. Benchmarks are artificial constructions (saturation at 100% by definition) and benchmark scores should not be mapped to these words when talking about overall progress.
Example - progress on ARC-AGI-3 at the moment is exponential, steeper than 2^t and e^t. Does that mean AI is progressing "exponentially" in the colloquial sense? No, it doesn't support or refute that colloquialism.
Likewise with MMLU saturation. We can't go above 100% by construction. Therefore we have a "sigmoid". Gah.
One example that always bugs me is when people point to "exponential" or "sigmoidal" progress on benchmarks. Benchmarks are artificial constructions (saturation at 100% by definition) and benchmark scores should not be mapped to these words when talking about overall progress.
Example - progress on ARC-AGI-3 at the moment is exponential, steeper than 2^t and e^t. Does that mean AI is progressing "exponentially" in the colloquial sense? No, it doesn't support or refute that colloquialism.
Likewise with MMLU saturation. We can't go above 100% by construction. Therefore we have a "sigmoid". Gah.
The colloquialism is not helpful to begin with.