People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps delivering more and more impressive results.
The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn, but I’ve noticed Fable to be quite a big step up there. How about politics? Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation? How about experiment driven domains like biology?
We will get much better at manipulation and better at people “writing” things to justify their own feelings.
What’s new about LLMs is that you can scalably manipulate people individually. It used to be that you could either have scale (speeches, tweets, interviews, website, etc.) or individual engagement (replying to mail/tweets/town hall questions.)
Now you can pull the history and preferences of an individual, then shape a message—in real time—to them, specifically. You can have conversations on social media with a single person and shape your message specifically to them.
Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
It does matter though. If you want to murder someone by hitting them with a plushie, you're not going to get charged with attempted murder because it's not possible that that would ever work. There must be justification that the choice will have the intended effect.
We should not be gung-ho to give the government more power to regulate speech.
It isn't so clear there is a reason for the crime of 'attempt', or what or how good the reason is. The Star Chamber, which cooked it up, is universally condemned as oppressive. You might say they anticipated science fiction tyranny, when they invented the world's first thought-crime: they were able to make a crime of 'conspiracy' because they imagined that the speech of the conspirators was an 'outward act'; inevitably its glorious future was e.g. to jail the left wingers in the McCarthy period.
The point of view that says 'there is a reason we punish attempts' has difficulty explaining why we punish 'success' _even more_. There is a sort of bad concience about it. The paradoxes are discussed in a characteristically brilliant and twisted work 'The Punishment that Leaves Something to Chance' by David Lewis, one of the greatest philosophers of the 20th c. It nominally defends the law of attempt but can as well be read as a catastophically destructive parody of it. https://andrewmbailey.com/dkl/Punishment_Chance.pdf
It's interesting to claim that it would be difficult to explain why we punish successful crime more. A successful murder creates more suffering (victim's friends and family), and ostensibly the loss of a productive member of society. These all seem like fairly common justifications for punishing murderers under the banner of retributive justice.
It’s unfortunate that you’re being downvoted because this comment is true. And it just goes to show how far we’ve sunk that people have forgotten the problems of ad tech in the age of LLMs.
You’re making a weird number of assumptions about people’s voting and extrapolating to assertions about Society.
Ads are not really the same. They can’t be as tightly targeted to what resonates with someone. Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
That’s where this is headed and people are not ready. I don’t think we could prepare them, anyway.
> Ads are not really the same. They can’t be as tightly targeted to what resonates with someone.
The EU referendum in the UK proved your point false.
> Programmatic ads are certainly much better and closer than, say, television advertising, but users can’t _engage_ with them. Like actually chat with them.
When people talk about “ad tech”, they’re not talking about TV ;)
And yes, people can and do engage with them. That’s how ads on social media works.
During the EU referendum, people were even resharing ads on Facebook without even realising they were ads.
> My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
I think that statement may itself highlight how prevalent manipulation is.
I fully anticipate all groups to continue maximal manipulation they can. One thing with LLMs is that it'll be a far less unified view, so a "divide and conquer" strategy is what I anticipate.
In US politics, the right is far, far better at winning elections than the left. This isn’t about personal preference. It’s objective political science.
Look at the most contentious issues in the US: abortion, climate change, taxing the wealthy, gun control, Affordable Healthcare Act.
The Democratic Party platform is aligned with national polling on every one. Every one of those issues has >60% support with voters and the Republican Party has blocked them all.
> Democratic Party platform is aligned with national polling on every one.
It depends on the pollster and where you're polling. I guarantee you rural Tennessee will not agree with downtown Washington DC on any of these issues. In contrast, rural California will likely agree with rural Tennessee. It's not as cut and dry as a homogeneous national poll of 2500 people. Every state, city, county are different. That's why there are federal, state, city, and county governments.
For instance, Abortions are legal nationally. States can individually decide how, or if, they wish to restrict it. This is as the constitution intends under the 10th amendment:
> powers not delegated to the federal government nor prohibited to the states are reserved to the states or the people.
This allows for democracy to take place at the local level, rather than having particular regions thousands of miles away from each other ultimately oppress the other.
To the point on LLMs, I think it's abundantly clear they will be used to propagandize and similar to social media will lock people in a bubble without alternative opinions. It'll be the worst of both worlds, the question is who's the puppet master. At some point soon, I imagine it'll be the AI.
I don't know what distance has to do with it. I would probably be voted out of existence by my town, if the same constitution hadn't convinced them that they have no right. Independence Hall is remote in space and the events still more in time.
The pretense of Dobbs is that laws against abortion are like laws prohibiting medical marijuana or setting speed limits. This contradicts the purpose of laws against abortion and the actual language of most of them. It was obviously a desperate decision, irrespective of the truth about abortion
So we need to "prohibit to the states" the power to enforce medical procedures (or the prohibition of a medical procedure) on individuals.
Which amendment protects someone's bodily autonomy or medical treatment from interference by the states?
If the Federal Congress passed a law providing that abortions are legal on demand up to the 22 week of gestation, after which it would require the corroboration of two doctors, would that stand, or would it be struck down by SCOTUS?
Same for legalization of marijuana, and many other "social" issues.
None of those are single issues, and the political fault lines are in the sub-issues. For example, while most Americans support "gun control," only 20% of Americans support a ban on handguns: https://news.gallup.com/poll/1645/guns.aspx. That figure has been trending steadily downward--from 38% in 1999 to 20% in 2024. That makes it much easier for Republicans to hold the line on that issue: portray all gun control efforts as a step towards confiscating handguns. That's hard for Democrats to defend against because most of the candidates, staffers, etc., who actually run the party probably are in that 20% who wants to ban handguns. That's simply logical, because handguns are used in the overwhelming majority of homicides committed with guns. It makes very little sense to have "gun control" without banning handguns.
The same thing for "taxing the wealthy." 59% of Americans think their own taxes are too high: https://news.gallup.com/poll/707951/americans-tax-views-rema.... And the difference isn't as big between parties as you think--49% of Democrats think their taxes are too high. So Democrats are in a position where they have to advocate for raising taxes on "the wealthy," without scaring any of their own voters into thinking that includes them.
Even a blind squirrel could find these nuts. Apart from that, republicans are ridiculously inept. For example, 2024 was the first time they spent real money trying to go after immigrant voters and minorities, and they made huge gains. But the on-the-ground operation disappeared after the election. Meanwhile, democrats are in these minority neighborhoods 365 days a year pushing their message.
> Apart from that, republicans are ridiculously inept.
They're not. They're really not. They're incredibly capable and are currently executing long term plans successfully, one after another. This goes all the way back to Regan and the disenfranchisement of education. They know exactly what they're doing to erode democracy.
Calling them inept is dangerously stupid at best, and at worst is just right wing propaganda to try and lull people into a false comfort and not act.
There's no "long term plans." It just seems that way because republicans are relatively ideologically homogenous, so it seems like there's much more top-down and long-term planning than there is.
Democrats, by contrast, have much more permanent political infrastructure. At the top law schools, for example, there's probably 10x as many liberal organizations as conservative ones. Democrats enjoy politics, so they are always out there organizing and engaging. Meanwhile, Republicans dislike the institutions through which politics is done, and so they gear up once every four years and then go back to their day jobs. Among the sort of more educated people who actually run the parties and serve as line staffers, republicans also self-select into private sector jobs, and out of public sector or political work.
Most importantly, putatively neutral institutions are de-facto aligned with democrats (because the professional class that runs these institutions is overwhelmingly democrats). Conservatives might have Fed Soc, but liberals own the ABA. Virtually every legal organization that isn’t expressly conservative or highly niche is de facto liberal and participates in support of democratic policies.
That presupposes that the left in the US wants to do the right thing. Something like government run grocery stores is not clearly correct and there is very little evidence supporting that it will work well yet it is a very popular leftist policy in New York.
Based on zero evidence. It’s very easy for a government to step in as a participant, ruin the profitability of a sector in an area, and offer a worse service.
They have no profit requirement or even revenue neutral requirement. So they can just operate poorly at a loss and still wreck other businesses because people will put up with breadlines to get bread for ultra cheap.
The general thing to watch out for with all of these “surely it can’t be evil to do nice thing X” is suicidal empathy. It can seem correct to your gut on the surface while it’s extremely destructive in the long term despite participants wanting to destroy something as an explicit goal.
Bad economic policy is subtly "evil", by way of allocating finite resources inefficiently. Usually this is unintentionally done by not appropriately taking second, third, ..., nth order effects into account.
They destroy grocery stores that offer variety in every neighborhood they operate in. They offer worse service and people put up with it because the rest of New York is subsidizing them through taxes.
The margins for groceries are objectively thin. The only way to provide food at lower prices is to provide a worse good or service, eg. less variety, less quality, less availability, etc. You will see all of these outcomes in NYC, if anyone even accepts the bids to begin with.
It's harder to convince people to all agree on the same, different from the norm, thing.
If you've got 5 people in a car and you play ABBA in every road trip, then one day you suggest to change, the problem is not in being okay with "something else", but on agreeing what that other thing should be, set against the already known thing.
That's why there's so much infighting in the left.
Biology would greatly benefit. We barely understand transcription and protein structure. And the straightforward systems that we know like insulin have complex post translational modifications. So while we have a map of the partial proteonome, we have barely scratched the surface on networks regulation and interactions.
> And the straightforward systems that we know like insulin have complex post translational modifications.
insulin is not straightforward, the way the insulin molecule interacts with its receptor is nuts. on the other hand its post translational modifications are simple and dont have anything particularly surprising (no glycoslation, disulfide bonds where you would expect, nothing special kex2 cuts, arent really defective in disease states even)
I think the question is less about where it's most beneficial but which knowledge structure lends itself to LLMs most. Since biology's "language" is way more complex and irregular than maths or natural language, it isn't particularly accessible.
> Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
Side note: it's manipulation either ways because you chose what to talk about, with a goal in mind.
Actually, no. "Manipulation" is a negatively loaded word, and you wouldn't use that word if f.ex. someone helpfully & truthfully helps others see they've misunderstood sth.
I disagree. Whether something is manipulation depends on whether you are trying to change someone’s opinion or behavior, not on whether the manipulator has “good” or “bad” intentions, since those judgments are not objectively universal.
I believe that the decentralization of manipulation (taken in a very broad sense as an effort to modify people's views) spells trouble for democracies. The mass media of old, with all its flaws, created a shared pool of information managed by well-educated people, some of whom understood that they had the power to keep democracy running (keeping populists away from the "manipulation machine," curbing blatant manipulation attempts, and so on).
Decentralized manipulation, by contrast, just runs amok creating echochambers and polarization.
The "left" as in the party, sure, not that much of an authoritarian leaning.
The "left" as in the pervasive group that crawled out of Tumblr, took hold of Twitter back in the day, and has a stronghold on Reddit now? Those do care about what you can say, think, watch and read, and the more they can control, the better. The US right can only dream to have half as much control as the left has had in the last three decades.
The parent comment wasn’t making that comparison. In fact, quite the opposite. The communism comment emphasized personal experience with a boot in one’s face, not that the boot was communist.
What even is the left though in the US? I haven’t watched any TV in decades, and the online content is entirely algo driven, so I don’t know what is happening.
its a bit like how big companies will pullin a CEO who;s job is to do all the unpopular stuff and then summarily dismiss him after (while keeping the policies he created in tact).
ideally we should be throwing our support behind democratic socialists like zorhan mamdani but old school moderate democrats are just as bad as trump. they are juts happy letting trump do the dirty work. At the end of the day they are both working for a certain foreign government and not the interests of the american people.
> If you think that the American Left is anything like communism
no, you completely missed my point. why would i talk about communism if in the same comment i'm saying that the left/right divide is a fake distraction... a red herring?
what i said is that what matters is the authoritarianism/freedom scale.
i just mentioned my past to emphasize that ideologies like communism are not to be identified by announcing themselves as 'communism'... but by recognizing their actual shape and actions. and i have a little first hand experience in that.
What's "scaremongering" to you is "life" to GP. Besides missing their point, you're also saying "you held socialism wrong, we can make it work, if only if it weren't for this pesky ... reality"... Sorry, you missed their comment, and I can't take yours seriously :)
This reminds me a lot of the proof by construction for the 4-color theorem. It was only enabled by the advancement of computers and dissatisfied many of the computer scientists and mathematicians since it was a "brute force" approach.
I wonder if AI will end up being similar. Certain theorems get proven by AI but others do not. We haven't reached the limits of this yet and I haven't found a good argument for where those limits will be (I do doubt that there are no limits).
Isn’t the point more “it’s easy to fall into the trap to believe that predicting when the sigmoid is going to bend is possible and the right heuristic is to instead extrapolate locally”?
That aside, I’d question whether applying the Lindy effect in particular to something that’s not really a life expectancy but more a growth rate is credible… or perhaps a bit circular since it “assumes away” the ceiling.
Predicting when the sigmoid bends is difficult and predicting how long until it unbends is equally difficult.
The simpler assumption is that over enough time, the S functions stack together for long enough that working backwards from exponential is a better predictor of reality.
These stacked S curves have continually been true with most technology.
The author of that post is a prominent Bay Area "rationalist," who have had a quasi-theistic relationship with the concept of all-powerful AIs for a couple decades now.
All of the point of that article is that most people that think something is a sigmoid think it'll bend just as they are publishing their analysis. And the article says, don't do that, assume it'll be related to how long we've been on the "goes up" part.
I dunno. They've been saying this for decades, including in a very well read Harry Potter fanfic, and I'd always dismissed them as kooks, but maybe they're right in the end.
Saying the same thing for decades, when things progress over those decades along the general trend line that you were worried it would, is decent evidence of a prescient prediction. Climate scientists also seemed pretty kooky in the 80s when they sketched out their trendlines, but now their concerns are all over the headlines as reality caught up, and the same is becoming true of the AI safety people.
Everything has limits, of course. But if you've not yet seen a deceleration, it's reasonable to expect that at the least you're in the middle of the sigmoid, not the top.
We have though. The velocity remains high but the acceleration is decreasing.
The improvement between gpt 2-4 where staggering. to 5, 5.6? Much less so.
The results improve of course, but the difference is no longer mind blowing.
Most of the improvement now has come from agentic harnessing, which is unrelated to the acceleration of the model but the tooling around the use of the model.
So we are seeing a deceleration and the speed is still high due to high prev acceleration but its not growing at the same rate as before and the edges are starting to show themselves
I literally don’t understand, and have a deep distrust of people who say we haven’t seen deceleration.
The diminishing returns are colossal. GPT 4 is more than 3 years old now, and while Sol is definitely better, it’s better at a tremendous cost. And 3 years of development. And it still fails in nearly all the same places as 4.0.
I’m not some “zero AI” person, I don’t think this industry will collapse into nothing and we will go back to a no LLM world.
But to look at this industry, which basically began in November 2022, almost 4 years and a trillion+ dollars later and say “oh yeah, definitely riding that exponential growth still” is some of the wildest shit I’ve ever seen, and it’s so common.
I'm the other way around. For doing real, useful work, these things were barely more than toys even just a year ago. Now they're very capable when used well, and still getting better at a fast pace.
That's a false binary. The right binary is whether it's a convergent function or a divergent function. A sigmoid is convergent, which is not well substantiated and more farfetched than a divergent function, even if it's true that 2^x is too optimistic.
I mean, "sure, technically correct is the best kind of correct" - but where we sit on this right now? It certainly feels exponential.
A lot of this sort these sorts of posts are just "appeals to geometry" (aka "cope"). This is coming, it's coming hard. Now you need to decide what you want to do with your life in a world where your smarts aren't as special as they used to be.
This is hard (believe me, I know). But what one ought do is not eschew progress and cling to the delusion that things don't change, what one ought to do is try to see how they can leverage these tools for greater and greater accomplishments.
My wife is working on her PhD in microbiology now, using OpenAI to implement her research ideas. Genetics is just too much data, and she eats through tokens like nobody's business. I thought I was careless with them, but she barely lasts a full day before exhausting her quota. There's definitely a lot of value there, but dealing with the data problem is a big obstacle in biology. I can only imagine what she could get done with more capacity, though...
This absolutely terrifies me, surely one misread piece of data or a hallucination here or there and a little “oh sorry about that, I guessed at this portion of the data to save time” and the data used is useless?
> The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn,
isn't it clearly split between verifiable not verifiable ? what is interesting about that question.
Programming has verifiable and non-verifiable aspects. Competitive programming, passing tests, and performance can all be verified. But translating English requirements into actual software, software architecture, taste, or UI design cannot. And yet over the last couple years we’ve seen huge lifts in all of these areas, not just the verifiable ones.
Verifiable areas I think are clearly seeing the most improvement, or are the quickest to see improvement. But we are seeing lots of progress in non-verifiable areas as well.
How much of the non-verifiable progress is a function of labs purchasing expert data vs. the models improving with compute is maybe another interesting question, but fundamentally I don’t see spend on expert data as something that can’t grow if AI revenues keep growing as well. And as models get better taste they can also help filter and generate new synthetic data for their next versions to train on. The limits of this approach are not so clear.
Labs spend billions hiring experts to generate new data, and better models can better filter existing training data and generate new synthetic data. There’s no reason for that to run out, it’s just expensive.
You could view this as just continually patching a leaky ship. But it seems to work.
I remember listening to Andrej Karpathy talk in a podcast about how synthetic data in particular is used to generate more data for pre-training. I see no reasons for that to have changed. I think it is likely a lot of the new data they are paying for contributes to pre-training as well.
I would also be very shocked if they weren't filtering or prioritising existing pre-training data as well, for example to do curriculum learning or to avoid data that degrades performance.
thats the whole point of turning to formal verification:
Imagine a hypothetical oracle, call her MyladyMath, imagine you can turn to MyladyMath, submit a correctly formed dossier of axioms and definitions, theorems with proofs, and then a newly putatively proven theorem T. MyladyMath will complain if your dossier is malformed, and point out where and why. If the dossier is not malformed it will eventually read in the claimed theorem T, evaluate its proof and then either point out a which step is erroneous and why, or ultimately accept the proof.
Instead of a large corpus of human authored text, this map from dossier/theorem -> accept / reject is a huge implicit array of bits, something fundamental, and this weird gigantic array of bits that effectively describe all accept/reject responses MyladyMath would return exactly. We would never run out of "corpus" when it comes to math, if humans had access to such an oracle.
And we do have access to this oracle, and possess compact algorithms that describe the accept / reject bits. One of them is called MetaMath, a minimalistic verifier, which keeps the concepts of prover and verifier separated, this choice results in concrete proof objects (a sequence of step label references).
The machines are going to comb through all possible paths of the next N steps, for progressively larger N, efficiently compress those results in the weights of an LLM and then use the gained experience as the intuition for guided "not-so-brute"-force proof search, using the prior iterations intuitions to grade the surprisal of the N+1't iteration of results, etc.
The money will not stop flowing in that direction: all power blocs, nation states, militaries, banks, ecommerce, ... depend on cryptography. And the machines will soon do more rigorous proof search grounded in more balanced and objective observations. There is no responsible disclosure mechanism for flawed hardness assumptions in cryptography. It's going to get rocky, and the common man will wonder why the gods have gone crazy, wonder why they don't just pull the plug out of the machines, but nobody will in a staring contest to see who dares keep the plug in the longest (and dominate global cybersecurity).
It's the end of the age of artisanal mathematics, it will now become industrialized mathematics.
That is because there is human annotated data there. Every session you or I used, then of course paid human feedback on repos (such as the recently famous example of meta forcing their employees to).
This is _much better_ data than 1/0 verification, it is as good as a gradient.
Automatically verifiable tasks improve faster since well, its automated.
I think you’re making a category error in your definition of politics here.
Certainly technologies can favour winners and losers, but the struggle is an inherently human one.
We definitely are not on an exponential. Don't say we are because this isn't up for debate. AI progress is logarithmic the million dollar question is 2x or 10x for linear improvement. The nearest qualitative shift would be very fast inference so people could start writing real software on top of LLMs. A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
> A 0.001% optimization on a packing problem just isn't interesting for the amount of investment.
I think you have completely misunderstood what OpenAI have accomplished here. Almost certainly no one cares about the specific concrete results achieved; they only care about (a) how difficult it would be for an intelligent human to achieve the same feat (ETA: the feat is the proof), which can be estimated by the amount of time the problem has remained open/a public conjecture, and (b) how general this artificial "intelligence" appears to be, which can be estimated by the diversity of topics where it was able to prove a difficult result.
It's as if I showed you a dog that I had taught to speak German fluently, and you remarked: "What point is a dog that can speak a language that less than 2% of the world speaks? Nothing to see here."
On the other hand, if provided the financial incentive would mathematicians have solved these problems? Its not hard to imagine a world where some hard problems were not selected by the sparse experts for whatever reason (lack of interest, whatever), which could have been solved if someone was throwing down millions for solutions.
nothing prevented us from defining cryptocurrencies, auto-rewarding new math in objective manners, grading theorem surprisal objectively, etc. before the advent of LLM's.
being lucky enough to receive the opportunity of hiding in some academic closet, poking your hand out begging for scraps, was a different, perfectly alternative path humans decided to take instead.
It's a bit late to start standing up for your rights when the robot overlords arrive.
Thats true, although once it stops being a party trick I expect the LLM proof mining to also be paid by someone else and be expensive as shit, the question same remains: who?
who? the state on advice of the intelligence community: machine learning inference bruteforcing not bits in private keys, but bruteforcing proof search space modulo machine intuition.
At some point cryptography evolves faster than human cryptographers can follow, at that point anyone can only trust cryptographic schemes proposed by their own LLM models, since they will be direct witness to useful responses for apparently unbreakable cryptoschemes with a mathematically known backdoor for the designing LLM.
Why would one power block accept usage of LLM-derived crypto from another power block, if they know it is likely backdoored? The same within a power block: why would one ally financially trust crypto from another ally in the same power block, if they are witness to how easy it will be to request new convincing crypto schemes with a backdoor? The same within a single nation state: why would one bank accept the cryptography proposed by another bank, if they are witness to how easy it would be to ensnare one another with LLM-designed crypto schemes with backdoors.
Its not that humans are incapable of pulling power plugs when the machines start organizing human-on-human violence, it's that no side, no party feels like they can afford a gentleman's agreement to not consult LLM's to find vulnerable hardness assumptions in mathematical cryptography. Perhaps many powerful entities would be more than willing to pull such a plug, but not before others have provably demonstrated pulling the plug,... which is basically waiting until no other human factions exist.
And there's who pays it, and they will pay private companies to compete in order to assure competitive prices for the government.
What is the y-axis in this claim about logarithmic improvements? Any exponential curve can be trivially turned into a logarithmic and vice-versa, and the y-axis redefined as "progress", with no loss of accuracy.
One example that always bugs me is when people point to "exponential" or "sigmoidal" progress on benchmarks. Benchmarks are artificial constructions (saturation at 100% by definition) and benchmark scores should not be mapped to these words when talking about overall progress.
Example - progress on ARC-AGI-3 at the moment is exponential, steeper than 2^t and e^t. Does that mean AI is progressing "exponentially" in the colloquial sense? No, it doesn't support or refute that colloquialism.
Likewise with MMLU saturation. We can't go above 100% by construction. Therefore we have a "sigmoid". Gah.
Yes, the rate and nature of the results being produced is impressive, and Anthropic have recently invested in building out more in-house capability to do life sciences research (see e.g. [1]).
Overall I'm excited for this acceleration in discovery, even if it's causing disruption to existing research workflows. I wrote a bit about it recently [2], after seeing Levent Alpöge's counterexample to the Jacobean conjecture.
The reason writing is hard might be that post-training pulls style into a particular direction.
In other words, big labs are much more interested in making "AGI" than in making a good writer, especially as what qualifies as "good writing" is rather subjective. E.g. before AI use of metaphors and rhetorical devices were generally a sign of a good writing. Of course, not if you keep spamming the same rhetorical device - but a stateless AI can't know which one it is over-using.
I expect we can squeeze a lot more exponential out of LLMs because they’ve basically shown that human “consciousness,” insofar as it’s composed of knowledge and rules for synthesizing that knowledge, can be represented mathematically in a very high dimensional space. Why does this “just work?” Nobody really knows, but it clearly does.
However, I also expect this squeeze will come at an increasingly expensive price — not just because of inefficient token usage, but because of fundamental limitations of LLMs as a model.
LLMs are letting us brute force our way through a lot of reasoning, but it’s hard to believe that such a generic model of intelligence will take us to the next frontier. We’ll need some fundamentally new approaches at some point. Maybe those will make achieving the exponential more efficient or maybe they’ll unlock even higher degrees of possibility. Who knows?
The transformer is Turing complete. It might be a tarpit though? I don't know.
I think a nice example is using them for arithmetic. It's a specialized deterministic process, so it's extremely wasteful to do it that way.
But they're good at finding solutions to things we don't know how to specialize yet.
So, to use metaphor, maybe the transformer-based models are like the FPGA, and then when we figure out the patterns in that system — all the different kinds of specialized reasoning — we can extract it into an ASIC?
I think this is one approach to AI safety and interpretability that could work, but would require labs to slow down to figure out how to extract circuits/algorithms out of trained LLMs rather than deploying the opaque artifact.
> Why does this “just work?” Nobody really knows, but it clearly does.
We know language has to be learnable by every human, so it needs to be really independent of any specific brain development particularities. If it was not accessible to babies there would be no more language next generation.
The distinction is between information flowing from people to power (elicitation), vs. it flowing from power to people (persuasion). These are not the same, even if they are closely related.
Does AI make real opinion easier to hear, or fake opinion easier to spread? Even if you believe wholly in manufactured consent, how easy it is to manufacture matters.
Today's models depend on inference time compute to get these results. The inference time compute available on any claude subscription is not comparable to the ones used to get some of these results (yes, in this case, it is 2000 USD total as noam confirmed, but some previous results took more).
In general, you can think of the process as generating massive rollouts in generation N, and then compiling in the verifier/human feedback("gradient") signal into generation N+1. The time taken to make the rollout in generation N, and separately the time taken to get the same rollout in generation N+1, each grows constant in some tasks, linear in more, and exponential in some.
In the end, this becomes bottlenecked by time. Today, we can make statements like "I generated all these successful trajectories with 2 weeks of compute, in the next model it will be able to do it in 7 hours of compute", but very soon you'll find yourself making statements like "I generated.... with 8 months of compute, in the next model it can do it in 6 months", which isn't really enticing the same way you can _technically_ brute force passwords but it just needs prohibitive amounts of time and money. That is the "plateau". Note that, this point is quite far away. For example, at any point if we agree it plateaus, today's known hardware techniques such as fixed function accelerators give you a 10-100x timeline reduction immediately allowing for a few more cycles of improvement. This is not to mention future innovations, but of course none of that is helping with the benchmarks where the time needed is growing superlinearly.
In many math and coding benchmarks, we are still in the constant phase. These are the massive improvements we see every few months. I'm not making any prediction of what will plateau and what will not as it's not possible to make an informed prediction about these things IMO. But the observed fact is that some have already plateaud as in, they don't improve with reasonable inference time (likely superlinear growth).
> will we need mathematicians to translate
Let's take a sudoku analogy. The model is initially just doing the random value algorithm, but lets say you the human are watching it. You make one of the usual reductions and interject "hey you can stop trying 8 here because of ....". Over enough examples, you get to a point where the model is _forced_ to learn the logical pattern. Next generation, it will skip that number. After this, you can peak the distribution using simple 1/0 RL. Doing _pure_ 1/0 RL works decent, but its not frontier as its a very sparse signal.
For that lift, human (or even a better LLM, but if you're trying to improve a frontier LLM, there is by definition no better LLM) feedback becomes necessary. This is _why_ it is crucial that these models interface in natural language and is also why the labs are hiring AI tutors by the hundreds. The "better LLM" case is what Kimi etc are doing by "distilling"(bad term for this) claude.
> But the long term is completely bewildering if you believe any of these trends can continue at a similar pace for the next few years.
For math and coding, for now we are in the phase where the times are just ... constant, so there's little reason to think it will stop soon. We still need humans to expand the frontier. It just becomes a matter of if its worth the cost of compute for running this generalized The Algorithm or not.
Given how well chess players internalized _many_ (not all) of alphazero's emergent chess knowledge, I am confident we wont have too much trouble figuring out any new math LLMs come up with, which will let us keep expanding the frontier by giving the LLM the next "lift". Only when we reach the stage where the time growth become exponential will this stop, IMO.
It's like Musks duplicitous argument about unlimited abundance. We have a lot of abundance now, we just keep accelerating it all into the hands of fewer and fewer people - whose response is only to want more, and more, and more.
Yeah, we really should just storm the facility where Musk is hoarding all the worlds bread and meat.
Wealth in terms of capital doesn't represent material goods, it represents the system's confidence in your ability to direct capital efficiently. But eventually efficient capital bottoms out at consumable goods. Someone like Musk with a lot of capital under his control is contributing to the end goal of unlimited abundance.
With his shitty cars, orbital garbage, or CSAM generator? Or his moneyed attacks on democracy and communications? Or his vandalism of government programs without insight, experience, or qualification?
In a free and fair market, his capital would be regarded as a deeply inefficient distortion.
In fact the opposite, look at talks by Peter Thiel. At least he’s being honest. These people are bastards, but for some reason moral goodness and wealth has been conflated leading us to venerate greed.
I've noticed that out of all LLMs I've ever used that Fable is the MOST LLM; the text it produces is abomination. It's impressive how much I hate it. It is such an awful writer - it assumes the reader has zero context and therefore gives every single bit of context and detail - which is nice if you're writing a legal document I suppose. But it uses, niche, $10 words to describe every facet of everything it's discussing. I had it re-write some docs and I ended up rewriting 1k lines of of Fable torment nexus text to around 100. Because guess what, someone reading highly technical docs has a knowledge base that allows us to compress the topic into a much tighter representation.
Just praised Fable in another comment but what you're saying is also insanely true.
I literally roll my eyes and cringe quite often at its output pretty much daily.
I don't like to overload my sessions with skills but I've been using a "write-normal" skill I made just to have it rewrite outputs that particularly piss me off.
"the test is whether the sentence would work as a pull quote or a LinkedIn post. If it would, rewrite it until it would not."
LinkedIn really became the default garbage example
Fable is much better at handling nuance. Opus/GPT 5.6 Sol are much more likely to miss the point you are trying to make, emphasise the wrong thing, exaggerate the importance of unimportant details, or introduce contradictions.
That said, Fable is still not a great writer, largely driven by it not knowing what it should exclude, and it still having the usual LLM-isms. But it’s better.
That's really what got everyone hooked in the first place.
5.6 Sol is great but there's a depth to the understanding that Fable exhibits that's unique to it currently.
Can I truly quantify this? I don't think so. Just that I spend a ton of time with various models and a certain point it's just a personal impression or a gut feeling.
In the days after Fable first came out I increased the amount of parallel planning of tasks that I was doing by 2-3x because it felt like I didn't need to be paranoid due to that handling of nuance.
Another interesting question is why the frontier labs are piling on pure maths, which has little direct economic value compared to something like law or improving the efficiency of their own models? How much OpenAI and Anthropic are paying to serve these models for ordinary users is the elephant in the room. A cynical take is that the frontier labs are trying their best to pump up their pre-IPO valuation through flashy headlines.
> Another interesting question is why the frontier labs are piling on pure maths
The reason is that the original scaling axes (parameters, training tokens, test-time compute) have saturated already, but RLVR (reinforcement learning from verifiable rewards) is still scaling well. And math has this nice property where you can synthetically generate arbitrary volumes of rewards to train the model, because math is self-contained and completely objective. Open-ended reasoning and analysis don't have that convenient property, and that is why progress is much slower outside of math and coding.
do we have any concrete idea of how well the models are scaling now? I agree with these 10 results being impressive, and it is easy to think "wow, and last year the models were barely able to solve IMOs problems". But for me it is perfectly possible that a non sofic group could be found by 100 good IMO students working on all the different strategies that have been proposed (OpenAIs solution was based on "expander graphs", which were introduced to solve the problem some years ago), so it could be that current AI is simply many (say 1000) old models working in parallel. This is linear scaling, not exponential. It could be I'm completely wrong also, the problem is that we have little information.
It's one of the few areas where you can verify results. That fits nicely into training models. They aren't just making judgement calls on what would be nice, it's "what can we do?".
If everyone publicly said that the models can only do things that humans have already done, but you know they can do more, wouldn’t you want to show them otherwise?
Math ability also helps with other things like making models more efficient.
Because it's a tool in search of a use case (or many use cases) and mathematics is the most natural use case for it. Mathematics is by definition the art of putting words on a page in a rigorously defined "correct manner" (i.e. in the form of a valid logical argument, a proof) and all LLMs do is put words on pages and evaluating if they're good words is by far easiest when there is a strict definition of right and wrong.
What's the best way to apply it to legal problems? Finding bugs in statutes? (there are often statutes with wording errors, missing negatives, things like that which don't get picked up for ages)
There are many math problems that are simply puzzles: intellectually interesting but nothing worth of value depends on it. To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that.
The results OpenAI demonstrated are impressive, but it also looks like they threw a lot of compute at it just to get results. How many tokens did they waste on problems they couldn't solve? Applying inference infrastructure on a large number of math problems at scale we haven't seen before to me doesn't demonstrate an exponential curve in model abilities.
"To me it would be more impressive if we could define hard problems that need to be solved up front and see how the models deal with that."
I was recently listening to BBC Radio 4's episode on the Poincare Conjecture[1] and the guests on the program were discussing how the problem that looked deceptively simple eluded the great mathematicians of the time (including Poincare himself) for nearly a century and how Grigori Perelman cleverly came up with the proof. It took other mathematicians working in groups years after Perelman's publication to understand and validate his proof. The mathematicians on the program were speaking of highly of his proofs and admiring the originality of his work. This made me think of one neat experiment where if we cut-off a frontier model's training data 2002 or anytime before Perelman posted his proofs on arXiv and check if it can come up with the solution by itself. That would surely be a great signal to see if these LLMs aren't just solving interesting puzzles and that they can came up with something truly novel.
P.S I highly recommend Misha Green's "Perfect Rigor" for anyone interested in the history of the problem and the genius behind the proofs of the conjecture - Perleman. I found it an entertaining read and could digest its description of the problem as a layperson (with undergrad level math).
The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn, but I’ve noticed Fable to be quite a big step up there. How about politics? Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation? How about experiment driven domains like biology?