Pretty cool. The impact of AI is getting undeniable, there aren’t many positions left to move the goalposts to at this stage, next they’ll have to be outside the stadium entirely.
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.
Never understood all this talk about moving goalposts - you understand that's how science works, right? We improve, we learn, we recalibrate our expectations based on what we've learned. If we never "moved the goalposts", we'd be stuck scoring the same goals over and over.
> We improve, we learn, we recalibrate our expectations based on what we've learned.
That's not what people mean when they say "moving the goalposts". It means that people are adamant that something wasn't important/hard/impressive once the "AI" solves it. And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
It's also very much not a new phenomenon. It's been happening since the 1980s. As you can see from this quote from GEB by Hofstadter:
> There is a related "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem: "AI is whatever hasn't been done yet."
Yes, this is exactly what is meant by “moving the goalposts”. And it’s a fairly well known expression applying wherever people retroactively change their requirements in reaction to those requirements having been met.
So you defined your own idiosyncratic version of "moving the goalposts" and used it to rebut his argument, with a condescending "you understand that's how science works, right?"--instead of being honest that it is you who are changing the definition, and not his failure to understand anything.
If it seems like I don't understand the meaning of that very well-known phrase, then clearly I have failed to make my point. I'll try again. And please note that I will use some generalizations to make my point more clearly, rather than because I don't understand nuance; kindly grant me a charitable reading.
In recent years, I have commonly seen the phrase "you're moving the goalposts" deployed by the "it might be sentient" crowd to shoot down the "it's a stochastic parrot" crowd when the latter respond to a new development with "OK but...". In a well-understood field of inquiry, that would be a clear case of goalpost-moving, in the commonly-understood meaning of the phrase where requirements are retroactively changed in response to them having been met. Thank you OP. 'Artificial Intelligence', and indeed intelligence in general, is very much not a well-understood field of inquiry - in fact we don't even have a common agreement about what 'intelligence' is. We are therefore learning as we go (even after all this time!) but making rapid progress in recent years. When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change? This is arguably one of the most pathological development projects ever - what are the requirements? 'It thinks like a human'? What does that mean? And the answer is we don't know what that means, and we're working it out as we go - moving the goalposts. If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
Side note that, in case it's not obvious, none of this detracts from how impressive LLMs are. They're a marvel of the modern age, all the problems notwithstanding. However I reserve the right to stay sceptical about their capabilities.
> When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change?
It's in how they change, not the fact that they change. The skeptics seem to have secret definitions for intelligence, sentience, consciousness, creativity, etc. that amounts to "a thing only humans have". Often that thing is equivalent to a soul. When yesterday's challenge (LLMs don't have X because they can't do Y!) is met, Y changes but X stays the same. This is not the process by which a field matures, it is a rhetorical technique used by skeptics to avoid honestly stating or confronting their internal definitions. That can be revealed by asking the skeptic the following:
"Forget LLMs. What if we made a completely physically accurate simulation of a human being?"
Many say no, that simulated human being still couldn't have (intelligence, consciousness, sentience, creativity, ...). This reveals that there is a necessary metaphysical component to those attributes, at which point any scientific-minded person will leave the debate.
OK, I see, thanks. That applies to some skeptics, though - I'm in the AI skeptic camp but only in the sense that I believe we should make sober assessments of what these things are actually capable of.
> If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
To me, the goalposts were already defined by the person you were responding to. "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
Note that this does not depend on things like AI sentience or defining "intelligence" more rigorously, it just depends on AI impact.
> "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
those are not goalposts - that's an incredible vague 'goal'. What are we moving here exactly?
The "goal" here, not in the sense of what AI scientists are shooting for, but in terms of "goal", "goalpost", and the idiom of "moving the goalposts", would be to invalidate the current argument. Typically, you "goal" when you successfully disprove the other person's argument.
Thus "moving the goalposts" is for the defenders to shift what argument that the other side is trying to disprove.
If the argument is "the impact of AI is getting undeniable", you'd try to show that impact of AI is not that high, and not looking to be that high in the near future.
The other side might move the goalposts by saying something like "well, it doesn't matter if AI is actually that broadly impactful right now, all that matters is whether it's impactful in the field of mathematics". At which point, you might be frustrated because the effort you put into engaging their previous argument is now moot.
> And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
The fundamental argument that I've personally made since the early days of this is that LLMs are not reasoning, in the way that word is commonly understood.
There are lots of reasons why that argument needs to evolve that could certainly appear to be "moving the goalposts", but let's take an example.
A lot of AIs were tripped up by the question "Should I walk or drive 50m to the carwash?" Several folks liked to use that as an example that illustrates that LLMs aren't reasoning, but as the models have been trained on that specific example, it's of course less useful. An AI can mostly nail it now.
So a different example is needed. A new demonstration of how these things fail at basic reasoning a child can do.
Did I move the goalposts? I don't think so. The fundamental argument stays the same. It's not hard to find lots of examples that trip up LLMs, because they are what they are: statistical inference machines. Nothing more and nothing less.
Useful, sure. But also commonly misapplied to areas for which they are inappropriate solutions.
The unified position which many folks deny is "AI is powerful" or, alternatively, "AI will be powerful soon".
(I'm personally still skeptical about this, but I'm being pulled towards accepting it).
"AI is useful" is too low of a bar, and "singularity is nigh" is too high. "AI is on its way to upending society" is about in the middle, and still vastly contentious among laypeople.
I think Ed Zitron is mentioned just because his name is Zitron. He just repeats ad nauseam opinions concentrated around one simple, very boring pole on a wild and interesting landscape of emerging reality. Anybody could be doing that. A lot of people do that. Yet no other is named Zitron. And that's why I heard name Ed Zitron hundred times. That's how you become a voice of (a part of) the generation. Just have a memorable name and repeat the same opinion over and over that people can flock around comfortably. Content is irrelevant.
Personally I prefer to follow explorers rather than swamp-sitters.
"it isn't clear whether generative AI actually provides much business value at all"
"cannot seem to find a product that people will pay for, in part because the results are so mediocre"
"Last week, we got our first real, definitive glimpse of what’s around that corner that future. And boy, was it underwhelming."
"OpenAI claims that o1 “performs similarly to PhD students on challenging benchmark tasks in physics, chemistry, and biology.” Just not in geography, it seems. Or basic elementary-level English language tests. Or math. Or programming. "
"Worse still, it's kind of hard to explain why anybody should give a shit about o1."
"o1 shows that OpenAI is both desperate and out of ideas."
"the software is not becoming more useful"
Honestly, every other line is quotable in this context.
Yep, he is a PR stunt guy, and the number of videos that come up when you type Ed Zitron into YouTube should tell you how many people are eager to feed their cognitive biases.
They aren't claiming that science doesn't progress by moving goal posts. They're talking about how critics of AI have claimed it isn't revolutionary/useful, then progressively changed what would it mean for AI to be actually revolutionary/useful.
Not long ago many folks were saying AI was the same as the crypto bubble. No real useful technology and only hype.
> Never understood all this talk about moving goalposts - you understand that's how science works, right?
I agree with the parent that we need to acknowledge that we're at a turning point in history. I lived through some of them (internet, ubiquitous personal computing). But it's somewhat difficult to comprehend the impact of this one for many people.
I do biomedical research at one of the top European research institutions. We're very well-funded, but I can clearly see the gap between us (say, top-100) and top-10. I also realize this gap is going to get so much wider unless we invest heavily in AI access (and I'm not so sure I can sell anything more expensive than $20 Claude subscription to the leadership).
I think people having 6-7 figure SOTA AI budgets will move exponentially faster than those who don't. That makes me worried.
So, for me, it's not a question of recalibrating expectations. We're way past that.
People will be broken out of their denial by actual economic growth. That's what this is all meant to be for... I think we might start seeing some surprising numbers.
Plenty of people are making money from AI? And I am not sure I've seen it "making things worse for most people" yet, do you have a source behind this?
If your source is AI layoffs, there were plenty of layoffs with the invention of the horseless carriage, but that doesn't mean it made humans worse off overall.
There are reasons to be skeptical about progress and AI and all that but the 'making things worse for most people' thing you mentioned seems yet to be based in any reality
most devs i talk to say that they would love to go back to coding without ai, if everyone had to do it, so there's that. Oh also scammers have gotten so much better with llms, so a significant amount of peoples lives are worse due to that
Is the amount of people making money from AI greater than the amount of people who lose money from AI? Is maximizing prosperity across humanity even a goal, at this point? Am I to be expected to believe, without doubting, that the end goal is the benefit of the many?
But who benefits from that economic growth? If it's only the capital owning class, and the rest of us are left in the dust, then the growth is irrelevant.
I ,for one, have read enough history to know that it's never the proles who end up benefiting.
I suspect that you have not read enough them. Ask someone in China (which yes, still has a capita owning class) if the proles were better off 50 years ago or now. The only real probes we have now in the west is that we have failed at housing, precisely because instead of growth, we have made many choices to increase returns for old people, and insufficient taxes for real estate.
I was a bit triggered by your China comment, if only given the fact that there is a very specific institution that drove many formerly prosperous cities and communities into abject poverty (the CCP in Mao's era) and then tries to claim all the credit for the market driving how many people they've lifted out of poverty.
And nowadays with Hong Kong (and probably soon Taiwan) they are proving they are perfectly happy to destroy economic growth as long as it benefits The Party
Can someone explain why we’re not using it to solve the obvious big conjectures/problems though? If this stuff is solved, why isn’t Riemann the first thing to go? Even if it means a fund of several hundred $k to let it churn on it.
Guys, please use critical thinking. The haters don't hate by default, we hate because we're gaslit about this stuff every day and it's annoying. Extraordinary claims require proof, and they're not giving us information that would be essential to knowing if this is actually significant or not.
AI already have an impact, and yes this is PR hype because this is a product. Yet both can be true at the same time. We're not blindly eating what's OpenAI is serving us as gold truth, we're just admitting it's doing remarkable progress.
Remember October 2024 Pelicans [1] ? It's been only less than 2 years.
We don't know what will come in the next 2 years. But the progress doesn't seem to stop for now.
People are skeptical of the announcement because the room include several PHDs in math and physics. The prompts are not published so we can see how generic the starting prompt is.
Nobody is saying it doesn't have an impact. What we're saying is the near-religious fervor isn't warranted. AI boosters always speak in the future tense, which is extremely telling because we have right now is basically "ok" not earth shattering.
If you think it's "ok", then you're probably not using it the right way.
I've been analyzing thousands of x86 traces and the AI made sense of them at a scale no previous human or algorithm could do before - I've been reading assembly everyday for half of my life. I'm doing prototypes in minutes what would take hours or days before. AI is answering questions in such a precise way in so many domains, learning something has never been easier.
This is "just ok" for you ? What do you need exactly ? god omniscience ? come on, can't we just appreciate for what it is and be excited about the future ?
Gary doesn't argue it's hype though. He argues 2 things: other people are getting carried away with the result, and we don't know enough about how it was reached to know where it falls on the impressive scale.
He literally says it's an impressive feat in the second article.
best point in that first link: openai doesn't tell us if they only tried to solve these 10 and each was solved (amazing) or if they asked it to solve a million problems and it got these 10. Either is great, one is more so.
Gary Marcus has been moving the goalposts since day 1. The guy is a psychologist. Why would anyone care what a psychologist has to say about AI? He's likely made good money from constantly moving the goalposts and being a denier, due to the publicity he gets.
He's simply a good phone number to have for journalists under time pressure who need to add the contrarian voice to their upcoming story. He delivers it reliably, then never reflects on how he was wrong in the past, just blasts forward as if nothing happened and just makes the next bonkers claims to the journalists who are very thankful for the prompt delivery of how AI is a nothingburger, and fake and won't ever do XYZ that it then proceeds to do in N months.
I remember the time when he insisted that diffusion-based image generators trained on Internet scale data will never be able to make an image of a horse riding an astronaut. Today you can generate 4K video of that.
Mate, if you don't like Gary Marcus that's your call, but you're completely just lying when you talk about his qualifications. He is not "just" a psychologist, he's done a lot of AI research and even started AI companies that were acquired. This is absolutely a person with the credentials to speak on this subject, and honestly all his takes I've read have probably been TOO nice to AI companies.
Your comment seems entirely disconnected from the posts you linked. It's impossible to deny the results, it is not PR hype that in the past couple of weeks LLMs have resolved problems that have been open in mathematics for many years. Some of those problems had remained unresolved despite keen interest from many humans.
The only way that is PR hype is if you're invoking the insane conspiracy that frontier AI labs are just buying off results that would otherwise be career defining for a mathematician, just for marketing.
The posts you linked are urging caution regarding the exaggerated e/acc-esque lies peddled by people like Musk, not that the models haven't proven themselves as having genuine ability to contribute to research in some areas.
I use these strange machines all the time. They have gotten notably smarter. That’s my personal experience.
They still do things that I find incredibly annoying and “dumb”. And I still have to clean up messes they make quite often.
But on the whole they are clearly smarter than before. No extraordinary claims needed. I just try to learn how the tool works and how to use it effectively.
In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
How does intellectual property work in an AI generated future? What about healthcare advances, who gets to own those?
What about meaning, what about purpose? How do we replace the work ethic that tells us we are our jobs and idleness is immoral? How do you replace “What do you do?” As one of the first questions you ask a new person?
I think this too is a kind of denial. As in, while some are in denial about the usefulness of AI, others are in denial about whose living standards are actually going to be uplifted.
And it's sad, really, because I think these two groups would make a great pairing if they could stop arguing against one another for a moment. They'll both be impacted about as much and probably have the same ultimate goals (to lead dignified lives).
But it seems these days everyone is more interested in Kayfabe and feeling like they're in the right than working together, so maybe I should just keep quiet rather than attract the ire of both groups...
> others are in denial about whose living standards are actually going to be uplifted.
I don't know if it is fair to say they're in denial. For my part, I don't expect life to get much better for regular people (especially short term), but that doesn't mean we shouldn't work to try to make it happen.
Asking questions about policy and values and pushing to have those resolved in positive ways is about as far away from denial as you can get. It’s possibly the only useful thing an ordinary person can do.
What a lot of people want to do, and I’m not saying that you’re one of them, is to assume that a positive outcome is impossible and either do nothing or loudly yell that the world is ending. Neither is particularly useful.
Or, as I said above, others just deny that there’s anything to see here and try to get people to move along.
The AI companies themselves, who are highly motivated to sell AI as overall positive for society, notwithstanding some security/etc risks, and who have economists on the payroll to think about things like this, do not seem to have found any possible positive outcome to present.
Shane Legg (DeepMind co-founder), one of the more intelligent and thoughtful people you'll find in the industry, could only offer "it's a tough problem - we need to think about it" when recently interviewed by Hannah Fry.
On the surface the most likely outcome for AI allowed to replace jobs is extraordinarily negative, especially since it is a general capability technology, not a specific one where displaced workers can just move to another field. Once AI becomes more capable it will be able to do the vast majority of white collar jobs, including any new ones that may appear as a result of AI. As Shane Legg put it, "if your job can be done remotely, sitting in front of a computer, then it can probably be replaced by AI".
Not only does AI threaten to replace ALL the white collar jobs, but it is rapidly going after blue collar (factory jobs, driving jobs) and pink collar ones (Japanese robotics for elder-care) as well.
If a positive outcome (which doesn't include putting displaced workers on welfare - UBI) is possible, then it sure would be nice to hear it, and the silence from the AI companies, and government for that matter, is deafening.
UBI probably is the positive outcome, though it may not seem like it to begin with. Initially it will likely be stigmatised and under-resourced, but as a larger proportion of people move out of work and onto UBI that stigma will drop and the resources should grow.
Eventually UBI will be the norm, and if the living standards of a person on UBI is as good as yours or mine today, that will be an enormous win for everyone. It’s like pensions, once these were only for the elderly poor, now they’re a right for everyone in most developed countries.
It’s also interesting that for most of human history leisure time was the point of life, and only in recent modernity has work come to be the meaning of someone’s existence.
UBI has to be commensurate with production being automated. That’s a big logistical problem, if you think building datacenters is a challenge try bringing about radical abundance, but even so it’s not insurmountable. It just needs to be taken on as project and not seen as an impossibility.
So much of this is not about what is possible so much as what people believe is possible. We can do anything if we try.
We don't see a progress towards UBI even inside individual countries and with AI we will need it on international. What people in Europe or Africa will do when all jobs there will be replaced by American or Chinese AI companies, where they will get money for UBI?
No, I think people will demand to not live in poverty. Capitalism allows people to escape poverty by personal effort. An AI future won’t allow it by any means other than collective effort, so that’s what we’ll do.
That will entail an immense amount of violence to be undertaken by the working class against those in power, unless you think they'll gladly give up all of their ill gotten gains out of the goodness of their black little hearts.
In fact I don't think that rich people are cartoon villains who enjoy watching the poors starve. If AI leads to greatly increased productivity, then at existing tax rates there will be more than enough to provide good living conditions for people who can't find jobs.
In some countries, especially less developed ones, poverty levels are declining, but in the US the trend seems to be the opposite. The middle class is being gutted and descending into survival mode if not poverty, while the top few percent become obscenely rich at their expense.
In any case historical trends are really irrelevant to the discussion, which is about the impact of AI, which threatens to take away almost ALL the jobs (this is what Dario's essay is assuming), which has no historical precedent. Some people like to bring up previous waves of job automation such as the industrial revolution, but the difference there is that automation took some jobs but created others. In the case of AI, AI will also be taking the new jobs that it creates.
If AI takes all the jobs - which is what the people like Dario who are creating it assume will happen - then it seems "UBI" is indeed the logical conclusion (other than those able to make a living working for themself, or via self-sufficiency), but this is not going to be utopia where we are all idle rich practicing our hobbies. What it really means is a welfare state, where formerly proud people capable of supporting themself become dependent on government handouts. What comes to mind is the movie "Soylent Green", not utopia.
I'm not sure how anyone imagines this would actually work. Is there any private enterprise left at all, or are all the means of production (AI datacenters and robotic factories) all controlled by the state. Instead of distributing Soylent Green, the state distributes UBI-scrip, essentially food-stamps, that can be exchanged at state stores for provisions?
Seriously, how could this actually work ?
An alternate future, no more optimistic, at least in the short term, but perhaps more realistic, is that in the fairly near future when unemployment and public pain becomes high enough, we reach a tipping point, and the pitchforks come out. Eventually the government concedes that AI is no more conducive to the public good than nuclear weapons, and heavily regulates it, banning the use of AI to replace jobs. This may sound extreme, but surely not a fraction as extreme at the UBI-based welfare state that Dario Amodei is fantasizing about as the best possible outcome (that is compatible with himself becoming enormously wealthy).
> No, I think people will demand to not live in poverty
What about other countries in the world where people... live in poverty?
> Capitalism allows people to escape poverty by personal effort
Isn't it possible that there is a future where AI makes the average value of human labor (or, "personal effort") plummet? Perhaps capitalism will lose some of its edge against a technology like this.
This seems more a fantasy than a considered likely outcome. He basically admits that humans will eventually mostly all be out of a job, replaced by AI, but then says (Gemini's summary) that there will be:
"Massive Economic Abundance: Because AI will exponentially grow the total economic pie, overall resource scarcity will diminish. The fundamental challenge shifts from producing wealth to distributing wealth."
So how do we go from everyone out of work, no income to spend on food, or the goods and services that the AI is producing, to "massive economic abundance"?!
"Massive Economic Abundance" implies massive increase in produced goods. This implies massive deflation, ceteris paribus. So step 3 could be simply printing money to pay for UBI. Deflation from AI productivity increase and inflation from UBI money printing will cancel out.
For money to work it has to represent some real value, something that has some scarcity to it such as potatoes or hours of human labor. Ultimately it is just a decoupler in a barter system, a universally recognized IOU.
Why would someone give me a car in exchange for UBI-scrip when that UBI-scrip has no inherent scarcity or value and can be produced in infinite supply by the government ?
> Why would someone give me a car in exchange for UBI-scrip when that UBI-scrip has no inherent scarcity or value and can be produced in infinite supply by the government ?
UBI script will have some value, I am proposing printing enough money just to combat AI productivity-induced deflation, not infinite UBI money.
You know, I think Dario should have kept the Richard Brautigan theme in his essay titles. I think that I was trying to describe you to someone is an absolutely perfect match for introducing an AGI that has been achieved. It’s basically the same idea in blank verse, about a spread of a different technology, and made personal along the way.
Asking politely is not how we got a 40-hour work week or workers' comp or most other labor standards we take for granted, just historically speaking. I think a positive outcome is very likely, and I think it will be a lot more work than loudly yelling, but I don't think anything will happen if we try to build everything up from first principles instead of taking a moment to consult history as many are wont to do in this AI era.
What pressure is there to push for any of these changes? I see AI advocates discussing the concept of UBI, but I can't imagine a world where the United States would ever pass this sort of legislation. I mean, congress can barely pass a budget each year.
If you are correct, I expect corporations to reap massive profits while most Americans try to find a way to survive in a world where they are obsolete.
There is whatever pressure you bring to bear on it. As long as you are still living in a democracy your vote is the pressure you can apply. If the parties that exist won’t represent you, make new ones that will. Do something rather than deciding it’s both impossible and up to other people anyway.
I also don’t understand why UBI is desirable. Putting everyone on welfare means everyone is poor. This won’t end well, and it’s certainly not the case that 99% of the population will let a tiny number of people remove their income in favor of pennies for all. Political violence will come first, easily.
Because the implied alternative of the comment is that people will carry on having jobs, which breaks the entire premise that there is AI that replaces all human labour. The comment is missing the point - UBI is the only suggestion anyone has so far that meets this fundamental question about the technology. We will be less poor than without UBI, unless we come up with a better plan.
Isn't it obvious if AI automates your job away, the AI company is going to reap the economic value, which is going to be less than what you cost, while giving you a pittance as UBI? If you received its full value there wouldn't be any point in anybody replacing you with AI.
Agreed, a LOT of UBI advocates gloss over the "B" in UBI. If AI increases human productivity overall, the only morally acceptable outcomes (imo) are that everyone's standard of living increases (or at least is the same without having to work) and wealth inequality decreases (if AI is doing ~all the work, there really isn't any sensible justification for some people having significantly more wealth than others). Frankly anything else seems like a recipe for massive social instability.
> In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
> We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
UBI in the United States is never going to happen in time. If it happens at all. We don’t even get universal healthcare. I think people who think AI will be a net positive for humanity are also in some sort of denial.
In a different US political climate I would entertain it. If these frontier labs weren’t so clearly going after the money, I would entertain it.
LLMs are clearly a step up for capitalists so I just can’t see any inclusion of LLMs move towards more progressive ideologies.
I guess we'll have to fall back on the plan of hoping one of the 7 Anthropic co-founders has secure power in this world, and decides to use EA principles to give us all a bit of their share of the future light cone ;) (The wink because I'm kinda not really joking that this seems the best plan, if we actually get superintelligence from a US AI company)
I don't understand why this got downvotes - simply extrapolating current trends leads to the need to answer all of these questions.
My own $0.02 on the economics piece - every country should have a sovereign wealth fund. Governments should block market access from automated[0] companies until those companies provide equity contributions to the wealth fund for that country. This aligns regulator and corporate interests. Dividends flow into the sovereign wealth funds and then can be allocated locally from there - UBI, job programs, etc. Let different jurisdictions explore different ways to structure a post-labor society.
On the broader social front - I think a lot of lack of meaning discussion boils down to the overemphasis we have on your job as your self-worth. We need to realign our societal expectations - and people need to spend more time with their families.
[0] for this to work, I think we would need well accepted metrics for 'how automated' a company is - and that probably needs a 3rd party auditing industry.
Nationalize the data centers, reserve enough inference to automate power generation, food production, transportation, and housing.
Is there any government that has gotten socialism correct for its citizens? I'd point to UAE/Qatar if they didn't depend on human servitude and inequality.
Everything you're saying is oddly simultaneously specific and hand-wavy at the same time. I say this because I'm not sure a lot of these things are solvable or, if they are, it's going to take lifetimes. For example:
> How do we replace the work ethic that tells us we are our jobs and idleness is immoral?
For many people it has nothing to do with morality, it's hardwired into their instincts. They want to work, and they will work.
For a lot of us who are not excited about this future it's that no one is trying to answer all the questions you laid out. Instead we have the disgusting people at the helm purposefully spreading doomerism and saying, "We'll figure it out." I think it's pretty problematic (to say the least) to care more about technological advancement than how that advancement is actually shaping up to effect people in the short term. But I know many people don't care, especially those who believe they won't be among the affected.
Beautifully put. This exactly summarizes my feelings about this particular tech. I actually find it hugely useful. But I fear we lack the wisdom to use this in a way that won't destroy us.
Why do you think the AI companies are going to let it be a “we” kind of decision, and since it’s obviously not going to be a “we” kind of decision, as getting to this point certainly has not been thanks to people like you, what makes you think living standards are going to be broadly uplifted?
The human zoo where the top ~250,000k humans live in a "human utopia" and the AI provides while mostly focusing on whatever it decides it's own goals are.
Humanity survives (but we reading this probably don't), the AI treats the living humans like the Emperor's favorite pets (probably a pretty good life), and then the AI does whatever else it deems important.
Realistically it means trying to start a small business of some sort, because AI is hugely advantageous to business owners and disadvantageous to workers. And it's something AIs can't do unless they get legal personhood, which may well not happen any time soon.
Hopefully doesn’t ever happen, since that’s one of the more plausible omnicide scenarios. I human like AI is almost certainly achievable without much research effort at this point, but we shouldn’t do it.
Logical mistake. There is a difference between every human dying eventually and humanity going extinct. The former doesn't imply the latter. The previous commenter clearly meant the latter.
Your 'serious' proposal is unlimited global military bombing campaign on civilian infrastructure by the United States (which is currently losing a war using the same strategy) to 'solve safety' preemptively against a 20% number you just made up?
And you accuse the 'other side' of 'suicidal apathy'??
You should put down the AI and do some self-reflection on how you came to hold these views.
> You should put down the AI and do some self-reflection on how you came to hold these views.
I've been in the field for almost 2 decades, and actively thinking about AI risk for over 15 years.
I've always held the controversial opinion that there may come a time where we might unfortunately have to consider using force to protect our civilisation from the threat of ASI, but I've always reframed from openly discussing that because I'm so personally against any use of force and there's always been time for more peaceful options.
Several months ago I changed my tone on AI risk to stop worrying about the optics of what I'm saying because it's too late in the day for me not to speak plainly at this point. Similarly over the last week I've decided I can no longer reframe from advocating for the use of force (if necessary) to stop rouge actors from playing Russian roulette with civilisation.
Please understand I don't want anyone to be harmed. Perhaps you don't agree with my suggestion, but my stance ultimately comes from a position of harm reduction.
> preemptively against a 20% number you just made up?
It's a probability estimate. Happy to expand in detail on why 20% specifically, but first you accept I can't tell you what's going to happen in the future with certainty right? To some degree any prediction you might have about the future will be "made up".
Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.
There are dozens of escalating, yet fundamentally less destructive means to prevent a data centers from continuing to operate (if that's the goal). Not starting at those more modest interventions is political malpractice and in opposition to your harm reduction narrative.
Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.
Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.
Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.
> Advocating for nuclear war as a means of 'harm reduction' should be causing you serious cognitive dissonance.
If you're referring to the "nuke the data centers" line in my bio, that's tongue in cheek. It's an expression of my sentiment towards AI at the extreme.
To be clear, I don't think any country should seriously be considering dropping nukes on data centers. Targeted strikes on data centers operated by rogue private companies or nation states is acceptable to me – assuming pre-warning of intent and appropriate regulatory foreshadowing.
I suspect we agree this is reasonable?
> Your P(doom) 'probability estimates' are almost certainly based on unbounded exponential growth curves.
> Nature abhors such things; I don't make any plans around their existence, and you shouldn't either. Limiting factors always emerge and dominate the curve, tamping it to a logistic at most.
No, they're not based on unbounded exponential growth curves at all... I'm actually somewhat skeptical of claims that there could exist an intelligence that far exceeds our own. My assumption is really just that AI will be far faster, far more knowledgable, far more persistent and marginally more generally intelligent than the average human. The first three are already true, the forth we're getting to.
We're on an exponential curve right now and I suspect the rate of progress will accelerate some more as a result of RSI. This will likely continue for 2-3 years (if we have that long) before starting to flatten out.
As for my doom prediction, my median concerns are around around the destabilising nature of ASI.
If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation. I can go into more detail here if you wish.
However, I'd still put these destabilisation risks at under 50% in total. My concern more broadly is the majority of ASI outcomes are doom scenarios. It's extremely hard to imagine a world in which you have a magic box that can grant any wish that intelligence can grant, but which we have a functional civilisation. You have to basically assume all potentially dangerous wishes won't be granted and probability the only feasible way to do that is restrict who can make the wishes – but then you have the concentrate of power risk.
I largely agree with the premise "If Anyone Builds It, Everyone Dies", so we must try to avoid building it. I think we can build ASI safely, but we're clearly not on the path right now and I suspect it might take decades (or more) to figure out how we can build ASI with an extremely high probability of a good outcome.
> Get involved with real politics by organizing with other human beings that share your values (posting alone and doomsaying is not praxis), or step back from tech and cherish the limited time you have with the people you love.
I do this. I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.
I don't think we realistically have a chance of diverting from this path at this point. I have made peace with this but it makes me very sad and haunts my dreams almost every night.
> If it can be controlled then it will concentrate power like never before, and I believe this will destabilise civilisation.
We already live in a reality with unprecedented concentration of wealth and power in the hands of the very few.
This process has produced horrific violence and ended civilization for tens of millions.
The actual impact of AI, so far, has been redistributive upwards (solely through speculation, not via productivity or displacement).
This kind of problem is a standard human social ill. Its solution is social too.
Redistributive economic and political action can equalize power and wealth disparity. AI is incidental in this story.
Your understanding of AI as an isolated perfected intellectual djinn lacks a consideration of the existing and ever-changing labour and social relations that give AI responses meaning in the first place.
It is a compressed, multi dimensional manifold containing the structure of relations between objects.
Responses are a reification of a point in that space.
Navigating the space can be interesting, but it is a poor substitute for the relations themselves.
Very few people yearn for the endless, frictionless solipsism silicon valley promises (AI + slave robots), and those that do are overrepresented in your social circles. The books they write are a study in psychological projection and limited political education.
> I've been writing endless comments more recently urging people worrying about their jobs/careers to stop and just spend the next few years with those they love.
This is good advice generally. But it isn't organizing. Join something, meet up, drink bad coffee, listen, take notes, advocate to fix just one bad thing in your community, feel your impact reverberate through the world, and then notice how much easier it is to shake off despair.
If you don't have a community, then no wonder everything feels like its ending.
If the thesis that "If anybody builds it, everyone dies" is true, or has any chance of becoming true, then it is the logical thing to do. As Geoffrey Hinton said, "If you want to know what it's like not to be the apex intelligence, ask a chicken."
It is not 'logical' to imagine a hypothetical doomsday scenario that justifies a preemptive nuclear war. (which is what the grandparent commenter's bio contemplates, 'nuke the datacenters' is their central credo).
Does it bother you that the people who are publicly cocksure that P(doom) is moments away are the same people that have profited most handsomely from that pronouncement?
That the 'humanists' that want to do 'altruism' for 'potential future humans' and are the same people that commit fraud and theft at a civilizational scale, then sell this 'intelligence' to any child-incinerating militaries with spare cash?
It's not wrong to want to do good, but if a system that is branded 'do (the most) good' commits great evils, you are morally and intellectually obligated to step back and reconsider how you are spending your time.
Also, I asked a chicken and a feral rock dove what it's like to be not be 'apex' and they burbled at me and kept eating millet and sunflower seeds.
Would you like me to follow up with them? I'm not sure what point you expected them to make.
I've removed the "nuke the data centers" line now.
It's was an expression of my sentiment, not a policy position I'd support. I wrongly assumed that was obvious, but some people are making bad-faith assumptions about me and my sanity.
Not that it should need to be said on a comment thread where I am express concern about civilisation in a post-ASI world, but I obviously don't want to see a nuclear war.
If the Judeo-Bolshevist theses were true, then Operation Barbarossa was the logical thing to do, and everything which came with it.
Powerful word, `if`. "You're not only wrong you're a fulminating psychopath" is a perfectly valid response to getting it wrong like a fulminating psychopath.
AI is a math & computer science problem. If AI can do advanced maths and computer science research, then it can begin to suggest useful algorithmic optimisations.
Right now I'm sure the vast majority of these will be junk, but occasionally, even with current limitations, they might occasionally stumble on something.
It's not really whether RSI is or isn't possible, it's really just whether it's the most efficient way for labs to improve their models today given they have limited compute to run AI-generated experiments and access to very intelligent humans who might have a better hit/miss ratio.
Do you disagree with anything I'm saying here? Do you not think current AIs can suggest algorithmic improvements or something?
A few years ago the majority of Hacker News dismissed LLMs as not much more than stochastic parrots who were decades away from doing any serious work, autonomously escaping containment and hacking Hugging Face, or outperforming most professional mathematicians at proving theorems. These things were labeled "insane" and "science fiction" despite the rapid progress we had already seen.
Now you are again postulating that there would be no more extreme progress in the near future. That's actually more "insane".
Not satire, but as you can see one is engaging in good faith and everyone is either engaging in character attacks or low-effort counters.
It's fine, and I'm very used to this, but if you want to have a dialogue I'd be very happy to. I'm a very reasonable and sane person outside of apparently holding some controversial views on AI risk =)
I'd love to know why you think what I said was insane?
I don't think this is a straw man, a huge number of people in my life (non CS people) think AI is a dead-end, that it's just a stochastic parrot, that it'll never be able to do many things that humans can do. I have had many arguments with people who told me that "AI will never be able to do X", and then 6 months later AI is able to do X. Then they will move the goal posts and say "well AI will definitely never be able to do Y".
The models are frequently getting worse at items that they aren’t being benchmarked for — and that’s happening more and more over time! Other people in other fields aren’t idiots, they are accurately perceiving the fact that these models are being hyper optimized for our industry, and are becoming less capable in other domains over time. Models of the same scale are massively worse at writing a broad variety of styles of prose than their equivalent from two years ago. (Models of increased scale are a mixed bag.)
Maybe you’re the one who needs breaking out of your cached beliefs.
you are working on coding. they are working on things like "creative writing" remember that gpt 4o was popular among those who had ai as a romantic partnet?
gpt4o & associated parasociality is considered an alignment failure and is actively trained out of the model, so that is a terrible example of regression
For customer support I don't think models have gotten better since gpt-4.1. The class of small models, with limited to no reasoning, that need to handle a complex issue with a touch of empathy, has not improved much.
I think most are actually worth, as agentic harnesses seem to optimize for solving poorly described problems rather than following complex procedures as written. In other words, instruction following maximizing models seem to make worse free-form agents, but they're really all that some domains need.
I understand the point (I don't agree with it; tool calling has gotten much better/reliable and that is very important for customer support) but consider: If you can get same for a lot less, that's an improvement. If we found a way to supply fresh water and electricity for -90% cost after 2 years, that would be fantastic.
You can do many more things, when stuff is cheaper, even if the stuff were otherwise unchanged.
"Small" was a poor choice of words here, "low compute budget" is more what I'm getting at.
In voice interactions, ttfat is actually relatively important. If you look at models with a <1s ttfat you eliminate almost every reasoning model, less some of the diffusion models and more obscure ddtree/dflash like speculative decoding implementations.
GPT-Live, which is coming to the API soon, responds instantly while reasoning in the background. So it can say "Hold on, I'll look that up for you" and continue to respond to the user conversationally while running an asynchronous reasoning task in the background.
It's not in the API yet, but it should be in the coming weeks. You'll see an enormous improvement compared to GPT-4.1.
I don't think any of the ARC-AGI-3 tests are very interesting. At least not as interesting as driving a car. Children literally do a similar task in go karts every day.
Another interesting task would be to take the AI in a robot body into a vegetable garden and teach it to pull weeds. This is another task that lots of children help out with.
Data 'compression' collapse.
People publish AI generated slop on the internet -> next generation of AI is trained on that data -> the lossy/fuzzy training make the output worse -> rinse and repeat.
I just don’t think it’s true, or is significant enough to matter to the direction of travel of AI even if there were something to it. It’s another cope post being lobbed at the idea of AI going somewhere and I’m sick of them.
every lab independently discovered that getting good at bit alchemy (coding and related tasks) should come first as it will enable the formation of training pipelines that will then solve everything else.
so far there is no end to this progress in sight so it's full steam ahead on this singular domain. once it plateaus you should expect to see the greatest disruptions in human endeavors ever as all the training flops will start flowing to other domains to disrupt and dominate.
I can deal with apathy, that’s the norm. What bothers me are all the people who think they can suppress AI by talking it down. That’s what’s counterproductive, just pretend the problem doesn’t exist. Tell other people it doesn’t exist either. I get it, it’s threatening socially, economically, maybe existentially. It’s also not going away.
So you think it’s a potentially existential threat but are bothered by people who maybe want to suppress it… Hopefully you acknowledge there is a bit of lack of self awareness here eh?
It is a fact of experience, and indeed effectively a theorem, that the better they get at coding and math, the dumber they are. These are the wages of RLVR etc
FWIW I was responding to the sarcastically expressed overt message that AI-math is a clear sign that AGI is here and anyone who disagrees is in denial.
My point being that AI math is a narrow skill just like AI chess and implies nothing about generality (AGI).
I agree. The most likely scenario is that this is just a "new normal" lift that is percolating through human endeavors and will saturate at some point. For example, the whole cyber-security bruhaha should ultimately resolve into higher standards for code published -- we can now cheaply find and fix a whole slew of minor bugs that weren't worth our time before.
The fact we see a lift is not the same as evidence that the lift is unbounded.
The lift being finite is supported by the fact improvements have come at the edges: improvements from human feedback, improvements in harnesses, improvements on model compatibility with harnesses, improvements in inference efficiency with new architectures, etc. If we were just training better models from scratch that would be one thing, but we are just making better use of a tool we've developed.
Not sure what your first sentence means, or why you are quoting AI. Many of these problems individually were math at a level approaching the highest possible for expert human mathematicians. These are not simple combinations of existing ideas, or following of human intuitions, or implementing something following specific human instructions. Then again maybe you mean that math is not knowledge and that all math is simply extending the basic axioms using known patterns, to which I would not agree.
as much as I love analogy with humans using super-shoes, not everybody can be a world-class expert in their industry. There should be a place at the table for average people to take part; otherwise, it won't be sustainable.
As a programmer, I am mostly interested in whether my role is sustainable long-term and whether the models will get better. I don't feel in jeopardy yet, but two more years like this and the calculus of hiring software engineers could shift even further. QAs are already overwhelmed with work
well not every runner can run world-class sub2 marathon or even vaguely close, not even in super-shoes
but with super-shoes more and more runners are qualifying for boston marathon and even olympic trials marathon where it would have been impossible for them previously
and that's what "AI" currently does, it allows average people to immediately "pick the brain" of every expert in every field, in every scientific paper, without previously reading a single other google result, something that would have been impossible for them previously (super-shoes for the brain? too far?)
but "AI" isn't creating new knowledge, it's just stitching together existing knowledge from patterns that would have taken years by human hand if even possible at all, it's going to "hit the wall" eventually (in its current form)
How do humans "invent knowledge"? Is your argument that the answer to these questions already existed in the training set? Why didn't any human recognize that before?
humans discover a lot of things by trial and error (aka what I meant by "invent knowledge")
basically everything Benjamin Franklin did was trial and error because no-one understood what electricity was in the slightest
almost everything Edison did was trial and error too, he had his lab try thousands of materials for his long lasting lightbulb filament
even the most advanced "AI" today is just machine-learning going through everything already known trying to piece together previously discovered facts, admittedly at levels and detail impossible by human hands
but that means there are limits and it's not really "AI"
Have you seen how it debugs? It's trial and error. As the most reasonable hypothesis fail, new data is found, and it keeps iterating. It's not a human intelligence, but if it has an issue, it's not some mysterious creativity anima in the heart of man.
I have seen it produce tentative genetics for experiments, just like a scientist does. Then the data comes in and it can evaluate the data from the experiment just as well. One just had to give it a lab budget.
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.