Yes, yes, I know there's like 1-2 companies that have highly modified vehicles that are pretty good, in a limited geofenced area, in good weather, at low speed, driving conservatively, local roads, most of the time. This is not "FSD".

They've been making very impressive incremental improvements every few years for sure. I had a Tesla for nearly 5 years and it was "wow" at first, and then "heh, I guess it's a little better" every year after that.

But when can I get in a taxi at JFK or on 5th Ave and get robotaxied through city streets, urban highway, off into the far suburbs? Could be a decade, if it happens. Just because we were able to make horses faster doesn't mean we flew horses to the moon.

Apply the same "sorta kinda almost" definition to AGI and yeah sure, maybe in 10 years. Really really actually solved? Hah.

But when you can get 5% returns just by sticking your money in the bank, why would you bother investing in software, where 99¢ is considered an exorbitant cost by its customers? That is no doubt why all of these creative industries are on the decline. There is, generally speaking, no money to be made.

Better tools like Copilot and ChatGPT reduce the amount of time it takes to deliver a feature, reducing the need to horizontally scale developers.

Why hire 10 when 3 could do the same work with AI tooling?

I think while tech is obviously contracting post-COVID from overhiring, it's also true that you just don't need as many people anymore.

Why hire 3 when you can hire 10 to do the work of 33? It was only a couple of years ago when businesses were boasting "We can't possibly hire enough."

But, of course, there isn't enough money to be made anymore, which is the real reason creative industries are on the decline.

Also, not to move the goal posts, but there has been a large economic shift since Q3 2022 with tech & finance belt tightening, inflation normalizing and generally slowing. So these things don't happen in a vacuum (which makes them hard to measure!).

For example my previous & current employer have done their first layoffs since pre-pandemic, and neither has anything to do with AI. They just overexpanded and need to shrink.

Well... maybe not computers, which seems relevant: https://en.wikipedia.org/wiki/Productivity_paradox#IT_unprod...

Thus, in some market segments, it's possible for real productivity to increase without having a significant impact on economically measured output.

How do you measure the output of a DEI department? Now assume those people automated their jobs with Chat GPT. How would you measure the change in productivity?

Having ridden in a lot of waymo's which can handle SF (urban stuff) and the phoenix area (highways and suburban stuff) perfectly well, I feel quite confident that that could happen right now.

So how to define AGI? I'm not sure economic value factors here. I would lean towards a definition around problem solving. When computers can solve general problems as well as humans, that's AGI. You want to find a drug for cancer, or drive a car, or prove a math theorem, or write a computer program to accomplish something, or whatever problems humans solve all the time. (EDIT: or reason about what problems need to be solved as part of addressing other problems.) There's already classes of problems, like chess, where computers outperform humans. But I mean calculators did that for arithmetic a long time ago. The "G" part is whether or not we have a generalized computer that excels at everything.

We'll have decently smart AIs before we nail down what that G actually means, should mean, absolutely cannot mean, etc. Which is usually what these threads on HN devolve into. Andrej Karpathy is basically side stepping that debate and using self driving as a case study for two simple reasons: 1) we're already doing it (which is getting hard to deny or nitpick about) and 2) it requires a certain level of understanding of things around us that goes beyond traditional automation.

You are dismissing self driving as mere "automation". But that of course applies to just about everything we do with computers. Driving is sufficiently hard that it seems to require the best minds many years to get there and we're basically getting people like Andreij Karpathy and his colleagues from Google, Waymo, Microsoft, Tesla, etc. bootstrapping a whole new field of AI as a side effect. The whole reason we're even talking about AGI is those people. The things you list, most people cannot do either. Well over 99% of the people I meet are completely useless for any of those things. But I wouldn't call them stupid for that reason.

Some people even go as far to say that we won't nail self driving without an AGI. But then since we already have some self driving cars that are definitely not that intelligent yet, they are probably wrong. For varying definitions of the G in AGI.

Except today the bit (which wasn’t really a debate in the sketch because everyone agreed) would start with real current negatives such as accelerating the spread of misinformation and getting artists fired. In your analogy, it would be as if they were asking “what have the Romans ever done for us” during the war. Doesn’t really work.

Misinformation, aka. propaganda, is as old as humanity. Probably even the Romans were whining about that back in the day. AIs are doing nothing new here. And it's not AIs spreading misinformation but people with an agenda that now use AIs as tools to generate it. People like that have always existed and they've always been creative users of whatever tools were available. We'll just have to deal with that as well and adapt.

Which, continuing the analogy, is like watching your neighbour be slaughtered and defending the war by saying we’ll be fine because those who won’t be will eventually die. Sure, in a few generations we could be better off, but there are people living right now to think about. Those who dismiss it are the lucky ones who (think they) won’t be affected. But spare some empathy for your fellow human beings, dismissing their plight because they’ll eventually “grow old and die off” is not a solution and could even be labelled as cruel. Surely you’re not expecting them to read your words and go “yeah, they’re right, I’ll just roll over and die”.

> If anything, people will have a lot more time to do artistic things. More than ever probably and possibly at a grander scale that past generations of artists could only dream about.

That’s an unproven utopian ideal with flimsy basis in reality. The owners of the technology think of one thing: personal profit. If humanity can benefit, that’s a side benefit. It’s definitely not something we should take for granted will happen.

> And it's not AIs spreading misinformation but people with an agenda that now use AIs as tools to generate it.

Correct. And they can do so at a much faster rate and higher accuracy than before. That is the issue. Dismissing that is like comparing a machine gun to a hand gun. The principle is the same but one of them is a bigger problem.

Even if the analogy were wrong, that wouldn’t make the point invalid. I know the point I’m making (and presumably so do you). Again, the analogy is for exemplification, it does not alter the original problem.

Yes, spam is worse with email, but we're still in a better place overall than before in my opinion.

Also, it may be that having powerful AI tools worsens the social problem by normalizing the generated art/misinformation.

I recall Norvig's AI book preaching decades ago that "intelligent" does not mean able to do everything, and that for an agent to be useful it was enough to solve a small problem.

Which in my mind is where the G came from.

And yet we now suddenly go back to the old narrow definition?

I still see no path from LLMs and autonomous driving to AGI.

That is exactly my view too. While LLMs and autonomous driving can be exceptionally good at what they do, they are also incredibly specialist, they completely lack anything along the lines of what you might call "common sense".

For example, (at least last time I looked) autonomous driving largely works off object detection at discreet time intervals, so objects can pop into and out of existence, whereas humans develop a sense of "object permanence" from a young age (i.e. know that just because something is no longer visible doesn't mean it is no longer there), and many humans also know about the laws of physics (i.e. know that if an object has a certain trajectory then there are probabilities and constraints on what can happen next).

I think you're basically right - incrementally automating aspects of one human job. However, it really ought to include AGI since I personally would never trust my life to an autonomous car they didn't have human-level ability to react appropriately to an out-of-training-set emergency.

It reduces it to "Can I fire 50% of my workforce? Then it must be AGI."

Now maybe this definition isn't so useful either, because a lot of work requires a body, to say, move physical goods, which has little to do with "intelligence" but I can see the appeal of looking for some sort of more objective measure of whether you have achieved AGI.

Well, no, that's job automation, and if it's job-specific then it's narrow AI at best (assuming this is a job requiring intelligence being automated, not just a weaving loom being invented), in other words specifically not AGI.

It's really pretty absurd that we've now got companies like OpenAI, Meta, Google (DeepMind) stating that their goal is to build AGI without actually defining what they mean. I guess it let's them declare success whenever they like .. Seems like OpenAI ("GPT-5 and AGI will be here soon!") is gearing up to declare GPT-5, or at least GPT-N as AGI, which is pretty sad.

Then don't call it AGI.

But the marketing types won't like that, will they. So here we go, let's keep hijacking.

OpenAI, back in 2018: https://openai.com/charter

It wasn't particularly controversial at the time - didn't get mentioned in the HN discussion: https://hackertimes.com/item?id=16794194

And a private company trying to hijack a term is not impressive or even merits any discussion. They just willed the term into existence. The rest of us are free to disagree with their "definition".

"These other people are useless, let's bypass them. But not me! I simply gain the ability to get anything I want."

The lack of second-order thinking is hilarious.

If we're all redundant, how do we live? On a pension that starts at ${debatable from conception to adulthood}. Who provides the production on which the pension is spent? The AI.

Even assuming UBI is great (small scale tests say so, but have necessary limits so we can't be sure), there's going to be a huge mess with most attempts to roll out such a huge change.

Now if AGI make people's work redundant, and makes economy grow 100-10000x times... what does that measure mean at all? Can produce lots of stuff not needed or affordable by anybody? So we just hand out welfare tickets to take care of the consumption of the ferocious production, a kind of paperclip-maximizer is doing? I suggest reading the novel Autofac, it might turn out prophetic.

Will that "growth" have any meaning then? Actually the current we print money and give it to the rich economic growth is pretty much this, so with algorithmic trading multiplying that money automatically... have we already achieved that inflection point?

Imagine a list of things many people wish to happen in physical reality. We’ll have more of that.

-Better healthcare

-Curing most things that destroy quality of life

-Curing aging and age-related death

-Much better treatment for all sources of mental suffering

-Far better and cheaper and reversible body modification

-More free time to spend at whatever you want

-Everything much cheaper

-Bigger and better homes and living spaces

-Bigger, faster, cheaper transport

-Easier to organize meaningful social interaction

-Better and more immersive entertainment

-More time to spend with close friends and loved ones

Regardless of if "illness" is or is not a terminological inexactitude, it looks like ageing is a chronic progressive terminal genetic disorder. I think "cure" is an appropriate term in this case.

Funny that this kind of ideological conflict will likely be a key fulcrum of the machine intelligence revolution. We will have a very loud minority that attempts to forcefully prevent all other humans from having the voluntary choice to avoid suffering.

Are you in it?

Or paper wasps: https://www.bloomberg.com/features/2017-biological-markets/

I disagree with the underlying presumption. We've been using animal labour since at least the domestication of wolves, and mechanical work since at least the ancient Greeks invented water mills. Even with regard to humans and incentives, slave labour (regardless of the name they want to give it) is still part of official US prison policy.

Economics is a way to allocate resources towards production, it isn't limited to just human labour as a resource to be allocated.

And it's capitalism specifically which is trying to equate(/combine?) the economy with incentives, not economics as a whole.

> Now if AGI make people's work redundant, and makes economy grow 100-10000x times... what does that measure mean at all?

From the point of view of a serf in 1700, the industrial revolution(s) did this.

Most of the population worked on farms back then, now it's something close to 1% of the population, and we've gone from a constant threat of famine and starvation, to such things almost never affecting developed nations, so x100 productivity output per worker is a decent approximation even in terms of just what the world of that era knew.

Same deal, at least if this goes well. What's your idea of supreme luxury? Super yacht? Mansion? Both at the same time, each with their own swimming pool and staff of cleaners and cooks, plus a helicopter to get between them? With a fully automated economy, all 8 billion of us can have that — plus other things beyond that, things as far beyond our current expectations as Google Translate's augmented reality mode is from the expectations of a completely illiterate literal peasant in 1700.

> Can produce lots of stuff not needed or affordable by anybody?

Note that while society does now have an obesity problem, we're not literally drowning in 100 times as much food as we can eat; instead, we became satisfied and the economy shifted, so that a large fraction of the population gained luxuries and time undreamed of to even the richest kings and emperors of 1700.

So "no" to "not needed".

I'm not sure what you mean by "or affordable" in this case? Who/what is setting the price of whatever it is you're imagining in this case, and why would they task an AI to make something at a price that nobody can pay?

> So we just hand out welfare tickets to take care of the consumption of the ferocious production, a kind of paperclip-maximizer is doing? I suggest reading the novel Autofac, it might turn out prophetic.

Could end up like that. Plenty of possible failure modes with AI. That's part of the whole AI alignment and AI safety topics.

But mainly, UBI is the other side of the equation: to take care of human needs in the world where we add zero economic value because AI is just better at everything.

We probably can't. I mean why stop at humans? Let's give every pet the same luxury, or ... in the limit we could give this to every living being. Ultimately someone is going to draw the line who gets what and who is useful or not "for the greater good".

It just happens that many living beings don't contribute to the goals of whoever is in charge and if they get in the way or cause resource waste nobody will care about them, humans or not.

Human rights and democracy is all cool, but I think we just witnessed enough workarounds that render human rights and democracy pretty much null and void.

Humans have rights insofar they're able to enforce them. Individually by withholding their labor (muscle or brain power), or collectively with pitchforks if need be.

Once labor is dime-a-dozen and pitchforks ineffective (OP's premise of "fully automated economy"), human rights and democracy go the way of dodo, inevitably. Nature loves to optimize away inefficiencies.

Although the "fully automated" bit is quite a stretch at the moment. The end-to-end supply chain required to produce & sustain advanced machinery and AI is too complex, a far cry from "LOL let's buy some GPU and run chatbots".

It's ahead of us, and that's good because we're not ready for it yet either.

But how far ahead? Nobody knows. For all its flaws, ChatGPT's capabilities were the stuff of SciFi three years ago.

We might hit a dead end, or have an investment bubble followed by a collapse, either of which may lead to another AI winter and us doing nothing interesting in this sector for 20 years. Or someone might already have a method of learning as quickly and from as few examples as humans manage, and they're keeping quiet until they figure out how to be sure it's not the equivalent of a dark triad personality in a human.

If I was forced to gamble (which I kinda am by thinking about a mortgage for a new house), I don't think we'll get a complete AGI in less than 6 years at the fastest. My modal guess is 10 years, with a long tail.

Even when we finally get AGI, there's a roll-out period of unclear duration, because the speed of rollout depends in part on how much hardware is needed to run the AGI, but also on the human reaction to it: if it needs the equivalent of a supercomputer, this will definitely be a slow rollout; but it still won't be instant even if it's an app that runs on a smartphone — it's amazing how many people don't know what theirs can already do.

Eh.

A line, drawn somewhere, sure.

Humans being humans, there's a good chance the rules on UBI will expand to exclude more and more people — we already see that with existing benefits systems.

But none of that means we couldn't do it.

Your example is pets. OK, give each pet their own mansion and servants, too. Why not? Hell, make it an entire O'Neill Cylinder each — if you've got full automation, it's no big deal, as (for reasonable assumptions on safety factors etc.) there's enough mass in Venus to make 500 billion O'Neill Cylinder of 8km radius by 32km length. Close to the order-of-magnitude best guess for the total number of individual mammals on Earth.

Web app to play with your size/safety/floor count/material options: https://spacecalcs.com/calcs/oneill-cylinder/

> It just happens that many living beings don't contribute to the goals of whoever is in charge and if they get in the way or cause resource waste nobody will care about them, humans or not.

Sure, yes, this is big part of AI alignment and AI safety: will it lead to humans being akin pets, or to something even less than pets? We don't care about termite mounds when we're building roads. A Vogon Constructor Fleet by any other name will be an equally bitter pill, and Earth is probably slightly easier to begin disassembling than Venus.

I don't. Even in the ideal state: aligned with who? Even if we knew what we were doing, which we don't, it's all the unsolved problems in ethics, law, governance, economics, and the meaning of the word "good", rolled into one.

> Without a doubt, one if the first use cases of the AI will be as a cyberweapon to hack and disrupt critical systems.

AI or AGI? You don't even need an LLM to automate hacking; even the Morris worm performed automated attacks.

> humans are capable of understanding and communicating the long term consequences of AI and unchecked power

The evidence does not support this as a generalisation over all humans: Even though I can see many possible ways AI might go wrong, the reason for my belief in the danger is that I expect at least one such long-term consequence to be missed.

But also, I'm not sure you got my point about humans being treated like pets: it's not a cause of a bad outcome, it is one of the better outcomes.

Kudos, unless we both turn out to be wrong of course.

If we introduce AGI but keep the system, people will be unemployed. If people aren't employed (and instead machines do their jobs), then they can't buy stuff. The whole system crumbles.

But it's possible that AGI will be disruptive enough to completely change the system. Let's hope it's a change for the better.

* Social media is decreasing the average attention span. TikTok is accelerating the trend of people not having time to look past a soundbite or headline in an endless scrolling feed. Intellectual depth and critical thinking vanishes.

* AI deep fakes make truth unknowable. Given the above, the majority of people will take these at face value, or they will give up, because "who can even know what's true anymore?"

* UBI (required because of the coming labor automation revolution) will keep everyone complacent. I'm happy, why would I care who gets elected, or what the government does, as long as I can still buy stuff and eat well?

The logical conclusion is that we fully transition from citizens into a herd of consumers with goldfish attention spans. Voter participation rates plummet. The populace is no longer able to hold government accountable.

There are only two possibilities that result:

1) We now live in a post-scarcity society where everyone self-actualizes and no one wants for anything.

2) We now live in a society where the small % of the population that owns the Replicators self-actualizes and wants for nothing while the remaining 99.9% of the population can f** off and die.

While we can get to a post scarcity society where people can live for free without a job, there are still going to be economies around "liberal arts". You cant realistically say "hey replicator, give me a usb filled with music that I like". You would have to find out which music you like, and random search on this is not really enjoyable, which would then means that there is economic opportunity for discovery, e.t.c.

However, once automation actually starts progressing, without "evil" parties trying to rent seek/get rich, the cost of living will essentially become zero. There is a very real future where the only economies that exist are those that appeal to human emotional side - entertainment, sports, concerts, e.t.c. Everything else is subsidized by the government with tax collected on the former.

For example, imagine a full self driving car trying to get out of a city that's flooding due to heavy rains, while having to compete with people fleeing to higher ground on foot. People can generalize that way but FSD is gonna take a shit, and if you don't know how to drive in that situation so are you.

"works" includes a failure mode of "alert a human and ask them to take over."

> when new areas of problem space are explored.

The problem space is that the "rules of the road" are both legal, technical and social. All of which have internal conflicts as well as conflicts among each other. Anyone who has driven in severe weather has realized this in one way or another.

> For example, imagine a full self driving car trying to get out of a city that's flooding due to heavy rains, while having to compete with people fleeing to higher ground on foot.

Why do I find this easier to imagine in the fictional setting of Elysium than on the real Earth?

People can't do that either. Some years ago there was a massive snowfall in Rome, where it seldom snows ever, people don't generally carry snow chains, and there's few snowplows and such.

Many people reacted by abandoning their cars in the middle of the road, which is basically what I'd expect any FSD vehicle to do.

I do think that once we start to investigate ML/AI structure in the direction of figuring out the correct solution rather than trying to just find functions for control algorithms based on input->output mappings, then a lot of these problems are going to disappear.

Mapping some complex input state to control actions is literally the definition of driving a vehicle.

Driving, at least in the way humans do it, is more then that. We have internal sim running in our head that allows us to deal with conditions that we have never seen before.