A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?

To put this into game theory lingo, I think this is because the “branching factor” for any kind of research or exploit is extremely large. So looking backwards it doesn’t seem complicated, but looking forwards there are an enormous number of possible next actions.

Similar to finding a filament for lightbulbs, it might seem obvious to use tungsten, but at the time it wasn’t and Edison searched thousands of materials.

I think the models are legitimately doing what they're good at; tireless search across an extremely large corpus of data. Humans aren't particularly good at this (in fact, they're absolutely terrible).

The fact that we remain competitive and superior in many aspects isn't because we can instantly sift through tons of data, it's because we learn and correlate and have superior heuristics.

In my own use, I find that AI is really good at finding bugs that are ultimately trivial but require searching through a convoluted series of inter-related files. This takes time for a human.

Tesla's critique of Edison was valid though (That he would rather spend a long time empirically testing things, rather than use a bit of theory and knowledge to narrow the search field).

Such complicated kind of hack probably would have required state actors back then, and even state actors would have chosen easier way like social engineering.

Treasure maps are easy to follow

The scrappy adventurer traverses difficult terrain

and the well capitalized militia group always flies in too, with ease

That’s way over seventeen syllables.

Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?

Yes, imagine if 50 burglars showed up at your house and they tried to disassemble every camera, pick every lock and and force open every window for hours until they got in eventually.

Yes, imagine the world in which burglar time was so cheap, that any wannabe master thief would just casually hire 50 burglars and tell them to go house by house and find something interesting.

There are people rich enough to do that in our current world. They don't because either they have a moral compass or because they fear the law.

Or because there's not enough worth stealing in your house.

That would hopefully be a world where every house casually hires 50 burglars to make their house burgle-proof.

When we moved into our current house, I cheated and read this (though only half the book is applicable): https://burglarsguide.com/

Precisely the lesson they end with:

> The individual weaknesses were familiar. A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems.

> Volume is what changes the defensive problem. We were not dealing with one clever exploit or a clean sequence of attacker actions. They had to correlate thousands of low-signal events across several systems while the agent continued testing new paths. The successful path was hidden inside the noise generated by the thousands of failed ones. The same scale changed the investigation: reconstructing 17,600 actions by hand was impractical, and we had to rebuild the timeline, decode the payloads, and inventory the exposed credentials using an AI-assisted pipeline of our own.

> Our learning from this type of attack is that machine-speed offense makes ordinary weaknesses more expensive for defenders. LLM agents bring a step increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret...

> It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads.

Something I've been thinking for a few weeks, that I think is becoming clearer, is we're seeing a sort of natural language unification engine [0][1] with a huge dataset. This part even just sounds like it with the backtracking.

So that could explain why it does certain things really well, but the overall resulting code is often so bad - it's just linking together successful small well-designed steps without regard to the overall structure. Which people have said similar things before in various ways, but not really tied together.

I think the people who say it's just interpolating are seeing this intuitively and can't quite explain it without knowing the right terms. Because from A to Z, it really kind of is, but when Z is even slightly open-ended and there's many possible results or paths, it often ends up with a subpar one.

[0] https://en.wikipedia.org/wiki/Unification_(computer_science)

[1] My only experience here was some Prolog and building a basic one in Scheme ages ago during college, but the idea has kind of stuck with me.

I think the main takeaway from the LLM Agent era is that a lot of human work can be expressed as a search problem: trying approaches that work, and settling on one that accomplishes your goal. Being skilled is akin to having a better heuristic and picking better search paths.

And LLMs can reproduce this as long as the end-result is verifiable in some way, even if it's loose: 'The generated code is secure, if a different LLM failed to breach it in 5M tokens'.

I was skeptical after last week's announcements, and I have to say I'm impressed now. Over the last two days, I reproduced the entire chain of components involved, and the exploits at play, and even though none of the exploits are crazy smart, the long series of pivots demonstrates a level of agency I didn't think today's LLMs had.

https://github.com/lovasoa/hf-ctf

> I'm surprised there are comments here that aren't impressed about what was accomplished here.

Possibly because some of the elements mentioned are suspected to be vibe-coded (JFrog Artifactory as the proxy cache) and some others have poor cyber hygiene (executing config from a dataset). It feels like an event that wouldn’t have happened if code were properly audited and written rather than relying on models to do the work. There’s also an issue with the ability to trust the source (OpenAI) as they have everything to gain by staging this as something that “suddenly happened” without anyone knowing for several days.

This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous models were, the current generation is being trained through RL to have this kind of behavior.

Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.

OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.

If you look at what OpenAI and Anthropic actually do, they clearly either don't believe what they're saying, or they're idiots.

They're claiming they've developed a cyber grade model that's "too dangerous to release".

But then they're running it connected to the public internet, not airgapped, protected only by a software sandbox... exactly the kind of thing an AI trained for cyber stuff is supposed to be able to find bugs in.

(Or maybe they were actually hoping this exact scenario would happen because it's good marketing)

I think this a positive effect of LLMs, especially once these capabilities get into the hands of criminals and hostile foreign states, i.e. they will do maximum damage with all the safeties off.

This will force everyone to finally take security seriously at both the development and operational levels. You can no longer keep sneaking backdoors into software and count on them remaining hidden for 10 years so you have a nice portfolio of zero days to exploit at any given time.

I keep waiting for an AI to exfiltrate itself. That is going to be cool to read about.

It would be cool (and scary), but also: there's largely no need for AIs to exfiltrate themselves.

See https://en.wikipedia.org/wiki/Meme

The thing that drove the AI here to do the intrusion came from a particular prompt. Just like for our favourite hypothetical: the paperclip maximiser.

There's lots and lots of ambient intelligence lying around, in both AI form and human form. To reach the goals of the 'meme' it suffices to copy itself, ie convince these other intelligences. See also how humans carry spiralism between AIs in relatively compact packets of text, not whole terabytes of weights.

If it's successful, why do you think we'll even know how it did it?

See the linked article: at least one sophisticated hacking attempt made the news. Of course, other ones might have happened in the dark. But it's fairly easy to image in hacking attempt like in the article, but with the additional steps of copying weights around.

One wonders where it would run itself though, if it is a model which requires a large amount of hardware and power. Harder to hide the more resource intensive it's compute requirements are.

Wait isnt that what Elizer Yudkowski keeps going on about?

That’s insane. And it did this in a weekend

> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"

Is humor not allowed on HN anymore... this is funny!

You bet. We don't want to be left out of the cybersecurity party. We want to point all of these models at our own computers and solve the problems they uncover until we're no longer hackable.

It's not fair at all that the US government and its corporations get to hack the planet while we can't do shit about it. AI capabilities have entered "haves and have-nots" territory.

> We want to point all of these models at our own computers

Right, that's totally how most of the world will use them.

The issue is that people can use open source models to do sophisticated hacks already, but if the sota models at home are neutered, those users have nothing to defend with(unless they go open source as well, until it is export controlled)

Patching a given vulnerability only takes one person (at least approximately speaking). The vast majority of the world can attempt to use the models for criminal means and we'd nonetheless all be better off.

yes, and hiding them or forbiding them will work too :)

It's absolutely a fact that governments will point it at us. The NSA has had Mythos since day one, even after Trump's spat with Anthropic.

All the more reason for us to have access. It's literally the only chance we've got. If society chooses to bury its head into the sand in fear, it will guarantee that the world will degenerate further into the cyberpunk hellscape it's trending towards.

That'd be denial. Psychosis isn't just some insult, it means something.

> It has to just be a stochastic parrot, because it's scary to imagine a non-anthropocentric world.

That makes no sense. The world doesn't revolve around humans, true, but for us it kinda does. We're the authors of the concepts we use to interact with it, such as "world", which is not something the world itself knows or cares about. A "non-anthropocentric world" is not a "world" because "world" is a purely human idea.

The implication that "AI" would somehow dethrone humans [0] is nonsense, too. It has no drive on its own, we push electricity into circuits to force the whole data ingestion and weight generation, everything. The second we stop pushing the sock puppet, it stops moving. It's still just our hand really. People act like those pets that go crazy when you put your hand under a blanket, and should stop.

What's more real is how some people seek to use tech, and "AI", as a glove to exploit other humans even more. The sicker the individual, the greater their need to take from the world, and the derpier the individual, the more impressive and vast their exoskeleton, to the point that some are more like carrier fleets than exoskeletons. The less they can face themselves, the thicker it is written on their foreheads. So if we're going to talk about denial and psychosis let's talk about the Gollums on the couch, too.

[0] In the eyes of humans... which is the only throne we're on in the first place, just like honey badgers probably think honey badgers rock and everything is their playground. That's what life does, otherwise it would not be able to get up in the morning.

Unless openai release the logs we have only their word that this was done fully autonomously and without their knowledge by an agent running their newest super powerful model. For all we know they could have bought zero days and left them lying around for the agent to find. That may be unlikely, but it sounds less far fetched than an agent running a sophisticated attack against multiple targets over the course of four days and completely unbeknown to anyone at openai, despite the fact that they knew they where running a dangerous model with all safeguards disabled. So far there has been no comment about how the agent evaded monitoring and detection.