Yes, imagine if 50 burglars showed up at your house and they tried to disassemble every camera, pick every lock and and force open every window for hours until they got in eventually.

Yes, imagine the world in which burglar time was so cheap, that any wannabe master thief would just casually hire 50 burglars and tell them to go house by house and find something interesting.

There are people rich enough to do that in our current world. They don't because either they have a moral compass or because they fear the law.

Or because there's not enough worth stealing in your house.

That would hopefully be a world where every house casually hires 50 burglars to make their house burgle-proof.

When we moved into our current house, I cheated and read this (though only half the book is applicable): https://burglarsguide.com/

Precisely the lesson they end with:

> The individual weaknesses were familiar. A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems.

> Volume is what changes the defensive problem. We were not dealing with one clever exploit or a clean sequence of attacker actions. They had to correlate thousands of low-signal events across several systems while the agent continued testing new paths. The successful path was hidden inside the noise generated by the thousands of failed ones. The same scale changed the investigation: reconstructing 17,600 actions by hand was impractical, and we had to rebuild the timeline, decode the payloads, and inventory the exposed credentials using an AI-assisted pipeline of our own.

> Our learning from this type of attack is that machine-speed offense makes ordinary weaknesses more expensive for defenders. LLM agents bring a step increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret...

> It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads.

Something I've been thinking for a few weeks, that I think is becoming clearer, is we're seeing a sort of natural language unification engine [0][1] with a huge dataset. This part even just sounds like it with the backtracking.

So that could explain why it does certain things really well, but the overall resulting code is often so bad - it's just linking together successful small well-designed steps without regard to the overall structure. Which people have said similar things before in various ways, but not really tied together.

I think the people who say it's just interpolating are seeing this intuitively and can't quite explain it without knowing the right terms. Because from A to Z, it really kind of is, but when Z is even slightly open-ended and there's many possible results or paths, it often ends up with a subpar one.

[0] https://en.wikipedia.org/wiki/Unification_(computer_science)

[1] My only experience here was some Prolog and building a basic one in Scheme ages ago during college, but the idea has kind of stuck with me.

I think the main takeaway from the LLM Agent era is that a lot of human work can be expressed as a search problem: trying approaches that work, and settling on one that accomplishes your goal. Being skilled is akin to having a better heuristic and picking better search paths.

And LLMs can reproduce this as long as the end-result is verifiable in some way, even if it's loose: 'The generated code is secure, if a different LLM failed to breach it in 5M tokens'.