It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
Correct. Basic security practices would have stopped most of this. Thus what is most insightful here is less that the attack was sophisticated but more that Hugging Face was running a setup that, by standards that are the norm elsewhere, would appear amateurish.
You might be able to say that HF made some amateurish mistakes, but this attack certainly was fairly sophisticated. It relied a number of separate exploits or vulnerabilities that it combined and used over the course of multiple days. Calling it otherwise seems to be downplaying it.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
> It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
In many if not most environments where I have seen these used, they are used to bridge an access boundary and are absolutely security critical. Frequently (and probably in this application, too), they are used for security enforcement in an architecture where CI doesn't have Internet access, but does need to download packages. This is a pretty big deal for a lot of big companies; historically they have relied on stuff like package proxies to help protect them from supply chain compromises like post-install script exploits by preventing CI from being able to detonate an arbitrary Internet payload; if a supply-chain worm were to incorporate this exploit, it would be a lot scarier and more potent. Actually, from the supply-chain worm standpoint, this is arguably a bigger finding than a Docker escape - I suspect we can look forward to more shenanigans in this space coming soon.
The deeper question though is why is someone running a model that’s supposedly so dangerous in an environment that can even get access to the internet. That just appears reckless.
There are ways to test supposedly dangerous things. What was on display here looks more amateur hour than serious testing.
My conspiracy theory is they purposely run these scary cyber intrusion tests on low security environments to bolster demand for LLM based cyber research, audits, and tooling.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
Well the fun part of AI is that the tools those "battle tested" sandboxes use are also becoming vibe coded more and more.
For example, Linux is accepting AI code. So if your sandbox depends on Linux kernel features, your "battle tested" sandbox is now partially vibe coded too.
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?
The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.