Theres a video of a recent talk Nicolas Carlini gave this past week on Youtube. It’s eye opening. If you don’t believe that LLMs are going to transform the cybersecurity space after watching that I can’t help you.
7 minutes in, he shows the SQLI he found in Ghost (the first sev:hi in the history of the project). If I'd remembered better, I would have mentioned in the post:
* it's a blind SQL injection
* Claude Code wrote an exploit for it. Not a POC. An exploit.
POC generally means “you can demonstrate unintentional behavior”.
“Exploit” means you can gain access or do something malicious.
It’s a fine line. Author’s point is that the LLM was able to demonstrate some malfeasance, not just unintended consequence. That’s a big deal considering that actual malicious intent generally requires more knowhow than raw POC.
Specifically: the exploit extracted the admin's credentials from the database. A blind SQLI POC would simply demonstrate the existence of a timing channel based on a pathological input.
One other commenter asked a decent question - does going lighter (Zig) or harder on memory safety (Rust) confer any meaningful advantages against the phenomenon you describe?
