>So you can then look at the diff and figure out what the vulnerabilities were.

It doesn't even take reading or understanding the vulnerabilities at all.

You just ask it to write tests and the tests themselves can be copied and pasted as bonafide exploits.