CWE-bench v2 opens 100 security-fix tasks across 91 CWEs; best model still fails 19% every time
Serguey Asael Shinder2 min readCollinear AI has released CWE-bench v2, a benchmark for the defensive side of AI security work. The public set holds 100 tasks that anyone can run. In each one an agent receives a checkout of a real open-source repository and a short instruction, and has to find and fix a security weakness without breaking existing functionality. Scoring is fully programmatic, so results are deterministic and reproducible.
The 100 tasks cover 91 different CWE classes. The team says every task was checked against a list of questions, among them whether the instruction reads like a real request rather than a test, whether the verifier checks behaviour rather than the shape of one particular fix, and whether the verifiers reject memorised solutions such as the known upstream patch. Every task was also reviewed by hand, and a held-out set is kept back for tasks no model has seen.
Eleven models have been evaluated. Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% and 78.0%. Ninety-five tasks are solved by at least one model, but even the leader fails 19% of the set on every attempt. On cost, DeepSeek V4.1 Flash reaches 57.0% pass@1 at $0.04 per rollout, against $1.83 for Opus and $3.92 for Gemini.
The tasks ship as Harbor tasks with a Docker environment and a deterministic verifier, and can be run straight from the Hugging Face dataset. They are also tagged as reinforcement-learning environments, so the same verifiers can reward training of defensive agents.
What it means
The useful number for teams is not the top score but the tail: a fifth of the tasks defeat the best model every time. Automated patching of known weakness classes is getting good, and the remaining failures are where human review still has to sit.