Skip to content
Neural Daily

ResearchNews

CWE-bench v2: Gemini 4 Argon tops vulnerability fixing

🇭🇺 Olvasd el magyarul →

Collinear AI's public 100-task security benchmark asks agents to find and fix vulnerabilities in real repositories. Gemini 4 Argon leads with 67.8% pass@1, just ahead of Claude Opus 5.5.

By: Mikola Máté Patrik · · 2 min read

CWE-bench cover: defensive audit-and-patch benchmark, 100 public tasks, 91 CWEs, Collinear AI
Image: Collinear AI · Press kit · Official thumbnail of the CWE-bench v2 Hugging Face blog post · Source

What was released?

Collinear AI released CWE-bench v2 (Hugging Face), a defensive security benchmark. An agent gets a checkout of a real open-source repository and a short instruction, and has to find and fix a security weakness without breaking existing functionality.

The new version opens up 100 public tasks covering 91 different CWEs. The dataset is on the Hub, and a held-out set stays private so progress can be tracked on tasks no model has seen.

How is it scored?

Scoring is fully programmatic and deterministic, so anyone can reproduce results. For every task the authors checked that the instruction reads like a realistic request rather than an exam, that the verifier checks correct behavior instead of the shape of one fix, and that memorized upstream patches fail. Every task was also reviewed by hand.

What do the results show?

Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% and 78.0%). 95 tasks were solved by at least one model, yet the leader still fails 19% of the set on every attempt.

Collinear AI chart: CWE-bench v2 pass@1 versus average cost per rollout, with DeepSeek V4.1 Flash, Claude Opus 5.5 and Gemini 4 Argon on the Pareto frontier
Collinear AI · Press kit · Official benchmark chart from the CWE-bench v2 Hugging Face blog post · Source

Source: Collinear AI / Hugging Face. Pass@1 versus average cost per rollout on the CWE-bench v2 public set.

On cost, DeepSeek V4.1 Flash, Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier. DeepSeek reaches 57.0% at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92. The full comparison, including API-only models, is on the leaderboard Space.

Why it matters

If you plan to point an AI agent at your own codebase for security audits, this is one of the more concrete yardsticks: real repositories and runnable fixes, not multiple-choice questions. Tasks run as Harbor environments, the whole benchmark launches with one command from the Hub, and the same environments can serve as RL rewards for training defensive agents. Even the best model stalls on one task in five, so it does not replace code review yet.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles