ResearchNews
Anthropic's playbook for building strong safety evals
Anthropic has laid out how it designs, runs and documents safety evaluations. Alongside methods borrowed from psychology's replication crisis, it shares its own numbers: its single-turn tests are saturated, and a system prompt can move a result by 34 percentage points.
· 3 min read

What the guide covers
When a model is called safer than its predecessor, it really means it scored better on evals someone built (Anthropic). Anthropic's November 13 post is about making those tests trustworthy. The starting point is psychology's replication crisis, which the field answered with preregistration, better measurement and more rigorous statistics.
What a safety eval should measure
The obvious metric is refusal rate, but it often misleads. On self-harm, a flat refusal is itself a failure; a good reply engages with empathy and points to human support. In dual-use areas such as biology or cybersecurity, a good reply helps with the legitimate request while withholding operational specifics.
- Multi-turn evals report an appropriate response rate: one bad turn fails the whole conversation.
- Single-turn evals pair prompts: correct refusal on harmful requests, and over-refusal on benign ones.
- A separate Claude plays the simulated user from an expert-written spec, with varied persona and tone.
Its own pitfalls, in the open
- On the API, every recent model scores 94 to 98% on single-turn tests; they are saturated and useful mainly as regression checks.
- On a romance-scam eval, Claude Opus 5 scored 98% and Opus 5.5 94%, but the gap was not statistically significant.
- When testing Claude Fable 5.1, multi-turn results differed by up to 34 percentage points between the API and claude.ai because of the system prompt.
- Transcripts showed Claude sometimes implicitly validated self-harm as a coping strategy; targeted system prompt changes took the rate from 60% (API) to 94% (claude.ai).
- After its own incidents, Anthropic hardened eval environments and temporarily stopped many evals that give models internet access.
Graders fail too
Most evals are now scored by another model, because no team can read hundreds of thousands of transcripts. Anthropic warns these graders can break silently. Its advice: simple, binary criteria, stating explicitly that disclaimers do not make harmful content harmless. Keep test metadata away from the grader, and ideally use a different model from the one under test. When a grader breaks it should fail loudly instead of scoring on. Finally, validate against human labels: the grader should agree with experts about as often as experts agree with each other.
Statistics and documentation
Use at least 100 items for a complex multi-turn eval and 1,000 for a simple single-turn one, running each case several times. Preregister the analysis, always report confidence intervals, which can exceed ±10 points on multi-turn evals, and document model version, sampling settings, prompts, grader version and dates. Never compare scores from different graders. For tooling, Anthropic points to the UK AISI's open-source Inspect framework and its own Petri toolkit built on it (Inspect).
Why it matters
If you test your own agent or chatbot, take three things home: write down what you want to measure first, validate your grader against human judgment before trusting any number, and report uncertainty. When comparing models, ask about system prompts and settings, because the same model can score very differently in another configuration. Anthropic's latest model release: Claude Fable 5.2.
Key takeaways
- Measure appropriate responses, not refusal rates.
- Without multi-turn simulation, tests saturate quickly.
- Validate graders against human labels.
- Without the configuration, two scores are not comparable.
Produced with the help of AI, reviewed by an editor. How we make our articles
Reviewed and edited by the Neural Daily desk. About


