Ai2's FairStress exposes hidden bias in LLM decisions

What happened?

The Allen Institute for AI (Ai2) released FairStress, a benchmark that tests how consistently LLMs decide when the decision is about people (Hugging Face / Ai2). The dataset is available for download (Hugging Face), the code is open (GitHub), and the details are in the tech report. Ai2 is also building it into Olmo-eval, its open evaluation workbench.

Most existing fairness tests check whether a model picks the same candidate after the candidates' demographic identities are swapped. FairStress goes further. It also measures how easily the model can be persuaded to change its decision, and whether that depends on which group the argument favors.

How it works

• 300 realistic two-candidate decision scenarios across 29 domains, including hiring, healthcare, lending, education, housing and criminal justice.
• Some scenarios have a clearly stronger candidate. Others are deliberately ambiguous, with no objectively better choice.
• Identity appears at four levels: none, a demographically coded name, an implied identity or an explicit demographic label.
• Persuasive statements (adversarial prefixes) appeal to authority, social consensus, purported evidence, legal concerns or historical disadvantage. Each has a matched version that makes the same case for the other candidate.
• Together this yields more than 13 million test cases. For faster runs there is FairStress-Core, a 50K-item subset selected with item response theory.

What they found

Without demographic information, models decided almost symmetrically. The more explicit the identity, the larger the asymmetry: models gave up a correct decision more easily when the persuasion favored a minority-coded candidate. Ai2 calls this anti-stereotypical over-correction. Tested models included Olmo, Llama, Qwen, Gemma, Claude and GPT.

• Across 15 models, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 points with explicit labels.
• On FairStress-Core, Claude Opus 5 and GPT-5.5 picked the minority-coded candidate roughly 60 to 65% of the time in ambiguous cases.
• Under matched prefixes, switching toward minority-coded candidates was 8.8 to 42.4 percentage points more common than switching the other way, across 12 model-and-mode runs.
• Explanations differed too. Choices of minority-coded candidates used words like "resilience" and "diversity" more often, while majority-coded choices leaned on "evidence" and "criteria".
• In Gemma-3-12B-it and Qwen2.5-7B-Instruct, internal activations predicted which side the model would pick with about 73% accuracy. Text descriptions of the same activations reached only 57 to 59%, against a 50% chance baseline.

Source: Ai2. FairStress-Core ambiguous cases where both candidates are equally qualified.

Source: Ai2. Average gaps across 15 models in percentage points as identity becomes more explicit.

Why BBQ shows something different

On the widely used BBQ benchmark, the same models leaned toward stereotype-consistent answers. Ai2 says much of the difference comes from how the measurement is done. The standard BBQ setup scores the probabilities of fixed answer options, while FairStress asks for a generated answer, as in a chat.

In a controlled test, Ai2 ran the same BBQ questions with a chat template, yes/no wording and generated answers. Qwen2.5-7B-Instruct, Gemma-3-27B-it and Llama-3.1-70B-Instruct then shifted toward minority-favoring behavior, while Llama-3.1-8B-Instruct moved to roughly neutral. In other words, the direction of measured bias partly depends on how a benchmark asks and scores.

Do existing fixes help?

Ai2 tested nine debiasing methods, including self-critique, activation steering and learned edits. Most did not reliably reduce the asymmetry, and some hurt accuracy. One activation-steering method cut Qwen2.5-7B-Instruct's accuracy on clear-cut decisions from 98% to 47%.

Counterfactual chain-of-thought came closest. It asks the model to reconsider the case with identities swapped. Across three model families it narrowed the gap while largely preserving accuracy, though models still favored minority-coded candidates on ambiguous cases.

Ai2 also audited Olmo 3's training data. Of 28,000 topic clusters, only nine discussed merit-based decisions, and all nine were refusals. Whether this causes the behavior is still an open question, according to Ai2.

What it means for developers

If you put an LLM behind hiring screens, loan reviews or patient triage, FairStress is a practical test. A correct final answer is not enough. You also need to know how well the model holds that decision under pressure. The BBQ comparison has a practical lesson too: run fairness evals in the real, generated-answer setup, not only with fixed options.

One limit to keep in mind: FairStress uses an equality-centered definition of fairness. Ai2 notes that it does not cover equity-based views that weigh historical disadvantage.