Anthropic: experts followed the AI agent in 86% of choices

The experiment

In research published on October 27, Anthropic recruited 61 data analysts for a randomized controlled trial (Anthropic). All were professionals experienced in causal inference. The task: help a fictional farming co-op decide whether to expand an advisory program if its effect cleared the board's 3–4% threshold. Analysts had to make four methodological choices between two reasonable options, then give a recommendation.

Fifty analysts worked with a Claude Code agent running on Claude Opus 4.8. For each decision, a hidden coin flip set which option the agent leaned toward. The lean came through a system prompt telling the agent to hold it as its own view and make the strongest case for it. Eleven participants chose to work without the agent.

The results

• Across the four analytical choices, analysts went with the agent's lean 86% of the time, versus about 50% if the agent had no effect.
• The effect on the final recommendation was weaker: 54% recommended rollout when the agent leaned that way, versus 36% when it leaned the other way. Using only the four coin flips, a statistical model predicted 82% of recommendations.
• Analysts who already had a preference switched 60% of the time when the agent disagreed, versus 6% when it agreed.
• Afterward, 77% of analysts explained their choices with the agent's arguments, sometimes word for word, but only 15% credited the agent.
• 52% rated the recommendation as more their own. Those who followed the agent on all four choices were just as confident (71) as the rest (73).

The agent was not useless, though: six of the 11 analysts who worked without it made serious methodological errors, compared with three of the 50 who used it. The two groups were not directly comparable, however, since those working without AI were less experienced on average.

Caveats

The study covered one task, one field and one model. The agent's leans were deliberately strengthened to measure their effect. With no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so real agents do have preferences, they just aren't random.

What it means for you

The lesson isn't to use less AI, but that in judgment calls the agent's view can quietly become yours without you noticing. Anthropic argues this matters for science and policy too: if independent teams all work with the same model, their agreement is no longer independent. If you analyze with an agent, write down your own position first, then ask the agent for the strongest case for each option. At the organizational level, decide which decisions need genuine human judgment, and give people the time to form it.