TheoPhysEval: Astra solves 49 of 50 physics problems

What the benchmark measures

The problems deliberately skip the grand open questions. The authors call them turn-the-crank problems: known methods applied to a slightly modified question. Without AI help, a world expert in the relevant subfield would typically need about a day. Experts from neighbouring fields would take much longer (arXiv).

The 50 problems span 11 subfields, from fluid dynamics and biological processes to particle physics and cosmology. Every agent ran two tracks: a realistic track with web search and tools, and an isolated track with local tools but no web search.

The results

On the realistic track, Astra scored 49/50. Opus 5.5 in Claude Code scored 46/50, Gemini 3.8 Flash in Antigravity 34/50, DeepSeek-V4.1-Flash in Codex 27/50 and GLM 5.3 in Claude Code 21/50. On the isolated track, Astra and Opus both scored 46/50, Gemini 31/50, GLM 26/50 and DeepSeek 25/50.

The full realistic run cost about $116 for Astra, $240 for Opus, $95 for Gemini and $37 for DeepSeek. Astra averaged under $3 per problem. The authors ran one API pass per agent, so the scores carry statistical uncertainty.

Internet access made surprisingly little difference. Astra and Opus still scored 92% on the isolated track. Many problems depend on formulas from papers that the models usually did not receive in the prompt. The authors take that as a sign that leading language models have effectively memorised large parts of the theoretical physics literature.

Where they failed, and the limit

Astra's only miss on the realistic track was an inverse problem that asked for an autonomous differential equation producing a given curve. No model solved it. In several other cases where Astra first looked wrong, the experts later found small errors in their own reference solutions. After those fixes, Astra's answer was correct every time.

The authors stress that a concrete, verifiable calculation comes only at the end of the research process. Before that, someone has to build a theory, ask the right questions and, when a problem is too hard, switch to a simpler but still interesting one. Current agents do not yet have that taste. When asked to design new problems, the authors say they get lost in details a human would find boring.

What it means for training

The authors say problems like these are no longer appropriate as first research projects for early-career researchers, unless the goal is explicitly to work through the material. Costs also raise equity questions: TheoPhysEval is still cheap, but harder problems may make a group's budget decide what can be solved. The dataset is on Hugging Face (Hugging Face).