IBM's ALTK-Evolve tops τ²-bench with learned agent memory

What happened

IBM Research showed that an agent built on an older, non-reasoning model can reach the top of a customer service benchmark if it learns from its own mistakes (Hugging Face / IBM Research). The tool is ALTK-Evolve, which is open source (GitHub).

The method is simple. The system reviews how the agent handled training tasks, then uses the same model to distill short behavioral rules from successes and failures. Examples: authenticate the customer first, get explicit approval before any action that changes the database, make only one tool call per turn. The rules are appended to the system prompt as a plain list. No fine-tuning, no human annotation.

What τ²-bench measures

Sierra's τ²-bench (GitHub) tests policy-bound customer service conversations. The agent talks to an LLM-simulated customer, works with tools (look up an order, issue a refund, rebook a flight) and must follow a written policy. A task only counts as solved if the final database state matches the expected one. IBM used the retail and airline domains with 4 trials per task. Pass^4 is the share of tasks the agent solves on all 4 attempts, so it measures reliability.

The numbers

• Retail, GPT-4.1, held-out test tasks: mean accuracy goes from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5% with all 292 learned rules.
• Airline, GPT-4.1: accuracy rises from 57.5% to 65%. Pass^4 is best with the smaller 14-rule set that keeps only rules extracted from at least 3 tasks: 35% to 45%.
• Airline with a GPT-5.5 agent: accuracy improves from 72.5% to 82.5% and Pass^4 from 55% to 70%. Frontier models still have something to learn.
• Leaderboard (full base split, October 12 snapshot): 67.5% on retail with GPT-4.1, ahead of RAFT-30B-A3B, a model fine-tuned on τ²-bench retail data, at 61.4%. On airline, 82% with GPT-5.5, while frontier entries range from 66% to 72%.

Source: IBM Research. Highlighted rows are ALTK-Evolve runs; the rest is a saved τ²-bench leaderboard snapshot from October 12, 2026.

What it costs

The longer prompt means more input tokens: on retail with the full rule list, cost per conversation rises from about $0.084 to $0.139 (agent and simulated customer combined). Conversations do not get longer, though. They get slightly shorter, 10.16 turns instead of 10.59 on retail. Since the rule block is identical on every call, prompt caching can recover most of the extra cost.

Why it matters for developers

If you build agents that follow a policy, this is a practical pattern: a rule list mined from your own trajectories is cheaper and easier to swap than fine-tuning. IBM says the right amount of guidance differs by domain, so tune it on your own tasks instead of hard-coding it. One caveat: the evaluation covers two domains with 20 and 40 test tasks, and the leaderboard results are IBM's own submissions.