Hugging Face's tiny-model contracts catch Transformers bugs

What happened?

On November 3, Hugging Face introduced transformers-mic, a proposed standard for packaging model integration tests in the Transformers library (Hugging Face). It started with a silent bug: on July 30, enabling SDPA attention for T5 let UMT5's decoder attend to future tokens, so during training it could peek at the answer. Nothing crashed, and the bug stayed in main for almost two months.

Transformers supports over 750 model types. Main runs more than 88,000 CPU tests, plus more than 200,000 GPU tests in a daily run. Running all of that on every PR would keep a lot of GPUs busy, so a regression often only shows up in a scheduled run, after it has already landed in main.

How does a MIC work?

A contract has three parts. The fixture is a shrunk model built from the real checkpoint's configuration, with fewer and narrower layers but every layer kind, and unchanged tokenizer and processor files. The usage path is a runnable Markdown example of a common task, such as generating text or running a training step. The baselines are that example's recorded outputs, reviewed by a human.

The contract holds if the usage path still produces the baselines on the fixture. On a PR, MICs are picked based on the files the patch changes. The experimental UMT5 fixture has 66 MB of weights, against 1,227 MB for the source checkpoint. Llama 4 Scout shrinks from 108.6 billion parameters to 44.7 million, with the vision stack cut from 34 layers to 2 and the language stack from 48 to 4, keeping its MoE routing style.

Replaying five months of commits, the small UMT5 fixture flagged the bug on every commit from its introduction until the September 28 fix. On a proof branch, the T5 contract's runner step took 13.4 seconds on CPU, excluding setup. One detail mattered: the bug only showed up on the default SDPA attention path, while the eager variant missed it.

Agents and overfit weights

The tmic CLI creates the contract and builds the fixture. For a new model, an LLM drafts the usage code and manifest, but it does not approve anything: changed expectations need human review. For Whisper, the random-weight fixture barely responded to audio changes, so the team overfit it on a few tones. In Hugging Face's study, random fixtures caught 18 of 24 injected faults, while overfit ones caught all 24.

What does it mean in practice?

As of November 1, the MIC job in Transformers CI reports failures without blocking PRs. Open questions remain: how many real bugs it catches on PRs, and how many false alarms it raises. If you integrate your own model into Transformers, you can run the tmic workflow locally without uploading anything. Hugging Face also wants Hub model pages to link to their MICs, so a developer or an agent could start from an example with tested, known outputs.