Skip to content
Neural Daily

ResearchNews

Zyphra's ZHA makes the sparse attention search sparse too

🇭🇺 Olvasd el magyarul →

Zyphra's ZHA organizes keys into a tree, so the sparse attention indexer no longer scans the whole context. It can be retrofitted onto existing models with about 4 billion tokens of training.

By: Mikola Máté Patrik · · 2 min read

ZHA: Zyphra Hierarchical Attention announcement image
Image: Zyphra · Press kit · Official social image of the Zyphra research post · Source

What happened?

Zyphra introduced Zyphra Hierarchical Attention (ZHA) (Zyphra). It targets a known gap in sparse attention: the final attention reads only a few keys, but the indexer that picks them still scores the entire context. Cited work shows indexing takes 60 to 90 percent of layer time near 1 million tokens.

How it works

ZHA splits keys into contiguous chunks and compresses each chunk into a summary key. Summaries are grouped and compressed again, level by level. At decode time, a query scores the top-level nodes and expands only the most promising branches, down to raw KV entries. The full KV cache is kept; the tree only decides which keys each query reads.

The method sits on top of a frozen base model, and only a small set of learned reference queries trains. For Qwen3.6-35B-A3B that is 81,920 parameters, trained on about 4 billion tokens at 32K context (MarkTechPost).

The numbers

All numbers are self-reported by Zyphra (arXiv). On Qwen3.6-35B-A3B, ZHA stays within a few points of dense attention on seven reasoning, coding and retrieval benchmarks. On LongBench v2, the deep tree scored 63.4% while reading about 1.1% of keys, versus 58.0% for dense attention. On RULER-13, both trees stayed above 93%, while a MiniMax Sparse Attention indexer trained on ten times as many tokens dropped to 77.1%.

There is a weak spot: on the NoPE-based Nemotron-3.5-Lightning-30B-A3B, the deep tree falls 25.3 points below dense attention on RULER-13. No optimized kernels were released, and speed was only modeled: at 256K tokens, Zyphra estimates 1.93x to 2.28x over dense attention.

Who should care

Teams working on long-context inference should keep an eye on it. The method is cheap to retrofit onto existing models, but without real kernels and independent measurements the speedup exists only on paper for now.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles