Skip to content
Neural Daily

ResearchNews

Memorilla: 16 memory tokens for a whole document store

🇭🇺 Olvasd el magyarul →

Stanford's SNAP group has open-sourced Memorilla, a memory module for frozen LLMs. It turns an entire document collection into exactly 16 memory tokens per query. On FactKG that beats putting every document in context, with 108x fewer tokens.

· 3 min read

A Memorilla architektúrája: Query és Document Encoder, K tanult latens, cross-attention rétegek és K memória latens, amelyek a befagyasztott LLM bemenetébe kerülnek
Image: Stanford University · Screenshot · Source

What is Memorilla?

Memorilla is a trainable memory module that sits in front of a frozen LLM (Stanford). A frozen encoder turns each document, such as a past chat or a book chapter, into an embedding once and stores it in a persistent memory store. For each question, learned memory slots read the question and the whole store and become 16 memory tokens, placed directly in the LLM's input embedding space.

The difference from RAG: instead of putting selected documents into the prompt as text, the module learns what the question needs from the entire collection. In the published setup the encoder is Qwen3-Embedding-4B and the decoder Qwen3-8B, both frozen. Only the 194M-parameter memory module is trained (MarkTechPost).

Pontdiagram: átlagos pontszám a tokenszám függvényében, a Memorilla 16 tokennel a Pareto-front tetején, a RAG, Mem0, MemoRAG és teljes kontextus alatta
Stanford University · Screenshot · Source

Source: Memorilla, Stanford University (arXiv)

What the numbers show

  • Across eight benchmarks Memorilla has the best average, ahead of Mem0 and MemoRAG, at 49.68, and tops five of the eight (
  • [object Object]
  • ).
  • FactKG: 84.33% accuracy versus 80.43% with full context, using 108x fewer document tokens.
  • PersonaMem-v2: 44.60% versus 27.92% for full context.
  • Latency: on one H100 with 512 documents (about 128K tokens) it answers in 0.74 seconds; full context takes 19.9 seconds, 27x slower.
  • TextWorld agent: success rises from 18.6% to 24.2%, as the agent wanders less between rooms it has already explored.

It does not win everywhere. On PubMedQA it scored 59.00% against 69.00% for LLMLingua-2, a sign that compression still loses detail on specialist text.

Before you build on it

The memory module was trained in stages, from Wikipedia paraphrasing through general question answering and summarization to the benchmarks' training splits. So the strong numbers come partly from task-tuned modules. The memory tokens are produced for Qwen3-8B's embedding space, so the published checkpoints belong to that decoder.

You need Linux, CUDA, Python 3.11 and one 80 GB GPU; the first run downloads about 25 GB of models (GitHub). Code, training recipes and checkpoints are available under Apache 2.0. No quantized variants have been released yet.

Why it matters

Long context is one of the main costs of agentic systems: every turn pays for the full history again. Memorilla reads at a fixed cost of 16 tokens, and the decoder's load does not grow with the number of documents. If this carries over to other models, memory moves from prompt assembly to a learned component. More on agent memory: Anthropic's agent memory building blocks and the FrontierMemory benchmark.

Key takeaways

  • 16 memory tokens instead of a whole document store for a frozen LLM.
  • More accurate than full context on FactKG with 108x fewer tokens.
  • 27x faster answers at 512 documents.
  • Apache 2.0, but tied to Qwen3-8B and an 80 GB GPU for now.

Related articles