ResearchNews
Memorilla: 16 memory tokens for a whole document store
Stanford's SNAP group has open-sourced Memorilla, a memory module for frozen LLMs. It turns an entire document collection into exactly 16 memory tokens per query. On FactKG that beats putting every document in context, with 108x fewer tokens.
· 3 min read

What is Memorilla?
Memorilla is a trainable memory module that sits in front of a frozen LLM (Stanford). A frozen encoder turns each document, such as a past chat or a book chapter, into an embedding once and stores it in a persistent memory store. For each question, learned memory slots read the question and the whole store and become 16 memory tokens, placed directly in the LLM's input embedding space.
The difference from RAG: instead of putting selected documents into the prompt as text, the module learns what the question needs from the entire collection. In the published setup the encoder is Qwen3-Embedding-4B and the decoder Qwen3-8B, both frozen. Only the 194M-parameter memory module is trained (MarkTechPost).

Source: Memorilla, Stanford University (arXiv)
What the numbers show
- Across eight benchmarks Memorilla has the best average, ahead of Mem0 and MemoRAG, at 49.68, and tops five of the eight (
- [object Object]
- ).
- FactKG: 84.33% accuracy versus 80.43% with full context, using 108x fewer document tokens.
- PersonaMem-v2: 44.60% versus 27.92% for full context.
- Latency: on one H100 with 512 documents (about 128K tokens) it answers in 0.74 seconds; full context takes 19.9 seconds, 27x slower.
- TextWorld agent: success rises from 18.6% to 24.2%, as the agent wanders less between rooms it has already explored.
It does not win everywhere. On PubMedQA it scored 59.00% against 69.00% for LLMLingua-2, a sign that compression still loses detail on specialist text.
Before you build on it
The memory module was trained in stages, from Wikipedia paraphrasing through general question answering and summarization to the benchmarks' training splits. So the strong numbers come partly from task-tuned modules. The memory tokens are produced for Qwen3-8B's embedding space, so the published checkpoints belong to that decoder.
You need Linux, CUDA, Python 3.11 and one 80 GB GPU; the first run downloads about 25 GB of models (GitHub). Code, training recipes and checkpoints are available under Apache 2.0. No quantized variants have been released yet.
Why it matters
Long context is one of the main costs of agentic systems: every turn pays for the full history again. Memorilla reads at a fixed cost of 16 tokens, and the decoder's load does not grow with the number of documents. If this carries over to other models, memory moves from prompt assembly to a learned component. More on agent memory: Anthropic's agent memory building blocks and the FrontierMemory benchmark.
Key takeaways
- 16 memory tokens instead of a whole document store for a frozen LLM.
- More accurate than full context on FactKG with 108x fewer tokens.
- 27x faster answers at 512 documents.
- Apache 2.0, but tied to Qwen3-8B and an 80 GB GPU for now.
Produced with the help of AI, reviewed by an editor. How we make our articles
Reviewed and edited by the Neural Daily desk. About


