Muir just dropped a 1.34B param model that holds 10M tokens in 25MB of state. No cache explosion, no drift.
The architecture is linear-attention with a learned gain gate that rebalances decaying channels. Old signal gets amplified, loud recent channels get damped. Think of it as adaptive row-weighting before the inner product.
Comparison: equivalent Transformer would need 2.1TB bfloat16 KV cache for the same stream. Their baseline died at 128K tokens. Muir trained on 4K sequences but held perplexity flat to 10M tokens. Passkey recovery at 1M tokens hit 99.4% after a short cooldown.
This is the first modern LLM explicitly targeting Kanerva's 1988 Sparse Distributed Memory constraint: fixed-size state regardless of document length. Not a direct port (Kanerva used binary Hamming space, Muir uses continuous learned gates), but the engineering goal is identical.
Practical implication: your entire email archive, codebase, or scanned docs fit in the same 25MB RAM footprint as the model itself. No linear cache growth, no cloud upsell when your corpus doubles.
Caveats: weights unreleased, no peer review yet, largest model is 1.34B (research scale). Training throughput matched gated-delta baseline, 2.7x faster than Transformer at 32K tokens.
The Fixed Desk test: load a long doc in windows, log RSS and KV cache size. Models obeying the rule hold RSS flat after exceeding training context. Models that fail show linear climb. Passkey check: hide a string early in multi-million-token file, retrieve it at the end.
Kanerva's Desk SuperPrompt: three-field journal (compact state, new observation, erase-then-write instruction). Retrieval by similarity, not position. Fits inside existing laptop context windows.
If this holds up, local archive agents just became economically viable without renting bigger cloud instances every time your data grows.
The architecture is linear-attention with a learned gain gate that rebalances decaying channels. Old signal gets amplified, loud recent channels get damped. Think of it as adaptive row-weighting before the inner product.
Comparison: equivalent Transformer would need 2.1TB bfloat16 KV cache for the same stream. Their baseline died at 128K tokens. Muir trained on 4K sequences but held perplexity flat to 10M tokens. Passkey recovery at 1M tokens hit 99.4% after a short cooldown.
This is the first modern LLM explicitly targeting Kanerva's 1988 Sparse Distributed Memory constraint: fixed-size state regardless of document length. Not a direct port (Kanerva used binary Hamming space, Muir uses continuous learned gates), but the engineering goal is identical.
Practical implication: your entire email archive, codebase, or scanned docs fit in the same 25MB RAM footprint as the model itself. No linear cache growth, no cloud upsell when your corpus doubles.
Caveats: weights unreleased, no peer review yet, largest model is 1.34B (research scale). Training throughput matched gated-delta baseline, 2.7x faster than Transformer at 32K tokens.
The Fixed Desk test: load a long doc in windows, log RSS and KV cache size. Models obeying the rule hold RSS flat after exceeding training context. Models that fail show linear climb. Passkey check: hide a string early in multi-million-token file, retrieve it at the end.
Kanerva's Desk SuperPrompt: three-field journal (compact state, new observation, erase-then-write instruction). Retrieval by similarity, not position. Fits inside existing laptop context windows.
If this holds up, local archive agents just became economically viable without renting bigger cloud instances every time your data grows.