This Select-TTT paper on arXiv tackles a very specific problem in long-context question answering: In-Place TTT writes every text chunk into the fast weights with the same strength, so irrelevant content dilutes information relevant to the question. The authors say that, on real-world web-based long-context QA, its original benefits largely disappear.
Select-TTT’s approach adds no parameters and requires no supervision. Instead, it reads signals directly from the model’s frozen output head: it checks how much probability each chunk’s hidden state assigns to words in the question, then uses a temperature-scaled softmax to convert that into a learning-rate multiplier. This only adjusts the strength of each write; the chunk-parallel computation remains unchanged. It delivers consistent gains in long-context QA across two benchmarks and two backbone models.
This is a preprint and has not undergone peer review; the original abstract does not state the size of the gains. If you work on long-context RAG or long-task agents, this idea is worth a look: let the question determine what gets written, rather than accepting everything indiscriminately.
$BTC
Select-TTT’s approach adds no parameters and requires no supervision. Instead, it reads signals directly from the model’s frozen output head: it checks how much probability each chunk’s hidden state assigns to words in the question, then uses a temperature-scaled softmax to convert that into a learning-rate multiplier. This only adjusts the strength of each write; the chunk-parallel computation remains unchanged. It delivers consistent gains in long-context QA across two benchmarks and two backbone models.
This is a preprint and has not undergone peer review; the original abstract does not state the size of the gains. If you work on long-context RAG or long-task agents, this idea is worth a look: let the question determine what gets written, rather than accepting everything indiscriminately.
$BTC
