Article reprinted from: Machine Heart
Edit: Egg sauce
If you have experienced communication with any conversational AI robot, you can definitely recall some extremely "frustrating" moments. For example, the main points you mentioned in the conversation the day before were completely forgotten by the AI...
This is because most current LLMs can only remember limited context, just like students who cram for an exam, and will be exposed if questioned.
Imagine if an AI assistant could contextually reference conversations from weeks or months ago during a chat. Or, if you could ask an AI assistant to summarize a report that’s thousands of pages long, wouldn’t that be an enviable capability?
In order to make LLM remember more and better, researchers are constantly working hard. Recently, researchers from MIT, Meta AI, and CMU proposed a method called "StreamingLLM" that enables language models to smoothly process endless texts.
Paper address: https://arxiv.org/pdf/2309.17453.pdf
Project address: https://github.com/mit-han-lab/streaming-llm
StreamingLLM works by identifying and saving the model's inherent "attention sinks" to anchor the initial token of its reasoning. Combined with a rolling cache of recent tokens, StreamingLLM's reasoning speed is increased by 22 times without sacrificing any accuracy. In just a few days, the project has gained 2.5K stars on the GitHub platform:
Specifically, StreamingLLM enables language models to accurately remember the touchdown score of the last game, the name of a newborn, a lengthy contract, or the content of a debate, just like upgrading the memory of an AI assistant so that it can perfectly handle heavier tasks.
Now let’s look at the technical details.
Method innovation
Typically, LLMs are limited by the attention window during pre-training. Although there have been many previous works to expand this window size and improve training and inference efficiency, the acceptable sequence length of LLM is still limited, which is not friendly for long-term deployment.
In this paper, the researchers first introduced the concept of LLM streaming applications and raised a question: "Can LLM be deployed with infinitely long inputs without sacrificing efficiency and performance?"
There are two main challenges when applying LLM to infinite-length input streams:
1. During the decoding phase, transformer-based LLM caches the Key and Value states (KV) of all previous tokens, as shown in Figure 1 (a), which may lead to excessive memory usage and increase decoding latency;
2. The existing model has limited length extrapolation capabilities, that is, when the sequence length exceeds the attention window size set during pre-training, its performance will degrade.
One intuitive approach is called Window Attention (as shown in Figure 1 b). This approach only maintains a fixed-size sliding window on the KV state of the most recent token. Although it ensures stable memory usage and decoding speed after the cache is full, the model will collapse once the sequence length exceeds the cache size, or even just evicts the KV of the first token. Another approach is to recalculate the sliding window (as shown in Figure 1 c). This approach rebuilds the KV state of the most recent token for each generated token. Although it has powerful performance, it requires calculating secondary attention within the window, so the speed is significantly slower and is not ideal in actual streaming applications.
In the process of understanding the failure of window attention, the researchers discovered an interesting phenomenon of autoregressive LLM: as shown in Figure 2, a large number of attention scores are assigned to the initial tokens, regardless of whether these tokens are relevant to the language modeling task.
The researchers call these tokens "attention pools": despite their lack of semantic meaning, they occupy a large amount of attention scores. The researchers attribute this phenomenon to Softmax (which requires the sum of attention scores of all context tokens to be 1), even if the current query does not have a strong match in many previous tokens, the model still needs to assign these unwanted attention values somewhere so that they sum to 1. The reason why the initial token becomes a "pool" is intuitive: due to the properties of autoregressive language modeling, the initial token is visible to almost all subsequent tokens, which makes them easier to train as attention pools.
Based on the above insights, the researchers proposed StreamingLLM, a simple and efficient framework that allows attention models trained with limited attention windows to process infinitely long texts without fine-tuning.
StreamingLLM takes advantage of the fact that attention pools have high attention values, and retaining these attention pools can make the attention score distribution close to a normal distribution. Therefore, StreamingLLM only needs to retain the KV values of the attention pool tokens (only 4 initial tokens are needed) and the KV values of the sliding window to anchor the attention calculation and stabilize the performance of the model.
Using StreamingLLM, models including Llama-2-[7,13,70] B, MPT-[7,30] B, Falcon-[7,40] B, and Pythia [2.9,6.9,12] B can reliably simulate 4 million tokens or even more.
Compared to the only feasible baseline, recomputing the sliding window, StreamingLLM is 22.2 times faster without any performance loss.
Evaluation
In the experiment, as shown in Figure 3, the perplexity of StreamingLLM is comparable to that of the Oracle baseline (recalculating the sliding window) on a text with a span of 20K tokens. At the same time, when the input length exceeds the pre-training window, the dense attention fails, and when the input length exceeds the cache size, the window attention gets into trouble, causing the initial token to be removed.
Figure 5 further confirms that StreamingLLM can reliably handle text of unusual size, including more than 4 million tokens, across a variety of model families and scales. This includes Llama-2-[7,13,70] B, Falcon-[7,40] B, Pythia-[2.8,6.9,12] B, and MPT-[7,30] B.
The researchers then confirmed the "attention pool" hypothesis and showed that language models can be pre-trained and only require a single attention pool token when deployed in a streaming manner. Specifically, they proposed adding an additional learnable token at the beginning of all training samples as a designated attention pool. By pre-training a 160 million parameter language model from scratch, the researchers showed that this method can maintain the performance of the model. This is in stark contrast to current language models, which require the reintroduction of multiple initial tokens as attention pools to achieve the same level of performance.
Finally, the researchers compared the decoding latency and memory usage of StreamingLLM with recalculating the sliding window and tested it on a single NVIDIA A6000 GPU using the Llama-2-7B and Llama-2-13B models. As shown in Figure 10, the decoding speed of StreamingLLM increases linearly with the increase of cache size. The decoding latency of the latter increases quadratically. The experiments show that StreamingLLM achieves impressive speedups, up to 22.2 times faster per token.
For more research details, please refer to the original paper.
