Reprinted from: AI Trends
Original: AIGC Open Community
The Massachusetts Institute of Technology and the Chinese University of Hong Kong jointly introduced LongLoRA, a new fine-tuning method that enhances the context capabilities of large language models without requiring extensive computing resources.
Generally, increasing the context-processing capability of a large language model requires more computing power. For example, expanding the context length from 2,048 to 8,192 requires 16 times more computing power.
LongLoRA was tested on the open-source LLaMA 2 models (7B/13B/70B), extending their original context length to 32K, 64K, and 100K while requiring very few computing resources.
Open-source repository: https://github.com/dvlab-research/LongLoRA
Paper: https://arxiv.org/abs/2309.12307
LongLoRA’s efficient fine-tuning method
According to the LongLoRA paper, efficient fine-tuning is achieved in two main steps. First, during training, a simpler form of attention (focused on specific information) is used, which the developers call Shift Short Attention (S2-Attn).
This new attention method helps save a great deal of computing power and is almost as effective as conventional attention, playing an important role during training.
Second, the researchers revisited an effective method for expanding the context (the amount of information used for training). The developers found that a method called LoRA works very well for this, especially when used with trainable embeddings and normalization.
LongLoRA has shown excellent results across a range of tasks and can be used with LLMs of different sizes. It can increase the amount of data used for training from 4k to 100k, and for another model, to 32k—all on a single powerful computer. In addition, it is highly compatible with other existing techniques and does not change the original model architecture.
Additionally, to make LongLoRA more practical and efficient, the developers compiled a dataset called LongQA, containing more than 3,000 question-and-answer pairs for training. This also enables LongLoRA to effectively improve the output capabilities of large language models.
The study found
The study evaluated different models on the Proof-pile and PG19 datasets. It found that the models performed better as the context size increased during training, demonstrating the effectiveness of its fine-tuning method.
Simply put, training with more information leads to better results. For example, when the context window size increases from 8192 to 32768, a model’s perplexity improves from 2.72 to 2.50.
The study also explored how much context these models can handle on a single machine. The developers extended the models to handle extremely long contexts and found that they still performed well, although performance declined somewhat when the context size was smaller.
In addition to language modeling, the study tested other tasks, including finding specific topics in very long conversations. The developers’ model performed similarly to state-of-the-art models on this task, and even better in some cases. Notably, compared with its competitors, the developers’ model was able to adapt to open-source data more effectively.
LongLoRA shows that the more information large models can process, the better they become at understanding language. It is not only good at handling long texts, but also excels at finding specific topics in long conversations. This suggests that it can handle complex and messy real-world tasks.
However, because it increases the context window, LongLoRA has some issues when processing shorter text segments. The authors have not yet identified the cause.
Overall, LongLoRA introduces an innovative approach to large language models. It makes fine-tuning these models easier and more efficient when processing large amounts of information, without requiring more computing resources.
This article is based on the LongLoRA paper. If you believe it infringes your rights, please contact us to have it removed.
