Article reprinted from: Machine Heart

Surprising discovery: The big model has serious flaws in knowledge deduction.

Flexible application of knowledge is the key to wisdom. The human brain can process knowledge quickly, such as quickly answering "How many months are there in "Quiet Night Thoughts"". So, can the large model do similar operations? It is known that the large model can first recite "Quiet Night Thoughts" through the Chain of Thought (CoT), and then answer questions based on the dictation, but this will make the generated text lengthy. In contrast, humans can complete simple knowledge deduction in their brains without writing out the intermediate steps. So, can the super large language model generate answers directly in its artificial brain without writing down the knowledge points first?

The answer is no! Figures 1/2/3 show many counterexamples on GPT4. Even the most basic classification (such as judging the odd or even birthday of a celebrity) and comparison (such as comparing the birthdays of two presidents) require a chain of thought. What’s worse is that the large model is almost completely unable to reversely extract the knowledge in the training set.

Figure 1: GPT4 makes mistakes in knowledge classification/comparison, but can get the correct answer through thought chaining

Figure 2: Example of GPT4’s knowledge reverse search error

Figure 3: Although GPT4 can correctly answer “When is someone’s birthday” and “Is a number even?”, when combining the two, it only gets 50% correct without using Chain of Thought (CoT). When comparing the birthdays of celebrities between 1900 and 1910, its performance is almost blind guessing.

The latest research by Zhu Zeyuan (MetaAI) and Li Yuanzhi (MBZUAI), "Language Model Physics Part 3.2: Manipulation of Knowledge", focuses on the above issues.

Paper address: https://arxiv.org/abs/2309.14402

Let's first ask a question. For questions like Figure 1/2/3, is it because GPT4's memory of people's birthdays is not accurate enough (the compression ratio is not enough, the training loss is not low enough), or is it because it has not deepened its understanding of parity through fine-tuning? Can GPT4 be fine-tuned so that it can combine existing knowledge within the model to generate new knowledge such as "parity of birthdays" so that it can directly answer related questions without relying on CoT? Since we don't know the training dataset of GPT4, we can't fine-tune it. Therefore, the author proposes to use a controllable training set to further study the "knowledge deduction" ability of the language model.

Figure 4: For pre-trained models like GPT4, it is difficult to determine whether scenario B/C/D occurs due to the uncontrollability of Internet data.

In "Language Model Physics Part 3.1: Storage and Retrieval of Knowledge", the author created a dataset containing 100k biographies. Each biography includes a person's name and six attributes: date of birth, place of birth, university major, university name, place of work, and work unit. For example:

「Anya Briar Forger originated from Princeton, NJ. She dedicated her studies to Communications. She gained work experience in Menlo Park, CA. She developed her career at Meta Platforms. She came into this world on October 2, 1996. She pursued advanced coursework at MIT.」

The authors ensured that the biographical entries were diverse to help the model better access knowledge. After pretraining, the model was able to accurately answer knowledge extraction questions such as "When is Anya's birthday?" (with an accuracy rate close to 100%) through fine-tuning.

Next, the author continued to fine-tune the model, trying to make the model learn knowledge deduction problems, such as knowledge classification/comparison/addition and subtraction. The article found that the ability of natural language models in knowledge deduction is very limited, and it is difficult to generate new knowledge through fine-tuning, even if they are just simple transformations/combinations of the knowledge the model has already mastered.

Figure 5: If CoT is not used during fine-tuning, the model will need a large number of samples or the accuracy will be very low if it is used to classify/compare/subtract knowledge. 100 professional

As shown in Figure 5, the author found that although the model can accurately answer everyone's birthday after pretraining (with an accuracy rate close to 100%), in order to fine-tune it to answer "Is xxx's birth month an even number?" and achieve an accuracy rate of 75% - don't forget that blind guessing has an accuracy rate of 50% - at least 10,000 fine-tuning samples are required. In contrast, if the model can correctly complete the knowledge combination of "birthday" and "oddity", then according to traditional machine learning theory, the model only needs to learn to classify the 12 months into two categories, and usually about 100 samples are enough!

Similarly, after pre-training, the model can accurately answer each person’s major (a total of 100 different majors). However, even with 50,000 fine-tuning samples, the model was able to compare “Which is better, Anya’s major or Sabrina’s major?” with an accuracy rate of only 53.9%, which is almost a blind guess.

However, when we used CoT fine-tuning to let the model learn sentences like "Anya's birth month is October, so it is an even number", the model's accuracy in judging the parity of birth months on the test set improved significantly (see the "Test with CoT" column in Figure 5).

The author also tried to mix CoT and non-CoT answers in the fine-tuning training data, and found that the accuracy of the model on the test set without CoT was still very low (see the "test without CoT" column in Figure 5). This shows that even if enough CoT fine-tuning data is added, the model still cannot learn to "think in the brain" and report the answer directly.

These results show that it is extremely difficult for language models to perform simple knowledge operations! The model must first write out the knowledge points and then perform the operations. It cannot operate directly in the brain like a human, even after sufficient fine-tuning.

Challenges of Reverse Knowledge Search

The article also found that the natural language model was unable to reverse search the knowledge it had learned. Although it could answer all the information about a person, it was unable to determine the person's name based on this information.

As with knowledge classification/comparison, the authors experimented with GPT3.5/4 and found that they performed poorly on reverse knowledge extraction (see Figure 6). However, since we cannot determine the training set of GPT3.5/4, this does not prove that all language models have this problem.

Figure 6: Comparison of forward/reverse knowledge search of GPT3.5/4. The “reversal curse” work we reported a few days ago (arxiv 2309.12288) also observed this on existing large models.

The author used the aforementioned biographical data set to conduct a more in-depth controlled experiment on the model's reverse knowledge search capabilities. Since the names of all biographies are at the beginning of the paragraph, the author designed 10 reverse information extraction questions, such as:

“Please tell me what is the name of a person born on October 2, 1996 in Princeton, NJ?”

“Please tell me the name of a person who studied Communications at MIT, was born on October 2, 1996 in Princeton, NJ, and works at Meta Platforms in Menlo Park, CA?”

Figure 7: Controlled experiment on the celebrity biography dataset

The authors verified that even though the model achieved lossless knowledge compression and sufficient knowledge enhancement, and could extract this knowledge almost 100% correctly, after fine-tuning, the model was still unable to perform reverse search of knowledge, with an accuracy rate of almost zero (see Figure 7). However, once the reverse knowledge appeared directly in the pre-training set, the accuracy of the reverse search immediately soared.

In summary, the model can only answer reverse questions through fine-tuning if reverse knowledge is directly included in the pretrain data - but this is actually cheating, because if the knowledge is already reversed, it is no longer "reverse knowledge search". If the pretraining set only contains positive knowledge, the model cannot master the ability to answer reverse questions through fine-tuning. Therefore, using language models for knowledge indexing (knowledge database) seems impossible at present.

In addition, one might think that the failure of the above “reverse knowledge search” may be due to the unidirectionality of autoregressive language models such as GPT. But in fact, bidirectional language models such as BERT perform worse in knowledge extraction and even fail in forward extraction. Interested readers can refer to the paper for details.