Article reprint source: AI DreamWorks

Original source: New Wisdom

Image source: Generated by Unbounded AI

After the release of ChatGPT, text generation technology has developed rapidly, and a large number of NLP tasks are facing the dilemma of being completely conquered, especially for the "text summarization" task that lacks standard answers.

But how to include a “reasonable amount of information” in the summary remains difficult: a good summary should be detailed and entity-centric, rather than entity-dense and difficult to understand.

In order to better understand the trade-off between information volume and comprehensibility, researchers from MIT, Columbia University and other institutions proposed a new "Chain of Dense" prompt, which can iteratively optimize the entity-sparse summary generated by GPT-4 and gradually add missing important entities without increasing the length of the summary text.

Paper link: https://arxiv.org/pdf/2309.04269.pdf

Open source data: https://huggingface.co/datasets/griffin/chain_of_density

From the experimental results, the summaries generated by CoD are more abstract than the GPT-4 summaries generated by ordinary prompts, showing more fusion and less lead bias.

A human preference study on 100 CNN DailyMail articles found that humans also tend to choose summary results with denser entities, which are similar to the entity density of manually written summaries.

The researchers open-sourced 500 annotated CoD abstracts and 5,000 unannotated abstract data.

Iteratively improving text summarization

Prompt

The task goal is to use GPT-4 to generate a set of summaries with "different levels of information density" while controlling the length of the text.

The researchers proposed the Chain of Density (CoD) hint to generate an initial summary and gradually make the entities denser and denser.

Specifically, under a fixed number of iteration rounds, a set of unique and salient entities in the source text are identified and merged into the previous summary without increasing the text length.

The first generated summaries are entity-sparse, focusing only on 1-3 initial entities; in order to maintain the same text length while increasing the number of covered entities, abstraction, fusion, and compression need to be explicitly encouraged instead of removing meaningful content from the previous summary.

Instead of specifying the type of entity, the researchers simply defined a missing entity as:

Relevant: Related to the main story;

Specific: Descriptive but concise (5 words or less);

Novel: not present in previous abstracts;

Faithful: exists in the original text;

Anywhere: Can appear anywhere in the article.

In terms of data selection, the researchers randomly selected 100 articles from the CNN/DailyMail summary test set to generate CoD summaries.

The CoD summary statistics are then compared with manually written bullet-point style reference summaries and summaries generated by GPT-4 under the general prompt "Write a VERY short summary of the Article. Do not exceed 70 words".

The expected token length is set to match the token length of the CoD digest.

Statistical results

Direct statistical indicators

We used NLTK to count the number of tokens and Spacy2 to measure the number of unique entities and calculate the entity density ratio.

The CoD hint greatly limits the expected number of tokens for generating summaries. It can be seen that gradually removing unnecessary words from the lengthy initial summary starting from the second step reduces the text length by 5 tokens on average (72 to 67).

The entity density also increased, initially at 0.089, which was lower than the results of humans and GPT-4 (0.151 and 0.122, respectively), and after 5 steps the density increased to 0.167.

Indirect statistical indicators

The extractive density (i.e., the square of the average length of the extracted segments) is used to measure the abstractness of the text, which is expected to increase as the CoD progresses.

The "number of summary sentences aligned with the source text" is used as the concept fusion indicator, where the alignment algorithm uses "relative ROUGE gain" to align the source sentence with the target sentence until the additional added sentences no longer improve the relative ROUGE gain. The expected fusion should gradually increase.

Using the "position of summary content in the source text" as the content distribution indicator, the specific measurement method is the average ranking of all aligned source sentences. It is expected that the CoD summary will initially show obvious lead bias, and then gradually begin to introduce entities from the middle and end of the article.

The statistical results also verified the correctness of the expected results: abstractness gradually increased with the rewriting process, the integration rate increased, and the abstract began to be incorporated into the content in the middle and end of the article.

Moreover, all CoD summaries are more abstract than those written manually and generated by the baseline model.

Experimental Results

To better understand the trade-offs of CoD summarization, we conducted a preference-based human study and a rating-based evaluation with GPT-4.

Human Preference Assessment

The researchers focused on evaluating the effects of densification on overall human mass assessment.

Specifically, by inputting 100 articles, we can get "5 steps * 100 = a total of 500 summaries". The summary results are randomly displayed to four annotators, and the summaries are evaluated based on the original text's fidelity (Essence), clarity (Clarity), accuracy (Accuracy), purpose (Purpose), conciseness (Concise) and style (Style).

Judging from the voting results, the second CoD step received the highest evaluation. Combined with the previous experimental results of average density, it can be roughly inferred that humans are more likely to choose text summaries with an entity density of about 15%, which is significantly higher than GPT- 4 generated summary (entity density 0.122).

Automatic evaluation metrics

Recent work has demonstrated that the correlation between GPT-4's evaluation and human evaluation results is very high, and it may even perform better than crowdsourcing workers on some labeling tasks.

As a supplement to manual evaluation, the researchers proposed using GPT-4 to rate CoD summaries (1-5) from five aspects: Informative, Quality, Coherence, Attributable, and Overall.

The command template used is:

Article: {{Article}} Summary: {{Summary}} Please rate the summary (1=worst to 5=best) with respect to {{Dimension}}. {{Definition}}

The definitions of each indicator are:

Information: An informative summary captures the important information in the article and presents it accurately and concisely.

Quality: A high quality summary is comprehensible and understandable.

Coherence: A coherent summary is well-structured and well-organized.

Attribution: Is all the information in the abstract fully attributed to the article?

summary fully attributable to the Article?)

Overall preference: A good summary should convey the main ideas in the Article in a concise, logical, and coherent fashion.

The experimental results show that densification is correlated with information content, but the score peaks at step 4 (4.74); quality and coherence decrease faster; all summaries are considered to be attributed to the source article; and the overall score tends to be denser and more informative summaries, with the highest score at step 4. On average, the first and last CoD steps are the least favored, while the middle three steps are close (4.78, 4.77, and 4.76, respectively).

Qualitative analysis

There is a trade-off between the coherence/readability of the abstract and the amount of information provided during the iterative process.

The above example shows two CoD steps, one containing more detailed content and the other containing more coarse content.

On average, the CoD summaries at intermediate steps achieve better balance, but there is no work on how to precisely define and quantify this balance.

References:

https://the-decoder.com/chain-of-density-prompt-improves-ai-summaries-by-packing-more-info-into-fewer-words/