MiniGPT-5, which unifies image and text generation, is here: Token becomes Voken, and the model can not only continue writing, but also automatically add pictures.

Source of article reprint: Heart of the Machine
 OpenAI's GPT-5 large model seems to be far away, but some researchers have taken the lead in launching the innovative visual and language cross-generation model MiniGPT-5. This has important implications for generating images with coherent textual descriptions.
Image source: Generated by Unbounded AI
Large models are making the leap between language and vision, promising to seamlessly understand and generate text and image content. In a series of recent studies, multimodal feature integration is not only a growing trend but has already led to key advances ranging from multimodal conversations to content creation tools. Large language models have demonstrated unparalleled capabilities in text understanding and generation. However, simultaneously generating images with coherent textual narratives is still an area to be developed.
Recently, a research team from the University of California, Santa Cruz proposed MiniGPT-5, an innovative interleaved visual language generation technology based on the concept of "generative voken".
Paper address: https://browse.arxiv.org/pdf/2310.02239v1.pdf
Project address: https://github.com/eric-ai-lab/MiniGPT-5
By combining the Stable Diffusion mechanism with LLM through a special visual token "generative voken", MiniGPT-5 heralds a new model for skilled multimodal generation. At the same time, the two-stage training method proposed in this paper emphasizes the importance of the description-free foundation stage, allowing the model to "thrive" even when data is scarce. The general stage of this method does not require domain-specific annotations, which makes this solution very different from existing methods. In order to ensure that the generated text and images are harmonious and consistent, the dual loss strategy of this paper comes into play, and the generative voken method and classification method further enhance this effect.
Building on these techniques, this work marks a transformative approach. By using ViT (Vision Transformer) and Qformer with large language models, the research team transforms multimodal inputs into generative vokens and seamlessly pairs with high-resolution Stable Diffusion2.1 to achieve context-aware image generation. This paper combines images as auxiliary inputs with instruction adjustment methods and pioneers the use of text and image generation losses, thus expanding the synergy between text and vision.
MiniGPT-5 matches models such as CLIP constraints, cleverly fuses the diffusion model with MiniGPT-4, and achieves good multimodal results without relying on domain-specific annotations. Most importantly, our strategy can leverage the progress of multimodal visual language foundation models and provide a new blueprint for enhancing multimodal generation capabilities.
As shown in the figure below, in addition to the original multimodal understanding and text generation capabilities, MiniGPT5 can also provide reasonable and coherent multimodal output:
The contributions of this article are reflected in three aspects:
We propose to use a multimodal encoder, which represents a novel general technique and has been shown to be more effective than LLM and inverse generative vokens, and combine it with Stable Diffusion to generate interleaved visual and language outputs (Multimodal Language Model for Multimodal Generation).
We focus on introducing a new two-stage training strategy for description-free multimodal generation. The unimodal alignment stage obtains high-quality text-aligned visual features from a large number of text-image pairs. The multimodal learning stage includes a novel training task, prompt context generation, which ensures that visual and textual prompts are well coordinated for generation. Including classifier-free guidance in the training stage further improves the generation quality.
Compared with other multimodal generative models, MiniGPT-5 achieves state-of-the-art performance on the CC3M dataset. MiniGPT-5 also establishes new benchmarks on well-known datasets such as VIST and MMDialog.
Next, let’s take a look at the details of the study.
Method Overview
In order to enable large language models to have multimodal generation capabilities, the researchers introduced a structured framework that integrates pre-trained multimodal large language models and text-to-image generation models. To address the differences between different model domains, they introduced special visual symbols "generative vokens" that can be trained directly on raw images. In addition, a two-stage training method was promoted and combined with a classifier-free guidance strategy to further improve the generation quality.
Multimodal input stage
Recent advances in multimodal large models such as MiniGPT-4 have focused on multimodal understanding, being able to process images as continuous input. To extend their capabilities to multimodal generation, researchers introduced generative vokens designed specifically for outputting visual features. In addition, they adopted parameter-efficient fine-tuning techniques within the Large Language Model (LLM) framework for multimodal output learning.
Multimodal output generation
To precisely align the generated tokens with the generative model, the researchers formulated a compact mapping module for dimensionality matching and incorporated several supervised losses, including text space loss and latent diffusion model loss. The text space loss helps the model learn the correct localization of tokens, while the latent diffusion loss directly aligns the tokens with appropriate visual features. Since the features of the generated tokens are directly guided by the image, this method does not require a comprehensive image description, thus achieving description-free learning.
Training strategy
Given the non-negligible domain shift between the text and image domains, we find that directly training on a limited dataset of interleaved text and images may lead to misalignment and image quality degradation.
Therefore, they adopted two different training strategies to alleviate this problem. The first strategy involves adopting a classifier-free bootstrapping technique to improve the effectiveness of generated tokens throughout the diffusion process; the second strategy is carried out in two stages: the initial pre-training stage focuses on coarse feature alignment, and the subsequent fine-tuning stage is dedicated to complex feature learning.
Experiments and Results
To evaluate the efficacy of the model, the researchers selected multiple benchmarks for a series of evaluations. The experiments aimed to address several key questions:
Can MiniGPT-5 generate believable images and plausible text?
How does MiniGPT-5 perform compared to other SOTA models in single-pass and multi-pass interleaved vision-language generation tasks?
What impact does the design of each module have on overall performance?
To evaluate the performance of the model on different benchmarks at different training stages, a quantitative analysis sample of MiniGPT-5 is shown in Figure 3 below:
The evaluation here spans both vision (image-related metrics) and language (text metrics) to demonstrate the versatility and robustness of the proposed model.
VIST Final-Step Evaluation
The first set of experiments involves single-step evaluation, i.e. generating the corresponding image based on the prompt model in the last step. The results are shown in Table 1.
In all three settings, MiniGPT-5 outperforms the fine-tuned SD 2. Notably, the CLIP score of the MiniGPT-5 (LoRA) model consistently outperforms other variants across multiple prompt types, especially when combining image and text prompts. On the other hand, the FID score highlights the competitiveness of the MiniGPT-5 (Prefix) model, suggesting that there may be a trade-off between the quality of image embeddings (reflected by the CLIP score) and the diversity and realism of images (reflected by the FID score). Compared to the model trained directly on VIST without including the unimodal registration stage (MiniGPT-5 w/o UAS), while the model retains the ability to generate meaningful images, the image quality and consistency are significantly degraded. This observation highlights the importance of the two-stage training strategy.
VIST Multi-Step Evaluation
In a more detailed and comprehensive evaluation, the researchers systematically provided the model with prior historical context and subsequently evaluated the generated images and narratives at each step.
Tables 2 and 3 summarize the results of these experiments, summarizing the performance of image and language metrics respectively. The experimental results show that MiniGPT-5 is able to generate coherent, high-quality images using long-horizontal multimodal input prompts in all data without compromising the multimodal understanding capabilities of the original model. This highlights the efficacy of MiniGPT-5 in different environments.
VIST Human Evaluation
As shown in Table 4, MiniGPT-5 generates more relevant textual narratives in 57.18% of cases, provides better image quality in 52.06% of cases, and generates more coherent multimodal outputs in 57.62% of scenarios. These data clearly demonstrate its stronger multimodal generation capabilities compared to the two-stage baseline that uses text-to-image prompt narratives without subjunctive mood.
MMDialog multiple rounds of evaluation
The results are shown in Table 5. MiniGPT-5 outperforms the baseline model Divter in generating more accurate text responses. Although the quality of the generated images is similar, MiniGPT-5 outperforms the baseline model in terms of MM correlation, indicating that it can better learn how to properly localize image generation and generate highly consistent multimodal responses.
How effective is it? Let’s take a look at the output of MiniGPT-5. Figure 7 below compares MiniGPT-5 with the baseline model on the CC3M validation set.
Figure 8 below shows the comparison between MiniGPT-5 and the baseline model on the VIST validation set.
Figure 9 below shows a comparison between MiniGPT-5 and the baseline model on the MMDialog test set.
For more research details, please refer to the original paper.