An open-source implementation of LLaVA-NeXT series for facilitating the large multi-modal model community.

Resources: [🤗HuggingFace]

• 🔥 All training data and checkpoints at each stage are open-sourced, friendly for research usage.
• 🔥 Able to reproduce the results of LLaVA-NeXT.
• 🔥 Based on the LLaVA codebase with minimal modification, easy to follow.

See more details in ModelZoo.md.

• Reproduce LLaVA-Next-LLaMA3-8B
• Integrate VLMEvalKit for convenient evaluation

1. Clone this repository and navigate to Open-LLaVA-NeXT folder

git clone https://github.com/xiaoachen98/Open-LLaVA-NeXT.git
cd Open-LLaVA-NeXT

2. Install Package

conda create -n llava-next python=3.10 -y
conda activate llava-next
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

1. Install additional packages for training

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

You should follow this instruction Data.md to manage the training datasets.

Open-LLaVA-NeXT training consists of two stages: (1) feature alignment stage: use 558K subset of the LAION-CC-SBU dataset to connect a frozen pretrained vision encoder to a frozen LLM; (2) visual instruction tuning stage: finetune the entire model with 1M completely open source data. Detailed data statics is provided in Visual Instruction Tuning. We take the Vicuna-v1.5-7B variant as example to present the training and evaluation details.

Open-LLaVA-NeXT series are trained on A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly. And utilizing DeepSpeed ZeRO-3 can further reduce the memory requirements. Always keep the global batch size the same: per_device_train_batch_size x gradient_accumulation_steps x num_gpus.

We use a same set of hyperparameters as LLaVA in finetuning. Both hyperparameters used in pretraining and finetuning are provided below.

1. Pretraining

2. Finetuning

Please download the 558K subset of the LAION-CC-SBU dataset with BLIP captions here.

Pretrain takes around 5 hours for Open-LLaVA-NeXT-7B on 16 x A100 (80G).

Training script with DeepSpeed ZeRO-2: pretrain.sh.

• --mm_projector_type mlp2x_gelu: the two-layer MLP vision-language connector.
• --vision_tower openai/clip-vit-large-patch14-336: CLIP ViT-L/14 336px.

1. Prepare data You should follow the instructions for data preparation in Data.
2. Prepare MLP projectors You may download our pretrained projectors in Model Zoo, or specify your own MLP projector after pre-training.
3. Start training Visual instruction tuning takes around 20 hours for Open-LLaVA-NeXT-7B on 16x A100 (80G).

Training script with DeepSpeed ZeRO-2: finetune.sh.

New options to note:

• --unfreeze_mm_vision_tower True: finetune vision tower.
• --mm_vision_tower_lr 2e-6: learning rate of vision tower.
• --image_aspect_ratio anyres: Process an image with variable resolutions.
• --mm_patch_merge_type spatial_unpad: This unpads a PyTorch tensor of a padded and resized image, and by inserting learnable newline vectors into image tokens, the model becomes aware of two-dimensional spatial information. This is used to process image token.

See Evaluation.md.

If you find this project useful in your research, please consider cite:

@misc{chen2024open,
  title={Open-LLaVA-NeXT: An open-source implementation of LLaVA-NeXT series for facilitating the large multi-modal model community.},
  author={Chen, Lin and Xing, Long},
  howpublished = {\url{https://github.com/xiaoachen98/Open-LLaVA-NeXT}},
  year={2024},
  doi={10.5281/zenodo.13935471}
}

• LLaVA: the codebase we built upon. Thanks for their brilliant contributions to the community! We just can't wait to use LLaVA-NeXT.
• ShareGPT4V: Thanks for their code about finetuning the vision tower.
• VLMEvalKit: the amazing open-sourced suit for evaluating various LMMs!