* Equal contributions

• Apr-30-24- LLaMA-3-V and Phi-3-V demos are now available via Hugging Face Spaces. Check them out at LLaMA-3-V & Phi-3-V 🔥🔥🔥
• Apr-28-24- Online demo of Phi-3-V and LLaMA-3-V are released, check them out at Online Demo 🔥🔥🔥
• Apr-28-24- LoRA, fully fine-tuned and S² fine-tuned models and results are added! 🔥🔥🔥
• Apr-27-24- Google Colab is released to chat with Phi-3-V-3.8B model, check it out at Google Colab 🔥🔥🔥
• Apr-26-24- Phi-3-V and LLaVA-3-V released: Excited to release the new integration of LLaVA with Phi-3 Mini Instruct and LLaMA-3 Instruct models! Hugging Face 🔥🔥🔥

This repository enhances the capabilities of the LLaVA 1.5 model, incorporating latest LLMs released this weak🔥, Phi-3 Mini Instruct 3.8B, and LLaMA-3 Instruct 8B.

• Average computed excluding MME, and second-best are underlined.

The following table provides an overview of the available models in our zoo. For each model, you can find links to its Hugging Face page.

git clone https://github.com/mbzuai-oryx/LLaVA-pp.git
cd LLaVA-pp
git submodule update --init --recursive

Packages you need to update from LLAVA:

pip install git+https://github.com/huggingface/transformers@a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3

To integrate Phi-3-V with LLaVA, follow these steps to update the codebase:

# Copy necessary files
cp Phi-3-V/train.py LLaVA/llava/train/train.py
cp Phi-3-V/llava_phi3.py LLaVA/llava/model/language_model/llava_phi3.py
cp Phi-3-V/builder.py LLaVA/llava/model/builder.py
cp Phi-3-V/model__init__.py LLaVA/llava/model/__init__.py
cp Phi-3-V/main__init__.py LLaVA/llava/__init__.py
cp Phi-3-V/conversation.py LLaVA/llava/conversation.py

# Training commands
cp scripts/Phi3-V_pretrain.sh LLaVA/Vi-phi3_pretrain.sh
cp scripts/Phi3-V_finetune_lora.sh LLaVA/Vi-phi3_finetune_lora.sh

1. Pre-train

cd LLaVA
bash Phi3-V_pretrain.sh

2. Finetune

cd LLaVA
bash Phi3-V_finetune_lora.sh

To integrate LLaMA-3-V with LLaVA, follow these steps to update the codebase:

# Copy necessary files
cp LLaMA-3-V/train.py LLaVA/llava/train/train.py
cp LLaMA-3-V/conversation.py LLaVA/llava/conversation.py
cp LLaMA-3-V/builder.py LLaVA/llava/model/builder.py
cp LLaMA-3-V/llava_llama.py LLaVA/llava/model/language_model/llava_llama.py

# Training commands
cp scripts/LLaMA3-V_pretrain.sh LLaVA/LLaMA3-V_pretrain.sh
cp scripts/LLaMA3-V_finetune_lora.sh LLaVA/LLaMA3-V_finetune_lora.sh

1. Pre-train

cd LLaVA
bash LLaMA3-V_pretrain.sh

2. Finetune

cd LLaVA
bash LLaMA3-V_finetune_lora.sh

We are thankful to LLaVA, lmms-eval and S²-Wrapper for releasing their models and code as open-source contributions.

In case if you face any issues or have any questions, please feel free to create an issue or reach out at hanoona.bangalath@mbzuai.ac.ae & muhammad.maaz@mbzuai.ac.ae.

  @misc{hanoona2024LLaVA++,
          title={LLaVA++: Extending Visual Capabilities with LLaMA-3 and Phi-3},
          author={Rasheed, Hanoona and Maaz, Muhammad and Khan, Salman and Khan, Fahad S.},
          url={https://github.com/mbzuai-oryx/LLaVA-pp},
          year={2024}
  }