Source code for our ACL 2024 paper : MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin

Click the links below to view our papers and checkpoints

If you find this work useful, please cite our paper and give us a shining star 🌟

@inproceedings{zhou2024marvel,
 title={MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin},
 author={Zhou, Tianshuo and Mei, Sen and Li, Xinze and Liu, Zhenghao and Xiong, Chenyan and Liu, Zhiyuan and Gu, Yu and Yu, Ge},
 booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics},
 year={2024}
}

MARVEL unlocks the multi-modal capability of dense retrieval via visual module plugin. It encodes queries and multi-modal documents with a unified encoder model to bridge the modality gap between images and texts, conducts retrieval, modality routing, and result fusion within a unified embedding space.

1. Install the following packages using Pip or Conda under this environment

Python==3.7
Pytorch
transformers
clip
faiss-cpu==1.7.0
tqdm
numpy
base64
Install the pytrec_eval from https://github.com/cvangysel/pytrec_eval

We provide the version file requirements.txt of all our used packages, if you have any problems configuring the environment, please refer to this document.

2. Prepare the pretrained CLIP and T5-ANCE

MARVEL is built on CLIP and T5-ANCE model.

• First, use git clone to download this project:

git clone https://github.com/OpenMatch/MARVEL
cd MARVEL

• Download link for our WebQA: WebQA. (❗️Note: For the imgs.tsv, you need to download the data from this link and run 7z x imgs.7z.001).
• Please refer to ClueWeb22-MM to obtain pretrain data and retrieval benchmark.
• Place the downloaded dataset in the data folder:

data/
├──WebQA/
│   ├── train.json
│   ├── dev.json
│   ├── test.json
│   ├── test_qrels.txt
│   ├── all_docs.json
│   ├── all_imgs.json
│   ├── imgs.tsv
│   └── imgs.lineidx.new
├──ClueWeb22-MM/
│   ├── train.parquet
│   ├── dev.parquet
│   ├── test.parquet
│   ├── test_qrels.txt
│   ├── text.parquet
│   └── image.parquet
└──pretrain/
    ├── train.parquet
    └── dev.parquet

Using the WebQA dataset as an example, I will show you how to reproduce the results in the MARVEL paper. The same is true for the ClueWeb22-MM dataset. Also, we provide the checkpoint for each step. You can skip a step and continue training.

• First step: Go to the pretrain folder and pretrain MARVEL's visual module checkpoint:

cd pretrain
bash train.sh

• Second step: Go to the DPR folder and train MARVEL-DPR using inbatch negatives checkpoint:

cd DPR
bash train_webqa.sh

• Third step: Then using MERVEL-DPR to generate hard negatives for training MARVEL-ANCE:

bash get_hn_webqa.sh

• Final step: Go to the ANCE folder and train MARVEL-ANCE using hard negatives checkpoint:

cd ANCE
bash train_ance_webqa.sh

• These experimental results are shown in Table 2 of our paper.
• Go to the DPR or ANCE folder and evaluate model performance as follow:

bash gen_embeds.sh
bash retrieval.sh

The results are shown as follows.

• WebQA

• ClueWeb22-MM

If you have questions, suggestions, and bug reports, please email:

zhoutianshuo@stumail.neu.edu.cn     meisen@stumail.neu.edu.cn