Bringing Back the Context: Camera Trap Species Identification as Link Prediction on Multimodal Knowledge Graphs
Paper: https://arxiv.org/pdf/2401.00608.pdf
Project webpage: https://osu-nlp-group.github.io/COSMO/
Authors: Vardaan Pahuja, Weidi Luo, Yu Gu, Cheng-Hao Tu, Hong-You Chen, Tanya Berger-Wolf, Charles Stewart, Song Gao, Wei-Lun Chao, and Yu Su
pip install -r requirements.txt
bash preprocess_iwildcam.sh
Note: The dir. data/iwildcam_v2.0/train/ contains images for all splits.
- Download snapshot_mountain_zebra.zip from this link and uncompress it into a directory
data/snapshot_mountain_zebra/. - Download images using the command
gsutil -m cp -r "gs://public-datasets-lila/snapshot-safari/MTZ/MTZ_public" data/snapshot_mountain_zebra/ - Run
bash preprocess_mountain_zebra.sh
Note: The below commands will use the DistMult model by default. Use the following hyperparameter configuration:
- For iWildCam2020-WILDS, set
DATA_DIRtodata/iwildcam_v2.0/,IMG_DIRtodata/iwildcam_v2.0/train/, andDATASETtoiwildcam - For Snapshot Mountain Zebra, set
DATA_DIRtodata/snapshot_mountain_zebra/andIMG_DIRtodata/snapshot_mountain_zebra/, andDATASETtomountain_zebra. - For ConvE, use
--kg-embed-model conve --embedding-dim 200in args.
python -u run_image_only_model.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR > CKPT_DIR/log.txt
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR > CKPT_DIR/log.txt
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id > CKPT_DIR/log.txt
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-image-location > CKPT_DIR/log.txt
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR/ --save-dir CKPT_DIR --add-image-time > CKPT_DIR/log.txt
python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id --add-image-time --add-image-location > CKPT_DIR/log.txt
python eval.py --ckpt-path <PATH TO COSMO CKPT> --split test --data-dir DATA_DIR --img-dir IMG_DIR
cd gen_utils/
python analyze_taxonomy_model.py --data-dir DATA_DIR --img-dir IMG_DIR --ckpt-1-path <PATH TO COSMO+TAXONOMY CKPT> --ckpt-2-path <PATH TO COSMO BASE CKPT>
cd gen_utils/
python analyze_img_loc.py --data-dir DATA_DIR
cd gen_utils/
python analyze_img_time.py --data-dir DATA_DIR
cd gen_utils/
python dump_imageonly_pred_specie_wise.py --ckpt-path <PATH TO IMAGE-ONLY MODEL> --split test --out-dir <OUT DIR>
cd gen_utils/
python dump_kge_pred_specie_wise.py --ckpt-path <PATH TO COSMO MODEL> --split test --out-dir <OUT DIR>
cd gen_utils/
python eval_kge_specie_wise.py --y-pred-path-1 <PATH TO PREDICTIONS JSON FILE OF BASELINE MODEL> --y-pred-path-2 <PATH TO COSMO PREDICTIONS JSON FILE>
@article{pahuja2023bringing,
title={Bringing Back the Context: Camera Trap Species Identification as Link Prediction on Multimodal Knowledge Graphs},
author={Pahuja, Vardaan and Luo, Weidi and Gu, Yu and Tu, Cheng-Hao and Chen, Hong-You and Berger-Wolf, Tanya and Stewart, Charles and Gao, Song and Chao, Wei-Lun and Su, Yu},
journal={arXiv preprint arXiv:2401.00608},
year={2023}
}