LLM101n: Let's build a Storyteller

The official code of "RWKV-CLIP: A Robust Vision-Language Representation Learner"

This repo contains evaluation code for the paper "BLINK: Multimodal Large Language Models Can See but Not Perceive". https://arxiv.org/abs/2404.12390

Dense Connector for MLLMs

FreeVA: Offline MLLM as Training-Free Video Assistant

[CVPR2024] ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

[ICLR'24] Consistency-guided Prompt Learning for Vision-Language Models

[ICML 2024] Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Data release for the ImageInWords (IIW) paper.

Library for corpus generation, reformatting, quality radaring, and deduplication

A framework for few-shot evaluation of language models.

Official repository for the paper PLLaVA

openpilot is an open source driver assistance system. openpilot performs the functions of Automated Lane Centering and Adaptive Cruise Control for 250+ supported car makes and models.

When do we not need larger vision models?

Official repository of Evolutionary Optimization of Model Merging Recipes

Dataset for the LREC-COLING 2024 paper "A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions"

The evaluation scripts of JMTEB (Japanese Massive Text Embedding Benchmark)

RealPersonaChat: A Realistic Persona Chat Corpus with Interlocutors' Own Personalities

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

[ECCV 2024] DriveLM: Driving with Graph Visual Question Answering

A framework for prompt tuning using Intent-based Prompt Calibration

A curated list of awesome vision and language resources (still under construction... stay tuned!)

Implementing a ChatGPT-like LLM in PyTorch from scratch, step by step

Strong and Open Vision Language Assistant for Mobile Devices