Artificial General Intelligence (AGI) requires comprehensive understanding and generation capabilities for a variety of tasks spanning different modalities and functionalities. Integrative AI is one important direction to approach AGI, through combining multiple models to tackle complex multimodal tasks. However, there is a lack of a flexible and composable platform to facilitate efficient and effective model composition and coordination.

In this paper, we propose the i-Code Studio, a configurable and composable framework for Integrative AI. The i-Code Studio orchestrates multiple pre-trained models in a finetuning-free fashion to conduct complex multimodal tasks. Instead of simple model composition, the i-Code Studio provides an integrative, flexible, and composable setting for developers to quickly and easily compose cutting-edge services and technologies tailored to their specific requirements. The i-Code Studio achieves impressive results on a variety of zero-shot multimodal tasks, such as video-to-text retrieval, speech-to-speech translation, and visual question answering. We also demonstrate how to quickly build a multimodal agent based on the i-Code Studio that can communicate and personalize for users.

More details can be found in the paper.

If you find our work useful, please consider citing:

@misc{fang2023icode,
      title={i-Code Studio: A Configurable and Composable Framework for Integrative AI}, 
      author={Yuwei Fang and Mahmoud Khademi and Chenguang Zhu and Ziyi Yang and Reid Pryzant and Yichong Xu and Yao Qian and Takuya Yoshioka and Lu Yuan and Michael Zeng and Xuedong Huang},
      year={2023},
      eprint={2305.13738},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}