Dolphin is a general video interaction platform based on large language models. Our team is trying to build a chatbot for video understanding, processing and generation.

The project 🐬 Dolphin is still under construction, we will continue to update it and welcome contributions/pull requests from the community.

• 2023/05/06: Code release & Online Demo

  • Video understanding: Q&A about the video.
  • Video processing: Basic functions such as trimming video, adding subtitles, extracting audio, and adding audio using moviepy. Video to pose/depth/canny also included.
  • Video generation: Text to video, pose/depth and text to video, and video pix2pix.

Prepare the project and environment:

# We recommend using conda to manage the environment and use python 3.8
conda create -n dolphin python=3.8
conda activate dolphin

# Clone the respository:
git clone https://github.com/BUAA-PrismGroup/dolphin.git
cd dolphin

# Install dependencies:
pip install -r requirements.txt

To start 🐬 Dolphin, you can specify the GPU/CPU assignment by --load, the parameter indicates which Video Foundation Model to use and where it will be loaded to. The model and device are separated by underline _, while the different models are separated by comma ,. The available Video Foundation Models can be found in the following table or configs/backends.yaml.

For example, if you want to load VideoCaptioning to cuda:0 and MoviepyInterface to cpu, you can use: VideoCaptioning_cuda:0,MoviepyInterface_cpu.

Some starting commands are as follows.

# Advice for CPU Users
python video_chatgpt.py --load VideoCaptioning_cpu,ImageCaptioning_cpu,ModelscopeT2V_cpu

# Advice for 4 Tesla V100 32GB (Full usage)
# You can specify the device where each model is loaded in `configs/backend.yaml`
python video_chatgpt.py

Our project framework is highly extensible for adding new features, including support for more video foundation models and more large language models.

For more video foundation models, you can add the inference code for new models under the modules directory. We recommend creating a new Python package for the new model within this directory and implementing the class in the package's __init__.py file (you can refer to ModelscopeT2V in modules/modelscope_t2v/init.py). Afterward, add the related information in configs/backends.yaml.

model_zoos:
  FaceText2Video:  # foundation model class
    target: modules.face2video.FaceText2Video # path of the class in project
    params: # params passed to the class
      device: cuda:0
tools:
  # - name: tool name
  #   desc: description about new tool
  #   instance: keep the name consistent with the one in the model_zoos section mentioned above
  #   func: inference function in foundation model class
  - name: Generate Video Condition On Face Video
    desc: "useful when you want to generate a new video from both the user description and a facial keypoints video. /
      like: generate a new video of a human face from this human face video, or can you generate a video based on both the text 'A boy is playing basketball.' and this face video. /
      The input to this tool should be a comma separated string of two, representing the video_path and the user description. "
    instance: FaceText2Video
    func: inference

For more large language models, you can refer to video_chatgpt.py, and create a new file like video_moss.py or video_stablelm.py in the project root directory to implement support for other large language models.

• Chatbot with video downstream works (video understanding, processing and generation)
• Pretrained unified video model with in-context learning
• Benchmark for emerging video tasks
• Service including Gradio, Web, API and Docker

We appreciate the open source of the following projects:

Hugging Face   LangChain   mPLUG   BLIP-2   MoviePy   Text2Video-Zero   damo/text-to-video-synthesis   bark

For help or issues using the 🐬 Dolphin, please submit a GitHub issue.

For other communications, please contact Zehuan Huang ([email protected]) or kaleido lab ([email protected]). Welcome to follow us in Twitter (@kaleido_lab).

If you find this repository useful, please consider citing:

@misc{dolphin,
    Author = {Zehuan Huang, Haoran Feng, Chongzhi Zhang, Lu Sheng, Ziwei Liu, Jing Shao},
    Year = {2023},
    Note = {https://github.com/kaleido-lab/dolphin},
    Title = {Dolphin: General Video Interaction Platform Based on LLMs}
}