Code and benchmark for the paper: "A Practitioner's Guide to Continual Multimodal Pretraining" [NeurIPS'24]

Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton

[NeurIPS 2024] ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization

Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more.

Pretrained deep learning models for Jax/Flax: StyleGAN2, GPT2, VGG, ResNet, etc.

Official repository of Evolutionary Optimization of Model Merging Recipes

Accelerating the development of large multimodal models (LMMs) with lmms-eval

(ECCV 2024) Code for V-IRL: Grounding Virtual Intelligence in Real Life

VLM Evaluation: Benchmark for VLMs, spanning text generation tasks from VQA to Captioning

A flexible and efficient codebase for training visually-conditioned language models (VLMs)

Official repository for "Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model" [ICLR 2024 spotlight]

Unofficial implementation of "SODA: Bottleneck Diffusion Models for Representation Learning"

[ICLR 2024] Official repository for "Vision-by-Language for Training-Free Compositional Image Retrieval"

✨✨Latest Advances on Multimodal Large Language Models

Consistency Distilled Diff VAE

DataComp: In search of the next generation of multimodal datasets

A curated list of plugins that you can add to your FiftyOne install!

Refine high-quality datasets and visual AI models

Python package to download and use the SSB datasets

{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch

This is the repository for the Photorealistic Unreal Graphics (PUG) datasets for representation learning.

Official implementation of "Controlling Text-to-Image Diffusion by Orthogonal Finetuning".

Implementation of Discrete Key / Value Bottleneck, in Pytorch

[ICML2023] Instant Soup Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models. Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, and Zhangyang Wang

An extension of the PyTorch library containing various tools for performing deep learning in hyperbolic space.