Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch

A concise but complete full-attention transformer with a set of promising experimental features from various papers

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

A simple way to keep track of an Exponential Moving Average (EMA) version of your Pytorch model

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

Implementation of π₀, the robotic foundation model architecture proposed by Physical Intelligence

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of DreamerV3 in Pytorch

Implementation of Soft Actor Critic and some of its improvements in Pytorch

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

Vector (and Scalar) Quantization, in Pytorch

Implementation of rectified flow and some of its followup research / improvements in Pytorch

Implementation of E2-TTS, "Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS", in Pytorch

An implementation of Phasic Policy Gradient, a proposed improvement of Proximal Policy Gradients, in Pytorch

Implementation of Alphafold 3 from Google Deepmind in Pytorch

Explorations into the proposed Streaming Deep Reinforcement Learning, from University of Alberta

Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

Implementation of the proposed minGRU in Pytorch

Implementation of LVSM, SOTA Large View Synthesis with Minimal 3d Inductive Bias, from Adobe Research

Implementation of the proposed MaskBit from Bytedance AI

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

Implementation of the proposed Spline-Based Transformer from Disney Research

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI

Implementation of Autoregressive Diffusion in Pytorch

Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

Explorations into improving ViTArc with Slot Attention