Pinned Loading

1. zo2 zo2 Public

   ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory [COLM2025]

   Python 198 17

2. Tiny-FSDP Tiny-FSDP Public

   Tiny-FSDP, a minimalistic re-implementation of the PyTorch FSDP

   Python 93 8

3. Tiny-DeepSpeed Tiny-DeepSpeed Public

   Tiny-DeepSpeed, a minimalistic re-implementation of the DeepSpeed library

   Python 49 10

4. Tiny-Megatron Tiny-Megatron Public

   Tiny-Megatron, a minimalistic re-implementation of the Megatron library

   Python 21 3

5. train-large-model-from-scratch train-large-model-from-scratch Public

   A minimal, hackable pre-training stack for GPT-style language models

   Python 5

6. Streaming-Dataloader Streaming-Dataloader Public

   A memory-efficient streaming data loader designed for LLM pretraining under limited CPU and GPU memory constraints

   Python 3