NeMo Lab is an example template for Generative AI with NVIDIA NeMo 2.0.

NVIDA NeMo is an accelerated end-to-end platform that is flexible and production ready. NeMo is comprised of several component frameworks which enable teams to build, customize, and deploy Generative AI solutions for:

• large language models
• vision language models
• video models
• speech models

NeMo Lab is inspired by NeMo tutorials and openhackathons-org/End-to-End-LLM; of which the later follows the below shown pipeline to guide hackathon participants through instruction tuning and deploying a Llama variant:

flowchart LR
id1(data processing) --> id2(model development) --> id3(model deployment)

NeMo Lab currently focuses on language models, and will expand into NeMo audio, vision, and multimodal capabilities when appropriate.

Data processing is task dependent, relative to pretraining or finetuning. When pretraining, we will use Hugging Face's cosmopedia dataset. When finetuning, we will use NeMo's default dataset – SquadDataModule – a variant of the Stanford QA dataset.

We will use NeMo to train Nemotron 3 4B on the cosmopedia dataset, and tune a Llama variant on the SQuAD dataset.

We will use NeMo interfaces to export models for inference with TensorRT-LLM and Triton Inference Server, or vLLM.

• Code profiling
• Logging training and tuning runs with Weights & Biases
• Model output control with NeMo Guardrails
• LLM traces with Arize
• Agents as DAGs with LangGraph
• Containerization with Docker
• System prompt design

The source code found in src/nemo_lab is used to provide examples of implementing concepts "from-scratch" with NeMo. For instance – how might we add a custom model, or our own training recipe given base interfaces and mixins found within the framework.

We will use NVIDIA and Meta models including, but not limited to:

• NVIDIA Llama variants, Mistral variants, Megatron distillations, and Minitron
• NVIDIA embedding, reranking, and retrieval models
• NVIDIA Cosmos tokenizers
• NeMo compatible Meta Llama variants

• a CUDA compatible OS and device (GPU) with at least 48GB of VRAM (e.g. an L40S).
• CUDA 12.1
• Python 3.10.10
• Pytorch 2.2.1

• NVIDIA Developer Program
• NVIDIA NGC for NeMo and TensorRT-LLM containers
• build.nvidia.com for API calls to NVIDIA hosted endpoints
• Hugging Face Hub for model weights and datasets
• Arize for tracing and observability
• Weights & Biases for experiment management during finetuning

To prepare a development environment, please run the following in terminal:

bash install_requirements.sh

Doing so will install nemo_lab along with the nemo_run, megatron_core 0.10.0rc0, and the nvidia/apex PyTorch extension.

Two Docker images have been created for the quick start tutorials. One for pretraining, and one for finetuning.

To run pretraining, do the following in terminal:

docker pull jxtngx/nemo-pretrain-nemotron3-4b
docker run --rm --gpus 1 -it jxtngx/nemo-pretrain-nemotron3-4b
python pretrain_nemotron3_4b.py

To run finetuning, do the following in terminal:

docker pull docker pull jxtngx/nemo-finetune-llama3-8b
docker run --rm --gpus 1 -it docker jxtngx/nemo-finetune-llama3-8b
huggingface-cli login
{ENTER HF KEY WHEN PROMPTED}
python finetune_llama3_8b.py

See Quickstart Studios and Images

Quickstart	Studio	Docker
Pretrain Nemotron 3 4B
Finetune Llama 3 8B

• NeMo documentation
• NeMo tutorials
• NeMo Guardrails documentation
• Deploy on a SLURM cluster
• Mixed Precision Training
• CPU Offloading
• Communication Overlap

• NVIDIA NIM (LLM) documentation
• langchain-nvidia-ai-endpoints documentation
• Arize documentation
• LangGraph documentation
• W&B documentation
• vLLM documentation
• cuVS (GPU accelerated vector search by NVIDIA Rapids)
• Weaviate documentation
• Gradio documentation

• Arize and LangGraph (Arize)
• Weaviate and LangChain (LangChain)
• vLLM and LangChain (LangChain)

• Generative AI Explained
• Deploying a Model for Inference at Production Scale
• Sizing LLM Inference Systems
• Building RAG Agents with LLMs
• Introduction to Deploying RAG Pipelines for Production at Scale
• Prompt Engineering with LLaMA-2

• Generative AI and LLMs
• Accelerated LLM Model Alignment and Deployment
• Beyond RAG Basics: Building Agents, Co-Pilots, Assistants, and More!
• Generative AI Essentials
• GTC 2024 - Latest in Generative AI

• Prompt Engineering and P-Tuning
• Scaling LLMs with NVIDIA Triton and NVIDIA TensorRT-LLM
• Turbocharging Meta Llama 3 Performance with NVIDIA TensorRT-LLM and NVIDIA Triton Inference Server
• Post-Training Quantization of LLMs with NVIDIA NeMo and NVIDIA TensorRT Model Optimizer
• Getting Started with Large Language Models for Enterprise Solutions
• Unlocking the Power of Enterprise-Ready LLMs with NVIDIA NeMo
• TensorRT-LLM KV Cache Early Reuse
• An Introduction to Model Merging for LLMs

• Attention is All You Need
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• LLaMA: Open and Efficient Foundation Language Models
• Compact Language Models via Pruning and Knowledge Distillation
• 8-bit Optimizers via Block-wise Quantization
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• QLoRA: Efficient Finetuning of Quantized LLMs
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Efficient Tool Use with Chain-of-Abstraction Reasoning
• ReAct: Synergizing Reasoning and Acting in Language Models
• Adaptive Mixtures of Local Experts

• Build a Large Language Model (Sebastian Raschka)
• Hands-On Large Language Models (Alammar et al)
• StatQuest (Josh Starmer)
• Coding a ChatGPT Like Transformer From Scratch in PyTorch (Josh Starmer)
• Serrano Academy (Luis Serrano)
• Intro to Large Language Models (Andrej Karpathy)
• Building LLMs from the Ground Up (Sebastian Raschka)
• Coding the Self-Attention Mechanism of LLMs (Sebastian Raschka)
• Neural networks (Grant Sanderson)
• Visualizing Transformers and Attention (Grant Sanderson)
• Agentic Design Patterns (Deep Learning AI)
• Intro to LangGraph (LangChain)
• Getting Beyond the Hype: A Guide to AI’s Potential (Stanford)