Tensor library for machine learning

⚠️ The quantization formats Q4 and Q8 have been updated: ggerganov/llama.cpp#1508 - requantize any old models

Note that this project is under development and not ready for production use.
Some of the development is currently happening in the llama.cpp and whisper.cpp repos

• Written in C
• 16-bit float support
• 4-bit integer quantization support
• Automatic differentiation (WIP in progress)
• ADAM and L-BFGS optimizers
• Optimized for Apple silicon via NEON intrinsics and Accelerate framework
• On x86 architectures utilzes AVX intrinsics
• No third-party dependencies
• Zero memory allocations during runtime

• Example of GPT-2 inference examples/gpt-2
• Example of GPT-J inference examples/gpt-j
• Example of Whisper inference examples/whisper
• Support 4-bit integer quantization #27
• Example of Cerebras-GPT inference examples/gpt-2
• Example of FLAN-T5 inference #12
• Example of LLaMA inference ggerganov/llama.cpp
• Example of LLaMA training ggerganov/llama.cpp/examples/baby-llama
• Example of BLOOM inference NouamaneTazi/bloomz.cpp
• Example of RWKV inference saharNooby/rwkv.cpp
• Example of SAM inference
• Idea for GPU support: ggerganov/llama.cpp#915
• Example of StableLM (GPT-NeoX) inference examples/gpt-neox
• Example of BERT inference skeskinen/bert.cpp
• Example of 💫 StarCoder inference examples/starcoder
• Example of MPT inference examples/mpt
• Example of Replit inference examples/replit

With ggml you can efficiently run Whisper inference on the CPU.

Memory requirements:

With ggml you can efficiently run GPT-2 and GPT-J inference on the CPU.

Here is how to run the example programs:

# Build ggml + examples
git clone https://github.com/ggerganov/ggml
cd ggml
mkdir build && cd build
cmake ..
make -j4 gpt-2 gpt-j

# Run the GPT-2 small 117M model
../examples/gpt-2/download-ggml-model.sh 117M
./bin/gpt-2 -m models/gpt-2-117M/ggml-model.bin -p "This is an example"

# Run the GPT-J 6B model (requires 12GB disk space and 16GB CPU RAM)
../examples/gpt-j/download-ggml-model.sh 6B
./bin/gpt-j -m models/gpt-j-6B/ggml-model.bin -p "This is an example"

# Run the Cerebras-GPT 111M model
# Download from: https://huggingface.co/cerebras
python3 ../examples/gpt-2/convert-cerebras-to-ggml.py /path/to/Cerebras-GPT-111M/
./bin/gpt-2 -m /path/to/Cerebras-GPT-111M/ggml-model-f16.bin -p "This is an example"

The inference speeds that I get for the different models on my 32GB MacBook M1 Pro are as follows:

For more information, checkout the corresponding programs in the examples folder.

# fix the path to point to your CUDA compiler
cmake -DGGML_CUBLAS=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.1/bin/nvcc ..

• GGML - Large Language Models for Everyone: a description of the GGML format provided by the maintainers of the llm Rust crate, which provides Rust bindings for GGML
• marella/ctransformers: Python bindings for GGML models.