On-Device Model Hub | Documentation | Discord | Blogs | X (Twitter)

Nexa SDK is a local on-device inference framework for ONNX and GGML models, supporting text generation, image generation, vision-language models (VLM), audio-language models, speech-to-text (ASR), and text-to-speech (TTS) capabilities. Installable via Python Package or Executable Installer.

• Device Support: CPU, GPU (CUDA, Metal, ROCm), iOS
• Server: OpenAI-compatible API, JSON schema for function calling and streaming support
• Local UI: Streamlit for interactive model deployment and testing

• Support Nexa AI's own vision language model (0.9B parameters): nexa run omnivision and audio language model (2.9B parameters): nexa run omniaudio
• Support audio language model: nexa run qwen2audio, we are the first open-source toolkit to support audio language model with GGML tensor library.
• Support iOS Swift binding for local inference on iOS mobile devices.
• Support embedding model: nexa embed <model_path> <prompt>
• Support pull and run supported Computer Vision models in GGUF format from HuggingFace or ModelScope: nexa run -hf <hf_model_id> -mt COMPUTER_VISION or nexa run -ms <ms_model_id> -mt COMPUTER_VISION
• Support pull and run NLP models in GGUF format from HuggingFace or ModelScope: nexa run -hf <hf_model_id> -mt NLP or nexa run -ms <ms_model_id> -mt NLP

Welcome to submit your requests through issues, we ship weekly.

macOS Installer

Windows Installer

Linux Installer

curl -fsSL https://public-storage.nexa4ai.com/install.sh | sh

Currently, executable installer does not support local UI. For local UI, please install using python package below.

nexa-exe <command>

We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our index page.

pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/cpu --extra-index-url https://pypi.org/simple --no-cache-dir

CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/metal --extra-index-url https://pypi.org/simple --no-cache-dir

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh
conda create -n nexasdk python=3.10
conda activate nexasdk
CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/metal --extra-index-url https://pypi.org/simple --no-cache-dir

CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

$env:CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON"; pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

set CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" & pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai

CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/rocm621 --extra-index-url https://pypi.org/simple --no-cache-dir

$env:CMAKE_ARGS="-DGGML_VULKAN=on"; pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/vulkan --extra-index-url https://pypi.org/simple --no-cache-dir

set CMAKE_ARGS="-DGGML_VULKAN=on" & pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/vulkan --extra-index-url https://pypi.org/simple --no-cache-dir

CMAKE_ARGS="-DGGML_VULKAN=on" pip install nexaai --prefer-binary --index-url https://github.nexa.ai/whl/vulkan --extra-index-url https://pypi.org/simple --no-cache-dir

git clone --recursive https://github.com/NexaAI/nexa-sdk

git submodule update --init --recursive

pip install -e .

Below is our differentiation from other similar tools:

Our on-device model hub offers all types of quantized models (text, image, audio, multimodal) with filters for RAM, file size, Tasks, etc. to help you easily explore models with UI. Explore on-device models at On-device Model Hub

Supported model examples (full list at Model Hub):

You can pull, convert (to .gguf), quantize and run llama.cpp supported text generation models from HF or MS with Nexa SDK.

Use nexa run -hf <hf-model-id> or nexa run -ms <ms-model-id> to run models with provided .gguf files:

nexa run -hf Qwen/Qwen2.5-Coder-7B-Instruct-GGUF

nexa run -ms Qwen/Qwen2.5-Coder-7B-Instruct-GGUF

Note: You will be prompted to select a single .gguf file. If your desired quantization version has multiple split files (like fp16-00001-of-00004), please use Nexa's conversion tool (see below) to convert and quantize the model locally.

Install Nexa Python package, and install Nexa conversion tool with pip install "nexaai[convert]", then convert models from huggingface with nexa convert <hf-model-id>:

nexa convert HuggingFaceTB/SmolLM2-135M-Instruct

Or you can convert models from ModelScope with nexa convert -ms <ms-model-id>:

nexa convert -ms Qwen/Qwen2.5-7B-Instruct

Note: Check our leaderboard for performance benchmarks of different quantized versions of mainstream language models and HuggingFace docs to learn about quantization options.

📋 You can view downloaded and converted models with nexa list

Here's a brief overview of the main CLI commands:

• nexa run: Run inference for various tasks using GGUF models.
• nexa onnx: Run inference for various tasks using ONNX models.
• nexa convert: Convert and quantize huggingface models to GGUF models.
• nexa server: Run the Nexa AI Text Generation Service.
• nexa eval: Run the Nexa AI Evaluation Tasks.
• nexa pull: Pull a model from official or hub.
• nexa remove: Remove a model from local machine.
• nexa clean: Clean up all model files.
• nexa list: List all models in the local machine.
• nexa login: Login to Nexa API.
• nexa whoami: Show current user information.
• nexa logout: Logout from Nexa API.

For detailed information on CLI commands and usage, please refer to the CLI Reference document.

To start a local server using models on your local computer, you can use the nexa server command. For detailed information on server setup, API endpoints, and usage examples, please refer to the Server Reference document.

Swift SDK: Provides a Swifty API, allowing Swift developers to easily integrate and use llama.cpp models in their projects.

More Docs

We would like to thank the following projects:

• llama.cpp
• stable-diffusion.cpp
• bark.cpp
• optimum