| Rust Documentation | Python Documentation | Discord | Matrix |

Mistral.rs is a cross-platform, highly-multimodal inference engine that brings you:

• All-in-one multimodal workflow: text↔text, text+vision↔text, text+vision+audio↔text, text→speech, text→image, text→embeddings
• APIs: Rust, Python, OpenAI HTTP server (with Chat Completions, Responses API), MCP server
• 🔗 MCP Client: Connect to external tools and services automatically (file systems, web search, databases, APIs)
• Performance: ISQ, PagedAttention, FlashAttention, per-layer topology optimization
• Support for embedding, speech generation, and image generation models

Please submit requests for new models here.

1. Install

2. Get models

3. Deploy with our easy to use APIs

   • Python
   • Rust
   • OpenAI-compatible HTTP server
   • Interactive mode
   • 🔗 MCP Client - Connect to external tools automatically

4. Try the web chat app for local in-browser conversation (text, vision, and speech support):

   • Quickstart here
   • Run the server and visit http://localhost:8080 by default.

🖥️ Web Chat App

Try our modern in-browser chat with text, vision, and speech support (TTS generation).

💻 Terminal Interactive Mode

Prefer the terminal? Use interactive mode for a classic CLI experience.

After following installation instructions

• 🔎 Generate embeddings with EmbeddingGemma or Qwen3 Embedding across APIs: EmbeddingGemma guide | Qwen3 guide | overview

  Show commands

  # HTTP API (OpenAI-compatible)
  ./mistralrs-server --port 1234 run -m google/embeddinggemma-300m
  
  # Python API example
  python examples/python/embedding_gemma.py
  
  # Rust API example
  cargo run --package mistralrs --example embedding_gemma
  
  # Qwen3 Embedding server
  ./mistralrs-server --port 1234 run -m Qwen/Qwen3-Embedding-0.6B
  
  # Qwen3 Embedding Python example
  python examples/python/qwen3_embedding.py
  
  # Qwen3 Embedding Rust example
  cargo run --package mistralrs --example qwen3_embedding

• 💎🪆💎🪆💎 Run the Gemma 3n family (E2B, E4B) with vision, audio, and MatFormer support: documentation

  Show commands

  Normal use, run the full model (E4B or E2B):

  ./mistralrs-server -i --isq 8 run -m google/gemma-3n-E4B-it

  Use MatFormer to get a balanced smaller model:

  ./mistralrs-server -i --isq 8 run -m google/gemma-3n-E4B-it \
    --matformer-config-path matformer_configs/gemma3n.csv \
    --matformer-slice-name "Config for E2.49B (block-level)"

• 🧠+📷 Run the Qwen 3 VL reasoning vision models with full tool-calling support: documentation

  Show command

  ./mistralrs-server -i --isq 8 run -m Qwen/Qwen3-VL-4B-Thinking

• 🤗🤗🤗 Run the SmolLM 3 long-context hybrid-reasoning model with full tool-calling support: documentation

  Show command

  Default, easiest:

  ./mistralrs-server -i --isq 8 run -m HuggingFaceTB/SmolLM3-3B

  UQFF prequantized:

  ./mistralrs-server -i run -m EricB/SmolLM3-3B-UQFF -f smollm33b-q4k-0.uqff

• 🔊 Run the Dia 1.6b model for highly-realistic dialogue generation: documentation

  Show command

  ./mistralrs-server -i speech -m nari-labs/Dia-1.6B -a dia

• 🦙 Run the Llama 3.* and Llama 4 models with long context & vision support: docs (llama 3.2), docs (llama 4)

  Show commands

  Llama 4:

  ./mistralrs-server -i --isq 4 run -m meta-llama/Llama-4-Scout-17B-16E-Instruct

  Llama 3.1/3.2/3.3:

  ./mistralrs-server -i --isq 8 run -m meta-llama/Llama-3.2-3B-Instruct
  

  Llama 3.2 vision:

  ./mistralrs-server -i --isq 8 run -m meta-llama/Llama-3.2-11B-Vision-Instruct
  

• 💎💎💎 Run the Gemma 3 family (1b, 4b, 12b, 27b) with 128k context & vision support: documentation

  Show command

  ./mistralrs-server -i --isq 8 run -m google/gemma-3-4b-it

• 🌲📷 Run the FLUX.1 diffusion model: documentation

  Show command

  ./mistralrs-server -i diffusion -m black-forest-labs/FLUX.1-schnell -a flux

• 🧠 Run the Qwen 3 hybrid-reasoning model with full tool-calling support: documentation

  Show command

  ./mistralrs-server -i --isq 8 run -m Qwen/Qwen3-8B

• 🔗 MCP Client - Connect to external tools and services automatically: Quick Start Guide

  Show examples

  1. Create config file (mcp-config.json):

  {
    "servers": [{
      "name": "Filesystem Tools",
      "source": {
        "type": "Process",
        "command": "npx",
        "args": ["@modelcontextprotocol/server-filesystem", "/tmp", "-y"]
      }
    }],
    "auto_register_tools": true
  }

  2. Start server with tools:

  ./mistralrs-server --mcp-config mcp-config.json --port 1234 run -m Qwen/Qwen3-4B

  3. Tools work automatically:

  curl -X POST http://localhost:1234/v1/chat/completions \
    -d '{"model":"Qwen/Qwen3-4B","messages":[{"role":"user","content":"List files in /tmp and create hello.txt"}]}'

  Python API:

  mcp_config = mistralrs.McpClientConfigPy(
      servers=[mistralrs.McpServerConfigPy(
          name="Filesystem",
          source=mistralrs.McpServerSourcePy.Process(
              command="npx",
              args=["@modelcontextprotocol/server-filesystem", "/tmp", "-y"]
          )
      )],
      auto_register_tools=True
  )
  
  runner = mistralrs.Runner(
      which=mistralrs.Which.Plain(model_id="Qwen/Qwen3-4B"),
      mcp_client_config=mcp_config
  )
  # Tools automatically available!

  Rust API:

  let model = TextModelBuilder::new("Qwen/Qwen3-4B")
      .with_mcp_client(mcp_config) // Tools automatically available!
      .build().await?;

• ⚡ Smart Per-Layer Optimization - Fine-tune quantization and device placement per layer: documentation

  Show examples

  Optimize memory usage with mixed quantization (fits large models in limited VRAM):

  # Use aggressive quantization on less important layers, preserve quality on critical ones
  ./mistralrs-server -i --topology topologies/isq.yml run -m meta-llama/Llama-3.2-8B-Instruct

  Example topology file (topologies/isq.yml):

  # Early layers: lower quantization for embeddings
  0-8:
    isq: Q3K
  # Middle layers: balanced quantization
  8-24:
    isq: Q4K
  # Final layers: higher quality for output
  24-32:
    isq: Q6K

  Advanced: Target specific components with regex patterns:

  # Quantize attention layers differently from FFN layers
  '/attn\.q_proj$/':
    isq: Q4K
  '/ffn_.*\.weight$/':
    isq: Q3K

  Multi-device deployment (split across GPUs/CPU):

  0-16:
    isq: Q4K
    device: cuda[0]
  16-32:
    isq: Q4K
    device: cuda[1]
  # Or offload some layers to CPU for very large models

  Python example:

  runner = mistralrs.Runner(
      which=mistralrs.Which.Plain(
          model_id="meta-llama/Llama-3.2-8B-Instruct",
          topology="topologies/isq.yml",
      ),
  )

mistral.rs is a blazing-fast, cross-platform LLM inference engine with support for text, vision, image generation, and speech.

Key Benefits:

1. Ease of Use

   • OpenAI-compatible HTTP server
   • Rust API & Python API
   • Automatic device mapping (multi-GPU, CPU)
   • Chat templates & tokenizer auto-detection
   • MCP server for structured, realtime tool calls
   • ⭐ MCP client to connect to external tools and services automatically

2. Performance

   • CPU acceleration (MKL, AVX, NEON, Accelerate)
   • GPU acceleration (CUDA with FlashAttention & cuDNN, Metal)
   • Automatic tensor parallelism for splitting models across multiple devices
     • CUDA-specialized NCCL
     • Heterogeneous, flexible Ring backend

3. Quantization & Optimization

   • ⭐ Per-layer topology: Fine-tune quantization per layer for optimal quality/speed balance
   • In-place quantization (ISQ) of Hugging Face models
   • GGML & GGUF support: 2–8 bit
   • GPTQ, AWQ, AFQ, HQQ, FP8, BNB (int8/fp4/nf4)
   • ⭐ Auto-select the fastest quant method
   • KV cache quantization

4. Flexibility

   • LoRA & X-LoRA adapters with weight merging
   • AnyMoE: create MoE models on any base model
   • Sampling & penalty options
   • Prompt chunking for large inputs
   • Integrated tool calling with customizable Python/Rust native tool and search callbacks

5. Advanced Features

   • High-throughput with PagedAttention & FlashAttention V2/V3
   • Prefix caching (including multimodal)
   • UQFF format for custom quantization
   • Speculative decoding across models
   • ⭐ Agentic web search integration

Rust multithreaded/async API for easy integration into any application.

• Docs
• Examples including MCP client integration
• To use: add mistralrs = { git = "https://github.com/EricLBuehler/mistral.rs.git" } to your Cargo.toml
• MCP Client: Connect to external tools automatically - Quick Start

Python API for mistral.rs.

• Installation including PyPI
• Docs
• Examples including MCP client usage
• Cookbook
• MCP Client: Full MCP integration - Quick Start

OpenAI API compatible API server

• API Docs - includes chat completions, completions, and Responses API for stateful conversations
• Launching the server or use the CLI
• Example
• Responses API examples - maintain conversation context without resending history
• Use or extend the server in other axum projects
• MCP Client: Configure via --mcp-config flag for automatic tool integration - Quick Start

Serve the same models over the open MCP (Model Context Protocol) in parallel to the HTTP API:

./mistralrs-server --mcp-port 4321 plain -m Qwen/Qwen3-4B

See the docs for feature flags, examples and limitations.

• Docs: https://docs.llamaindex.ai/en/stable/examples/llm/mistral_rs/

To enable one or more features, pass them to Cargo. For example:

cargo build --release --features "cuda flash-attn cudnn"

Note for Linux users: The metal feature is macOS-only and should not be used on Linux. Use --features "cuda flash-attn cudnn" for NVIDIA GPUs or --features mkl for Intel CPUs instead of --all-features.

Note: You can use our Docker containers here. Learn more about running Docker containers: https://docs.docker.com/engine/reference/run/

• Install the Python package here.
• The Python package has wheels on PyPi!

1. Install required packages:

   • OpenSSL (Example on Ubuntu: sudo apt install libssl-dev)
   • Linux only: pkg-config (Example on Ubuntu: sudo apt install pkg-config)

2. Install Rust: https://rustup.rs/

   Example on Ubuntu:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   source $HOME/.cargo/env

3. Optional: Set HF token correctly (skip if already set or your model is not gated, or if you want to use the token_source parameters in Python or the command line.)

   • Note: you can install huggingface-cli as documented here.

   huggingface-cli login

4. Download the code:

   git clone https://github.com/EricLBuehler/mistral.rs.git
   cd mistral.rs

5. Build or install mistralrs-server:

   • Build the mistralrs-server binary, which can be found at target/release/mistralrs-server.

     cargo build --release --features <specify feature(s) here>

   • Install with cargo install for easy command line usage

     Pass the same values to --features as you would for cargo build

     cargo install --path mistralrs-server --features <specify feature(s) here>

6. (If you used cargo build) The build process will output a binary mistralrs-server at ./target/release/mistralrs-server. We can switch to that directory so that the binary can be accessed as ./mistralrs-server with the following command:

   Example on Ubuntu:

   cd target/release
   

7. Use our APIs and integrations:

   APIs and integrations list

Mistral.rs uses subcommands to control the model type. Please run ./mistralrs-server --help to see the subcommands which categorize the models by kind.

🚨 Important: The run subcommand (alias for plain/vision-plain) only auto-detects and runs text and vision models. It does not support diffusion or speech models. To run a diffusion model (e.g. FLUX series), use the diffusion subcommand:

mistralrs-server -i diffusion -m <model-id> [options]

To run a speech model (e.g. Dia), use the speech subcommand:

mistralrs-server -i speech -m <model-id> [options]

If you attempt to use run with diffusion or speech models, model loading will fail.

These options apply across all subcommands:

Llama 3.2 3B running on an M3 Max with 8-bit ISQ:

You can launch interactive mode, a simple chat application running in the terminal, by passing -i:

./mistralrs-server -i plain -m meta-llama/Llama-3.2-3B-Instruct

Vision models work seamlessly:

./mistralrs-server -i vision-plain -m lamm-mit/Cephalo-Llama-3.2-11B-Vision-Instruct-128k

Diffusion models can be run too (quantization and adapters are not yet supported):

./mistralrs-server -i diffusion -m black-forest-labs/FLUX.1-schnell -a flux

And you can run speech generation in your terminal!

./mistralrs-server -i speech -m nari-labs/Dia-1.6B -a dia

You can launch an HTTP server by replacing -i with --port <port>. For instance:

./mistralrs-server --port 1234 run -m microsoft/Phi-3.5-MoE-instruct

You can find documentation about the server itself here.

Serve multiple models simultaneously from a single server instance. This is useful for comparing models, A/B testing, or serving different models for different use cases.

./mistralrs-server --port 1234 multi-model --config example-multi-model-config.json --default-model-id meta-llama/Llama-3.2-3B-Instruct

Select models in your requests using the model parameter:

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.2-3B-Instruct", "messages": [{"role": "user", "content": "Hello!"}]}'

📖 Complete multi-model documentation →

We provide a method to select models with a .toml file. The keys are the same as the command line, with no_kv_cache and tokenizer_json being "global" keys.

Example:

./mistralrs-server --port 1234 toml -f toml-selectors/gguf.toml

Note: for plain models, you can specify the data type to load and run in. This must be one of f32, f16, bf16 or auto to choose based on the device. This is specified in the --dype/-d parameter after the model architecture (plain). For quantized models (gguf/ggml), you may specify data type of f32 or bf16 (f16 is not recommended due to its lower precision in quantized inference).

If you do not specify the architecture, an attempt will be made to use the model's config. If this fails, please raise an issue.

Note: for vision models, you can specify the data type to load and run in. This must be one of f32, f16, bf16 or auto to choose based on the device. This is specified in the --dype/-d parameter after the model architecture (vision-plain).

Note: for embedding models, you can specify the data type to load and run in. This must be one of f32, f16, bf16 or auto to choose based on the device. This is specified in the --dype/-d parameter after the model architecture (vision-plain).

Please submit more benchmarks via raising an issue!

To use a derivative or adapter model (e.g., quantized, LoRA, X-LoRA, vision, etc.), select the correct architecture subcommand and pass the required arguments—typically model id, and for quantized/adapters, also the quantization filename, tokenizer, or adapter ordering if needed.

• See all options: Run ./mistralrs-server <subcommand> --help
• Docs: Adapter models, Chat templates

./mistralrs-server --port 1234 --log output.txt gguf -t HuggingFaceH4/zephyr-7b-beta -m TheBloke/zephyr-7B-beta-GGUF -f zephyr-7b-beta.Q5_0.gguf

Chat template and tokenizer are usually auto-detected.
If you need to override, see the chat templates doc.

An adapter model is a model with X-LoRA or LoRA. X-LoRA support is provided by selecting the x-lora-* architecture, and LoRA support by selecting the lora-* architecture. Please find docs for adapter models here. Examples may be found here.

Mistral.rs will attempt to automatically load a chat template and tokenizer. This enables high flexibility across models and ensures accurate and flexible chat templating. However, this behavior can be customized. Please find detailed documentation here.

Thank you for contributing! If you have any problems or want to contribute something, please raise an issue or pull request. If you want to add a new model, please contact us via an issue and we can coordinate how to do this.

• Debugging with the environment variable MISTRALRS_DEBUG=1 causes the following things
  • If loading a GGUF or GGML model, this will output a file containing the names, shapes, and types of each tensor.
    • mistralrs_gguf_tensors.txt or mistralrs_ggml_tensors.txt
  • More logging.
• Setting the CUDA compiler path:
  • Set the NVCC_CCBIN environment variable during build.
• Error: recompile with -fPIE:
  • Some Linux distributions require compiling with -fPIE.
  • Set the CUDA_NVCC_FLAGS environment variable to -fPIE during build: CUDA_NVCC_FLAGS=-fPIE
• Error CUDA_ERROR_NOT_FOUND or symbol not found when using a normal or vison model:
  • For non-quantized models, you can specify the data type to load and run in. This must be one of f32, f16, bf16 or auto to choose based on the device.
• What is the minimum supported CUDA compute cap?
  • The minimum CUDA compute cap is 5.3.
• Metal not found (error: unable to find utility "metal", not a developer tool or in PATH)
  1. Install Xcode: xcode-select --install
  2. Set the active developer directory: sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
• Disabling Metal kernel precompilation:
  • By default, Metal kernels are precompiled during build time for better performance
  • To skip Metal kernel precompilation (useful for CI or when Metal is not needed), set MISTRALRS_METAL_PRECOMPILE=0 or MISTRALRS_METAL_PRECOMPILE=false
  • Example: MISTRALRS_METAL_PRECOMPILE=0 cargo build --release --features metal
• Disabling mmap loading
  • Set MISTRALRS_NO_MMAP=1 to disable mmap during loading.

For multi-node setups, configure the head node and workers:

Head Node:

Worker Nodes:

When running the HTTP server, these defaults apply:

When a text or vision model is loaded in a multi-threaded runtime, mistral.rs automatically performs a warmup ("dummy") run:

• Sends a short completion request ("hello" with max 1 token) to initialize CUDA kernels and caches
• Logs "Beginning dummy run." when starting and "Dummy run completed in Xs." when finished
• Helps ensure more consistent performance for the first real user request
• Only runs for text and vision models (not diffusion/speech)

If the inference engine thread dies unexpectedly (e.g., due to a panic), mistral.rs can automatically recover:

• Detects dead engine threads when sending requests
• Automatically reboots the engine using saved configuration
• Logs "Engine {model_id} is dead, rebooting" followed by "Successfully rebooted engine {model_id}"
• Preserves all original configuration including KV cache settings, prefix cache, and tool callbacks

This ensures high availability without manual intervention.

This project would not be possible without the excellent work at candle. Additionally, thank you to all contributors! Contributing can range from raising an issue or suggesting a feature to adding some new functionality.

⬆️ Back to Top