FlashInfer: Kernel Library for LLM Serving
-
Updated
Mar 25, 2026 - Python
#
distributed-inference
Here are 20 public repositories matching this topic...
A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Mar 25, 2026 - Python
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
-
Updated
Aug 24, 2025 - JavaScript
Code for paper "JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks"
compression deep-learning distributed-computing inference dnn neural-networks pruning quantization dnn-optimization edge-computing mobilenet edge-cloud ai-models dnn-partition distributed-inference
-
Updated
Aug 24, 2025 - Python
Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.
csv s3 pandas data-engineering parquet data-processing data-pipelines observability synthetic-data unstructured-data mlops batch-inference polars llm-inference evals distributed-inference
-
Updated
Mar 17, 2026 - Python
Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads.
-
Updated
Jan 21, 2026 - Python
Source code of the paper "Private Collaborative Edge Inference via Over-the-Air Computation".
machine-learning ensemble-learning wireless-communications differential-privacy multi-view-learning edge-inference over-the-air-computation collaborative-inference distributed-inference
-
Updated
Jan 14, 2025 - Python
Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics
python caching machine-learning streaming performance ai multi-node http2 load-balancer rpc asyncio ray model-serving gpu-monitoring inference-optimization llm llama-cpp ollama distributed-inference hybrid-routing
-
Updated
Nov 14, 2025 - Python
Pool your CUDA + ROCm GPUs into one OpenAI-compatible API. Speculative decoding proxy gives you 2-3x faster inference — for free, using hardware you already own. Stop renting GPU clouds. Be a tightwad.
python open-source machine-learning ai amd gpu cuda inference self-hosted nvidia homelab rocm inference-engine llm llm-inference local-ai speculative-decoding distributed-inference openai-compatible ollama-alternative
-
Updated
Feb 20, 2026 - Python
Web UI for orchestrating distributed llama.cpp RPC GPU clusters with auto node discovery, telemetry, and one-click deployment.
python flask rpc rpc-server rpc-client gpu-cluster llm llama-cpp gguf distributed-inference ai-cluster gguf-models ai-cluster-automation
-
Updated
Mar 8, 2026 - JavaScript
Official impl. of ACM MM paper "Identity-Aware Attribute Recognition via Real-Time Distributed Inference in Mobile Edge Clouds". A distributed inference model for pedestrian attribute recognition with re-ID in an MEC-enabled camera monitoring system. Jointly training of pedestrian attribute recognition and Re-ID.
mec re-identification edge-computing re-id attribute-recognition mobile-edge-computing distributed-inference
-
Updated
Apr 26, 2020 - Python
Turn any Kubernetes Cluster into a private LLM endpoint. One Helm command deploys distributed inference across commodity hardware. Raspberry Pi's, old servers, mixed architectures. OpenAI-Compatible API Powered by llama.cpp RPC
docker kubernetes open-source raspberry-pi machine-learning ai gpu helm decentralized distributed-computing inference self-hosted homelab cluster-computing edge-computing large-language-models llm llm-inference ollama distributed-inference
-
Updated
Mar 4, 2026 - Dockerfile
A comprehensive framework for multi-node, multi-GPU scalable LLM inference on HPC systems using vLLM and Ollama. Includes distributed deployment templates, benchmarking workflows, and chatbot/RAG pipelines for high-throughput, production-grade AI services
hpc rag ai-inference vllm llm-inference ollama distributed-inference vllm-serve ai-inference-server multinode-training
-
Updated
Dec 10, 2025 - Python
Encrypted Decentralized Inference and Learning (E.D.I.L.)
deep-neural-networks deep-learning he federated-learning decentralized-training distributed-inference ratio1
-
Updated
Jan 20, 2026 - Python
A cache-centric architecture, compatibility contracts, and protocols for KV cache handoff in LLM inference.
-
Updated
Mar 22, 2026
Distributed LLM inference across multiple machines. A central server routes OpenAI-compatible requests to llama.cpp client nodes, with automatic model distribution and mutual TLS security.
docker golang grpc openai inference-server llm llm-serving llama-cpp llm-inference local-ai gguf distributed-inference
-
Updated
Mar 11, 2026 - Go
🚀 Master GPU kernel programming and optimization for high-performance AI systems with this comprehensive learning guide and resource hub.
android java docker opengl gpu interview perf performance-analysis gpuimage puzzles profiling rocm ascend llm llm-serving genai llm-inference qwen distributed-inference mindie
-
Updated
Mar 26, 2026
Distributed inference across heterogeneous hardware.
python pytorch model-parallelism heterogeneous-computing pipeline-parallelism llm distributed-inference
-
Updated
Mar 13, 2026 - Python
Practical guide to clustering NVIDIA DGX Spark nodes for multi-node vLLM inference (NCCL, RoCE, Ray), with troubleshooting playbooks and step-by-step notebooks.
-
Updated
Mar 25, 2026 - Jupyter Notebook
Improve this page
Add a description, image, and links to the distributed-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the distributed-inference topic, visit your repo's landing page and select "manage topics."