CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

Multimodal LLM inference gateway with KV-cache-aware routing and LMCache offload. OpenAI-compatible, benchmarked on GPUs.

Collection of benchmarks, benchmark drivers, harnesses, plotting, and various scripts for KV cache offloading in vLLM.

Benchmarking LMCache under simulated RTT

Provide fast, memory-efficient caching for language models to improve response times and reduce redundant computations.