semantic-router

Manages Unified Access to Generative AI Services built on Envoy Gateway

Fastest enterprise AI gateway (50x faster than LiteLLM) with adaptive load balancer, cluster mode, guardrails, 1000+ models support & <100 µs overhead at 5k RPS.

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

🏗️ Fine-tune, build, and deploy open-source LLMs easily!