lmdeploy

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

中文LLaMA-2 & Alpaca-2大模型二期项目 + 64K超长上下文模型 (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs