Mooncake

Community maintained hardware plugin for vLLM on Ascend

Production ready toolkit to run AI locally

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, CPU only, lossless, on stock llama.cpp