PowerInfer

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

A highly optimized LLM inference acceleration engine for Llama and its variants.

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.