Distribute and run LLMs with a single file.
LLM speculative inference server for heterogeneous hardware & consumer GPUs
jevos is an open-source alternative to Jev for yes/no decisions that runs on your laptop.
Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, lossless, on stock llama.cpp
Port of OpenAI's Whisper model in C/C++