← Back to List
⚠
Notice:This resource is provided by a third-party author. Please review the code with AI tools or manually before use to ensure security and compatibility.
Rusthuggingface/tokenizers

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

89.4/100
★ 11.2KForks: 1.2K
View on GitHub →Homepage →
Loading report...

Similar Projects

jan

91

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.

Rust★ 44.8K

memvid

75

Memory layer for AI Agents. Replace complex RAG pipelines with a serverless, single-file memory layer. Give your agents instant retrieval and long-term memory.

Rust★ 16.6K

tensorzero

74

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.

Rust★ 11.7K

code2prompt

85

A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.

Rust★ 7.7K
← Back to List