← Back to List
⚠
Notice:This resource is provided by a third-party author. Please review the code with AI tools or manually before use to ensure security and compatibility.
Rusthuggingface/tokenizers

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

89.3/100
★ 10.9KForks: 1.2K
View on GitHub →Homepage →
Loading report...

Similar Projects

memvid

87

Memory layer for AI Agents. Replace complex RAG pipelines with a serverless, single-file memory layer. Give your agents instant retrieval and long-term memory.

Rust★ 16.0K

tensorzero

81

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.

Rust★ 11.7K

code2prompt

81

A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.

Rust★ 7.5K

tokscale

88

🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with quadrillions of tokens tracked.

Rust★ 4.6K
← Back to List