Back to List
Notice:This resource is provided by a third-party author. Please review the code with AI tools or manually before use to ensure security and compatibility.
Rusthuggingface/tokenizers

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

89.4/100
10.7KForks: 1.1K
View on GitHubHomepage →
Loading report...

Similar Projects

rust-bert

59

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

Rust3.1K

memvid

80

Memory layer for AI Agents. Replace complex RAG pipelines with a serverless, single-file memory layer. Give your agents instant retrieval and long-term memory.

Rust15.1K

tensorzero

89

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.

Rust11.3K

code2prompt

85

A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.

Rust7.3K
Back to List