Back to List
Notice:This resource is provided by a third-party author. Please review the code with AI tools or manually before use to ensure security and compatibility.
Rusthuggingface/tokenizers

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

86.7/100
10.5KForks: 1.0K
View on GitHubHomepage →
Loading report...

Similar Projects

rust-bert

64

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

Rust3.1K

memvid

88

Memory layer for AI Agents. Replace complex RAG pipelines with a serverless, single-file memory layer. Give your agents instant retrieval and long-term memory.

Rust13.3K

tensorzero

89

TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluation, and experimentation.

Rust11.1K

code2prompt

85

A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.

Rust7.2K
Back to List