EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
Learning BPE embeddings by first learning a segmentation model and then training word2vec
A curated list of tokenizer libraries for blazing-fast NLP processing.
Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.
Detect whether the text is AI-generated by training a new tokenizer and combining it with tree classification models or by training language models on a large dataset of human & AI-generated texts.
Byte Pair Encoding (BPE)
WordPiece Tokenizer for BERT models.
See exactly how AI reads your text. Watch GPT-5.6, Claude Fable 5, Gemini 3.1, Llama 4, Kimi K3 & 20+ models split your prompt into tokens - live, color-coded, byte-exact token IDs. 100% client-side, zero dependencies, no API keys, free forever.
A pure Dart implementation of the BERT-compatible WordPiece tokenizer.
Fast wordpiece, sentencepiece tokenizer by Trie, OpenMP, SIMD, MemoryPool
A framework for generating subword vocabulary from a tensorflow dataset and building custom BERT tokenizer models.
A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeniser. Unlike traditional BPE, this tokeniser selects merge candidates based on a normalized likelihood score, improving tokenisation quality by prioritising statistically significant merges.
⚡ Blazing-fast, universal tokenization engine in Rust. Unifies BPE, WordPiece, and Unigram into a single trie-walker automaton with AVX2 SIMD acceleration. Achieves 3.02+ GB/s multi-core throughput on consumer hardware. Features native HuggingFace, tiktoken, and GGUF support.
From-scratch Java port of BERT WordPiece tokenization for on-device/Android NLP preprocessing.
To associate your repository with the wordpiece topic, visit your repo's landing page and select "manage topics."