tok

tok is an R package that provides bindings to the Hugging Face tokenizers library, letting you tokenize and detokenize text directly in R using the same Rust backend that powers the Python implementation.

The package can load pretrained tokenizers from JSON files or pull them directly from the Hugging Face Hub using from_pretrained. It supports encoding text into token IDs and decoding them back, making it useful for preparing inputs to transformer models. Because it relies on Rust via extendr rather than a pure-R implementation, it offers performance comparable to the Python tokenizers library.