python3Packages.toktok-rs
toktok-rs 0.5.0 · pkgs/development/python-modules/toktok-rs/default.nix
Health
Risk
risk 0 ok
Version
Update bot
Channels
Hydra, nixos-26.05
Advisories
none open
Packaging
meta complete
Packagewhat each field means
Description
Fast, exact BPE tokenizer for OpenAI encodings, with a Rust core
Long description
toktok is a Rust port of quicktok, producing token ids identical to
OpenAI's tiktoken for the cl100k, o200k and o200k_harmony encodings,
with the vocabulary tables compiled into the extension module so
nothing is downloaded at import time.
License
Platforms
31 platforms: Linux, Darwin
Source provenance
not set: built from source
Outputs
out distStatus
not broken, not insecure, free
Maintainers
Version bumps 0
all open bumpsNone open.
Pull requests 0
None open.
Closed or merged in the last 60 days 4
treewide: remove redundant structuredAttrs from Python packages treewide · 62 packages
#584769 by SFrijters · merged 2w ago
Issues 0
None open.