python3Packages.toktok-rs

Health

Risk
risk 0 ok
Version
0.5.0 no newer release in its git tagsrepology ↗nixkeeper ↗
Update bot
succeeded 2d ago · queued 25,791 of 32,800, next attempt in about 12 days · found by its update script · logs
Channels
master 0.5.0 · nixos-unstable 0.5.0 · nixos-26.05 absent no channel is behind master
Advisories
none open
Packaging
meta complete
Description
Fast, exact BPE tokenizer for OpenAI encodings, with a Rust core
Long description
toktok is a Rust port of quicktok, producing token ids identical to OpenAI's tiktoken for the cl100k, o200k and o200k_harmony encodings, with the vocabulary tables compiled into the extension module so nothing is downloaded at import time.
License
Platforms
31 platforms: Linux, Darwin
Source provenance
not set: built from source
Outputs
out dist
Status
not broken, not insecure, free
Maintainers

Version bumps 0

all open bumps
None open.

Pull requests 0

None open.
Closed or merged in the last 60 days 4
hindsight-api-slim: init at 1.1.0
#570629 by brudel · merged 3w ago
treewide: remove redundant structuredAttrs from Python packages treewide · 62 packages
#584769 by SFrijters · merged 2w ago

Issues 0

None open.