Skip to content

Instantly share code, notes, and snippets.

View xenova's full-sized avatar

Joshua Lochner xenova

View GitHub Profile
@xenova
xenova / tiktoken-to-hf.ipynb
Last active December 3, 2024 13:51
Convert tiktoken tokenizers to the Hugging Face tokenizers format
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.