Support newer merges format in tokenizer.json files
#392
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`tokenizer.json` files generated using the newest versions of the HuggingFace tools represent the BPE merge list as an array of pairs instead of an array of space-separated strings. Update the `tokenizer.json` parser and `Bpe` model to support this. A `merge_pairs_from_lines` function has been added to convert the old format into the new format for code using the `Bpe` model directly. Fixes #391.
This was generated using transformers v4.45 and tokenizers v0.20.1.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tokenizer.jsonfiles generated using the newest versions of the HuggingFacetools represent the BPE merge list as an array of pairs of strings instead of an array of
space-separated strings. Update the
tokenizer.jsonparser andBpemodel tosupport this. A
merge_pairs_from_linesfunction has been added to convert theold format into the new format for code using the
Bpemodel directly.Fixes #391.