Add special_tokens_in_strings
Arg to byte_pair_tokenizer.
#1546
Google CLA / cla/google
succeeded
Apr 2, 2024 in 1s
✅ All contributors are covered under a CLA with Google
See https://cla.developers.google.com/ for more info about Google's Contributor License Agreement (CLA).
ℹ️ Googlers: Go here to view more details and manage scans for this pull request.
Details
The following contributors were found for this pull request:
✅ a2bc5de Author: @abuelnasr0 <abu******70@gmail.com>
(Only the first commit for a unique contributor is listed.)
Loading