Rare Tokens

[JustAnOkapi]

https://gist.github.com/JustAnOkapi/80be6959476599f1ab899f948a140eba
I filtered and organized the list of all tokens sd can read. It includes longer than one token words (merged).

wouldnt the best training option be to add a word to vocab.json that it doesnt know?
the word ive seen used ohwx is number 943, it is known just not very well.

[TheLastBen]

the text encoder and vocab.json go together I think, you can't simply add a token to vocab

[JustAnOkapi]

Ill try it

[JustAnOkapi]

oh yea to update,
it kills the model

[JustAnOkapi]

even replacing a token. 💀

[JustAnOkapi]

trying again with
tokenizer.add_tokens(list(['<token>']))
text_encoder.resize_token_embeddings(len(tokenizer))

[JustAnOkapi]

trained one step on each, testing if model still works.

[JustAnOkapi]

nope