Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test to verify eos_token is working as expected #166

Open
gkumbhat opened this issue Sep 1, 2023 · 0 comments
Open

Add test to verify eos_token is working as expected #166

gkumbhat opened this issue Sep 1, 2023 · 0 comments
Labels
good first issue Good for newcomers

Comments

@gkumbhat
Copy link
Collaborator

gkumbhat commented Sep 1, 2023

Description

In this PR, we identified that the eos_token can be set to None if the model is not "caikit" format converted. This can cause some confusion. We need to better handle eos_token situation, and whether not set it in our modules and always fetch it from the tokenizer or if we do set it then set it consistently and add tests for it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
Status: ToDo
Development

No branches or pull requests

2 participants