Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Select function always selects the same option for a specific prompt. #1082

Open
Haseeb-BSAA opened this issue Nov 23, 2024 · 0 comments
Open

Comments

@Haseeb-BSAA
Copy link

Haseeb-BSAA commented Nov 23, 2024

I'm new to LLMs, so I'm sorry if I'm missing something obvious. However, when I run this simple code:
from guidance import models, gen, select, user, assistant, system, substring
model_path = "Llama-3.2-3B-Instruct-Q4_K_M.gguf"
llm = models.LlamaCpp(model_path, n_gpu_layers=-1)
llm + f'Pick a random number: ' + select(['One',"Two","Three","Four",'Five',"Six","Seven","Eight","Nine","Ten"])

I always get the same output, which is Eight.
If I change the prompt a tiny bit, for example if I remove the colon and the space, it would always get Nine.
What's wrong?

I also tried running the example code on guidance's main github page:
llama2 + f'Do you want a joke or a poem? A ' + select(['joke', 'poem'])
Same thing here, it always picks the same option.

I tried changing top_k and temperature parameters when loading the model, but nothing seems to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant