inputs must have less than 1000 tokens.

[rohan-uiuc]

I am trying to create a QA bot where I have 2 options to get the answer from OpenAI and FLAN-UL2.
I am creating index by scraping a website and from a pdf. As usually done, I am splitting the document and using load_qa_with_sources_chain to create the chain using the created index.
Earlier I used simple chain() to get the response using the similarity_search which worked perfectly for OpenAI GPT.

However, when I use the same for FLAN-UL2, it throws the following error:
ValueError: Error raised by inference API: Input validation error: inputs must have less than 1000 tokens. Given: 2023

I tried the same with VectorDBQAWithSourcesChain and it results in same.

I understood that inputs variable is actually the prompts that is being passed to the _call_ method in llms.base.py but that is created internally. How do I change the size of that? Am I missing something or doing something wrong?

Here is the link to the source https://github.com/rohan-uiuc/makerlab-bot/tree/compare

Any inputs are appreciated.

[Kamalabot]

Stuck with the same issue as above. I have tried many other hugging face models, the issue is persisting across models. Looking at the base.py is not providing any clue as to how to modify the length of the document or tokens fed to the Hugging face LLM.

[karndeepsingh]

Stuck with the same issue as above. I have tried many other hugging face models, the issue is persisting across models. Looking at the base.py is not providing any clue as to how to modify the length of the document or tokens fed to the Hugging face LLM.

@Kamalabot @rohan-uiuc Did you find any solution for the same issue?

[rohan-uiuc]

Token limit for HF models is small as compared to GPT. You can try map_reduce chain instead of stuff. That worked for me, however the responses were not as accurate as compared to GPT3.5.

[Kamalabot]

I simply dumped Chromadb and moved to FAISS. Its more realiable. It works without issue.

[ArchieLuxton]

In my experience, adding line breaks in the text solves this. I.e., there should be a long list of statements instead of one continuous block of text.

[Kamalabot]

After a lot of trial error, the solution is using the persist() method on chromaDB python variable. After persisting, I am able to load it with ease.