Response includes unwanted examples

[momassimo]

Hi,

I'm using AzureOpenAI with gpt 3.5 Turbo and simply want to receive a product categorization for a given product description.
Unfortunately the output I receive always contains some kind of examples for other (fictional) product descriptions.
Here is an example:
Prompt:

You are an AI great at categorising product descriptions to categories. Given a product description, return the most detailed suggested category in German. The Output should constist of ONLY the final category and not any higher level categories, nore any examples.
Desired format:
<string_without_any_examples>

Input: 'slush calippo strohhalm 5000st slush calippo straw rot trinkhalm karton 5000 stück trocken'
Output:

Chat output:

'Einweg-Trinkhalme'

Input: 'kaffeebecher 0,2l 50 stück kaffeebecher 0,2l 50 stück'
Output (restricted to max tokens = 200):
'Kaffeebecher'

Input: 'kaffeebecher 0,2l 50 stück kaffeebecher 0,2l 50 stück kaffeebecher 0,2l 50 stück'
Output:
'Kaffeebecher'

Input: 'kaffeebecher 0,2l 50 stück kaffeebecher 0,2l 50 stück kaffeebecher 0,2l 50 stück kaffeebecher 0,2l 50 stück'
Output:
'Kaffeebecher'

Input: 'kaffeebecher 0,2l 50 stück kaffeebecher 0,2l

The fuction to generate the output is the following:

def generate_text(self, text: Text, artikel: Text) -> Text:
    prompt = PromptTemplate(template=text, input_variables=["artikel_text"])
    llm_chain = LLMChain(prompt=prompt, llm=model)
    response = llm_chain.run(artikel).strip()
    return response

template = '''
You are an AI great at categorising product descriptions to categories. Given a product description, return the most detailed suggested category in German. The Output should constist of ONLY the final category and not any higher level categories, nore any examples.
Desired format:
<string_without_any_examples>

Input: '{artikel_text}'
Output:
'''

artikel = "slush calippo strohhalm 5000st slush calippo straw rot trinkhalm karton 5000 stück trocken"
generate_text(text=template, artikel=value)

Does anybody know how to cope with this issue? (besides parsing the result from the long output, as this will produce unnecessary tokens -> costs)

[dosubot[bot]]

🤖

Hello,

Based on your description, it seems like you want to stop the generation of text once the final category in German is produced. To achieve this, you can use the stop parameter in the _create_message_dicts method of the ChatOpenAI class in the openai.py file. This parameter accepts a list of strings to stop generation when encountered.

You can set the stop parameter to include the tokens that signify the end of the category in German. This way, the model will stop generating text once it encounters these tokens, ensuring that only the final category is included in the output.

Here is an example of how you can modify the _create_message_dicts method:

def _create_message_dicts(
        self, messages: List[BaseMessage], stop: Optional[List[str]]
    ) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
        params = self._client_params
        if stop is not None:
            if "stop" in params:
                raise ValueError("`stop` found in both the input and default params.")
            params["stop"] = stop
        message_dicts = [convert_message_to_dict(m) for m in messages]
        return message_dicts, params

In the above code, you can set the stop parameter when calling the model like this:

stop_tokens = ["<end_of_category_in_German>"]
response = model(prompt, n_predict=55, stop=stop_tokens)

Please replace <end_of_category_in_German> with the actual token that signifies the end of the category in German.

You can find more details about this in the libs/langchain/langchain/llms/gpt4all.py and libs/langchain/langchain/chat_models/openai.py files.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/langchain/langchain/llms/gpt4all.py
• libs/langchain/langchain/chat_models/openai.py
• libs/langchain/langchain/llms/textgen.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mspronesti]

Are you using the ChatCompletion API (i.e. your model is of type langchain.chat_models.OpenAIChat or the completion API (i.e yyour model is of type langchain.llms.OpenAI)?