with_structured_output not working with models on retrieval chains. Its only working on a raw chain I create myself

[AryanMadhavVerma]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

### This works
model = ChatOpenAI(
            model="###",
            temperature=0.5,
        ).with_structured_output(PydanticTemplate, method="json_mode")
prompt = ChatPromptTemplate.from_messages([(...), ("...Response in json...")])
chain = prompt | model


### This doesn't work 
model = ChatOpenAI(
            model="###",
            temperature=0.5,
        ).with_structured_output(PydanticTemplate, method="json_mode")
prompt = ChatPromptTemplate.from_template("""
        Answer the user's question:
        Context: {context}
        Question: {input}
        Respond in json
        """
       )
chain = create_stuff_documents_chain(
            llm=model,
            prompt=prompt,
        )
retriever = vector_store.as_retriever()
retrieval_chain = create_retrieval_chain(
            retriever,
            chain
        )

Description

with_structured_output() is not working when I use it in a model which I pipe with a combination of create_stuff_documents_chain and create_retrieval_chain

It doesn't respond with the Pydantic I've specified and gives error :-
{"detail": "Error invoking retriever chain: 1 validation error for Generation\ntext\n str type expected (type=type_error.str)"}

The method with_structured_output() is working well with my other chain but is creating an issue in my retrieval chain.

Any help would be appreciated!

System Info

langchain==0.1.9
langchain-community==0.0.24
langchain-core==0.1.26
langchain-openai==0.0.7

[dosubot[bot]]

🤖

Based on the error message you provided, it seems like the output from the retrieval model is not matching the expected schema defined by with_structured_output(). This is likely causing the validation error you're seeing.

In the LangChain framework, the with_structured_output() function is designed to work with pydantic models (BaseModel) or dictionaries that define the schema of the expected output. These schemas are then used to parse and validate the output from the language model (LLM). When using raw chains, the output can be directly parsed and validated against these schemas without much complication.

However, when using models on retrieval chains, the output from the retrieval model may not directly match the expected schema, leading to validation errors. This is because the retrieval model's output is expected to be a string (the retrieved text), but the structured output mechanism expects the output to match the defined schema (e.g., a pydantic model or a dictionary defining a JSON schema). When the output does not match the expected schema, a validation error occurs, indicating that a string was expected but a different type was received.

To resolve this issue, you need to ensure that the output from the retrieval model is appropriately formatted or transformed to match the expected schema before it is passed to the structured output validation mechanism. This may involve additional processing steps to convert the retrieval model's output into a format that conforms to the expected structured output schema.

For more details on how the with_structured_output() function works in the LangChain codebase, you can refer to the structured_output.ipynb guide in the LangChain documentation.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• libs/langchain/langchain/chains/structured_output/base.py
• docs/docs/guides/structured_output.ipynb

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nav-dev-2000]

Its help full thankyou

[viren-vii]

The structure_output.ipynb is now moved to https://github.com/langchain-ai/langchain/blob/master/docs/docs/how_to/structured_output.ipynb

[AryanMadhavVerma]

This sounds good. Thanks!

[lelleib]

@AryanMadhavVerma I am facing this exact problem. What was your solution?

[xihuai18]

How can I retry with error message?