Retry chain with updated prompt including validation errors

[LoopKarma]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import logging
from datetime import datetime, date, time
from typing import Optional
from langchain_openai.chat_models import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field, field_validator, model_validator, ValidationError


class MatchingResult(BaseModel):
    """Response from the LLM agent.

       Args:
           id: identifier matching the input string
           confidence: confidence score of the match
           reason: reason for the match
    """
    id: Optional[int] = Field(description="identifier matching the input string")
    confidence: int = Field(description="confidence score of the match")
    reason: str = Field(description="reason for the match")

    @field_validator('id')
    def check_location_id(cls, value):
        possible_values = [1, 2, 3]
        if value is not None and value not in possible_values:
            raise ValueError(f'id {value} is not a valid value')
        return value


class MatchingInput(BaseModel):
    content: str = Field(description="input string")
    options_map: dict[int, str] = Field(description="map of options for matching")


class Matcher:
    def __init__(self):
        self.llm = (ChatOpenAI(
            model="gpt-4o-mini",
            temperature=0,
            request_timeout=60
        ).with_structured_output(MatchingResult)
        .with_retry(
            retry_if_exception_type=(ValidationError,),
            stop_after_attempt=3,
        ))
        self.prompt = ChatPromptTemplate.from_messages(
            [
                ("system", "You are a helpful assistant"),
                ("user",
                 "You should match the input string:\n"
                 "{content}\n to the provided options map."
                 "**HERE SOME RULES FOR MATCHING**\n"
                 "Map of options to select from IDs to names:\n"
                 "{options_map}\n"),
            ]
        )

        self.chain = (
            (self.prompt | self.llm)
            .with_types(input_type=MatchingInput, output_type=MatchingResult)
        )

    async def match_location(self, content: str, locations: dict[int, str]) -> (LocationMatchingResult, str):
        try:
            result = await self.chain.ainvoke(LocationMatchingInput(content=content, locations_map=locations))
            return result, None
        except ValidationError as e:
            logging.error(f"Error matching  from string {content}. Error(s): {e.errors()} ({type(e)})")
            # Here I want to change to prompt and include the error message and repeat the request
            return None, e.json()
        except Exception as e:
            logging.error(f"Error while matching from string {content} to string: Error {e} ({type(e)})")
            return None, str(e)

Description

Often model fails to return correct result and post validation of the result object can spot this issue. My question is what is the most clear way to react on a validation exception by including errors in the prompt and request LLM again assuming last validation error.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.2.0: Fri Dec 6 19:02:41 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6030
Python Version: 3.12.7 (main, Oct 16 2024, 12:05:52) [Clang 16.0.0 (clang-1600.0.26.3)]

Package Information

langchain_core: 0.3.21
langchain: 0.3.9
langchain_community: 0.3.9
langsmith: 0.1.147
langchain_aws: 0.2.9
langchain_openai: 0.2.11
langchain_pinecone: 0.2.0
langchain_text_splitters: 0.3.2
langgraph_sdk: 0.1.42
langserve: 0.3.0

Other Dependencies

aiohttp: 3.9.5
async-timeout: Installed. No version info available.
boto3: 1.35.75
dataclasses-json: 0.6.7
fastapi: 0.115.6
httpx: 0.27.2
httpx-sse: 0.4.0
jsonpatch: 1.33
langsmith-pyo3: Installed. No version info available.
numpy: 1.26.4
openai: 1.56.2
orjson: 3.10.12
packaging: 24.2
pinecone-client: 5.0.1
pydantic: 2.10.3
pydantic-settings: 2.6.1
PyYAML: 6.0.2
requests: 2.32.3
requests-toolbelt: 1.0.0
SQLAlchemy: 2.0.36
sse-starlette: 2.1.3
tenacity: 9.0.0
tiktoken: 0.8.0
typing-extensions: 4.12.2