Expose prompt variables in callback

[AveshCSingh]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Consider this simple prompt with variables context and question:

template = """
You are a question answer bot. 

Given this context: ```{context}```

Answer this question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)

When this prompt is used within a chain and a CallbackHandler is provided, the callback interfaces exposes the prompt passed to the LLM, but interpolates variables. For example, suppose you define:

class MyCallbackHandler(BaseCallbackHandler):
  def on_chat_model_start(
      self,
      serialized: Dict[str, Any],
      messages: List[List[BaseMessage]],
  ) -> Any:
    print(messages)

chain = setup_and_retrieval | prompt | model | output_parser

handler = MyCallbackHandler()
response = chain.invoke(
    "What did the president say about the economy?",
    config={"callbacks": [handler]}
)

MyCallbackHandler sees the LLM input:

You are a question answer bot. 

Given this context: Document(page_content='Vice President Harris and I ran for office with a new economic vision for America. \n\nInvest in America. Educate Americans. Grow the workforce. Build the economy from the bottom up  \nand the middle out, not from the top down.  \n\nBecause we know that when the middle class grows, the poor have a ladder up and the wealthy do very well. \n\nAmerica used to have the best roads, bridges, and airports on Earth. \n\nNow our infrastructure is ranked 13th in the world. \n\nWe won’t be able to compete for the jobs of the 21st Century if we don’t fix that. \n\nThat’s why it was so important to pass the Bipartisan Infrastructure Law—the most sweeping investment to rebuild America in history. \n\nThis was a bipartisan effort, and I want to thank the members of both parties who worked to make it happen. \n\nWe’re done talking about infrastructure weeks. \n\nWe’re going to have an infrastructure decade.', metadata={'source': 'state_of_the_union.txt', 'id': 'a0251297-78e4-407c-aeb4-7bb6410ccc2d'})

Answer this question: What did the president say about the economy?

This feature request is to provide callbacks with the PromptTemplate and variables, for example, the following could be passed to on_chat_model_start:

prompts=[{"template": "You are a question answer bot.\n\nGiven this context:````{context}```\n\nAnswer this question: {question}.", "input_variables": {"context": "...", "question": "..."})]

I am new to LangChain, so please let me know if there's a more natural interface to pass this information to the CallbackHandler.

Motivation

Tuning RAG applications involves prompt engineering, updating chunking, updating retrieval, re-ranking, etc.

By separating out the context from the variables, applications that aid in RAG tuning can use LangChain callback handlers to identify which variables are being modified between experimental "branches".

Proposal (If applicable)

No response