Summary Evaluator not showing in Langsmith UI

[whatever-afk]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

results = evaluate(
    lambda inputs: label_query(inputs["text"]),
    data=dataset_name,
    evaluators=[correct_label],
    summary_evaluators=[summary_eval],
    experiment_prefix="Toxic Queries",
)

Description

I am trying to replicate the example in the documentation for the summary_evaluator. I am only able to see the accuracy metric not the precision in the LangSmtih UI.
In the documentation label_query is not found only label_text function.
Am i missing something?

System Info

System Information

OS: Windows
OS Version: 10.0.22621
Python Version: 3.9.2rc1 (tags/v3.9.2rc1:4064156, Feb 17 2021, 11:25:18) [MSC v.1928 64 bit (AMD64)]

Package Information

langchain_core: 0.2.10
langchain: 0.2.6
langchain_community: 0.2.6
langsmith: 0.1.82
langchain_experimental: 0.0.62
langchain_groq: 0.1.5
langchain_huggingface: 0.0.3
langchain_text_splitters: 0.2.2
langchainhub: 0.1.20

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[dosubot[bot]]

Hello @whatever-afk! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together while we wait for a human maintainer to assist us.

To ensure the summary_evaluator includes the precision metric in the LangSmith UI, you need to configure it properly. Here is an example of how you can set up the summary_evaluator to include the precision metric:

from langchain.smith.evaluation.config import RunEvalConfig, EvaluatorType

# Create a configuration for the summary evaluator
summary_evaluator_config = RunEvalConfig(
    evaluators=[
        {
            "evaluator_type": EvaluatorType.SUMMARY,
            "metrics": ["precision"]
        }
    ]
)

# Use this configuration in your evaluate function
results = evaluate(
    lambda inputs: label_query(inputs["text"]),
    data=dataset_name,
    evaluators=[correct_label],
    summary_evaluators=[summary_evaluator_config],
    experiment_prefix="Toxic Queries",
)

This configuration specifies that the summary_evaluator should include the precision metric. Ensure that you replace summary_eval with summary_evaluator_config in your evaluate function call.

Regarding the label_query function, it seems you might be referring to the embed_query function. Here is the implementation of the embed_query function:

def embed_query(self, text: str) -> List[float]:
    if self.query_instruction:
        text = self.query_instruction + text
    return self._embed_text([text])[0]

The embed_query function embeds a single query text, optionally prepending a query_instruction if it is provided. It differs from the embed_documents function, which embeds a list of text documents and handles batching of the documents ^[1].

Make sure your configuration and function names are correctly set up to see the desired metrics in the LangSmith UI.

To continue talking to Dosu, mention @dosu.