How can I get the metadata from evaluators programmatically using the API?

[adocherty]

Question:
I have an evaluation function that returns metadata like this:

{'results': {'key': 'f_strict_valid',
  'score': 0,
  'reasoning': "Expecting ',' delimiter: line 8 column 5 (char 625)"}}

I can get the score using the list_feedback API call, but is there any way I can pull the reasoning field (namely, the error) in the API?

I can see this in the LangSmith web app, but not in the API.

More Context

I have written code that evaluates the validity of JSON strings using the LangChain evaluator JsonValidityEvaluator as follows

import os
import uuid
from langchain.evaluation import JsonValidityEvaluator
from langsmith.schemas import Example, Run
from datetime import datetime
from pydantic import BaseModel, Field

class EvalJsonOutput:
    def __init__(self, object: BaseModel):
        self._reference = object
        self._eval_json_valid = JsonValidityEvaluator()

    def __call__(self, root_run: Run, example: Example) -> dict:
        json_output = root_run.outputs.get("output")

        results = dict(
                key="f_strict_valid",
                **self._eval_json_valid.evaluate_strings(prediction=json_output),
            )
        return dict(results=results)

json_string_test_correct = """
{
  "title": "The Curious Case of the Vanishing Cookies"
  "problem": "This is definitely a problem for the cookie monster"
}
"""

json_string_test_invalid = """
{
  "title": "The Curious Case of the Vanishing Cookies",
  "problem": "This is definitely a problem for the cookie monster"
}
"""

eval_json_valid = JsonValidityEvaluator()
eval_json_schema = JsonSchemaEvaluator()

Running this on a test example like below produces the expected output, except for when the string is invalid JSON an extra piece of metadata reasoning is returned, like so:

{'results': {'key': 'f_strict_valid',
  'score': 0,
  'reasoning': "Expecting ',' delimiter: line 8 column 5 (char 625)"}}

This works fine, and in LangSmith I can see the score in the experiment:

To get the metadata from the evaluator I can see it in the “Rendered Output” section when clicking the Evalutor button above.

My questions is that now I’d like to collect this metadata programatically; however, in the API I can’t find how to get this data

I can load the project with stats, but these are only aggregate:

p_with_stats = client.read_project(project_id=[project.id](http://project.id/), include_stats=True)

I load all feedback associated with the project

runs = list(client.list_runs(project_id=[project.id](http://project.id/)))
feedback_list = list(client.list_feedback(run_id=[[r.id](http://r.id/) for r in runs], feedback_key="f_strict_valid"))

But this only contains, the score and I can’t find the reasoning metadata anywhere:

{
  "id": "...",
  "run_id": "...",
  "key": "f_strict_valid",
  "score": 0.0,
  "value": null,
  "comment": "",
  "correction": null,
  "feedback_source": {
    "type": "model",
    "metadata": {
      "__run": {
        "run_id": "..."
      }
    }
  },
  "session_id": "...",
  "comparative_experiment_id": null,
  "feedback_group_id": null
}

Is there any way I can pull out the metadata from the evaluation (namely, the error) in the API?

[hinthornw]

If you return it as a "comment" (instead of "reasoning") it'll be passed through

[adocherty]

Thanks! I'll give that a go.

Out of curiosity, I also tried using LangChainStringEvaluator("json_validity") that wraps the LangChain evaluators; however this doesn't return any metadata on the errors. Is there any particular reason for this?