Local LLM pydantic error

[amaloney]

ALL software version info

(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc should be added within the dropdown below.)

Software Version Info

Python 3.11.10
Panel 1.5.4a1
panel-graphic-walker 0.5.0
lumen[ai-llama,sql] 0.7.0a35 (build from main)

Description of expected behavior and the observed behavior

• Expected a response from a local LLM
• Observed pydantic error using the Tell me about the dataset. pill

Complete, minimal, self-contained example code that reproduces the issue

Clone the repo, and install using pixi

git clone https://github.com/holoviz/lumen && cd lumen
pixi && pixi shell --environment build
pip install --editable .[tests,sql,ai-llama] && pip install panel-graphic-walker[kernel]

Download the penguins data to the same folder you cloned the repo to

wget https://datasets.holoviz.org/penguins/v1/penguins.csv

Next run the app, and load the penguins data into it.

# app.py
import lumen.ai as lmai
llm = lmai.llm.Llama()
lmai.ExplorerUI(["./penguins.csv"], llm=llm, agents=[lmai.agents.VegaLiteAgent]).servable()

panel serve app.py --show

Loaded data in the app

click on the pill to tell you about the data

Stack traceback and/or browser JavaScript console output

Traceback (most recent call last):

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/lumen/ai/coordinator.py", line 671, in _compute_execution_graph
    raise e

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/lumen/ai/coordinator.py", line 663, in _compute_execution_graph
    plan = await self._make_plan(
           ^^^^^^^^^^^^^^^^^^^^^^

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/lumen/ai/coordinator.py", line 591, in _make_plan
    async for reasoning in self.llm.stream(

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/lumen/ai/llm.py", line 199, in stream
    for chunk in chunks:

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/.pixi/envs/build/lib/python3.11/site-packages/instructor/dsl/partial.py", line 115, in from_streaming_response
    yield from cls.model_from_chunks(json_chunks, **kwargs)

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/.pixi/envs/build/lib/python3.11/site-packages/instructor/dsl/partial.py", line 142, in model_from_chunks
    obj = partial_model.model_validate(obj, strict=None, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/Users/rmaloney/Development/open-source/holoviz-dev/lumen/.pixi/envs/build/lib/python3.11/site-packages/pydantic/main.py", line 596, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

pydantic_core._pydantic_core.ValidationError: 1 validation error for PartialReasoning
tables.0
  Input should be './penguins.csv' [type=literal_error, input_value='', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/literal_error

Screenshots or screencasts of the bug in action

• I may be interested in making a pull request to address this

[philippjfr]

Yes, we've been dealing with this error a lot recently. Will tag a new dev release now with some prompting improvements but it seems like we may have to validate the tables manually rather than using Literal types for validation.

[ahuang11]

I think this is partially because Mistral-Instruct 7B does not work well with structured output (indicated by the empty output input_value='' in Input should be './penguins.csv' [type=literal_error, input_value=''

Newer models like Qwen 7B seem to work much better, and faster as implemented in #780. If you have the chance, please test that branch and let me know if that fixes this problem!