Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix serialization of pydantic.BaseModel fields with pathlib.Path dict keys #1157

Merged
merged 7 commits into from
Nov 5, 2024

Conversation

brendanator
Copy link
Contributor

Apply _simple_default serialization on dict keys to resolve this exception

Traceback (most recent call last):
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 281, in _dumps_json_single
    return orjson.dumps(
           ^^^^^^^^^^^^^
TypeError: Dict key must a type serializable with OPT_NON_STR_KEYS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 5802, in _tracing_thread_handle_batch
    client.multipart_ingest_runs(create=create, update=update, pre_sampled=True)
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 1675, in multipart_ingest_runs
    valb = _dumps_json(value)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 321, in _dumps_json
    return _dumps_json_single(obj, _serialize_json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 292, in _dumps_json_single
    result = json.dumps(
             ^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
TypeError: keys must be str, int, float, bool or None, not PosixPath

@hinthornw
Copy link
Collaborator

Thanks for flagging and proposing a fix! Do you have an example test case you could add? :)

elif hasattr(obj, "dict") and callable(obj.dict):
return obj.dict()
elif hasattr(obj, "_asdict") and callable(obj._asdict):
return obj._asdict()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed these are they are dead code - namedtuple, and pydantic.BaseModel are handled in _serialize_json

@brendanator brendanator changed the title Fix orjson serialization of dict keys Fix serialization of pydantic.BaseModel fields with pathlib.Path dict keys Oct 31, 2024
("to_dict", False), # dataclasses-json
(
"model_dump",
{"exclude_none": True, "mode": "json"},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mode="json" convert pathlib.Path dict keys to str

Apply `_simple_default` serialization on dict keys to resolve this exception

```python
Traceback (most recent call last):
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 281, in _dumps_json_single
    return orjson.dumps(
           ^^^^^^^^^^^^^
TypeError: Dict key must a type serializable with OPT_NON_STR_KEYS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 5802, in _tracing_thread_handle_batch
    client.multipart_ingest_runs(create=create, update=update, pre_sampled=True)
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 1675, in multipart_ingest_runs
    valb = _dumps_json(value)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 321, in _dumps_json
    return _dumps_json_single(obj, _serialize_json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/brendan/.local/share/virtualenvs/core-bRkH1-KB/lib/python3.11/site-packages/langsmith/client.py", line 292, in _dumps_json_single
    result = json.dumps(
             ^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
TypeError: keys must be str, int, float, bool or None, not PosixPath
```
@brendanator
Copy link
Contributor Author

@hinthornw Added test cases, and then cleaned up my implementation

@brendanator
Copy link
Contributor Author

@hinthornw Is there anything I need to do to get this in? Happy to make any follow up changes you want

@brendanator
Copy link
Contributor Author

I've merged latest main and updated the tests

@hinthornw hinthornw changed the base branch from main to wfh/fix_serde_class November 5, 2024 18:57
@hinthornw hinthornw merged commit 270d1a4 into langchain-ai:wfh/fix_serde_class Nov 5, 2024
6 of 9 checks passed
@brendanator brendanator deleted the patch-1 branch November 8, 2024 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants