[Bug] TypeError: Object of type Table is not JSON serializable

[K-Oxon]

Is this a new bug in dbt-common?

• I believe this is a new bug in dbt-common
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When I try to parse the return value dbtRunnerResult of a dbt seed command with JSONEncoder, I get the following error:

TypeError: Object of type Table is not JSON serializable

Expected Behavior

The JSONEncoder can handle agate.Table objects without raising an error.

Steps To Reproduce

1. Run a dbt seed (or build) command.
2. Attempt to parse the return value dbtRunnerResult with JSONEncoder.

from dataclasses import asdict

from dbt.cli.main import dbtRunner, dbtRunnerResult
from dbt_common.utils.encoding import JSONEncoder

dbt = dbtRunner()
# run the command
res: dbtRunnerResult = dbt.invoke(["seed"])
print(json.dumps(asdict(res), cls=JSONEncoder))

Relevant log output

TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 pprint.pprint(json.dumps(asdict(res), cls=JSONEncoder))

File ~/.local/share/mise/installs/python/3.9.6/lib/python3.9/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    232 if cls is None:
    233     cls = JSONEncoder
--> 234 return cls(
    235     skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    236     check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237     separators=separators, default=default, sort_keys=sort_keys,
    238     **kw).encode(obj)

File ~/.local/share/mise/installs/python/3.9.6/lib/python3.9/json/encoder.py:199, in JSONEncoder.encode(self, o)
    195         return encode_basestring(o)
    196 # This doesn't pass the iterator directly to ''.join() because the
    197 # exceptions aren't as detailed.  The list call should be roughly
    198 # equivalent to the PySequence_Fast that ''.join() would do.
--> 199 chunks = self.iterencode(o, _one_shot=True)
    200 if not isinstance(chunks, (list, tuple)):
    201     chunks = list(chunks)

File ~/.local/share/mise/installs/python/3.9.6/lib/python3.9/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot)
    252 else:
...
    178     """
--> 179     raise TypeError(f'Object of type {o.__class__.__name__} '
    180                     f'is not JSON serializable')

TypeError: Object of type Table is not JSON serializable

Additional Context

I think we can solve this problem by adding the following to the default method:

if isinstance(obj, agate.Table):
    return None

Reference:

https://github.com/dbt-labs/dbt-core/blob/5d32aa8b62af4f664db2801ebd1ecd7efc730eb3/core/dbt/artifacts/schemas/run/v5/run.py#L37

[dbeatty10]

Thanks for reaching out @K-Oxon ! 🤩

First, did you happen to try out the ForgivingJSONEncoder?

from dbt_common.utils.encoding import ForgivingJSONEncoder
# ...
print(json.dumps(asdict(res), cls=ForgivingJSONEncoder))

Second, it sounds like you're proposing that we update the code here
to add the following?

if isinstance(obj, agate.Table):
    return None

It sounds like you're proposing that we update the following code?

dbt-common/dbt_common/utils/encoding.py

Lines 26 to 48 in 60ffb06

	class JSONEncoder(json.JSONEncoder):
	"""A 'custom' json encoder.
	A 'custom' json encoder that does normal json encoder things, but also
	handles `Decimal`s and `Undefined`s. Decimals can lose precision because
	they get converted to floats. Undefined's are serialized to an empty string
	"""
	def default(self, obj):
	if isinstance(obj, DECIMALS):
	return float(obj)
	elif isinstance(obj, (datetime.datetime, datetime.date, datetime.time)):
	return obj.isoformat()
	elif isinstance(obj, jinja2.Undefined):
	return ""
	elif isinstance(obj, Exception):
	return repr(obj)
	elif hasattr(obj, "to_dict"):
	# if we have a to_dict we should try to serialize the result of
	# that!
	return obj.to_dict(omit_none=True)
	else:
	return super().default(obj)

And just add this additional conditional in there somewhere?

(el)if isinstance(obj, agate.Table):
    return None

If so, did you try it out and see if it worked in your scenario?

[K-Oxon]

@dbeatty10
Thanks for the answer.

ForgivingJSONEncoder worked!

from dbt_common.utils.encoding import ForgivingJSONEncoder
# ...
print(json.dumps(asdict(res), cls=ForgivingJSONEncoder))

The output is:

# ...
'agate_table': '| column   | data_type |\n| -------- | --------- |\n| id       | Number    |\n| keyword  | Text      |\n| category | Text      |\n'}],

I implemented the following CustomJSONEncoder and it worked, but normally it seems to be better to use ForgivingJSONEncoder.

class CustomJSONEncoder(JSONEncoder):
    def default(self, obj):
        if isinstance(obj, agate.Table):
            return None
        return super().default(obj)

[dbeatty10]

Awesome @K-Oxon very glad ForgivingJSONEncoder worked for you!

I'm going to close this as not a bug since ForgivingJSONEncoder works. But if you think dbt should provide an additional encoder (or an option to disable serialization of the agate table), please open a feature request.