10.3.0 doesn't send any (?) data to New Relic's servers due to dependency version serialization errors

[sh-at-cs]

Description
After upgrading to newrelic-python-agent (newrelic PyPI package) 10.3.0 (released 15 hours ago), the agent stopped sending any (?) data to New Relic's servers.

This is the red region in all these graphs, while the region afterwards where it goes back to normal is when I downgraded back to 10.2:

During this time, the agent logs errors from thread NR-Harvest-Thread, saying Failed to harvest data for [name of our application] with tracebacks like these:

Traceback

Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/newrelic/core/agent.py", line 645, in _harvest_default
    application.harvest(shutdown, flexible=False)
  File "/app/.venv/lib/python3.11/site-packages/newrelic/core/application.py", line 1273, in harvest
    self._active_session.send_loaded_modules(self.modules)
  File "/app/.venv/lib/python3.11/site-packages/newrelic/core/data_collector.py", line 267, in send_loaded_modules
    return self._protocol.send("update_loaded_modules", payload)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/newrelic/core/agent_protocol.py", line 229, in send
    params, headers, payload = self._to_http(method, payload)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/newrelic/core/agent_protocol.py", line 274, in _to_http
    return params, self._headers, json_encode(payload).encode("utf-8")
                                  ^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/newrelic/common/encoding_utils.py", line 91, in json_encode
    return json.dumps(obj, **_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/newrelic/common/encoding_utils.py", line 78, in _encode
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <module 'pypdfium2.version' from '/app/.venv/lib/python3.11/site-packages/pypdfium2/version.py'> is not JSON serializable

So presumably that has something to do with it?

Expected Behavior
The agent sends data to New Relic's servers as before.

Steps to Reproduce
Upgrade a FastAPI app deployment which also has a dependency on e.g. pypdfium2 (or other packages, see other users' comments below) to newrelic-python-agent 10.3.0.

Your Environment

• Python 3.11
• FastAPI 0.115
• newrelic-python-agent (newrelic PyPI package) 10.3.0
• pypdfium2 4.30.0

Additional context

It could be that there is something users need to configure differently in 10.3, but I wouldn't know because there are no release notes for 10.3 yet.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-341483

[cix-code]

Similar issue on my end as well.

Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/newrelic/core/agent.py", line 645, in _harvest_default
    application.harvest(shutdown, flexible=False)
  File "/opt/venv/lib/python3.8/site-packages/newrelic/core/application.py", line 1273, in harvest
    self._active_session.send_loaded_modules(self.modules)
  File "/opt/venv/lib/python3.8/site-packages/newrelic/core/data_collector.py", line 267, in send_loaded_modules
    return self._protocol.send("update_loaded_modules", payload)
  File "/opt/venv/lib/python3.8/site-packages/newrelic/core/agent_protocol.py", line 229, in send
    params, headers, payload = self._to_http(method, payload)
  File "/opt/venv/lib/python3.8/site-packages/newrelic/core/agent_protocol.py", line 274, in _to_http
    return params, self._headers, json_encode(payload).encode("utf-8")
  File "/opt/venv/lib/python3.8/site-packages/newrelic/common/encoding_utils.py", line 91, in json_encode
    return json.dumps(obj, **_kwargs)
  File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/venv/lib/python3.8/site-packages/newrelic/common/encoding_utils.py", line 78, in _encode
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <module 'googleapiclient.version' from '/opt/venv/lib/python3.8/site-packages/googleapiclient/version.py'> is not JSON serializable

[lrafeei]

Thanks for bringing this issue to our attention--This is definitely a bug and I'll look into this now.

Meanwhile, there seems to be a delay with the docs publishing to the New Relic docs site, so until then, the release notes can be found here: https://github.com/newrelic/newrelic-python-agent/releases/tag/v10.3.0

[lwang]

Getting the same issue when using version 10.3.0 in a Lambda function. gremlin_python seems like an included module from the aws-sdk-pandas Lambda layer.

Edit: seems like it only happens for old versions of the Lambda layer AWSSDKPandas-Python39 with version < 10

{
  "errorMessage": "<module 'gremlin_python.__version__' from '/opt/python/gremlin_python/__version__.py'> is not JSON serializable",
  "errorType": "TypeError",
  "requestId": "7660537c-b8b0-4cbe-93fb-7d39c31e63a1",
  "stackTrace": [
    "  File \"/var/task/newrelic/api/lambda_handler.py\", line 151, in _nr_lambda_handler_wrapper_\n    return result\n",
    "  File \"/var/task/newrelic/api/transaction.py\", line 660, in __exit__\n    self._application.record_transaction(node)\n",
    "  File \"/var/task/newrelic/api/application.py\", line 161, in record_transaction\n    self._agent.record_transaction(self._name, data)\n",
    "  File \"/var/task/newrelic/core/agent.py\", line 586, in record_transaction\n    application.harvest(flexible=False)\n",
    "  File \"/var/task/newrelic/core/application.py\", line 1273, in harvest\n    self._active_session.send_loaded_modules(self.modules)\n",
    "  File \"/var/task/newrelic/core/data_collector.py\", line 267, in send_loaded_modules\n    return self._protocol.send(\"update_loaded_modules\", payload)\n",
    "  File \"/var/task/newrelic/core/agent_protocol.py\", line 229, in send\n    params, headers, payload = self._to_http(method, payload)\n",
    "  File \"/var/task/newrelic/core/agent_protocol.py\", line 274, in _to_http\n    return params, self._headers, json_encode(payload).encode(\"utf-8\")\n",
    "  File \"/var/task/newrelic/common/encoding_utils.py\", line 91, in json_encode\n    return json.dumps(obj, **_kwargs)\n",
    "  File \"/var/lang/lib/python3.9/json/__init__.py\", line 234, in dumps\n    return cls(\n",
    "  File \"/var/lang/lib/python3.9/json/encoder.py\", line 199, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n",
    "  File \"/var/lang/lib/python3.9/json/encoder.py\", line 257, in iterencode\n    return _iterencode(o, 0)\n",
    "  File \"/var/task/newrelic/common/encoding_utils.py\", line 78, in _encode\n    raise TypeError(repr(o) + \" is not JSON serializable\")\n"
  ]
}

[lrafeei]

Hello! I reproduced this on my end and I think I have a fix. Just to make sure I don't miss any use cases, could you try installing New Relic from this branch and seeing if that fixes the issue?

https://github.com/newrelic/newrelic-python-agent/tree/package-json-fix

If so, I can clean this up and put it in the next release

[lwang]

Fixed it for me

[sh-at-cs]

Looks good on my end as well 👍