[BUG] x509_v2 module returns binary data in the return payload, which crashes state.event event stream listeners

[OrlandoArcapix]

Description
When using x509_v2 in salt 3006.9, the x509.certificate_managed state returns binary data in the payload. This causes any api event stream client to fail (including salt-run state.event), with error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 1: invalid start byte

Traceback

Exception occurred in runner state.event: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 388, in low
    data["return"] = func(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/runners/state.py", line 312, in event
    return statemod["state.event"](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 2594, in event
    salt.utils.data.decode(ret["data"]),
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/data.py", line 252, in decode
    return decode_dict(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/data.py", line 365, in decode_dict
    value = decode_list(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/data.py", line 482, in decode_list
    item = decode_dict(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/data.py", line 411, in decode_dict
    value = decode(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/data.py", line 293, in decode
    data = _decode_func(data, encoding, errors, normalize)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/stringutils.py", line 113, in to_unicode
    return _normalize(to_str(s, encoding, errors))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/stringutils.py", line 89, in to_str
    raise exc  # pylint: disable=raising-bad-type
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/stringutils.py", line 82, in to_str
    return _normalize(s.decode(enc, errors))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 1: invalid start byte

Setup

• Install salt, salt-master, salt-api and salt-minion rpms on a Rocky 8 VM
• Configure the minion for x509_v2
• Add configuration for signing x509 certificates
• Set up a local certificate authority
• Create a salt state that will issue a signed certificate

/etc/salt/master.d/test.conf

user: root
id: saltdev
log_level: debug
peer:
  .*:
    - x509.sign_remote_certificate

/etc/salt/minion.d/test.conf

master: saltdev
id: saltdev
x509_signing_policies:
  sslpolicy:
    - signing_cert: /etc/pki/sslpolicy/ca/ca_ca_cert.crt/saltdev/saltdev_ca_cert.crt
    - signing_private_key: /etc/pki/sslpolicy/ca/ca_ca_cert.crt/saltdev/saltdev_ca_cert.key
    - C: UK
    - ST: Scotland
    - O: Salt Dev
    - OU: SSL Test
    - basicConstraints: "critical CA:false"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - copypath: /etc/pki/sslpolicy/remote
features:
  x509_v2: true

/srv/salt/setup.sls

Create a directory to store a copy of granted certificates:
  file.directory:
    - name: /etc/pki/sslpolicy/remote
    - makedirs: True

Create CA cert dir:
  file.directory:
    - name: /etc/pki/sslpolicy/ca
    - makedirs: True

Create Certificate Authority:
  module.run:
    - tls.create_ca:
        - ca_name: saltdev
        - cacert_path: /etc/pki/sslpolicy/ca/ca_ca_cert.crt
        - replace: False

Create a TLS private key:
  x509.private_key_managed:
    - name: "/etc/pki/tls/private/test.key"
    - keysize: 4096

/srv/salt/test.sls

Request TLS a certificate:
  x509.certificate_managed:
    - name: /etc/pki/tls/certs/test.crt
    - ca_server: saltdev
    - signing_policy: sslpolicy
    - private_key: /etc/pki/tls/private/test.key
    - CN: "DNS:saltdev"
    - subjectAltName: "DNS:saltdev"
    - days_valid: 39700
    - days_remaining: 90

• Start the salt-master: systemctl start salt-master
• Start the salt-minion, and accept the key: systemctl start salt-minion; sleep 11; salt-key -A
• Apply the setup state to create the CA files: salt-call state.apply setup; systemctl restart salt-master; systemctl restart salt-minion

Steps to Reproduce the behavior

• Start an event stream listener: salt-run state.event pretty=True
• Apply a state to issue a certificate: salt-call state.apply test

Expected behavior

Certificate is issued, event stream does not crash

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.16.1.el8_10.cloud.0.1.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

Additional context
Add any other context about the problem here.

[OrlandoArcapix]

Confirmed that this issue also affects Salt 3007.1

[OrlandoArcapix]

Functional workaround to just discard the data that can't be decoded:

--- /opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py	2024-05-19 13:12:38.000000000 +0000
+++ /opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py.patched	2025-02-11 14:17:09.095269981 +0000
@@ -2587,11 +2587,16 @@

             if salt.utils.stringutils.expr_match(ret["tag"], tagmatch):
                 if not quiet:
+                    try:
+                        serial_data = salt.utils.data.decode(ret["data"])
+                    except UnicodeDecodeError as e:
+                        log.error(f'Could not serialise data: {ret["data"]}; error: {e}')
+                        serial_data = {}
                     salt.utils.stringutils.print_cli(
                         "{}\t{}".format(
                             salt.utils.stringutils.to_str(ret["tag"]),
                             salt.utils.json.dumps(
-                                salt.utils.data.decode(ret["data"]),
+                                serial_data,
                                 sort_keys=pretty,
                                 indent=None if not pretty else 4,
                             ),

[lkubb]

There's a similar issue for returners (#59012).

Imho this should be handled when outputting the data since the x509_v2 execution module is just a common example of a module that sends binary event data, not the only one (for reference, it's a DER-encoded certificate). Using PEM for this – while not fixing the root issue of false assumptions about the event data format – is likely fine though, it should just add a bit of overhead.