can't load matrix.py #357

bloosanbi · 2023-07-25T07:11:51Z

python: loading script "/home/user/.weechat/python/matrix.py"
python: stdout/stderr (?): Traceback (most recent call last):
python: stdout/stderr (?):   File "/home/user/.weechat/python/matrix.py", line 43, in <module>
python: stdout/stderr (?):     import OpenSSL.crypto as crypto
python: stdout/stderr (?):   File "/home/user/.local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
python: stdout/stderr (?):     from OpenSSL import SSL, crypto
python: stdout/stderr (?):   File "/home/user/.local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 9, in <module>
python: stdout/stderr (?):     from OpenSSL._util import (
python: stdout/stderr (?):   File "/home/user/.local/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module>
python: stdout/stderr (?):     from cryptography.hazmat.bindings.openssl.binding import Binding
python: stdout/stderr (?):   File
"/home/user/.local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module>
python: stdout/stderr (?):     from cryptography.exceptions import InternalError
python: stdout/stderr (?):   File "/home/user/.local/lib/python3.9/site-packages/cryptography/exceptions.py", line 9, in
<module>
python: stdout/stderr (?):     from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
python: stdout/stderr (?): ImportError: PyO3 modules may only be initialized once per interpreter process
python: unable to parse file "/home/user/.weechat/python/matrix.py"

Complete the installation and run /script load matrix.py, I got the following error. What should I do?

poljar · 2023-07-25T08:15:02Z

Well, seems like OpenSSL started to use Rust and PyO3 for their Python bindings. And you seem to be loading multiple WeeChat Python scripts which use OpenSSL, which PyO3 doesn't seem to support.

I'm not sure that there's much we can do.

ikwyl6 · 2024-09-04T00:53:03Z

I have similar error and cannot load matrix.py. This is being tracked in pyo3 at PyO3/pyo3#576

My error is:

21:01 python: loading script "/mnt/wd1000/BTSync/RPi conf files/weechat/python/autoload/matrix.py"
21:01 python: stdout/stderr (?): Traceback (most recent call last):
21:01 python: stdout/stderr (?):   File "/mnt/wd1000/BTSync/RPi conf files/weechat/python/autoload/matrix.py", line 43, in <module>
21:01 python: stdout/stderr (?):     import OpenSSL.crypto as crypto
21:01 python: stdout/stderr (?):   File "/usr/lib/python3.12/site-packages/OpenSSL/__init__.py", line 8, in <module>
21:01 python: stdout/stderr (?):     from OpenSSL import SSL, crypto
21:01 python: stdout/stderr (?):   File "/usr/lib/python3.12/site-packages/OpenSSL/SSL.py", line 11, in <module>
21:01 python: stdout/stderr (?):     from OpenSSL._util import (
21:01 python: stdout/stderr (?):   File "/usr/lib/python3.12/site-packages/OpenSSL/_util.py", line 6, in <module>
21:01 python: stdout/stderr (?):     from cryptography.hazmat.bindings.openssl.binding import Binding
21:01 python: stdout/stderr (?):   File "/usr/lib/python3.12/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module>
21:01 python: stdout/stderr (?):     from cryptography.exceptions import InternalError
21:01 python: stdout/stderr (?):   File "/usr/lib/python3.12/site-packages/cryptography/exceptions.py", line 9, in <module>
21:01 python: stdout/stderr (?):     from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
21:01 python: stdout/stderr (?): ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576

nexy7574 · 2024-09-05T16:25:08Z

Getting the same traceback on a mostly fresh install on Ubuntu 22.04 in WSL (Python 3.10.12) (Same error on Ubuntu 24.04 LTS/Python 3.12.3)

kqr · 2024-09-12T09:40:10Z

I am also having this problem – does anyone know how I would find out whether I am

loading multiple WeeChat Python scripts which use OpenSSL

?

Because of the few Python scripts in the list of loaded scripts, none are particularly networky or anything beyond things like topicdiff, auto_away, etc.

Oooh wait, I'm dumb. I get a separate error when I first load the plugin, then when I reload it I get this error because it conflicts with itself – or something?

ikwyl6 · 2024-11-03T14:29:31Z

Well, seems like OpenSSL started to use Rust and PyO3 for their Python bindings. And you seem to be loading multiple WeeChat Python scripts which use OpenSSL, which PyO3 doesn't seem to support.

I'm not sure that there's much we can do.

Hi @poljar So is this basically the question if Py03 starts supporting openssl or they don't? Or as well, if pyOpenSSL starts including another package than Py03? I'm just trying to understand the changes that were made and who made them upstream.. Or I guess if you support and include a different openssl package then pyOpenSSL in your code? Thanks for any insight..

On the README for pyOpenSSL it says Note: The Python Cryptographic Authority strongly suggests the use of [pyca/cryptography](https://github.com/pyca/cryptography) where possible. If you are using pyOpenSSL for anything other than making a TLS connection you should move to cryptography and drop your pyOpenSSL dependency.

LorenzoBianconi · 2024-11-24T22:09:52Z

@poljar @ikwyl6 any news on this issue?

ikwyl6 · 2024-11-25T04:40:08Z

@LorenzoBianconi waiting for @poljar...
Or someone to write a PR to switch to pyca/cryptography pkg..

trygveaa · 2024-11-25T09:40:28Z

Or someone to write a PR to switch to pyca/cryptography pkg..

That won't help. pyOpenSSL depends on pyca/cryptography, and pyca/cryptography is the one depending on PyO3, not pyOpenSSL directly. So switching to pyca/cryptography won't change anything in regards to this issue.

LorenzoBianconi · 2024-11-25T21:47:03Z

Or someone to write a PR to switch to pyca/cryptography pkg..

That won't help. pyOpenSSL depends on pyca/cryptography, and pyca/cryptography is the one depending on PyO3, not pyOpenSSL directly. So switching to pyca/cryptography won't change anything in regards to this issue.

@trygveaa how do you suggest to fix it in this case?

trygveaa · 2024-11-26T20:07:00Z

As far as I can see pyOpenSSL is only used to print certificate info, so could just avoid printing that if using a version that has this problem.

However, I see that matrix-nio depends on jsonschema which transitively depends on another package that is also using PyO3 (rpds-py). This is used for schema validation of responses and some other data. So would have to either replace that with something else or drop validation.

LorenzoBianconi · 2024-12-23T07:57:46Z

@trygveaa @poljar is there any way to move on with this issue? this is blocking any updated to new Fedora or Ubuntu versions.

zer0def · 2024-12-23T08:45:44Z

To the best of my understanding, there are three ways to progress from here:

PyO3 implements subinterpreter support: Support sub-interpreters PyO3/pyo3#576 → Tracking Issue: Sub-Interpreter Support PyO3/pyo3#3451 → RFC: Implement Multi-Phase Module Initialization as per PEP 489 PyO3/pyo3#4162 (unlikely to timely happen)
matrix-nio refactors it's dependency on jsonschema into pydantic and hopefully that's the only dependency indirectly dependent on PyO3: Validate Responses and Events using pydantic instead of json-schema matrix-nio/matrix-nio#506 (the extent of this draft at the time of writing raises an eyebrow, so "unlikely to timely happen" is charitable)
switch over to the Rust-based plugin, which sees occasional spats of life

As of now, neither looks particularly appealing.

The cryptography package (which pyOpenSSL depends on) doesn't support being loaded in multiple sub interpreters from version 41.0.0. Doing it in previous versions seemed to work, but could apparenly cause soundness issues, so it errors out from version 41.0.0. Note that this isn't a change in the cryptography package, but in pyo3 from version 0.17.0, which cryptography uses. This causes the script to fail when reloading it. However, since pyOpenSSL is only used to print certificate info, wrap it in a try and skip printing info instead of crashing if pyOpenSSL can't be imported. There is another issue with a dependency of matrix-nio using pyo3 (rpds-py via jsonschema), but this at least helps with part of the problem. See poljar#357 for more details.

trygveaa · 2024-12-23T12:11:16Z

Yeah, that's a good summary @zer0def. I see two other options as well:

Pin jsonschema to version 4.17.3. This is the last version before they added the dependency on rpds-py. This is not a great option though, and may break in the future.
Open a request for jsonschema for a pure python version, stating that the current dependency causes problems with sub interpreters. A pure python fallback is apparently on the TODO list, but not prioritized: https://github.com/orgs/python-jsonschema/discussions/1275

As for the issue with pyOpenSSL, I opened a PR now to print certificate info conditionally as a workaround to this part of the problem: #367

zer0def · 2024-12-23T12:14:39Z

Just for clarity, this is the unavoidable output users are likely to bump into, even if they work around pyca/cryptography's complaints stemming from pyOpenSSL's usage:

python: script "/home/weeeeee/.weechat/python/autoload/_matrix.py" not found
python: stdout/stderr (?): Traceback (most recent call last):
python: stdout/stderr (?):   File "/home/weeeeee/.weechat/python/autoload/weechat-matrix.py", line 53, in <module>
python: stdout/stderr (?):     from nio import RemoteProtocolError, RemoteTransportError, TransportType
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/nio/__init__.py", line 9, in <module>
python: stdout/stderr (?):     from .client import *
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/nio/client/__init__.py", line 3, in <module>
python: stdout/stderr (?):     from .async_client import AsyncClient, AsyncClientConfig, DataProvider
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/nio/client/async_client.py", line 72, in <module>
python: stdout/stderr (?):     from ..crypto import (
python: stdout/stderr (?):     ...<4 lines>...
python: stdout/stderr (?):     )
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/nio/crypto/__init__.py", line 33, in <module>
python: stdout/stderr (?):     from .key_request import OutgoingKeyRequest
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/nio/crypto/key_request.py", line 20, in <module>
python: stdout/stderr (?):     from ..responses import RoomKeyRequestResponse
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/nio/responses.py", line 25, in <module>
python: stdout/stderr (?):     from jsonschema.exceptions import SchemaError, ValidationError
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/jsonschema/__init__.py", line 13, in <module>
python: stdout/stderr (?):     from jsonschema._format import FormatChecker
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/jsonschema/_format.py", line 11, in <module>
python: stdout/stderr (?):     from jsonschema.exceptions import FormatError
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/jsonschema/exceptions.py", line 15, in <module>
python: stdout/stderr (?):     from referencing.exceptions import Unresolvable as _Unresolvable
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/referencing/__init__.py", line 5, in <module>
python: stdout/stderr (?):     from referencing._core import Anchor, Registry, Resource, Specification
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/referencing/_core.py", line 9, in <module>
python: stdout/stderr (?):     from rpds import HashTrieMap, HashTrieSet, List
python: stdout/stderr (?):   File "/usr/lib/python3.13/site-packages/rpds/__init__.py", line 1, in <module>
python: stdout/stderr (?):     from .rpds import *
python: stdout/stderr (?): ImportError: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576
python: unable to parse file "/home/weeeeee/.weechat/python/autoload/weechat-matrix.py"

The cryptography package (which pyOpenSSL depends on) doesn't support being loaded in multiple sub interpreters from version 41.0.0. Doing it in previous versions seemed to work, but could apparenly cause soundness issues, so it errors out from version 41.0.0. Note that this isn't a change in the cryptography package, but in pyo3 from version 0.17.0, which cryptography uses. This causes the script to fail when reloading it, or if you load it after another script with the same dependency. However, since pyOpenSSL is only used to print certificate info, wrap it in a try and skip printing info instead of crashing if pyOpenSSL can't be imported. There is another issue with a dependency of matrix-nio using pyo3 (rpds-py via jsonschema), but this at least helps with part of the problem. See poljar#357 for more details.

zer0def · 2024-12-23T12:53:19Z

Confirming that #367 and rolling back to jsonschema==4.17.3 mitigates the issue, even if it's unfeasible from a distro packaging standpoint.

ikwyl6 · 2024-12-23T22:42:53Z

Confirming that #367 and rolling back to jsonschema==4.17.3 mitigates the issue, even if it's unfeasible from a distro packaging standpoint.

Can users do this workflow if they use venv instead of typical package upgrade stream? I assume if this is the case the user has to run weechat under the venv as well..?

trygveaa · 2024-12-24T01:32:42Z

Can users do this workflow if they use venv instead of typical package upgrade stream? I assume if this is the case the user has to run weechat under the venv as well..?

Yes, you can use a venv for the dependencies of weechat-matrix. If it's a directory named venv that's placed next to the script and it's created by virtualenv (not pyvenv or the python3 venv module) it will be automatically loaded by weechat-matrix. Otherwise you can use it by activating it before starting WeeChat or to avoid that you can modify the weechat-matrix script to add <path-to-venv>/lib/python3.12/site-packages to sys.path which will load the venv when weechat-matrix is loaded.

LorenzoBianconi · 2024-12-26T10:18:52Z

To the best of my understanding, there are three ways to progress from here:

* PyO3 implements subinterpreter support: [Support sub-interpreters PyO3/pyo3#576](https://github.com/PyO3/pyo3/issues/576) → [Tracking Issue: Sub-Interpreter Support PyO3/pyo3#3451](https://github.com/PyO3/pyo3/issues/3451) → [RFC: Implement Multi-Phase Module Initialization as per PEP 489 PyO3/pyo3#4162](https://github.com/PyO3/pyo3/pull/4162) (unlikely to timely happen)

* matrix-nio refactors it's dependency on jsonschema into pydantic and hopefully that's the only dependency indirectly dependent on PyO3: [Validate Responses and Events using pydantic instead of json-schema matrix-nio/matrix-nio#506](https://github.com/matrix-nio/matrix-nio/pull/506) (the extent of this draft at the time of writing raises an eyebrow, so "unlikely to timely happen" is charitable)

* switch over to the [Rust-based plugin](https://github.com/poljar/weechat-matrix-rs), which sees occasional spats of life

As of now, neither looks particularly appealing.

the rust plugin does not work with latest weechat packaged in fedora

trygveaa mentioned this issue Dec 23, 2024

Conditionally print certificate info #367

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can't load matrix.py #357

can't load matrix.py #357

bloosanbi commented Jul 25, 2023 •

edited

Loading

poljar commented Jul 25, 2023

ikwyl6 commented Sep 4, 2024

nexy7574 commented Sep 5, 2024 •

edited

Loading

kqr commented Sep 12, 2024 •

edited

Loading

ikwyl6 commented Nov 3, 2024 •

edited

Loading

LorenzoBianconi commented Nov 24, 2024

ikwyl6 commented Nov 25, 2024

trygveaa commented Nov 25, 2024 •

edited

Loading

LorenzoBianconi commented Nov 25, 2024

trygveaa commented Nov 26, 2024

LorenzoBianconi commented Dec 23, 2024

zer0def commented Dec 23, 2024 •

edited

Loading

trygveaa commented Dec 23, 2024

zer0def commented Dec 23, 2024 •

edited

Loading

zer0def commented Dec 23, 2024

ikwyl6 commented Dec 23, 2024

trygveaa commented Dec 24, 2024

LorenzoBianconi commented Dec 26, 2024

can't load matrix.py #357

can't load matrix.py #357

Comments

bloosanbi commented Jul 25, 2023 • edited Loading

poljar commented Jul 25, 2023

ikwyl6 commented Sep 4, 2024

nexy7574 commented Sep 5, 2024 • edited Loading

kqr commented Sep 12, 2024 • edited Loading

ikwyl6 commented Nov 3, 2024 • edited Loading

LorenzoBianconi commented Nov 24, 2024

ikwyl6 commented Nov 25, 2024

trygveaa commented Nov 25, 2024 • edited Loading

LorenzoBianconi commented Nov 25, 2024

trygveaa commented Nov 26, 2024

LorenzoBianconi commented Dec 23, 2024

zer0def commented Dec 23, 2024 • edited Loading

trygveaa commented Dec 23, 2024

zer0def commented Dec 23, 2024 • edited Loading

zer0def commented Dec 23, 2024

ikwyl6 commented Dec 23, 2024

trygveaa commented Dec 24, 2024

LorenzoBianconi commented Dec 26, 2024

bloosanbi commented Jul 25, 2023 •

edited

Loading

nexy7574 commented Sep 5, 2024 •

edited

Loading

kqr commented Sep 12, 2024 •

edited

Loading

ikwyl6 commented Nov 3, 2024 •

edited

Loading

trygveaa commented Nov 25, 2024 •

edited

Loading

zer0def commented Dec 23, 2024 •

edited

Loading

zer0def commented Dec 23, 2024 •

edited

Loading