Using trio with httpx in a basic example works. In a basic locust script it fails with NotImplementedError: unsupported platform

[dkelsey]

Description:

• I have a basic example (testTrio.py) that uses httpx.AsyncClient with trio. This works correctly when I run it.
• I have a basic locustfile.py that also uses httpx.AsyncClient. It fails with NotImplementedError: unsupported platform.

Working Script testTrio.py

import httpx
import trio

url = 'https://puginarug.com/'

async def main():
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        print(response)

trio.run(main)

Output

<Response [200 OK]>

Non-working locustfile.py

from locust import HttpUser, task
import trio, httpx

class MyUser(HttpUser):
    wait_time = between(1, 3)
    url = 'https://puginarug.com/'

    async def on_start(self):
        # Create an httpx.AsyncClient session at the start of the user's lifecycle
        self.client = httpx.AsyncClient(backend='asyncio')

    async def on_stop(self):
        # Close the httpx.AsyncClient session at the end of the user's lifecycle
        await self.client.aclose()

    @task
    async def my_task(self):
        # Use httpx.AsyncClient to make asynchronous requests
        response = await self.client.get(url)
        print(f"Response: {response.text}")

Output

Traceback (most recent call last):
  File "/Users/dkelsey/.pyenv/versions/HTTPX/bin/locust", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/locust/main.py", line 100, in main
    docstring, _user_classes, shape_classes = load_locustfile(_locustfile)
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/locust/util/load_locustfile.py", line 70, in load_locustfile
    loader.exec_module(imported)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/dkelsey/Development/DES/load-testing-tools/experiments/AsyncHTTXClient/locustfile.2.py", line 2, in <module>
    import trio, httpx
  File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/__init__.py", line 22, in <module>
    from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED  # isort: split
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/_core/__init__.py", line 21, in <module>
    from ._local import RunVar, RunVarToken
  File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/_core/_local.py", line 9, in <module>
    from . import _run
  File "/Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages/trio/_core/_run.py", line 2800, in <module>
    raise NotImplementedError("unsupported platform")
NotImplementedError: unsupported platform

Environment:

OS: Mac M1
Python: 3.11.4 (venv)
Locust:

Name: locust
Version: 2.23.1
Summary: Developer friendly load testing framework
Home-page:
Author:
Author-email:
License: MIT
Location: /Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages
Requires: ConfigArgParse, flask, Flask-Cors, Flask-Login, gevent, geventhttpclient, msgpack, psutil, pyzmq, requests, roundrobin, Werkzeug
Required-by:

httpx:

Name: httpx
Version: 0.27.0
Summary: The next generation HTTP client.
Home-page:
Author:
Author-email: Tom Christie <[email protected]>
License:
Location: /Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages
Requires: anyio, certifi, httpcore, idna, sniffio
Required-by:

trio:

Name: trio
Version: 0.24.0
Summary: A friendly Python library for async concurrency and I/O
Home-page:
Author:
Author-email: "Nathaniel J. Smith" <[email protected]>
License: MIT OR Apache-2.0
Location: /Users/dkelsey/.pyenv/versions/HTTPX/lib/python3.11/site-packages
Requires: attrs, idna, outcome, sniffio, sortedcontainers
Required-by:

Line 2800 of trio/_core/_run.py

Questions

• Why would the basic example work and locust not?
• Is this related to Gunicorn worker fails to boot due to NotImplementedError  #2960

[A5rocks]

I've had success using httpx with more complicated trio applications, so I'm thinking locust must do something very strange. Probably not the same thing as gunicorn though (re your question: gunicorn fails because it removes the primitives we need to wait for files).

EDIT: nevermind, locust does use gevent. see https://github.com/locustio/locust/blob/baf007f1c9f243a550621ecc99720d835513e220/docs/changelog.rst#L789-L790

There's no satisfying solution IMO because if trio reaches into gevent's API to pull out the original kqueue, then I think we're blocking the greenlet? Which is obviously not what the user wants IMO.

Maybe we should just fall back to gevent's patched version of something? I'm not too familiar with the underlying mechanics of trio to say what. Hopefully someone else has an answer!

EDIT 2: Heh, looks like they did some special casing on their end re: gevent w/ trio! See locustio/locust@aedc3b9

[jakkdl]

sounds like something to bring up on their end in that case. Either to re-fix their workaround, or to see if they know how we should work around the patch on our end.
Is this a thing that's broken in one of our recent releases, given that we're getting several reports now? Or new version of gevent/other?

[CoolCat467]

Is it possible to do backend='trio'?

[dkelsey]

@CoolCat467 is this something you are asking me to try?

I added backent'trio' as a param to httpx.AsyncClient
I get the same errror.

[dkelsey]

I ran both scripts in debug (pyCharm).

TestTrio.py

It lands on this part of trio/_core/_run.py

...
elif TYPE_CHECKING or hasattr(select, "kqueue"):
    from ._generated_io_kqueue import *
    from ._io_kqueue import (
        EventResult as EventResult,
        KqueueIOManager as TheIOManager,
        _KqueueStatistics as IOStatistics,
    )

• TYPE_CHECKING is False
• The call to hasattr inspects select-cpython-311-darwin.so and finds kqueue exists.

<module 'select' from '/Users/dkelsey/.pyenv/versions/3.11.4/lib/python3.11/lib-dynload/select.cpython-311-darwin.so'>

• it exists.
• it works.

locustfile.py

I place a break point in _run.py and ran the locust script in debug.
The same check occurs:

<module 'select' from '/Users/dkelsey/.pyenv/versions/3.11.4/lib/python3.11/lib-dynload/select.cpython-311-darwin.so'>

• no kqueue
• It fails.

So where did kqueue go?

It was monkey patched away?

Note:

I followed this to get debugging working with locust/gevent in pyCharm:

locustio/locust#613 (comment)

Also added this to the locust script at the end (and imported it):

if __name__ == "__main__":
    run_single_user(MyUser)

After all this... I re-read the comments from @A5rocks and @jakkdl
I now understand what you wrote.

So how to get httpx.AsyncClient or trio to work in locust?

[A5rocks]

A hacky solution would be to edit /Users/dkelsey/.pyenv/versions/HTTPX/bin/locust to try import trio before anything else, which will cache trio. Obviously this is a no-go for various reasons.

The best solution here (IMO) is to use non-async httpx, and use gevent if you need "async" stuff. Unfortunately it looks like locust is built around this.

As for fixing this in trio, I'm starting to lean more towards the broken-solution to this where we reach into gevent and pull out kqueue (there's a way to do that) but this probably requires more thought.

[A5rocks]

Actually I thought about this a bit more, and select.kqueue = gevent.monkey.get_original("select", "kqueue") before you import trio (with the necessary imports, of course) might work in your locustfile.py? I'm on mobile so I can't check whether you can assign to module contents like that.

Obviously this is a bad solution but less so than my previous one where you had to patch locust!

[dkelsey]

Found this in the locust docs here: testing other systems

It is important that the protocol libraries you use can be monkey-patched by gevent.

[dkelsey]

A hacky solution would be to edit /Users/dkelsey/.pyenv/versions/HTTPX/bin/locust to try import trio before anything else, which will cache trio. Obviously this is a no-go for various reasons.

The best solution here (IMO) is to use non-async httpx, and use gevent if you need "async" stuff. Unfortunately it looks like locust is built around this.

As for fixing this in trio, I'm starting to lean more towards the broken-solution to this where we reach into gevent and pull out kqueue (there's a way to do that) but this probably requires more thought.

My intention is to send multiple multiplexed requests for resources through a single connection.

[jakkdl]

I'll reiterate that I think you should open an issue in locust and see if they can re-fix their workaround. Getting kqueue support into gevent or general gevent support in trio both sound like quite tough tasks.

[dkelsey]

Thanks @jakkdl those are good recommendations.

I found a way to achieve what I want useing pycurl in locust, writing a client and user class following the locust docs on testingnother services.

I'm satisfied. Thanks again.