How does aiohttp client works internally? Detailed writeup?

[lb1mg]

Is there possibly any detailed writeup on the working on aiohttp?

aiohttp loads only the headers when .get() is executed

• How does .get() only fetches headers?

await rest.json() # <-- what if some problem occurs here

• What if something breaks at read time? How is it handled?

Can any contributor do a writeup on the general mechanism by which aiohttp partially loads the response as headers and responses.status is available before awaiting reading of content?
Any bare bones example (maybe at socket level) would be appreciated!

[Dreamsorcerer]

I'm not entirely sure I understand your questions/confusion, but:

* How does `.get()` only fetches headers?

Surely by reading the response until it reaches the start of the message content.

* What if something breaks at read time? How is it handled?

Same as anything else, an exception would be raised: https://docs.aiohttp.org/en/stable/client_reference.html#client-exceptions

Can any contributor do a writeup on the general mechanism by which aiohttp partially loads the response as headers and responses.status is available before awaiting reading of content?

I'm afraid nobody has time, but I'm also not sure if you need more information than the above. Let me know if there is more information you need, but I don't think there's anything unusual or non-obvious here.

[lb1mg]

I'm really sry. It could've been posed more clearly.

aiohttp loads only the headers when .get() is executed

This is from official doc : The aiohttp Request Lifecycle

letting you decide to pay the cost of loading the body afterward, in a second asynchronous operation. Hence the await response.text().

async with aiohttp.ClientSession() does not perform I/O when entering the block, but at the end of it

I found these very confusing.

Now as you have replied,

Surely by reading the response until it reaches the start of the message content.

So has the entire response been downloaded from the server at this point?

For a simple request:

async with aiohttp.ClientSession() as session:#1
        async with session.get(url=url) as resp:#2
            await resp.json()#3

• I don't have any doubts about line 1.
• At line 2, how does it send a request with GET method and fetch only headers? Will server not send entire body too?
  I tried to check if it's sending a HEAD request first, using a dummy server, but its sending a GET only.
• and so aiohttp then requests body after partially fetching headers and status code? again how

I am really curious abt it and I tried reading source too but I thought if I could find any already existing resource on it.

Thanks @Dreamsorcerer

[Dreamsorcerer]

I found these very confusing.

Feel free to suggest any improvements in a PR as you make sense of it.

So has the entire response been downloaded from the server at this point?

Well, as the doc suggests (you don't pay the cost of loading the body), not necessarily. I think the exact answer to that question depends on a few factors, but most likely if it is a small response (e.g. fits within a packet or similar), then yes, whereas if it is a large body, then most likely it has not been loaded.

In short, you should assume that the body has probably not been read until you've returned from a method that reads it, and you should assume that the connection may still be open until you exit the async with block (of the .get() call).

* At line 2, how does it send a request with GET method and fetch only headers? Will server not send entire body too?

Yes, the server would try to, it is just a GET request.
I don't know the exact details, but my rough thinking is that data is sent to the OS socket. This buffers a certain amount of data it receives, but will probably start rejecting packets if the buffer is full. This will cause the server to retry the packets at a slower interval. Therefore, if the buffer is full, the server won't have completed sending the full response until you start reading from it.

That's at the OS level, which is out of the scope of aiohttp. Aiohttp is simply going to read from the OS socket until it reaches the content, then wait until you ask it to read the rest of the data from the socket.

* and so `aiohttp` then requests body after partially fetching headers and status code?

As above, it simply resumes reading from the socket. I think it's also the full headers that are loaded initially, not partial. It should read until the content starts (in HTTP, that's a \r\n\r\n sequence).

I am really curious abt it and I tried reading source too but I thought if I could find any already existing resource on it.

Yeah, I just tried to check a couple of my assumptions in the source code, but there's rather a lot going on. :P

[lb1mg]

Yes, the server would try to, it is just a GET request.

Yep, I think it's actually impressive that despite this aiohttp is reading only headers first then content later.

That's at the OS level, which is out of the scope of aiohttp. Aiohttp is simply going to read from the OS socket until it reaches the content, then wait until you ask it to read the rest of the data from the socket.

I think this might be the gist of it, I would give it a try with a simple implementation.

I don't know the exact details, but my rough thinking is that data is sent to the OS socket.
Yeah, I just tried to check a couple of my assumptions in the source code, but there's rather a lot going on. :P

Thanks for the help @Dreamsorcerer
I hope we can keep this discussion open for someone might possibly want to add something