Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slowest consumer broadcasting? #235

Open
goodboy opened this issue Sep 3, 2021 · 0 comments
Open

Slowest consumer broadcasting? #235

goodboy opened this issue Sep 3, 2021 · 0 comments
Labels
experiment Exploratory design and testing help wanted Extra attention is needed

Comments

@goodboy
Copy link
Owner

goodboy commented Sep 3, 2021

We now have fastest-consumer (aka tokio style) from #229 but there seems to be demand from others as mentioned in python-trio/trio#987 for a non-lossy variant which performs strict backpressure on the laggers.


I had some brief notes in #229 (though not sure they're relevant any more after refining that patch):

If we wanted to also support slowest consumer style, I'm pretty sure there's an easy hack to just inherit and proxy through reads to trio._channel.MemoryChannelState.data (presuming the queue size to broadcast_receiver() and the underlying mem chan's deque are the same size).

The thinking is, there's a length check to determine if the sender should block in MemorySendChannel.send_nowait():
https://github.com/python-trio/trio/blob/master/trio/_channel.py#L147

If .data here proxied to lambda: max(self._subs.values()) and the order of these branches were reversed, it might just work no?

        elif len(self._state.data) >= self._state.max_buffer_size:
            # causes ``.send()`` to block
            raise trio.WouldBlock
        else:
            self._state.data.append(value)

On further thought I think this may require a deeper investment to hook into the send side of the underlying receive channel to get this behavior without relying on implementation details of trio's builting mem chans.

Ideas from lurkers welcome.

@goodboy goodboy added help wanted Extra attention is needed experiment Exploratory design and testing labels Sep 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experiment Exploratory design and testing help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant