Cached feeds #161
Merged
Cached feeds #161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Was actually good cause there was a conflict from the windows patches stuff. |
goodboy
commented
Apr 6, 2021
| # count: int = 20, # NOTE: any more and we'll overrun underlying buffer | ||
| count: int = 6, # NOTE: any more and we'll overrun the underlying buffer | ||
| count: int = 10, # NOTE: any more and we'll overrun the underlying buffer |
There was a problem hiding this comment.
iirc this was < 6 to avoid tws throttling on another system I tried.
We might want to include this change as a default?
goodboy
commented
Apr 6, 2021
piker/data/_buffer.py
Outdated
| if total_s % delay_s != 0: | ||
| continue | ||
|
|
||
| # TODO: numa this! |
There was a problem hiding this comment.
Ahh right, that was the last thing we should do. Stick this in a new _sampling.py module instead of the current data._buffer which is a lame name.
goodboy
commented
Apr 6, 2021
| subs.remove(ctx) | ||
|
|
||
|
|
||
| async def sample_and_broadcast( |
There was a problem hiding this comment.
There I think isolating these shm sampling routines is much more easy for code readers.
Move all feed/stream agnostic logic and shared mem writing into a new set of routines inside the ``data`` sub-package. This lets us move toward a more standard API for broker and data backends to provide cache-able persistent streams to client apps. The data layer now takes care of - starting a single background brokerd task to start a stream for as symbol if none yet exists and register that stream for later lookups - the existing broker backend actor is now always re-used if possible if it can be found in a service tree - synchronization with the brokerd stream's startup sequence is now oriented around fast startup concurrency such that client code gets a handle to historical data and quote schema as fast as possible - historical data loading is delegated to the backend more formally by starting a ``backfill_bars()`` task - write shared mem in the brokerd task and only destruct it once requested either from the parent actor or further clients - fully de-duplicate stream data by using a dynamic pub-sub strategy where new clients register for copies of the same quote set per symbol This new API is entirely working with the IB backend; others will need to be ported. That's to come shortly.
Avoid bothering with a trio event and expect the caller to do manual shm registering with the write loop. Provide OHLC sample period indexing through a re-branded pub-sub func ``iter_ohlc_periods()``.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Copy of #158 but merging into
master😂This is a rework on data feed machinery to allow for persistent real-time feeds to remain running after created by some data client where that client detaches but we want to keep the feed live for later use.
For now this means spawning a background feed task which also writes to shared memory buffers and can be accessed by any (returning) data client. This has the immediately useful benefit of allowing for super fast chart loading for pre-existing feeds.
Try it out by,
Both the
ibandkrakenbackends have been ported thus far.Questrade can likely come in a new PR since the single-connection-multi-symbol-subscription style can also be accomplished with kraken's websocket feed and we'll probably want to iron out details of how to request such things with both backends in view.
Some other stuff got slapped in here, namely engaging the Qt hidpi detection on windows (which reportedly works?)