Agg feedz #414

goodboy · 2022-11-04T21:13:37Z

Enhances our piker.open_feed() real-time quotes and history managment layer piker.data.feed to accept multi-fqsn inputs to deliver multi-symbol quote streams and a new internal data streaming abstraction/API: .data.Flume which provides the basis for real-time stream mangement, access and measure for the needs of real-time data-flow management and orchestration.

Synopsis

The final core-UX feature you always wanted as a chart trader is probably something like:

mult-instrument overlayed real-time and historical data feeds with simultaneous interaction and "current symbol" selectable order mode control..

well, this is finally within reach 😎 and this patch add the "backend" work making it possible 🏄🏼

Notes for manual testing

Ideally reviewers run the new feeds test set with pytest tests/test_feeds.py.
Note that you'll need to install the piker_pin branch of tractor in order for the test set to run green:

the requirements file was updates in this patch: https://github.com/pikers/piker/pull/414/files#diff-4d7c51b1efe9043e44439a949dfd92e5827321b34082903477fd04876edb7552
you can also simply switch to the remote piker_pin branch for tractor if installed in dev mode locally.

to land

fill out commit msg for 7abcb3e which was initial (half-working) patch to get basic funtionality
port all consumer code in clearing, order mode, charting/graphics layer to expect this adjusted Feed api.
add basic per-brokerd multi-symbol real-time feeds working
- enables piker.open_feed(fqsns=['btcusdt.binance', 'ethusdt.binance']) as feed) where the delivered Feed now has a .flumes: dict[str, Flume] which enables per-fqsn data flow access, mgmt, measure (see the historical flume for idea behind this abstraction terminology)
- add a test in test_feeds.py
  - binance multi-symbol case
  - kraken multi-symbol case
  - kraken currently seems to depend on a brokers.toml existing? we should fix this..
add cross-brokerd multi-feeds such that `piker.open_feed(fqsns=['btcusdt.binance', 'xbtusdt.kraken']) will work with an aggregate receive channel delivering quotes from both backends?
- multi-brokerd multi-sym case (kraken+binance`)

Test suite TODO: see two comments below

This slipped in early from #414 before merge and was likely due to cherry-picking from #417.

goodboy · 2022-11-17T20:42:34Z

Le's try to land #421 before this.

goodboy · 2022-12-13T19:07:12Z

Test suite TODO:

we can't run binance tests from CI due to ip blacklisting on their end.. see the CI error
- instead only run kraken tests in CI (using fixture logic for detection) and expect devs to run the full suite manually on PRs
the initial fix for test failures on teardown was from the combo commits in Don't unset actor global on root teardown goodboy/tractor#347, but those fixes compeletely break tractor 😂, so we need either a better fix or to adjust the feeds tests to expect the streaming teardown gunk.
good news, seems like this issue went away with the new open_test_pikerd() addition

Set each quote-stream by matching the provider for each `Flume` and thus results in some flumes mapping to the same (multiplexed) stream. Monkey-patch the equivalent `tractor.MsgStream._ctx: tractor.Context` on each broadcast-receiver subscription to allow use by feed bus methods as well as other internals which need to reference IPC channel/portal info. Start a `_FeedsBus` subscription management API: - add `.get_subs()` which returns the list of tuples registered for the given key (normally the fqsn). - add `.remove_sub()` which allows removing by key and tuple value and provides encapsulation for sampler task(s) which deal with dropped connections/subscribers.

Previously we would only detect overruns and drop subscriptions on non-throttled feed subs, however you can get the same issue with a wrapping throttler task: - the intermediate mem chan can be blocked either by the throttler task being too slow, in which case we still want to warn about it - the stream's IPC channel actually breaks and we still want to drop the connection and subscription so it doesn't be come a source of stale backpressure.

Allows using `set` ops for subscription management and guarantees no duplicates per `brokerd` actor. New API is simpler for dynamic pause/resume changes per `Feed`: - `_FeedsBus.add_subs()`, `.get_subs()`, `.remove_subs()` all accept multi-sub `set` inputs. - `Feed.pause()` / `.resume()` encapsulates management of *only* sending a msg on each unique underlying IPC msg stream. Use new api in sampler task.

This reverts commit 02fbc0a.

Instead of requiring any `-b` try to import all built-in broker backend python modules by default and only load those detected from the input symbol list's fqsn values. In other words the `piker chart` cmd can be run sin `-b` now and that flag is only required if you only want to load a subset of the built-ins or are trying to load a specific not-yet-builtin backend.

Hopefully helps resolve #435

Seems that by default their history indexing rounds down/back to the previous time step, so make sure we add a minute inside `Client.bars()` when the `end_dt=None`, indicating "get the latest bar". Add a breakpoint block that should trigger whenever the latest bar vs. the latest epoch time is mismatched; we'll remove this after some testing verifying the history bars issue is resolved. Further this drops the legacy `backfill_bars()` endpoint which has been deprecated and unused for a while.

…e stack

goodboy mentioned this pull request Nov 4, 2022

PlotItemOverlay: an alt approach for #1359 pyqtgraph/pyqtgraph#2162

Draft

5 tasks

Base automatically changed from pg_exts_fork to master November 8, 2022 17:47

goodboy force-pushed the agg_feedz branch from c111bae to 54d7d0d Compare November 10, 2022 03:31

goodboy requested a review from guilledk November 10, 2022 03:45

goodboy force-pushed the agg_feedz branch 2 times, most recently from 56d312d to 091329d Compare November 10, 2022 18:19

goodboy added data-layer real-time and historical data processing and storage broker-backend `brokerd`/`datad` related backend tech brokers-can-smbz features that suits should provide labels Nov 10, 2022

goodboy force-pushed the agg_feedz branch 3 times, most recently from 9ca5152 to df6df2e Compare November 12, 2022 18:47

goodboy added a commit that referenced this pull request Nov 12, 2022

Fix _main() arg back to sym: str

02fbc0a

This slipped in early from #414 before merge and was likely due to cherry-picking from #417.

goodboy mentioned this pull request Nov 12, 2022

HOTFIX: Fix _main() arg back to sym: str #419

Merged

goodboy requested a review from iamzoltan November 14, 2022 23:16

goodboy mentioned this pull request Nov 15, 2022

Multichartz #420

Closed

43 tasks

goodboy force-pushed the agg_feedz branch 3 times, most recently from 701de9f to 4948bae Compare November 17, 2022 20:40

goodboy changed the base branch from master to ib_contract_updates November 17, 2022 20:42

Base automatically changed from ib_contract_updates to master November 17, 2022 21:38

goodboy force-pushed the agg_feedz branch 2 times, most recently from 4a35fcb to 13d81eb Compare December 10, 2022 21:15

goodboy mentioned this pull request Dec 13, 2022

OSError: [Errno 9] Bad file descriptor in cluster test goodboy/tractor#345

Open

2 tasks

goodboy force-pushed the agg_feedz branch from 9722aad to 791d214 Compare January 10, 2023 00:01

goodboy requested a review from algorandpa January 10, 2023 00:08

goodboy added 2 commits January 10, 2023 11:09

Only log pos size errors for ib

4000034

Drop duplicate live gateway from compose file for now

508de61

goodboy added 23 commits January 10, 2023 11:09

Not all accounts will have API trade transactions this session..

0a959c1

Use (a new) piker_pin branch in tractor (again)

ddbba76

Add an sdist job to CI

79fcbcc

Always touch config file dir if dne

c088963

Move maybe_open_feed() above for readability

1e96ca3

Set brokers: list[st] from mods when not provided..

88870fd

Adjust built-in backend list to known working

967e28b

Revert "Fix _main() arg back to sym: str"

c944db5

This reverts commit 02fbc0a.

Bump up timeout on multi-feed test for CI

5d021ff

Use ETH on kraken to ensure enough quotes

b7e1443

Add ci_env detector fixture

f232d6d

Always force lowercase on binance symbol keys

76f920a

Hopefully helps resolve #435

Set global registry addr after first entry point spawns pikerd

81585d9

Reset ._registry_addr to any passed in value from caller

008ae47

Add a open_test_pikerd() acm fixture for easy booting of the servic…

0662210

…e stack

Only run kraken feed tests in CI, use open_test_pikerd()

593db0e

Adjust cli mod imports of daemon sockaddr vars

55de9ab

Go back to Feed.pause/resume(), new flume APIs coming later

963e5bd

goodboy force-pushed the agg_feedz branch from 791d214 to 963e5bd Compare January 10, 2023 16:09

goodboy requested review from esmegl and removed request for iamzoltan January 10, 2023 16:13

guilledk approved these changes Jan 11, 2023

View reviewed changes

goodboy merged commit 8d1eb81 into master Jan 13, 2023

goodboy deleted the agg_feedz branch January 13, 2023 17:20

goodboy mentioned this pull request Feb 14, 2023

Overlays and multicharts UX: follow ups to #420 (and friends) #461

Open

77 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Agg feedz #414

Agg feedz #414

goodboy commented Nov 4, 2022 •

edited

Loading

goodboy commented Nov 17, 2022

goodboy commented Dec 13, 2022 •

edited

Loading

Agg feedz #414

Agg feedz #414

Conversation

goodboy commented Nov 4, 2022 • edited Loading

Synopsis

Notes for manual testing

to land

Test suite TODO: see two comments below

goodboy commented Nov 17, 2022

goodboy commented Dec 13, 2022 • edited Loading

Test suite TODO:

goodboy commented Nov 4, 2022 •

edited

Loading

goodboy commented Dec 13, 2022 •

edited

Loading