Overlays and multicharts UX: follow ups to #420 (and friends)

[goodboy]

The UI follow up to #414, it was originally a PR #420 but ended up being
a sequence of smaller (factored) PRs which landed finishing up with #455
📚

Hopefully this is the final core-UX feature you always wanted 🏄🏼

mult-instrument overlayed real-time and historical data feeds with
simultaneous interaction and "current symbol" selectable order mode
control

---

To run and test the UI:

piker chart btcusdt.binance ethusdt.binance xmrusdt.binance

This should (currently) spawn a chart with all real-time quote feeds and history overlayed on both the fast and slow chart and boot using the method=loglin_ref_to_curve' overlay method by default.

---

First draft feature set included:

The first MVP for overlay charts lands in #455 and contained a plethora of precursor PRs including:

• full multi-fqsn support in our data feed layer with multiple symbols per brokerd feed (Agg feedz #414)
• new samplerd daemon introduced to conduct singleton sampler step event synchronization across multiple flow-actor processors (samplerd service #440)
• mega rework to the pre-graphics ShmArray -> QPainterPath rendering pipeline by adding a formalized IncrementalFormatter layer which allows incremental update (and cache) of source data to pre-graphics formats suitable for minimal conversion to a painter path (Pregraphics formatters: IncrementalFormatter #447)
• internally adjusted Axis sticky label APIs (moved to lower level PlotItem in Axis sticky api, PlotItem is the new "chart" #448)
• multi-symbol input to the real-time graphics system(s) supporting first draft of multicharts (Multi symbol input (support) #449)
• epoch-time index alignment of multiple time series and a graphics-layer driven "data viz" sub-sys (Epoch indexing and dataviz layer #451)
• drastically simplified L1 labels to enable tight (horizontal) stacking of y-axis co-domains (Compact L1 labels #452)
• initial multichart latency tuning (Overlays interaction latency tuning #453) through reworks of the display loop's:
  • maxmin() y-ranging and better support for only draw last uppx pixel column style
  • only update from last framed (by type) tick
  • distribution of throttle rate uniformly over number of input fqsns
  • better Viz API pipeline usage to speed up interaction handlers
  • pre-graphics 1-d data caching and use in only draw last uppx pixel column style updates
• Log linearized curve overlays #455 algorithm which allows sanely viewing multiple time series with co-domains normalized via a returns transform:
  • includes support for incremental yrange update from display loop via inputs to ChartView.interact_graphics_loop()
  • removal of all Qt.Signal stuff from view box handlers instead opting to call our .interact_graphics_cycle() manually from mouse event callbacks
  • better leverage of new Viz update methods for faster pipeline updates from incremental quote driven updates in the .ui._display loop.
  • support for multiple overlay methods as finalized in this commit:

    method: Literal[
        'loglin_ref_to_curve',
        'loglin_ref_to_first',
        'mxmn',
    
    ] = 'loglin_ref_to_curve',

---

Init basic funtionality / usage:

• basic initial usage is to start a chart with multiple input symbols:
  piker chart btcusdt.binance ethusdt.binance xmrusdt.binance
• overlay method control for how to align overlays to one another.
• L1 labels can be simplified
  • we decided to make them overlay on their respective y-axis
• how should last datum graphics be offset
  • currently all last datums are time step aligned.

History view

• curves need to be epoch time aligned instead shm index aligned
  since obviously it will result in non-time aligned data sets as it
  is now using the 'index' basis..
• fix pg.LinearRegion item to show all background overlay curves transparently.

Graphics loop

• use single Feed.open_multi_steam() for the update task and have it look up appropriate graphics update apis per symbol-quote
• refactor a bunch of the charting/graphics apis to mostly to remove
  the global-ish ChartPlotWidget instance and API calls, instead
  moving all this state into Flow / Flume items and possibly add
  a Viz type to replace the Flow and instead make the Flow
  a table of associated src data -> procesor pipelines?
  • more to write up as individual subtasks here

Search UI/UX:

• make search results and caching work with multi-symbol selections by the GodWidget
• all selected symbols should be shown in fqsn form in the window title bar

---

Follow up ToDo task list after the last PR (#455) lands above:

These include outstanding todo features which are still in discussion / research.

Charting graphics layer: flows mgmt, backendy controls, viewlists, schema:

(Edit-able) list of rt-feeds on chart: the ViewList:

• more or less very similar to the legacy idea of loading and saving
  watchlists but instead is a more sophisticated abstraction: a list
  of trade-able asset pairs that maps to a display of rt-data-flows
  shown overlayed in a single chart view
• obviusly a declarative syntax much like you'd expect for
  a legacy watchlist but with support for additional metadata:
  [{'btcusdt.binance: {'pin_to': 'qqq.nasdaq.ib'}}, 'qqq.nasdaq.ib']
  or something or other.. (we need to figure out the generic
  data-structure syntax form(s)
• a way to edit multiple lists within the chart UI using (probably)
  vi-like controls similar to those in the ranger file manager
  except for moving entries between multiple stacked view lists in the
  search pane.

bugs^

• current cachelist is not in same order as that passed on CLI?

common axes for common dst-asset:

• how to support same-dst but cross-broker markets, eg. xbtusdt.kraken and btcusdt.binance
  • should we use a single axis?
  • how to determine this automatically?
    • Do a in the same range calc ?
    • have some kinda of adhoc asset name mapping?

OHLC chart overlays:

• for ohlc main chart overlays how should we do coloring?
  • currently we pull from a sequence of colors in .ui._display but
    we should likely make this configurable, possibly through the UI,
    definitely through a config file.
• how to handle overlaid (levered) derivatives where the %returns
  are substantially larger due to the lever profile?
  
  • it might make sense in these cases to instead use a "leverage
    and delta adjusted" scalar for the opt contract so that you can see
    a "somewhat linear" translation of the deriv price vs. the underlying?
    • the issue is that the call option is clearly causing the bitcoin
      move to be mega squashed in the y-range.
• L1 labels were simplified (we decided to make them stack on
  respective y-axis):
  • only show (humanized) size on bid/ask not price?
• in terms of last datum bars, could they be placed in some kind
  of no-exactly-aligned (reverse) order of the y-axis from
  left-to-right?
• des-synced last bar issue on 1Min chart seemingly due to startup
  races where there are N > 2 overlay curves from diff providers?
  • is this something we can fix in the .data.feed shm allocation
    logic?
  • 

FSP (subcharts)

• should fsps like vlm which require a subchart be overlayed or only
  the currently selected shown by default?
  • we can make this configurable and maybe offer a sidepane drop down
    showing all combos and/or a toggle set of symbols from the overlay
    set?
  • can we offer a pair selector to make it possible to show the fsp
    subchart for any market (in the viewlist) not currently selected
    on the OHLC chart(s)?
    • this would probably help perf issues when using binance 😂

=> bugs^

• we still have some strange issue on our (vlm) FSP charts where
  when multiple overlays are being backfilled the chart will get all
  mucked up with strange lines from offscreen connecting to every step?
  (TODO: PUT PIC HERE from @guilledk!)

---

Widget UIs: panes and controls, search..:

UI for "pair selection"

• dynamic management and selection of a current market pair overlay on a chart as per
  the ViewList idea from above:
  • (right-click) context menu options to add/remove some chart
    from/to a diff viewlist?
  • editing a ViewList:
    • mouse UX (shift-click-drag-release?) to create a group from a single/flat results tree?
    • hot keys for adding a new search result to a group instead of a new widget set (maybe ctl-o or ctl-enter on result row?)
    • API allowing for the currently selected market which would
      activate functionality for the order mode and possibly (or
      independently) the current FSP processing (in subchart).
  • current market-feed selection controls:
    • highlight the feed's curve vs. the other overlays
    • highlight the matching y-axis?
    • re-order the axes, selected should be closest to curve?
    • show feed meta-data in sidepane label for selected
    • allow selection by y-axis?
      • double click on axis?
    • or, double click on curve?
    • or, select by hot key (tab or space global hotkey?)
    • activate order mode control for that symbol (see more details in section below)

UI for overlay-curve config:

• support config (meta) data to choose the pin-method (to
  .ui.view_mode.overlay_viewlists()) as well as parameters for each
  mode if supported (eg. 'loglin_ref_to_curve' can take a target curve
  to pin to)
• unify this with the current "feed config" controls: eg. throttle
  rate, shm buffer sizing, host discovery, what daemon to use as
  source.. etc?

Search UI/UX:

• what viewlist(s) should be loaded by default?
  • obvs make it configurable
  • how lazily should we load the rt feed for each?
• a new result should be added to existing viewlist?
  • obvs make this configurable through toml file.
  • how would this play with the idea of an alt-search for
    derivatives?
• how to open/start a new view list?
• how to search through existing viewlists
  • if you pick a result from one do you load the whole thing?

=> bugs^:

• current results order is not the same as CLI input order..

better distinction of pairs on axes?

• draw little line from each last label to curve?
• some kind of label system to easily know which curve is which feed?
  • last label should include fqsn?
  • each y-axis should have fqsn?

---

Order mode

order mode client multi-market API support:

• allow passing a fqsns: list[str] to open_order_mode() and
  having this setup the client loop and the ems tasks to use
  feed.open_feed() with the same fqsns passed through?
  • what should be the handle delivered back? a table of clients
    or should we add some kind of client-selectable API already that
    strat code can (eventually) use for easy arbing?

selection oriented UX, show each mkt's EMS state by axis:

• pp line arrow-marker should sit on the specific y-axis
  • depending on "selected symbol" only show that pp?
• how/when to show all pps?
  • show active order summary markers (for those not in view) on each sub-yaxis
  • maybe only pinned arrow at ppu level always?
• support submitting orders only for current selection:
  • eg. alert submitted should show on only that axis's y-domain
  • cc cancelling should only cancel current instrument's orders?
  • tie this in with the viewlist / search result pane, show
    active orders indicator of some sort?