Ib py311 fixes #535
Draft
Ib py311 fixes #535
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As per the hot tip from the edgecases.md, https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md#modulenotfounderror-no-module-named-packagename Factor all the (mostly `setuptools`) overrides into a `pypkgs-build-requirements` set and `.extend()` in any `preferWheel` additions (`polars`, `pyqt`, etc.) before passing to to `mkPoetryApplication(overrides=<it>)`. Add a buncha todos for improving the poetry2nix pkging including: - adding the override requirements to the json file for all our deps in the `pypkgs-build-requirement` set. - maybe propose docs for the edgecases.md to show how to do the auto-gen set (via func) AND extend with further overrides like `preferWheel`? - task to support `polars` build from src (by copying `cryptography` stuff) instead of only from a wheel? - get pyqt5 building from wheel since it seems to be taking forever from src.. - get pyqt6 working in general - going to require taking stuff from nixpkgs and applying it in the overrides of p2n.
NB: for now this is linking to a presumed local clone of the `poetry2nix` repo since part of fixing what was adjusted here needs to be patched upstream, which means hackin on the p2n repo in tandem B) Since there's some dependency build issues we need to tweak the following to get baseline `nix develop` working: - drop `python-levenshtein` (required by `fuzzywuzzy[speedup]`) for now since the overlay and/or wheel install needs to be properly figured out. - build `pyqt5` from src for the moment (since `preferWheel` doesn't seem to be workin?) despite it taking forever XD - add in the `flake.lock` file.
Makes it easier to pass the overrides to multiple p2n functions (like hopefully `.mkPoetryEnv`). Also, add some commented attempts at using `mkPoetryEnv` and todo list for "why", remove the `poetry` CLI main point from the pyproject.toml, bump the poetry lock file.
…os are not allowed..
Since we eventually want to allow users to minimally deploy `pikerd` service-tree (aka distributed cross host) installs, we need to offer a "headless" deps group. Really this is just the core dep set minus Qt and some aux search related libs (for now). The new `.dev` group is for adding hacking and testing tools including `xonsh` since that will eventually be our REPL of choice more then likely B) Oh, and fix the namespace path (was a typo) for the `ledger` CLI and of course bump the lock file.
…specifies to raise
So that pos msgs from the ems are correctly loaded..
Of course I missed this first try but, we need to use the ws market pair symbology set (since apparently kraken loves redundancy at least 3 times XD) when processing transactions that arrive from live clears since it's an entirely different `LTC/EUR` style key then the `XLTCEUR` style delivered from the ReST eps.. As part of this: - add `Client._altnames`, `._wsnames` as `dict[str, Pair]` tables, leaving the `._AssetPairs` table as is keyed by the "xname"s. - Change `Pair.respname: str` -> `.xname` since these keys all just seem to have a weird 'X' prefix. - do the appropriately keyed pair table lookup via a new `api_name_set: str` to `norm_trade_records()` and set is correctly in the ws live txn handler task.
Since `poetry` doesn't seem to actually mark optional group deps as such in the lock file (!?) manually generate a `poetry.lock` with the optional groups commented out in the `pyproject.toml`; this is all in an attempt at trying to make `poetry2nix` build without any UI components which seem to be the source of much frustration without hacking on p2n and/or nixpkgs repos.. Further drop all the old build system files including the setup.py and requirements.txt files.
Change the deats inside the method and have the `brokerd` search task just call it as needed since we already do internal mem caching on the lookup table. APIs changed so we need to make some tweaks as per: - https://github.com/maxbachmann/RapidFuzz/blob/main/api_differences.md - https://github.com/maxbachmann/RapidFuzz/blob/main/api_differences.md#differences-in-processor-functions The main motivation is to get better wheel pkging support (for nixos), better impl in C++, and a more simply licensed dep.
Previously we were assuming that the `Client._contracts: dict[str, Contract]` would suffice this directly, which obviously isn't true XD Also, - add the `NSE` venue to skip list. - use new `rapidfuzz.process.extract()` lib API. - only get con deats for non null exchange names..
A helper for scanning a "pairs table" that most backends should expose as part of their (internal) symbology set using `rapidfuzz` over a `dict[str, Struct]` input table. Also expose the `data.types.Struct` at the subpkg top level.
Apparently they're being massive cucks and changing their futes pair schema again now adding a `NEXT_QUARTER` contract type which we weren't handling at all. The good news is falling back to an old symcache file would have prevented this from crashing. Add a new `FutesPair.expiry: str` field so that `.bs_fqme` can simply call it during the summary FQME-ification output rendering..
Just like the others but also flip to using a `Client.get_mkt_pairs()` meth name for consistency across clients.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So more or less this started as a set of patches to get py3.11 working in general despite there being some weird differences with
asyncio.wait()and usage of it in theibbackend, but now it also includes a buncha fixes which are part of getting things working more properly on nixos. There might also be some rando fixes that need to get backported to other upstream devs branches that I'll detail below.Currently the diff is against #528 but likely to change substantially; think of this as the dev dev dev branch 😂
Commits to backport
e9887cb needs to be backported to Basic buy bot #526
to Account tests #528
to pull into Poetry2nix #533, though i don't know if it's going to be possible due to path dependence.. i think a bunch of these may have already been cherried? gotta triple check..
back to Start
piker.storagesubsys: cross-(ts)db middlewares #486oof, and i guess i should try to bring in the new symcache stuff as a separate patch set?