Rekt pps? problem? => `piker.accounting` #489

goodboy · 2023-03-10T23:07:47Z

Even moar fixes after #470, #475, #484 which mega broke the old
pps.toml management for live backend 💥 😶‍🌫️ 🤕

Turns out this might be the reason to start assaulting the revamp of
our accounting tooling as per our now long list of desirables:
accounting prolly positioning: the accounting of "what/when (is) owned"

Testing enhancements (some tasks deferred)

rework test set to use pytest.mark.parametrize so we can more
easily use common harness logic for darks and alert testing.
reorg the ahab API docker supervisor endpoints to be decoupled
from open_pikerd() and able to accept arbitrary config overrides for
both testing and distributed setups.
~~dark order and alert clearing~~
- Dark order and alert clearing tests #466 and what we're planning to do for offline storage of level
  lines
~~add a basic accounting of position only test set?~~
- deferred to Ledger (syncing) CLI and offline toolset #512 testing section

tasks and bugs getting deferred..

pre-existing live order loading is borked on first chart edit from
order mode: Editting initial live order load fails? #502
final removal of Symbol and .symbol semantics -> summarized in Final removal of .accounting._mktinfo.Symbol #517
- drop from clearing msg set.
- order_mode .symbol final usage..?
make a follow up issue to add a Flume.Cascade type for fsps
which describes stream IO between flumes:
deribit still need full get_mkt_info() and FeedInit
sync up the new data.validate stuff with old brokercheck idea in
Backend test suite #387, Brokercheck an all backends #393 -> add an actual test?

dependency changes needed for this to land

move to new TOML libs (tomlkit for styling and tomli-w for
fast writing) B)
- break the shoddy toml writing hacks we have into a new submod..
- actually remove the toml dep and go with tomli, tomli-w
  and tomlkit B)
- move to using tomlkit which does support writing inline
  tables (as well as lotsa other great stuff) as requested in Better TOML libs -> tomlkit for style, tomli(-w) for speed #496 B)
  - For now pin to our fork!
    (https://github.com/pikers/tomlkit/tree/piker_pin)
    - need to land all local changes and pin the requirement!
  - make PR to upstream -> Support Decimal and MutableMapping codecs, add a multiline-Array indent control python-poetry/tomlkit#294
Context semantics for cross-actor-task cancellation and overruns goodboy/tractor#357
- at least move the piker pin to this history to get CI green

Unrelated to `.accounting` changes

complete rework of .data._web_bs implementation to avoid the use
of backend code using async generators within async context managers.
- see 59743b7 and follow up commits
  - yeah probably should have broken this out into a separate PR but
    too many devs are too busy to review it rn so there's not a lot of
    point 😂

NEW SUBSYSTEM: the `piker.accounting` (subpkg) 🏄

Outstanding/Related issues/bugs that should be resolved here:

kraken: balance <-> position disparities with crypto/crypto pairs #373 closed for Position accounting refinements, bug fixes, src-asset-pnls, ledgerd daemon #515
Pps toml header leaking after zeroing positions #472 fixed here and prior
Better TOML libs -> tomlkit for style, tomli(-w) for speed #496
- linked in our PR and issues for follow up B)
ib forex is currently borked since the move to MktPair (b44b091)
- fix no-vlm relay as well so data layer / sampling / charts
  behave as before!
  - it's actually fine graphically as is (an empty vlm plot) but it's
    burning a shm allocation...
- fix cmdty and other non-vlm contract live feed startup.. bad con
  type matching.
kucoin borked on search result feed selection-load. (ff74d47)

discovered BUGs tofix, issues likely to re-issued and defer, new feature ideas:

kraken outstanding stuff from kraken position tracking gotchas #263 -> moved to Position accounting refinements, bug fixes, src-asset-pnls, ledgerd daemon #515

extra follow ToDos (birds with stones):

follow up .accounting subsys summary refinements and bug fixes
in Position accounting refinements, bug fixes, src-asset-pnls, ledgerd daemon #515
EMS layer: unify the messaging layer to a single msg set: Unify EMS messaging set #514
CLI piker ledger (sync) development and nice-to-haves
- make follow up issue: Ledger (syncing) CLI and offline toolset #512
- make follow up to move to typer for all CLIs: Move to typer for all clis #513

goodboy · 2023-04-16T00:19:38Z

Lul, gotta drop that pytest_trio import..

To be compat with the `Symbol` (for now) and generally allow for reading the (derivative) contract specific part of the fqme. Adjust `contract_info: list[str]` and make `src: str = ''` by default.

Initial attempt at getting the sampling and shm layer to use the new mkt info meta-data type. Draft out a potential `BackendInitMsg: msgspec.Struct` for validating the init msg returned from the `stream_quotes()` start value; obvs don't actually use it yet.

To make nested `msgspec.Struct`s work we need to tell the codec that the `.symbol` is some struct def, since we don't really need to enforce that (yet) we're just going to enc/dec as `str` until we further formalize and/or need something more complex.

…n order

goodboy · 2023-05-26T19:05:03Z

piker/accounting/_pos.py

-# TODO: instead see if we can hack tomli and tomli-w to do the same:
-# - https://github.com/hukkin/tomli
-# - https://github.com/hukkin/tomli-w
-class PpsEncoder(toml.TomlEncoder):


entirely replaced by tomlkit

goodboy · 2023-05-26T19:08:18Z

piker/accounting/_pos.py

+    conf_path: Path
+    conf, conf_path = config.load_account(brokername, acctid)
+
+    if brokername in conf:


this was originally for converting legacy account files: pps.toml but we can probably remove it in next patch set for this subsys.

goodboy · 2023-05-26T19:10:31Z

piker/brokers/binance.py



+@async_lifo_cache()
+async def get_mkt_info(


this is the new required endpoint for all backends which delivers the MktPair (ideally with embedded .src/.dst attrs which are Assets) translation for any searched market/symbol/ticker pattern.

goodboy · 2023-05-26T19:16:23Z

piker/brokers/ib/feed.py



+@async_lifo_cache()
+async def get_mkt_info(


I'm the least happy with this implementation (in terms of existing backends) just because as a legacy market broker ib tends to assume a source asset for every exchange..

Ideally we can work toward getting this more refined going forward particularly if we're going to be re-doing our tsdb subsys backend choice(s) and don't have to rely on the table keys in the existing replicated data sets..

longer run i would like to see mnq/usd.cme.ib over no usd in the fqme.

goodboy · 2023-05-26T19:17:56Z

piker/brokers/ib/feed.py

+    # but likely we want all backends to eventually handle
+    # ``dst/src.venue.`` style !?
+    src: str | Asset = ''
+    if atype == 'forex':


bleh, yeah we need some backend specific tests i think to be able to get this to the new proper fqme nomenclature i'm thinking...

…ged it!?

… the test

goodboy · 2023-05-28T19:38:39Z

Ok so as per discussion in dev chat I'm going to cowboy land this massive change set without extensive review bc,

there aren't enough core devs around to give it the attention rn
the bug fix creep got too big and would require a lot of commit factoring and sub-PRs to be made (which i don't really have the time or patience to go through rn 😂 ) along the scale of what happened in Multichartz #420 ..
there's enough critical backend stuff that was fixed and the new .accounting pkg is important enough to get on the main branch asap.
there is a test set which has caught pretty much all regressions and all such issues have been resolved.
everything else that wasn't finalized has been documented in follow up issues.
i've been using this code in production for weeks and am aware of any oustanding critical live trading problems.

Hopefully this won't happen again 😹

goodboy requested a review from guilledk March 10, 2023 23:07

goodboy force-pushed the rekt_pps branch 3 times, most recently from 3fa212f to fa3cdbe Compare March 13, 2023 20:38

This was referenced Mar 23, 2023

FQME: fully qualified market endpoint, (or f#$% me) the thing no suit could imagine.. #467

Open

We should add tab completion for our cli #22

Open

goodboy mentioned this pull request Apr 2, 2023

deribit: L1 labels for options have too little precision, y-range oddness.. #480

Open

2 tasks

goodboy mentioned this pull request Apr 13, 2023

kucoin backend #494

Merged

goodboy force-pushed the rekt_pps branch 2 times, most recently from 2e2f49e to c913adc Compare April 21, 2023 17:43

jaredgoldman mentioned this pull request Apr 22, 2023

kucoin backend: live order mgmt #499

Open

8 tasks

This was referenced Apr 23, 2023

Start piker.storage subsys: cross-(ts)db middlewares #486

Open

Better TOML libs -> tomlkit for style, tomli(-w) for speed #496

Open

Caching subsys: db, filesystem, do we even need something? #504

Open

goodboy force-pushed the rekt_pps branch from 89207f2 to 81ea1c8 Compare May 9, 2023 18:47

goodboy added 14 commits May 9, 2023 14:49

Optionally load MktPair in Flumes

452cd7d

Add MktPair.suffix: str read from contract info

d4a5a30

To be compat with the `Symbol` (for now) and generally allow for reading the (derivative) contract specific part of the fqme. Adjust `contract_info: list[str]` and make `src: str = ''` by default.

.clearing: broad rename of fqsn -> fqme

1d08ee6

Expect new MktPair.tick_size: Decimal attr in ems

580165f

kraken: parse our source asset key and set on MktPair.src: str

d62fb65

Typecast OrderMode.staged.symbol: str before .copy()!

b6df83a

Drop use of legacy Symbol.broker_info in display startup

7be85a8

'kraken: finally, use new MktPair in 'mkt_info' init msg field!'

dc2332c

Drop more Optional usage on our Struct

6272cae

Rename fqsn -> fqme in paper engine

406565f

Add .__str__() to mktpair and symbol types, fix MktPair.fqme toke…

fd9e484

…n order

Use str(cmd.symbol) for fqme on cancels, add _nowait() method names

0694662

goodboy added 4 commits May 26, 2023 14:05

ib: don't log-emit ib pp msg when none exists..

3d8c1a7

conftest: be explicit about which config we touch

40c5f39

Ensure quote last price is a float

3b5bd8f

tmpconfdir: always assert brokers config created

5f1d0fc

goodboy commented May 26, 2023

View reviewed changes

goodboy force-pushed the rekt_pps branch 3 times, most recently from c72b998 to 7011187 Compare May 26, 2023 20:21

goodboy added 4 commits May 26, 2023 16:50

Adjust config.load() to handle CI git checkout dir, seems they chan…

5e2107f

…ged it!?

Drop elastic search container build for now since we're also skipping…

9ec664f

… the test

add in [kucoin] section to brokers conf

024cf8b

Change to new context-cancelled msg contents: pikerd is canceller

4f67ac0

goodboy force-pushed the rekt_pps branch from 7011187 to 4f67ac0 Compare May 26, 2023 21:17

This was referenced May 26, 2023

Forex fqsn fun.. #403

Closed

Backend test suite #387

Closed

binance: use built-in anext() add note about new ws ep URL, fix agen streaming within NoBsWs usage #473

Merged

goodboy added 4 commits May 28, 2023 12:41

Add nonlocal annots to satisfy ruff

d6331ce

Fix _digits attr names in order mode..

41aa87f

Always allocate a new OrderClient per open_ems() call

f6549fc

Parametrize ems service test to cancel with API and kbi

abd3cef

goodboy mentioned this pull request May 28, 2023

Final removal of .accounting._mktinfo.Symbol #517

Open

4 tasks

goodboy merged commit e51ba40 into master May 28, 2023

goodboy deleted the rekt_pps branch May 28, 2023 19:41

goodboy mentioned this pull request May 31, 2023

Fix price label precision as MktPair.price_tick_digits #518

Merged

goodboy mentioned this pull request Jun 21, 2023

the boss, conf.toml #221

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rekt pps? problem? => `piker.accounting` #489

Rekt pps? problem? => `piker.accounting` #489

Uh oh!

goodboy commented Mar 10, 2023 •

edited

Loading

Uh oh!

goodboy commented Apr 16, 2023

Uh oh!

goodboy May 26, 2023

Uh oh!

goodboy May 26, 2023

Uh oh!

goodboy May 26, 2023

Uh oh!

goodboy May 26, 2023 •

edited

Loading

Uh oh!

goodboy May 26, 2023

Uh oh!

goodboy commented May 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		@async_lifo_cache()
		async def get_mkt_info(



		@async_lifo_cache()
		async def get_mkt_info(

Rekt pps? problem? => piker.accounting #489

Rekt pps? problem? => piker.accounting #489

Uh oh!

Conversation

goodboy commented Mar 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing enhancements (some tasks deferred)

tasks and bugs getting deferred..

dependency changes needed for this to land

Unrelated to .accounting changes

NEW SUBSYSTEM: the piker.accounting (subpkg) 🏄

Outstanding/Related issues/bugs that should be resolved here:

discovered BUGs tofix, issues likely to re-issued and defer, new feature ideas:

extra follow ToDos (birds with stones):

Uh oh!

goodboy commented Apr 16, 2023

Uh oh!

goodboy May 26, 2023

Choose a reason for hiding this comment

Uh oh!

goodboy May 26, 2023

Choose a reason for hiding this comment

Uh oh!

goodboy May 26, 2023

Choose a reason for hiding this comment

Uh oh!

goodboy May 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goodboy May 26, 2023

Choose a reason for hiding this comment

Uh oh!

goodboy commented May 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rekt pps? problem? => `piker.accounting` #489

Rekt pps? problem? => `piker.accounting` #489

goodboy commented Mar 10, 2023 •

edited

Loading

Unrelated to `.accounting` changes

NEW SUBSYSTEM: the `piker.accounting` (subpkg) 🏄

goodboy May 26, 2023 •

edited

Loading