Skip to content

Commit 2153b0b

Browse files
authored
fix(gl): paginate gl-clone Arweave recovery and make /encrypted-blobs parsing schema-strict (#49) (#70)
* fix(gl): make /encrypted-blobs recovery parsing schema-strict Deserialize the /encrypted-blobs 200 body into a typed EncryptedBlobsResponse instead of poking at a serde_json::Value and defaulting blobs to [] via unwrap_or_default(). A server schema regression now fails closed (a hard parse error) rather than silently yielding nothing to recover, mirroring the existing WithheldPathsResponse convention. The sole caller still does not abort the clone but now surfaces the error instead of swallowing it into an empty result. Part of #49. * fix(gl): paginate Arweave manifest discovery and rank ties by block height Discovery hard-coded a single first:100 GraphQL query with no pagination, so a repo with more than 100 encrypted-manifest anchors (one per push) silently lost recovery coverage for any blob whose most recent seal fell outside that window. Follow cursor pagination (after + pageInfo) until coverage is complete, deduping transaction ids and bounding the loop (1000 pages / 10000 ids, degenerate-cursor guard) so a hostile or buggy gateway cannot grow the downstream per-id fetch loop without limit. A full page returned without pageInfo, a cap hit, or a failure after discovery began now warns; a page-1 failure (unconfigured/unreachable gateway) stays silent as before. Pagination concatenates pages in gateway order, which made merge_manifests' equal-timestamp tie-break (previously insertion-order, i.e. accidentally newest on single-page HEIGHT_DESC) gateway-dependent. Rank candidates by the composite key (timestamp, Arweave block height, cid): timestamp primary keeps the existing distinct-timestamp latest-wins; block height breaks equal-timestamp ties by anchor recency (a not-yet-mined anchor counts as newest), preserving the pre-pagination behavior; cid is a final deterministic tiebreak for same-block anchors. Height is parsed defensively so a present-but-unreadable height (e.g. a gateway stringifying large ints) cannot masquerade as a pending/newest anchor. Part of #49. * fix(review): bound edges per Arweave page and cover degenerate-cursor/dedup Code-review follow-ups on the #49 recovery change: - Cap edges parsed per GraphQL page at ARWEAVE_PAGE_SIZE so a hostile gateway returning one oversized page cannot defeat the MAX_TX_IDS budget or flood the downstream per-id manifest fetch loop (adversarial review). - Collapse the two identical gateway-failure arms in the discovery loop into a single warn site, and hoist PAGE_SIZE to a named const tied to the query's first:100 (maintainability). - Add tests: per-page edge bound, non-advancing-cursor termination (infinite-loop guard), and cross-page tx-id dedup fetched exactly once (testing). Part of #49. * fix(review): make the degenerate-cursor test assert, and pin the page-size invariant Round-2 review follow-ups: - The non-advancing-cursor test bound its mock to a _-prefixed handle with expect_at_most(3) and never asserted it; mockito defaults assert_on_drop=false, so the test passed even if the guard were removed. Bind it, tighten to expect(2) (the guard breaks on the second page, so discovery makes exactly two POSTs), and assert_async. - Correct the parse_tx_page docstring: the per-page take() bounds the TxRef vec and fetch loop, not the response-body deserialization (an unbounded-body concern shared by every gateway read here). - Hoist the discovery query to a const and add query_first_matches_page_size so the query first:N literal and ARWEAVE_PAGE_SIZE are kept equal by a test rather than a comment. Part of #49.
1 parent 96fcfdb commit 2153b0b

1 file changed

Lines changed: 838 additions & 64 deletions

File tree

0 commit comments

Comments
 (0)