feat(pagination): plumb max_pages through public *_all() methods#135
Conversation
Code Review — PR #135:
|
| Bug | Lazy validation in yield from generators — _validate_max_pages doesn't fire at call time for ~6 methods |
| Design | None → 1000 silently truncates callers expecting unbounded iteration |
| Housekeeping | Missing async FQNs in _MAX_PAGES_FQNS; inline duplicate test helper |
| ✅ | Mechanical plumbing is correct, consistent, and well-tested for the non-generator paths |
| ✅ | ExclusionKind = "client_only" is the right way to handle this in the drift framework |
| ✅ | _validate_max_pages as a standalone helper at the public boundary is clean |
Block on the silent-truncation question — that's a user-facing behavior decision. The lazy-validation bug should be fixed before merge.
|
Both bot findings addressed; plus the related async generator issue that surfaced while fixing the sync one. Fixed
Regression coverage added
Skipped (justified)
Verify: 1577 passed (+5 over agent baseline), ruff clean, mypy |
…thods Per bot review on PR #135: ### Behavior change: max_pages=None is now truly unbounded Dropped the silent 1000-page cap. None iterates until the server returns no cursor; the existing cursor-repeat guard remains the safety net against infinite loops (cap was never part of the public contract). Bot's recommended path: most users expect None = unbounded and a silent truncation post-release would be a harder break to fix. ### Fix: 7 wrapper methods had lazy validation Methods using `yield from self._list_all(...)` (5 sync) or `async for item in ... yield item` (2 async) were silently generators — their body, including _validate_max_pages, did not execute until the caller advanced the iterator. milestones.list_all(max_pages=0) returned a generator without raising; the ValueError only fired on the first next(). Fixed by converting to `return self._list_all(...)`: - communications.{Sync,Async}.list_all_rfqs - incentive_programs.list_all - milestones.list_all - structured_targets.list_all - subaccounts.{Sync,Async}.list_all_transfers Async ones additionally need `def` (not `async def`) so they're plain functions returning the AsyncIterator, not coroutines. ### Async drift FQNs Added all 19 async resource FQNs to _MAX_PAGES_FQNS in tests/_contract_support.py so the drift framework recognizes the kwarg if coverage is ever extended to async classes. ### Regression tests - TestMaxPagesEagerValidation × 4 (sync + async, both styles) — asserts ValueError fires at call time, not on first __next__. - TestMaxPagesNoneIsUnbounded × 1 — server returns 1100 pages, suite iterates all 1100 (proves the 1000 cap is gone). Verify: 1577 passed (+5). ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code Review — PR #135:
|
|
Second-pass items addressed in latest commit:
Skipped (justified):
|
Per second-pass bot review on PR #135: - CHANGELOG.md [Unreleased]: added Added entry for max_pages kwarg + RateLimit export, and Changed entry calling out that *_all() is now unbounded by default (was silently capped at 1000 pages). - _base.py async _list_all docstring: restored the "Raises KalshiError on repeated cursor; see sync docstring." note that got trimmed in the previous refactor. - TestMaxPagesNoneIsUnbounded: 1100 → 1010. Still exceeds the old 1000-cap by enough to prove unboundedness; 90% fewer mock requests. Skipped: - nonlocal-vs-dict counter style nit — bot called this take-or-leave; the existing pattern is consistent across the test file. uv run ruff check . clean. uv run pytest tests/test_base_helpers.py::TestMaxPagesNoneIsUnbounded passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code Review — PR #135:
|
Per third-pass bot review on PR #135: - _validate_max_pages docstring: "None (use default cap)" was stale after dropping the 1000-page default. Now reads "None (unbounded — iterates until the server returns no cursor)". - TestMaxPagesNoneIsUnbounded: added test_async_iterates_past_1000_pages as the async counterpart to test_sync_iterates_past_1000_pages. Prevents the cap regression from silently re-appearing in only one of the two code paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Third-pass items addressed in latest commit + PR-body rewrite:
Skipped (non-blocking):
|
Code Review — PR #135:
|
| Method | Eager test? |
|---|---|
MilestonesResource.list_all |
✅ |
SubaccountsResource.list_all_transfers |
✅ |
AsyncSubaccountsResource.list_all_transfers |
✅ |
AsyncCommunicationsResource.list_all_rfqs |
✅ |
IncentiveProgramsResource.list_all |
❌ |
StructuredTargetsResource.list_all |
❌ |
CommunicationsResource.list_all_rfqs (sync) |
❌ |
Since all three use the same return self._list_all(...) pattern, the regression is structurally impossible, but adding the three missing cases would make the test class exhaustive and avoid any doubt in future audits. Low priority.
2. _validate_max_pages docstring violates project conventions.
CLAUDE.md says: "Never write multi-paragraph docstrings or multi-line comment blocks — one short line max." The helper has a 5-line docstring. Suggested trim:
def _validate_max_pages(max_pages: int | None) -> None:
# Zero or negative would produce an empty iterator silently deep in _list_all.
if max_pages is not None and max_pages <= 0:
raise ValueError(f"max_pages must be positive or None, got {max_pages}")3. Behavior change — silent truncation removed.
The removal of the 1000-page cap is technically a breaking change for any caller that relied on it as a safety budget (even inadvertently). The CHANGELOG calls it out clearly under ### Changed, which is the right place. Consider whether a [Unreleased] → Breaking section or semver bump note is needed if this SDK follows strict semver — removing a guarantee (even an undocumented one) typically warrants at least a minor version bump signal.
4. _fresh_cursor_side_effect naming.
The function returns a callable (closure) to use as a respx side_effect, not a side-effect directly. The name implies it returns the effect itself. Consider _make_fresh_cursor_responder or just inline the counter into each test class. Minor.
Performance
The TestMaxPagesNoneIsUnbounded tests make 1010 iterations. Since these use respx.mock (in-process), not real HTTP, this is fine. The inline comment explains the choice — appreciated.
Test Coverage
The new test suite is solid:
TestValidateMaxPages— all four boundary cases (None, positive, zero, negative)TestMaxPagesEagerValidation— the tricky generator-deferral regressionTestMaxPagesNoneIsUnbounded— proves the hard cap is gone in both pathsTestSyncListAllMaxPagesCap/TestAsyncListAllMaxPagesCap— verifies the counttest_max_pages_caps_fetch/test_max_pages_zero_rejected— end-to-end signal on the public surface
The empty-cursor regression guard (test_empty_cursor_stops_after_one_request) is a nice bonus.
Summary
| Category | Finding |
|---|---|
| Correctness | ✅ Logic is sound; eager-validation fix is right |
| Tests | ✅ Good coverage; 3 eager-validation cases not yet exercised (low risk) |
| Conventions | _validate_max_pages |
| Breaking change | ℹ️ Documented, but verify semver intent |
| Code quality | ✅ Clean, mechanical, follows existing patterns |
Two action items worth addressing before merge:
- Trim
_validate_max_pagesdocstring to one line. - Optionally add the 3 missing eager-validation test cases for completeness.
Everything else looks ready.
Plumb max_pages: int | None = None through all 19 public *_all() paginator methods (sync + async, 38 signatures) so callers can cap fetch depth from outside without subclassing the resource. Previously the cap existed only on the internal _list_all and silently defaulted to 1000 pages. - _list_all: accept max_pages=None (default 1000), positive int caps page fetches at that count. - _validate_max_pages: rejects max_pages <= 0 with ValueError at the public boundary; None and positive ints pass through. - Add "client_only" ExclusionKind for kwargs that have no wire counterpart (max_pages is purely a client-side knob). EXCLUSIONS registers max_pages for every *_all FQN so TestRequestParamDrift doesn't flag the new kwarg as REMOVE drift. - Tests cover the bare numeric cap (cap=1 / cap=3 with a fresh-cursor side effect so cursor-loop detection can't trip), empty-cursor termination (F-Q-17 regression), and ValueError on max_pages=0 at the public list_all boundary (markets). Closes #98 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…thods Per bot review on PR #135: ### Behavior change: max_pages=None is now truly unbounded Dropped the silent 1000-page cap. None iterates until the server returns no cursor; the existing cursor-repeat guard remains the safety net against infinite loops (cap was never part of the public contract). Bot's recommended path: most users expect None = unbounded and a silent truncation post-release would be a harder break to fix. ### Fix: 7 wrapper methods had lazy validation Methods using `yield from self._list_all(...)` (5 sync) or `async for item in ... yield item` (2 async) were silently generators — their body, including _validate_max_pages, did not execute until the caller advanced the iterator. milestones.list_all(max_pages=0) returned a generator without raising; the ValueError only fired on the first next(). Fixed by converting to `return self._list_all(...)`: - communications.{Sync,Async}.list_all_rfqs - incentive_programs.list_all - milestones.list_all - structured_targets.list_all - subaccounts.{Sync,Async}.list_all_transfers Async ones additionally need `def` (not `async def`) so they're plain functions returning the AsyncIterator, not coroutines. ### Async drift FQNs Added all 19 async resource FQNs to _MAX_PAGES_FQNS in tests/_contract_support.py so the drift framework recognizes the kwarg if coverage is ever extended to async classes. ### Regression tests - TestMaxPagesEagerValidation × 4 (sync + async, both styles) — asserts ValueError fires at call time, not on first __next__. - TestMaxPagesNoneIsUnbounded × 1 — server returns 1100 pages, suite iterates all 1100 (proves the 1000 cap is gone). Verify: 1577 passed (+5). ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per second-pass bot review on PR #135: - CHANGELOG.md [Unreleased]: added Added entry for max_pages kwarg + RateLimit export, and Changed entry calling out that *_all() is now unbounded by default (was silently capped at 1000 pages). - _base.py async _list_all docstring: restored the "Raises KalshiError on repeated cursor; see sync docstring." note that got trimmed in the previous refactor. - TestMaxPagesNoneIsUnbounded: 1100 → 1010. Still exceeds the old 1000-cap by enough to prove unboundedness; 90% fewer mock requests. Skipped: - nonlocal-vs-dict counter style nit — bot called this take-or-leave; the existing pattern is consistent across the test file. uv run ruff check . clean. uv run pytest tests/test_base_helpers.py::TestMaxPagesNoneIsUnbounded passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per third-pass bot review on PR #135: - _validate_max_pages docstring: "None (use default cap)" was stale after dropping the 1000-page default. Now reads "None (unbounded — iterates until the server returns no cursor)". - TestMaxPagesNoneIsUnbounded: added test_async_iterates_past_1000_pages as the async counterpart to test_sync_iterates_past_1000_pages. Prevents the cap regression from silently re-appearing in only one of the two code paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0666938 to
dc55905
Compare
Code Review — PR #135:
|
Summary
Wave 2 #98: exposes the previously-internal
_list_allpagination cap as amax_pages: int | None = Nonekwarg on every public*_all()resource method (sync + async), and removes the silent 1000-page hard cap soNoneis truly unbounded.Scope
kalshi/resources/_base.py—_list_allacceptsmax_pages: int | None = None(sync + async). The cap is now only applied when the caller passes an int;Noneiterates until the server returns no cursor. New_validate_max_pageshelper raisesValueErrorat the public boundary on<= 0.yield from self._list_all(...)(5 sync) orasync for item in ...: yield item(2 async) were silently generators — their body didn't execute until firstnext(), so_validate_max_pagesran lazily. Converted toreturn self._list_all(...). The 2 async wrappers additionally switched fromasync def→defso they return theAsyncIteratorimmediately (otherwise the coroutine wrapper itself defers the body).tests/_contract_support.py— newExclusionKind = "client_only"for SDK-only kwargs (vs.paginator_handledwhich describes spec params the SDK hides). All 38 FQNs (sync + async) registered in_MAX_PAGES_FQNS.tests/test_contracts.py— addedclient_onlytosig_mismatch_kinds.Behavior change (was the open question — now resolved)
Before:
_list_allhad an invisible 1000-page hard cap. Users iterating beyond ~100k items were silently truncated.After:
max_pages=Noneiterates until the server returns no cursor. The existing cursor-repeat guard (KalshiErroron a repeating cursor value) provides the runaway protection — that's what stops infinite loops if a server bug emits stuck cursors. Users wanting a cap passmax_pages=Nexplicitly.CHANGELOG
[Unreleased] → Changedcalls this out. The cap was never part of the public contract; the change matches user expectation forNoneand is the right long-term direction (silent truncation post-release is a worse compat problem to unwind later).Cap-of-0
Rejected at the public boundary with
ValueError("max_pages must be positive or None").Test deltas
TestValidateMaxPages— None / positive / zero / negative pathsTestMaxPagesEagerValidation(×4) — sync milestones, sync subaccounts, async subaccounts (AsyncIterator-returningdef), async communications. Each assertsValueErrorfires at call time, not on first__next__.TestMaxPagesNoneIsUnbounded(×2) — sync + async. Server returns 1010 pages; suite iterates all 1010, proving the 1000 cap is gone in both code paths.test_markets.py::test_max_pages_caps_fetchandtest_max_pages_zero_rejectedas end-to-end signal on the public surface.Total: +17 tests over baseline.
Closes #98
Test plan
uv run ruff check .cleanuv run mypy kalshi/clean (strict; 75 files)