Per-discrepancy test infrastructure#2512
Open
jucor wants to merge 1 commit into
Open
Conversation
This was referenced Mar 30, 2026
jucor
changed the title
jucor
force-pushed
the
spr/edge/bdc830db
branch
3 times, most recently
from
7a35d30 to
d06ba7f
Compare
ballPointPenguin
approved these changes
Apr 26, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds per-discrepancy (Python vs Clojure) test scaffolding and supporting utilities to enable TDD-driven parity fixes, including dataset+blob variant parametrization and cold-start blob generation improvements.
Changes:
- Introduces
test_discrepancy_fixes.pywith per-discrepancy, dataset+blob-parametrized tests (mostly xfail-targeted for staged parity work). - Adds blob-variant discovery (
incrementalvscold_start) and updates legacy regression/comparison tests to run against specific blob variants. - Improves cold-start blob generation script coordination (parallel runs, container cleanup) and adjusts pytest execution defaults to better leverage new caching.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| delphi/tests/test_legacy_repness_comparison.py | Update legacy repness comparison tests to use composite dataset+blob IDs and load the matching Clojure blob. |
| delphi/tests/test_legacy_clojure_regression.py | Parametrize legacy Clojure regression tests over dataset+blob variants and reuse cached Conversations. |
| delphi/tests/test_discrepancy_fixes.py | New per-discrepancy parity test suite with dataset+blob parametrization and helper utilities. |
| delphi/tests/conftest.py | Adds session-scoped Conversation cache, blob-aware dataset parametrization, and dataset-grouped test reordering. |
| delphi/scripts/generate_cold_start_clojure.py | Adds coordination for parallel cold-start runs (marker file + pause/unpause logic), plus improved DB-copy mechanics and container cleanup. |
| delphi/scripts/clojure_comparer.py | Switch votes loader to non-test utility and update messaging. |
| delphi/pyproject.toml | Default pytest opts to sequential execution to exploit the shared Conversation cache. |
| delphi/polismath/regression/datasets.py | Adds blob_type selection to get_dataset_files() and introduces get_blob_variants(). |
| delphi/polismath/regression/init.py | Exports get_blob_variants. |
| delphi/docs/PLAN_DISCREPANCY_FIXES.md | Documents PR naming convention. |
| delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md | Adds the parity-fix work journal. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+90
to
+94
| subprocess.run(['docker', 'pause', container], capture_output=True) | ||
|
|
||
| # Create marker so the last coldstart run knows to unpause | ||
| PAUSE_MARKER.touch() | ||
| click.echo(f" ✓ Paused {len(workers)} container(s)") |
| 'comments': comments, | ||
| } | ||
|
|
||
| return deepcopy(_SESSION_CONV_CACHE[dataset_name]) |
Comment on lines
+31
to
+35
| import numpy as np | ||
| import pytest | ||
| import pytest_check as check | ||
|
|
||
| from polismath.conversation.conversation import Conversation |
## Summary Per-discrepancy test infrastructure for TDD fixing of Python-Clojure differences. ### Changes - Add per-discrepancy test markers and parametrized test infrastructure - Cold-start recorder: coordinate parallel runs with marker file, auto-pause math workers - Update journal with xpassed test breakdown across all datasets - Address Copilot review: remove unused import, fix script issues - Add naming convention documentation ## Test plan - [x] 223 passed, 4 skipped, 22 xfailed, 7 xpassed, 0 failures 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Squashed commits - Address Copilot review feedback and fix pre-existing test failures - Add per-discrepancy test infrastructure and fix journal - Fix cold-start recorder: auto-pause math workers and clean up containers - Cold-start recorder: coordinate parallel runs with marker file - Address Copilot review: remove unused import, fix script issues - Update journal: xpassed test breakdown with all 7 datasets - Add naming convention - Test both incremental and cold-start Clojure blobs - Add repness blob comparison tests and fix tid type mismatches - Address Copilot review on PR #2420: fix stale references and clean up imports commit-id:bdc830db
jucor
changed the title
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per-discrepancy test infrastructure for TDD fixing of Python-Clojure differences.
Changes
Test plan
🤖 Generated with Claude Code
Squashed commits
commit-id:bdc830db
Stack: