Add uv-managed pytest test foundation#108
Merged
Merged
Conversation
Foundation for CLAUDE_IMPROVEMENTS.md #4 (Testing Infrastructure). Establishes the shared, uv-managed pytest harness that the DAG-validation (Plan A) and setup-script (Plan B) suites build on. - pyproject.toml: pin requires-python to >=3.13,<3.14 (Airflow 3.2.1 does not support 3.14) and add a `test` dependency group (pytest + Airflow + the curated provider set) plus pytest config and markers. - uv.lock: locked resolution for reproducible installs. - tests/conftest.py: airflow-free shared anchors (REPO_ROOT, EXAMPLES_DIR) and hermetic Airflow env vars, so lightweight suites run with only pytest. - tests/test_foundation.py: smoke tests giving the CI job a green baseline. - .github/workflows/whirl-ci.yml: add a fast, Docker-free `unit-tests` job. - CONTRIBUTING.md: document the `uv sync --group test` / `uv run pytest` workflow. Verified locally: `uv sync --group test` resolves and installs Airflow 3.2.1 on Python 3.13, `import airflow` + DagBag work, and `uv run pytest` is green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the DAG-validation half of CLAUDE_IMPROVEMENTS.md #4, on top of the uv test foundation. - tests/test_dag_validation.py: for every examples/*/dag.py, build an Airflow DagBag and assert it parses with no import errors and yields >=1 DAG (DagBag also enforces unique task ids / no cycles). A second test asserts each dag_id stays greppable the way `whirl ci` auto-detects it. - Explicit SKIP_PARSE map for examples needing packages outside the curated test group (dbt-example, dbt-spark-example -> airflow_dbt_python; spark-delta-sharing -> delta_sharing) so skips are visible and a new example with missing deps fails loudly. - Two examples import local packages (custom, custom_plugins); validated after installing them. CI installs them (mkdir /tmp/custom_build for the timetable plugin's egg_base) before running pytest. Local run: 31 passed, 3 skipped on Python 3.13. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the setup-script half of CLAUDE_IMPROVEMENTS.md #4, on top of the uv test foundation. B1 — Fix shellcheck findings in setup scripts so they pass at warning severity: - SC1083: `-w ''%{http_code}''` -> `-w '%{http_code}'` (empty-quote pairs left the braces unquoted) across the add_connection_* scripts. - SC2155: split `local X=$(...)` / `export SPARK_HOME=$(...)` declare+assign. - SC2046: quote the `--key` value containing `$(date ...)` in add_mockdata. - SC2128: `${BASH_SOURCE}` -> `${BASH_SOURCE[0]}`. - SC2044: justified `# shellcheck disable` in the sftp mock-data script (controlled, space-free filenames). B2 — CI: extend the shellcheck job to lint every setup script at `--severity=warning` (the main `whirl` script is still linted with `-x`). Info-level quoting noise (SC2086) is intentionally deferred. B3 — tests/test_setup_scripts.py: airflow-free pytest conventions asserting each setup script has the bash shebang and NN_name.sh naming, and that every whirl.setup.d script enables `set -e` near the top (locking in the Topic 3 invariant). compose.* scripts are sourced under whirl's own `set -e`, so the errexit check is scoped to whirl.setup.d. Verified: `shellcheck --severity=warning` over all 68 setup scripts is clean; all modified scripts pass `bash -n`; the pytest conventions suite (191 checks) passes with a bare pytest (no airflow needed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add setup-script checks: shellcheck-all + pytest conventions (Plan B)
Add DAG parsing & validation test suite (Plan A)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of three PRs implementing CLAUDE_IMPROVEMENTS.md #4 (Testing Infrastructure). This establishes the shared,
uv-managed pytest harness; the DAG-validation and setup-script suites stack on top of it.What's here
pyproject.toml— pinsrequires-python = ">=3.13,<3.14"(Airflow 3.2.1 doesn't support 3.14) and adds a PEP 735testdependency group (pytest + Airflow 3.2.1 + curated providers + pendulum/pandas/ephem), plus[tool.pytest.ini_options]and markers.uv.lock— locked resolution for reproducible installs.tests/conftest.py— airflow-free shared anchors (REPO_ROOT,EXAMPLES_DIR,example_dirs_with_dag()) + hermetic Airflow env vars, so lightweight suites run with only pytest.tests/test_foundation.py— smoke tests for a green baseline.unit-testsjob (uv sync --group test→uv run pytest).uv sync --group test/uv run pytestworkflow.Validation
Local run on Python 3.13:
uv sync --group testresolves & installs Airflow 3.2.1,import airflow+DagBagwork,uv run pytestis green (2 passed).Follow-ups (stacked on this branch)
tests/test_dag_validation.py).tests/test_setup_scripts.py).🤖 Generated with Claude Code