Skip to content
Nightly hardening

Nightly hardening #53

Sign in to view logs

Nightly hardening

Nightly hardening #53

Workflow file for this run

.github/workflows/nightly-hardening.yml at 7ff1bbc
name: Nightly hardening
# Runs nightly (never on push/PR) plus on manual dispatch. This is deliberately
# separate from the `CI` workflow (ci.yml): it exists to catch the *rare*
# teardown races and leaks that a single small stress run in the PR gate is
# unlikely to surface, at a cost (full OS matrix x free-threaded interpreter x
# repeated suite runs, with scaled-up stress iterations) that would be too slow
# to pay on every push/PR. A failure here shows up as a red status for this
# workflow in the Actions tab, and GitHub additionally emails the actor who
# last modified this workflow file when a schedule-triggered run fails — both
# are informative-only signals; this workflow never gates a PR (it has no
# `pull_request`/`push` trigger, and nothing in `ci.yml` depends on it).
on:
schedule:
# 03:17 UTC daily. Off-the-hour and distinct from codeql.yml's weekly
# '32 5 * * 1' slot, so the two scheduled workflows don't contend for
# runner capacity at the same instant.
- cron: '17 3 * * *'
workflow_dispatch:
concurrency:
group: nightly-hardening
cancel-in-progress: true
permissions:
contents: read
jobs:
memory-leaks:
# Process-level RSS is meaningful only when the scenarios run serially in
# one interpreter. Keep this independent from the scaled/repeated stress
# stage below: leak thresholds use their own fixed warmup and iteration set.
name: memory leaks (${{ matrix.os }}, ${{ matrix.python || 'abi3' }})
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python: ['', '3.14t']
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # stable
with:
toolchain: stable
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
enable-cache: true
- name: Run memory leak tests
run: >-
uv run ${{ matrix.python && format('--python {0} ', matrix.python) }}pytest
tests/test_memory_leaks.py -p no:xdist
-o addopts="-ra --strict-markers --strict-config --import-mode=importlib"
stress:
# Same (OS x interpreter) matrix as `ci.yml`'s `test` job (full OS coverage
# + the free-threaded 3.14t build), but every entry here additionally
# scales up the hardening tests' iteration counts and repeats the whole
# suite several times per run, to raise the odds of tripping a rare,
# timing-dependent teardown race that a single fast pass would miss.
name: stress (${{ matrix.os }}, ${{ matrix.python || 'abi3' }})
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm, windows-latest, windows-11-arm, macos-latest]
python: ['', '3.14t']
runs-on: ${{ matrix.os }}
# `PROCESSKIT_STRESS_SCALE` multiplies the iteration/spawn counts in
# tests/test_hardening.py (default 1, i.e. the fast PR-gate values); this
# does not change PR-gate behavior since `ci.yml` never sets this variable.
env:
PROCESSKIT_STRESS_SCALE: '10'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # stable
with:
toolchain: stable
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
enable-cache: true
# Repeat the full suite several times in one job run (rather than only
# scaling iteration counts within a single run) to also catch flaky
# races that depend on process/OS scheduling variance between separate
# interpreter invocations, not just on in-process repetition count.
# `shell: bash` runs the same POSIX loop on every OS in the matrix,
# including both Windows architectures (Actions ships Git Bash there).
- name: Run the suite repeatedly with scaled-up stress iterations
if: matrix.os != 'windows-11-arm' || matrix.python != '3.14t'
shell: bash
run: |
set -e
for i in $(seq 1 5); do
echo "::group::Suite run $i/5"
uv run ${{ matrix.python && format('--python {0} ', matrix.python) }}pytest
echo "::endgroup::"
done
- name: Stress the native Windows ARM64 free-threaded wheel
if: matrix.os == 'windows-11-arm' && matrix.python == '3.14t'
env:
CIBW_ARCHS_WINDOWS: ARM64
CIBW_BUILD: cp314t-win_arm64
CIBW_TEST_COMMAND: >-
python -m pytest {project}/tests
--ignore={project}/tests/test_api_reference.py
--ignore={project}/tests/test_docs_snippets.py &&
python -m pytest {project}/tests
--ignore={project}/tests/test_api_reference.py
--ignore={project}/tests/test_docs_snippets.py &&
python -m pytest {project}/tests
--ignore={project}/tests/test_api_reference.py
--ignore={project}/tests/test_docs_snippets.py &&
python -m pytest {project}/tests
--ignore={project}/tests/test_api_reference.py
--ignore={project}/tests/test_docs_snippets.py &&
python -m pytest {project}/tests
--ignore={project}/tests/test_api_reference.py
--ignore={project}/tests/test_docs_snippets.py
CIBW_TEST_REQUIRES: >-
pytest~=9.1
pytest-timeout~=2.4
pytest-xdist~=3.8
hypothesis~=6.156
griffelib~=2.0
run: uvx 'cibuildwheel>=4,<5' --output-dir wheelhouse
bench:
# `benchmarks/` (pytest-benchmark: overhead vs subprocess/asyncio.subprocess,
# ProcessGroup start/exit, line-streaming throughput, output_all at varying
# concurrency — see benchmarks/README.md). One (ubuntu, default interpreter)
# runner, not the full stress matrix: this is a performance trend to watch
# across nights, not a correctness check that needs every OS/interpreter
# combination, and a single consistent runner keeps night-to-night numbers
# comparable (a matrix would split the trend across differently-noisy
# hosts). Never gates a PR — same `schedule`/`workflow_dispatch`-only
# triggers as `stress` above.
#
# History/regression-alert decision (see benchmarks/README.md for the
# user-facing writeup): each run's `benchmark.json` is appended as a data
# point on the orphan `gh-pages` branch via `github-action-benchmark`
# (dev/bench/data.js), rather than a scratch external service or a
# committed-to-main history file — this is the tool's own designed-for
# storage model and keeps benchmark history entirely out of `main`'s
# commit history. A >=30%-slower run trips an informative-only commit
# comment (never a job failure — `fail-on-alert: false`, consistent with
# this workflow's "never a traceback" philosophy above); it does not gate
# `ci.yml`/any PR.
name: bench
# Job-scoped write access (least privilege): only this job needs to push
# to `gh-pages`, so the write grant is local to it rather than widening
# the workflow-level `permissions:` (which stays `contents: read` for
# `stress`/`mutmut` above). Job-level `permissions:` fully replaces the
# workflow-level block for this job (not additive), so this is also the
# complete permission set `bench` runs with.
permissions:
contents: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # stable
with:
toolchain: stable
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
enable-cache: true
- run: uv sync --group bench
# Directory discovery intentionally includes every benchmark module,
# including test_pty.py, test_lifecycle_events.py,
# test_aoutput_as_completed.py, and test_supervisor.py. Keeping one
# directory-wide invocation makes the JSON passed to the existing history
# and regression-alert action cover the complete benchmark set.
# `-p no:xdist -o addopts=""` drops the repo's default xdist options:
# pytest-benchmark needs to run un-distributed (single worker) to
# produce real timings — under xdist it silently skips measuring
# instead (see benchmarks/README.md).
- name: Run benchmarks
run: >
uv run pytest benchmarks/ --benchmark-only -p no:xdist -o addopts=""
--benchmark-json=benchmark.json
- name: Publish results to the job summary
shell: bash
run: |
uv run python - <<'EOF' >> "$GITHUB_STEP_SUMMARY"
import json
with open("benchmark.json") as f:
data = json.load(f)
print("# Benchmark results\n")
print("| Benchmark | Mean | Min | Max | Rounds |")
print("| --- | --- | --- | --- | --- |")
for bench in data["benchmarks"]:
stats = bench["stats"]
def fmt(seconds: float) -> str:
return f"{seconds * 1000:.3f} ms"
print(
f"| `{bench['fullname'].removeprefix('benchmarks/')}` "
f"| {fmt(stats['mean'])} | {fmt(stats['min'])} | {fmt(stats['max'])} "
f"| {stats['rounds']} |"
)
EOF
# `github-action-benchmark` (below) reads/writes `dev/bench/data.js` on
# this orphan branch via a plain `git fetch`/`git switch` — it does not
# create the branch itself if missing, only pushes updates to an
# already-existing one. Self-heal here instead of requiring a human to
# run `git checkout --orphan gh-pages && git push` out-of-band once
# before the first nightly run.
- name: Ensure the gh-pages branch exists
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -e
if git ls-remote --exit-code --heads origin gh-pages > /dev/null; then
echo "gh-pages already exists, nothing to do"
exit 0
fi
echo "gh-pages missing, creating an empty orphan branch"
git checkout --orphan gh-pages
git rm -rf . > /dev/null
git -c user.name='github-actions[bot]' \
-c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
commit --allow-empty -m "Initialize gh-pages branch for benchmark history"
git push "https://x-access-token:${GITHUB_TOKEN}@github.com/${{ github.repository }}.git" gh-pages:gh-pages
git checkout --detach "$GITHUB_SHA"
# Appends this run's result as a new data point on `gh-pages`
# (`dev/bench/data.js`) and compares it against the previous point,
# leaving an informative-only commit comment on regression — see the
# `bench` job's header comment above and benchmarks/README.md for the
# storage/threshold decision. Pinned per K-002 (full commit SHA +
# version comment, not a floating tag).
- name: Track benchmark history and alert on regression
uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1
with:
name: processkit benchmarks
tool: 'pytest'
output-file-path: benchmark.json
gh-pages-branch: gh-pages
benchmark-data-dir-path: dev/bench
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: true
# 130%: this run's mean is >=30% slower than the previous run's.
# Deliberately looser than the tool's 200% default but well above
# ordinary shared-runner noise, since a single-night jump this
# large is worth a look even though it never blocks anything --
# see benchmarks/README.md for what to do when it fires and how
# smaller, slower creep across many nights is instead caught by
# eyeballing the trend chart, not this per-run threshold.
alert-threshold: '130%'
comment-on-alert: true
fail-on-alert: false
summary-always: true
mutmut:
# Mutation testing for the pure-Python binding layer -- see `[tool.mutmut]` in
# pyproject.toml for the exact scope (deadlines/cancellation in `_aio.py`,
# cassette-mode/naming in `pytest_plugin.py`, exit-code mapping in
# `__main__.py`, the release helper scripts) and the reasoning for keeping it
# narrow (the compiled `_processkit` extension is native code coverage.py-style
# tools can't instrument; the sibling `processkit` Rust crate already runs
# cargo-mutants over that logic in its own repository). Line-coverage (the
# `test` job's `--cov`) only proves
# code ran, not that a test would actually notice it behaving differently --
# mutmut proves the latter for this subset. A single ubuntu runner, like
# `bench` above: mutmut's mutant-execution model is `os.fork()`-based (POSIX
# only, no native Windows support), so there is no OS matrix here. Never gates
# a PR -- same `schedule`/`workflow_dispatch`-only triggers as the jobs above;
# a surviving mutant is expected, ordinary output (see the summary step below),
# not a job failure -- `mutmut run` itself only exits non-zero for a genuine
# setup/environment problem (e.g. the baseline test run failing outright).
name: mutmut
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # stable
with:
toolchain: stable
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
enable-cache: true
- run: uv sync
- name: Run mutation testing
run: uv run mutmut run
# `!cancelled()` (rather than the GitHub Actions default, implicit
# `success()`) on this step and the two below: if `mutmut run` itself
# fails (a genuine setup/environment problem, not an ordinary surviving
# mutant -- see the comment above), these steps still run instead of
# being silently skipped, so the job summary/artifact carry whatever
# diagnostic output `mutmut results`/`export-cicd-stats` can produce
# instead of the failure going dark. Only an explicit workflow
# cancellation suppresses them.
- name: Publish surviving mutants to the job summary
if: ${{ !cancelled() }}
shell: bash
run: |
{
echo '## Mutation testing (mutmut)'
# Literal Markdown backticks inside single quotes, not command substitution.
# shellcheck disable=SC2016
echo 'Scope: `src/processkit/_aio.py`, `pytest_plugin.py`, `__main__.py`, `scripts/release/*.py` -- see `[tool.mutmut]` in pyproject.toml. Killed mutants are omitted; only surviving/no-tests/timeout/etc. are listed below (empty output means every mutant was killed).'
echo '```'
uv run mutmut results
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Export mutation stats
if: ${{ !cancelled() }}
run: uv run mutmut export-cicd-stats
- if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mutmut-cicd-stats
path: mutants/mutmut-cicd-stats.json
if-no-files-found: error
cargo-mutants:
# Mutation testing for the Rust binding layer. The crate's default Cargo
# build intentionally leaves `extension-module` disabled, so unit tests can
# link the cdylib as documented in Cargo.toml. Scope and per-mutant timeouts
# are fixed in .cargo/mutants.toml; the matrix only partitions that scope.
# This scheduled/manual signal never gates pull requests. Surviving and
# timed-out mutants are reported, but do not hide genuine setup failures.
name: cargo-mutants (shard ${{ matrix.shard }}/32)
runs-on: ubuntu-latest
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # stable
with:
toolchain: stable
- name: Install cargo-mutants
uses: taiki-e/install-action@7f4eb899022d8fe70b20c4f3de697aa85c309026 # v2
with:
tool: cargo-mutants
- name: Run mutation testing shard
id: mutants
shell: bash
run: |
set +e
cargo mutants --shard ${{ matrix.shard }}/32
status=$?
set -e
if [ "$status" -ne 0 ] && [ "$status" -ne 2 ] && [ "$status" -ne 3 ]; then
echo "::error::cargo-mutants failed with exit code ${status}"
exit "$status"
fi
if [ "$status" -eq 2 ] && [ ! -s mutants.out/missed.txt ] && [ ! -s mutants.out/timeout.txt ]; then
echo "::error::cargo-mutants returned an outcome code without outcome files"
exit 1
fi
if [ "$status" -eq 3 ] && [ ! -s mutants.out/timeout.txt ]; then
echo "::error::cargo-mutants returned a timeout code without timeout.txt"
exit 1
fi
echo "cargo-mutants completed with outcome code ${status}"
- name: Publish surviving mutants to the job summary
if: ${{ !cancelled() }}
shell: bash
run: |
{
echo '## Mutation testing (cargo-mutants)'
echo "Scope: \`src/*.rs\` without the \`extension-module\` feature."
for outcome in missed timeout; do
echo
echo "### ${outcome}"
echo '```'
if [ -s "mutants.out/${outcome}.txt" ]; then
cat "mutants.out/${outcome}.txt"
else
echo '(none)'
fi
echo '```'
done
} >> "$GITHUB_STEP_SUMMARY"
- name: Export mutation statistics
if: ${{ !cancelled() }}
shell: bash
run: |
mkdir -p mutation-stats
cp mutants.out/*.txt mutation-stats/ 2>/dev/null || true
cp mutants.out/outcomes.json mutation-stats/ 2>/dev/null || true
- if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: cargo-mutants-stats-${{ matrix.shard }}-of-32
path: mutation-stats/
if-no-files-found: error