Promotion 2026-09-08 prod (#8291)

.claude/skills/5xx-investigation.md

@@ -0,0 +1,69 @@
+---
+name: 5xx-investigation
+description: Investigate 5xx alarms from CloudWatch API Gateway logs
+user_invocable: true
+---
+
+# 5xx Alarm Investigation
+
+The user will typically start by giving you a CloudWatch Logs query already
+preset for the time frame in question. Each query result represents an incident.
+The goal is to classify the incidents into categories based on similarity, and
+for each category identify if there is a pre-existing GitHub issue, in which
+case a comment recording the incidents needs to be made, or if a new issue needs
+to be created.
+
+These incidents will have triggered CloudWatch alarms. When these alarms trip, a
+notification is sent to a Google Group, starting a conversation there. The user needs to triage these alarms which is
+why they are asking you to do this. Multiple incidents occurring in quick
+succession may have been grouped into a single alarm trip. The user will know
+how many alarms occurred during the timeframe in question. There should be at
+least as many incidents.
+
+## Step 1: Run the query
+
+Run the query. Each result represents an incident.
+
+## Step 2: Investigate each incident
+
+If the query is against the API Gateway logs, you will very likely need to
+retrieve the corresponding application log entries, based on the integration
+request ID. Once you found an error message or a stack trace, move to the next
+step.
+
+## Step 3: Classify
+
+Classify the issue into categories, one per error message or stack trace.
+
+## Step 4: Identify existing issue
+
+For each category, find preexisting issues. Include closed issues in this
+search. Print a table that associates each incident category with the issues you
+think match that category. Print issues as links so that the user can open them
+easily. Work with the user to refine the list. The user needs to explicitly
+approve the creation of a new issue for unmatched categories. Don't make any
+changes to GitHub yet, this step is just about aggreeing on the contents of that
+association table. Once approved, move to the next step.
+
+## Step 5: Report the incidents on the existing issues
+
+Post a short comment to each issue, reporting the recurrence. Keep the comment
+short.
+
+## Step 6: Create new issues for unmatched categories
+
+For the title and the description focus on the symptom, the reproduction and the
+error message / stack trace. Offer to also post your hypothesis as to the root
+cause, and if the user agrees, post that as a special comment. As always, use
+the aggreed upon attribution clause in issue descriptions and comments.
+
+## Step 7: Generate the triage summary
+
+The user will paste the triage summary into the Google Groups conversation for
+each alarm. They will paste the same summary into each conversation for the time
+frame in question. The triage summary should be plain text and have an entry for
+each category. Each entry should consist of lines of the form "name: value". For
+each category, include a title, the dates and times (local timezone) of the
+incidents in that category and links to the issues or issue comments you
+created. The dates and times will allow other users to correlate each alarm with
+the category should they want to.

.claude/skills/pr-review.md

@@ -0,0 +1,123 @@
+---
+name: pr-review
+description: "Review an Azul PR"
+user_invocable: true
+---
+
+# PR review
+
+Use this skill when asked to review a PR, to check whether a PR actually fixes
+its linked issues, or to post review findings to GitHub.
+
+## Ground rules
+
+- Ask for confirmation before proceeding with a review of a PR whose branch is
+  either A) not checked out locally, B) locally checked out but not up-to-date
+  with the PR HEAD or C) has uncommitted changes locally
+
+- Do not post anything to GitHub until explicitly asked. Report the verdict in
+  chat first. A request to review is not a request to post.
+
+- Never submit a review event. Approving, requesting changes or commenting is
+  the user's call, not yours; see step 4.
+
+- Distinguish fact from assumption in every finding. "Confirmed: 6 of 6
+  requests returned 500" and "I suspect this path is unguarded" are different
+  claims and must read differently.
+
+- Cite locations as full `file:line`, never a bare line number.
+
+## Step 1: General evaluation
+
+Perform whatever analysis you see fit. 
+
+## Step 2: Routine checks
+
+- **Checklist audit.** Compare the description's checked items against
+  reality: labels actually present on the PR, `p`/`r`/`a`/`A`/`u`/`R`/`F` tags
+  actually on commit titles, the `app.py` API version bump against whether the
+  change is backwards compatible, and whether a minor bump is defensible. Tags
+  go on commit titles; PRs carry labels. Report mismatches — a checked box
+  asserting state that does not exist is itself a finding.
+
+- **Reindex and mirror labels.** Analyse the actual changes rather than
+  applying these mechanically.
+
+## Step 3: Report the verdict
+
+Report in chat before touching GitHub. Separate blocking from non-blocking,
+most severe first. Every blocking finding states the claim, the evidence, and
+how it was verified. Say plainly what passed, too — lint, type check and test
+status belong in the report even when they are clean, because their being
+clean is part of the story.
+
+## Step 4: Post as a pending review, only when asked
+
+### Pending versus published
+
+Omitting `event` from the review POST creates a **PENDING** review: the
+comments are drafts, visible only to their author, exactly as when composing a
+review in the web UI. The user submits it themselves.
+
+```
+gh api repos/DataBiosphere/azul/pulls/<N>/reviews --method POST --input review.json \
+  -q '"review id: \(.id)  state: \(.state)"'
+```
+
+Confirm the response says `state: PENDING`. Caveats:
+
+- Only one pending review per user per PR.
+
+- The review is attributed to the authenticated account; check
+  `gh api user -q .login` and tell the user, since the review will read as
+  theirs even though each comment carries the Claude Code attribution.
+
+- To discard, `DELETE /pulls/<N>/reviews/<review_id>`.
+
+### Anchoring inline comments
+
+Inline comments only anchor to lines that appear in the PR's diff. The locally
+checked out branch *should* be up-to-date and the working copy *should* be
+clean but just to be sure, compute anchors from GitHub's diff:
+
+```
+gh pr diff <N> | awk '
+/^\+\+\+ /   { path = substr($0, 7); next }
+/^@@/        { match($0, /\+[0-9]+/); n = substr($0, RSTART + 1, RLENGTH - 1) + 0; next }
+/^[-\\]/     { next }
+/^\+/        { printf "%s:%d + %s\n", path, n, substr($0, 2); n++; next }
+/^ /         { n++; next }'
+```
+
+Then verify the content at each chosen anchor before posting, where
+`<headRefOid>` comes from `gh pr view <N> --json headRefOid -q .headRefOid`:
+
+```
+git show <headRefOid>:<path> | sed -n '<start>,<end>p'
+```
+
+Line numbers cited in the comment *text* must also be PR-head numbers, so
+verify those the same way. Further rules:
+
+- Pass `commit_id` explicitly, set to the PR head.
+
+- Multi-line comments use `start_line` plus `line`, with `start_side` and
+  `side` both `RIGHT`.
+
+- Warn the user that a force-push to the PR branch before they submit will
+  mark the comments outdated, and that re-anchoring means deleting and
+  recreating the review. This holds regardless of the state of the local
+  checkout.
+
+### Post conventions
+
+- `*(Posted by Claude Code)*` as the first line of the review body **and** of
+  every inline comment; each comment is a separate post.
+
+- No hard-coded line breaks; let GitHub wrap.
+
+- Build the payload with the Write tool into a JSON file and pass it via
+  `gh api --input`. Do not inline it as a heredoc.
+
+- Fold closely related points into one comment on a shared anchor rather than
+  stacking several comments on adjacent lines.

.claude/skills/promotion-base.md

@@ -0,0 +1,71 @@
+---
+description: "Find the develop commit that a stable branch was last promoted from. Works for `prod`, `anvilprod`, or any branch that follows the promotion merge workflow."
+---
+
+# Promotion base
+
+Use this skill when asked to find the develop commit that was promoted to a
+stable branch, or to determine what version of develop a stable branch is based
+on.
+
+When the user specifies multiple stable branches, apply the procedure to each
+one independently and report results for all of them.
+
+## Background
+
+A promotion merges a promotion branch (e.g., `promotions/2026-08-15-prod`) into
+the target stable branch. The promotion branch is created from a commit on
+`develop`, then the target stable branch is merged into it (to reconcile
+history), and optionally hotfixes are applied on top. The resulting merge commit
+on the stable branch has:
+
+- Parent 1: the previous tip of the stable branch
+- Parent 2: the tip of the promotion branch
+
+The promotion branch itself contains a merge commit titled "Merge branch
+'<target>' into promotions/<date>-<target>". That merge's first parent is the
+develop commit the promotion was based on.
+
+## Procedure
+
+Apply the following steps to each target stable branch (e.g., `prod`,
+`anvilprod`):
+
+1. Get the two parents of the tip commit on the target branch:
+
+   ```
+   git cat-file -p <target-branch>
+   ```
+
+   Parent 1 is the previous stable tip. Parent 2 is the promotion branch tip.
+
+2. Walk parent 2's history to find the merge of the target branch into the
+   promotion branch. Look for a commit message matching "Merge branch
+   '<target>' into promotions/":
+
+   ```
+   git log --oneline <parent2> --grep="Merge branch '<target>' into promotions/"
+   ```
+
+3. Get the parents of that merge commit:
+
+   ```
+   git cat-file -p <merge-commit>
+   ```
+
+   Parent 1 is the develop commit the promotion was based on.
+
+4. Verify the result is on `develop`:
+
+   ```
+   git branch --contains <develop-commit> | grep -w develop
+   ```
+
+## Pitfall: backported commits
+
+Commits that originated on a stable branch and were backported to `develop`
+appear in `git log develop` and pass `git branch --contains`, but they are not
+develop-native commits. Their parents include prior promotion merge commits
+(e.g., a previous "Promotion ... anvilprod" commit as a parent). Always trace
+through the promotion branch merge to find the true develop base rather than
+relying on `git merge-base` or `git log` alone.

.dockerignore

@@ -1,9 +1,9 @@
 *
 !Makefile
 !common.mk
-!requirements*.txt
-!environment
+!pyproject.toml
+!uv.lock
 !bin/keys/awscli-public-key.asc
 !bin/keys/docker-apt-keyring.pgp
 !bin/checksums/gh_checksums.txt
-requirements.all.txt
+!bin/checksums/uv_checksums.txt

.github/ISSUE_TEMPLATE/upgrade.md

@@ -35,9 +35,9 @@ _period: 14 days
   - [ ] GH Action workflow succeeded
   - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator/tags) 
 - [ ] Create Azul PR, linked to this issue, with … 
-    - [ ] … changes to `requirements*.txt` from open Dependabot PRs, one commit per PR
+    - [ ] … changes to `pyproject.toml` and `uv.lock` from open Dependabot PRs, one commit per PR
     - [ ] … upgrade direct Python dependencies, [reference the operator manual](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#upgrade-direct-python-dependencies) for instructions <sub>or not applicable</sub>
-    - [ ] … update to [Python](https://hub.docker.com/_/python/tags) (only patch versions) <sub>or no update available</sub>
+    - [ ] … update to [Python](https://hub.docker.com/_/python/tags) (only patch versions), following the instructions next to the pin in `pyproject.toml` <sub>or no update available</sub>
     - [ ] … update to [Terraform](https://hub.docker.com/r/hashicorp/terraform/tags) (only patch versions) <sub>or no update available</sub>
     - [ ] … update to Terraform provider (only minor and patch versions) …
         - [ ] … [hashicorp/aws](https://registry.terraform.io/providers/hashicorp/aws/latest) <sub>or no update available</sub>
@@ -54,7 +54,8 @@ _period: 14 days
     - [ ] … update to [AL2023 release](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#updating-software-packages-via-release-version-upgrade-in-al2023-instances) <sub>or no update available</sub>
     - [ ] … update to [Swagger UI](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#updating-the-swagger-ui) <sub>or no update available</sub>
     - [ ] … update to [AWS CLI v2](https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst) <sub>or no update available</sub>
-    - [ ] … update to [GitHub CLI](https://github.com/cli/cli/releases) <sub>or no update available</sub>
+    - [ ] … update to [GitHub CLI](https://github.com/cli/cli/releases), having run `make gh_checksums` afterwards <sub>or no update available</sub>
+    - [ ] … update to [uv](https://github.com/astral-sh/uv/releases), following the instructions next to the pin in `pyproject.toml` <sub>or no update available</sub>
 - [ ] Created issues for any deferred updates to …
   - [ ] … the next major or minor Python version <sub>or such an issue already exists</sub>
   - [ ] … the next major Docker version <sub>or such an issue already exists</sub>

.github/PULL_REQUEST_TEMPLATE/anvilprod-hotfix.md

@@ -34,9 +34,9 @@ Linked issue: #0000
 ### Author (before every review)
 
 - [ ] Rebased PR branch on `anvilprod`, squashed fixups from prior reviews
-- [ ] Ran `make requirements_update` <sub>or this PR does not modify `Dockerfile`, `environment`, `requirements*.txt`, `common.mk`, `Makefile` or `environment.boot`</sub>
-- [ ] Added `R` tag to commit title <sub>or this PR does not modify `requirements*.txt`</sub>
-- [ ] This PR is labeled `reqs` <sub>or does not modify `requirements*.txt`</sub>
+- [ ] Ran `make requirements_update` <sub>or this PR does not modify `pyproject.toml`</sub>
+- [ ] Added `R` tag to commit title <sub>or this PR does not modify `uv.lock`</sub>
+- [ ] This PR is labeled `reqs` <sub>or does not modify `uv.lock`</sub>
 - [ ] PR is not a draft
 - [ ] PR is awaiting requested review from system administrator
 - [ ] Status of PR is *Review requested*

.github/PULL_REQUEST_TEMPLATE/backport.md

@@ -2,6 +2,8 @@
 This is the PR template for backport PRs against `develop`.
 -->
 
+Linked issues: #0000
+
 
 ## Checklist
 
@@ -12,7 +14,8 @@ This is the PR template for backport PRs against `develop`.
 - [ ] Status of PR is *In progress*
 - [ ] Target branch is `develop`
 - [ ] Name of PR branch matches `backports/<7-digit SHA1 of most recent backported commit>`
-- [ ] Status of linked issue is *Stable*
+- [ ] PR is linked to the issues it backports
+- [ ] Status of linked issues is *Stable*
 - [ ] PR title contains the 7-digit SHA1 of the backported commits
 - [ ] PR title references the issues relating to the backported commits
 - [ ] PR title references the PRs that introduced the backported commits
@@ -21,9 +24,9 @@ This is the PR template for backport PRs against `develop`.
 ### Author (before every review)
 
 - [ ] PR branch is up to date (if not, merge `develop` into PR branch to integrate upstream changes)
-- [ ] Ran `make requirements_update` <sub>or this PR does not modify `Dockerfile`, `environment`, `requirements*.txt`, `common.mk`, `Makefile` or `environment.boot`</sub>
-- [ ] Added `R` tag to commit title <sub>or this PR does not modify `requirements*.txt`</sub>
-- [ ] This PR is labeled `reqs` <sub>or does not modify `requirements*.txt`</sub>
+- [ ] Ran `make requirements_update` <sub>or this PR does not modify `pyproject.toml`</sub>
+- [ ] Added `R` tag to commit title <sub>or this PR does not modify `uv.lock`</sub>
+- [ ] This PR is labeled `reqs` <sub>or does not modify `uv.lock`</sub>
 - [ ] PR is not a draft
 - [ ] PR is awaiting requested review from system administrator
 - [ ] Status of PR is *Review requested*
@@ -95,7 +98,7 @@ This is the PR template for backport PRs against `develop`.
 - [ ] PR is assigned to only the operator
 - [ ] Deleted PR branch from GitLab `dev`
 - [ ] Deleted PR branch from GitLab `anvildev`
-- [ ] Status of linked issue is *Stable*
+- [ ] Status of linked issues is *Stable*
 
 
 ### Operator

.github/PULL_REQUEST_TEMPLATE/prod-hotfix.md

@@ -34,9 +34,9 @@ Linked issue: #0000
 ### Author (before every review)
 
 - [ ] Rebased PR branch on `prod`, squashed fixups from prior reviews
-- [ ] Ran `make requirements_update` <sub>or this PR does not modify `Dockerfile`, `environment`, `requirements*.txt`, `common.mk`, `Makefile` or `environment.boot`</sub>
-- [ ] Added `R` tag to commit title <sub>or this PR does not modify `requirements*.txt`</sub>
-- [ ] This PR is labeled `reqs` <sub>or does not modify `requirements*.txt`</sub>
+- [ ] Ran `make requirements_update` <sub>or this PR does not modify `pyproject.toml`</sub>
+- [ ] Added `R` tag to commit title <sub>or this PR does not modify `uv.lock`</sub>
+- [ ] This PR is labeled `reqs` <sub>or does not modify `uv.lock`</sub>
 - [ ] PR is not a draft
 - [ ] PR is awaiting requested review from system administrator
 - [ ] Status of PR is *Review requested*

.github/PULL_REQUEST_TEMPLATE/upgrade.md

@@ -36,9 +36,9 @@ Linked issue: #0000
 ### Author (before every review)
 
 - [ ] Rebased PR branch on `develop`, squashed fixups from prior reviews
-- [ ] Ran `make requirements_update` <sub>or this PR does not modify `Dockerfile`, `environment`, `requirements*.txt`, `common.mk`, `Makefile` or `environment.boot`</sub>
-- [ ] Added `R` tag to commit title <sub>or this PR does not modify `requirements*.txt`</sub>
-- [ ] This PR is labeled `reqs` <sub>or does not modify `requirements*.txt`</sub>
+- [ ] Ran `make requirements_update` <sub>or this PR does not modify `pyproject.toml`</sub>
+- [ ] Added `R` tag to commit title <sub>or this PR does not modify `uv.lock`</sub>
+- [ ] This PR is labeled `reqs` <sub>or does not modify `uv.lock`</sub>
 - [ ] Updated the `AL2023_release` variable in [gitlab.tf.json.template.py](../blob/develop/terraform/gitlab/gitlab.tf.json.template.py) to the most recent [AL2023 release](../blob/develop/OPERATOR.rst#updating-software-packages-via-release-version-upgrade-in-al2023-instances) <sub>or no update is available</sub>
 - [ ] `make integration_test` passes in personal deployment <sub>or this PR does not modify functionality that could affect the IT outcome</sub>
 - [ ] PR is not a draft

.github/dependabot.yml

@@ -0,0 +1,11 @@
+version: 2
+
+updates:
+  - package-ecosystem: 'uv'
+    directory: '/'
+    target-branch: 'develop'
+    # Disable routine dependency updates, but keep security updates active
+    open-pull-requests-limit: 0
+    # Ignored if open-pull-requests-limit is 0
+    schedule:
+      interval: 'weekly'