Skip to content

fix(deploy): resolve the reserved address by the VM using it - #83

Merged
andres-pcg merged 3 commits into
mainfrom
fix/reserved-address-lookup
Jul 28, 2026
Merged

fix(deploy): resolve the reserved address by the VM using it#83
andres-pcg merged 3 commits into
mainfrom
fix/reserved-address-lookup

Conversation

@andres-pcg

@andres-pcg andres-pcg commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

The first full-fleet roll aborted in preflight — fail-closed, no VM touched:

ERROR: (gcloud.compute.addresses.describe) The resource
  'projects/.../regions/.../addresses/...' was not found
error: ... has no reserved regional address in ...

addr() resolved the reserved address with addresses describe <VM name>, which assumes every address is named after the VM it serves. That holds for most of the fleet but not all of it, so the roll aborted on the first VM where the names differ. The address itself is correct and in use — only the lookup assumption was wrong.

All four callers want the address value, never its name, so the lookup now identifies the address by the VM using it. It lists the addresses once and matches in the script: a server-side --filter over users would be a filter on a repeated field, the same shape that made the firewall preflight fail with Invalid list filter expression on the runner, so the script no longer passes --filter to gcloud anywhere. A name match remains as a fallback for an address reserved before its VM exists, which is the --create path, and a second address for the same VM fails closed rather than being resolved by picking one.

Why it stayed hidden: --only narrows the preflight to the selected VM, so every earlier single-node roll audited a VM whose names happen to match.

Verified against the live fleet and with fixtures:

  • all six nodes resolve to the IP their ns<N>.zfnd.org record points at
  • an unknown VM and a wrong-region lookup both return empty instead of erroring
  • an address reserved but not yet in use still resolves by name (the --create path)
  • two addresses on one VM fails closed
  • --audit clears every infra-scope check on all six, and a full --dry-run --roll plans ns2→ns6 with ns1 last

The preflight looked the address up by the VM's name, which assumes every
address is named after the VM it serves. That does not hold across the fleet, so
a full roll aborted on the first VM where the names differ. Every caller wants
the address value, so identify it by its user instead, keeping a name lookup for
an address reserved before its VM exists.

Latent until now: --only narrows the preflight to the selected VM, so earlier
single-node rolls never audited the one that breaks the assumption.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the GCP deployment/audit script (deploy/gcp/seeders.sh) to resolve a VM’s reserved static IP by looking up the address resource via the VM that is using it, instead of assuming the address resource name matches the VM name. This prevents full-fleet roll preflights from aborting when a VM and its reserved address have different names.

Changes:

  • Update addr() to first find the reserved IP via gcloud compute addresses list filtered by users:(<vm>) in the target region.
  • Keep the existing addresses describe <name> lookup as a fallback for the --create path (reserved address created before the VM exists).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread deploy/gcp/seeders.sh Outdated
Comment on lines +115 to +116
ip="$(get_cmd gcloud compute addresses list --project="${PROJECT}" \
--filter="region:(${region}) AND users:(${name})" --format='value(address)' | head -n1)"
A server-side --filter over a repeated field is rejected as an invalid list
filter expression by some API versions, which is what broke the firewall
preflight. List the addresses once and match in the script instead, and treat
more than one address per VM as a misconfiguration rather than picking one.
@andres-pcg

Copy link
Copy Markdown
Contributor Author

Both points addressed in the latest commit, and the ambiguity one led to dropping the server-side filter entirely.

On head -n1: agreed — it masked exactly the misconfiguration it should surface. The lookup now counts the addresses a VM uses and fails closed when there is more than one, rather than picking an arbitrary match.

On the filter itself: the first version matched with --filter="... users:(...)", and users is a repeated field — the same shape that made the firewall preflight fail with Invalid list filter expression on the runner. Locally it worked, but the runner's gcloud is not necessarily the same build, and that failure mode has already cost us one aborted roll. The lookup now lists the addresses once and matches in the script, mirroring firewall_ok. The script no longer passes --filter to gcloud anywhere.

Verified against the live fleet and with fixtures:

  • all six nodes resolve to the IP their ns<N>.zfnd.org record points at
  • an unknown VM and a wrong-region lookup both return empty instead of erroring
  • an address reserved but not yet in use still resolves by name (the --create path)
  • two addresses on one VM fails closed with has 2 reserved addresses ... expected one
  • --audit clears every infra-scope check on all six, and a full --dry-run --roll plans ns2→ns6 with ns1 last

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

deploy/gcp/seeders.sh:116

  • The PR description says the new lookup uses a server-side gcloud ... addresses list --filter="region:(...) AND users:(...)", but this implementation explicitly avoids --filter and does client-side matching instead (see the note about API versions rejecting list filters over users). Please update the PR description (or implement the described server-side filter with a safe fallback) so the documented behavior matches what actually ships.
  # A server-side --filter over the repeated `users` field is rejected as an
  # invalid list filter expression by some API versions, so match client-side.
  while IFS=';' read -r a_name a_ip a_region a_users; do

@andres-pcg

Copy link
Copy Markdown
Contributor Author

Good catch — the description was stale and contradicted the code.

It described the first version of this branch, which did use --filter="region:(...) AND users:(...)". That was dropped precisely because users is a repeated field: the same shape rejected as Invalid list filter expression when the firewall preflight ran on the runner. The lookup lists once and matches client-side, and the script now passes no --filter to gcloud at all.

Description updated to match what ships.

@andres-pcg
andres-pcg merged commit 1e39dde into main Jul 28, 2026
8 checks passed
@andres-pcg
andres-pcg deleted the fix/reserved-address-lookup branch July 28, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants