Skip to content

feat(storage): give a convergence the budget its caller can afford to wait - #1419

Draft
aparajon wants to merge 24 commits into
armand/storage-schema-guidefrom
armand/storage-apply-own-budget
Draft

aparajon wants to merge 24 commits into
armand/storage-schema-guidefrom
armand/storage-apply-own-budget

Conversation

@aparajon

@aparajon aparajon commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

SchemaBot converges its own storage schema on every boot, under an advisory lock that keeps two instances from converging at once. That convergence is bounded by a five-minute budget, and the budget is short for an availability reason rather than a DDL one: a pod converging is a pod not yet serving, and while it holds the lock no other instance can start either.

storage apply lets an operator run that same bootstrap deliberately, from a terminal, ahead of a roll. Because it is the same code, it inherited the same five minutes — which took the work out of the roll but not out of the ceiling. The case the command exists for, an index over a table with a long history, still timed out from the terminal and still had to be built by hand.

The budget now follows the path that asked rather than the code that runs. It is an option threaded through both bootstrappers' contexts and both advisory-lock waits, defaulting to the boot budget so a caller that names none keeps a boot's guarantee, and seeded to an hour in front of the operator path's options so a caller's own budget still wins. On PostgreSQL the per-statement DDL budget derives from the effective budget rather than once from the boot constant, so a raised ceiling is not silently re-capped per statement. --timeout lowers it for a run that should fail fast.

An operator runs storage apply ahead of a roll. The only outstanding DDL is an index over a table with eight years of rows, and the build takes 22 minutes.

Before                                       After

┌──────────────────────────────┐             ┌──────────────────────────────┐
│ storage apply                │             │ storage apply                │
│ budget: 5m — a boot's        │             │ budget: 1h — an operator's   │
└──────────────┬───────────────┘             └──────────────┬───────────────┘
               │ index build, 22m                           │ index build, 22m
               ▼                                            ▼
┌──────────────────────────────┐             ┌──────────────────────────────┐
│ 5m: context deadline         │             │ 22m: the DDL completes       │
│ the copy is abandoned        │  ✗          │ the lock is released         │
└──────────────┬───────────────┘             └──────────────┬───────────────┘
               │                                            │
               ▼                                            ▼
┌──────────────────────────────┐             ┌──────────────────────────────┐
│ storage still unconverged    │             │ storage is converged         │
│ the index is built by hand   │             │ the roll finds nothing to do │
└──────────────────────────────┘             └──────────────────────────────┘

The maximum is held at the default. A convergence cannot yet be stopped once it starts — the context is built from Background so a caller hanging up cannot abandon a table copy half-done, which means the budget is the only thing that ends one early. An hour of held lock is already the outer edge of what should be untakeable-back. Raising the cap is the payoff of the follow-up that adds storage cancel, not something to do before it.

Out of range is refused rather than clamped, in both directions, so a command never reports a budget it did not get. That includes a duration the wire's whole seconds cannot carry: --timeout 500ms would truncate to the zero that means "no preference" and come back as the hour.

A budget above the maximum
$ schemabot storage apply --timeout 2h
Error: --timeout: a convergence budget of 2h0m0s exceeds the maximum of 1h0m0s; a convergence holds the storage bootstrap lock for its whole budget and cannot yet be stopped, so pods booting in that window will not come up
A budget the request cannot carry
$ schemabot storage apply --timeout 500ms
Error: --timeout: a convergence budget of 500ms is shorter than the one second the request carries; name a whole number of seconds

Invariants

  • Establishes AV-11 — a storage convergence is bounded by what its caller can afford to wait. Enforced by the non-positive refusal at the shared entry point before any dialect is dispatched to, the operator path's seeded default, and apitypes.ResolveStorageApplyTimeout at both ends of the wire.
  • Establishes AV-12 — a storage convergence records nothing in the storage it converges. This one is not new behavior: the convergence entry points take a connection string rather than a store, so no row is written today. What is new is that something holds it there. Raising the operator budget to an hour makes a convergence a long-lived thing running against a live serving instance, which is exactly when someone reaches for "record that this happened" — and that write would land in a table being rewritten, or on a first boot in a table that does not exist yet. Enforced on both dialects by converging a seeded database and requiring the row count of every table in the live catalog to come through unchanged; the census reads the catalog rather than a list, so a table added to the embedded schema is covered without anyone extending the test. Both tests were checked against an injected write on each bootstrapper and fail on it.
  • Upholds AV-9 unchanged. The budget is not a schema source and the destructive gate is untouched; this PR modifies all four files AV-9's *Enforced:* line names, and enforcement stays in each of them.

⚠️ ID collision to resolve: #1389 also takes AV-11. Whichever of the two merges second needs to renumber — to AV-13 if this PR lands first, since it takes AV-12 as well.

Opened by Claude Code (Opus 5).

aparajon and others added 24 commits September 16, 2026 18:54
An interactive `storage apply` stopped when its preview found the catalog
already matching, which made it do less than the same command with
--auto-approve: the bootstrap also clears the schema change engine's leftover
tables, which outlive an interrupted convergence and are invisible to a catalog
diff, so they stayed on the database. It now runs the convergence either way,
and the prompt asks for the run it is about to do rather than for statements
there are none of.

A PostgreSQL-only repair command now refuses on the storage family before the
DSN is fetched. Fetching is not free of consequence — storage dsn_from reads a
secret — and refusing afterwards reported whatever went wrong resolving a
connection the command was never going to open, instead of the family that does
not apply to it.

Both halves of a direct convergence are attributed the way the preview's report
is, so a run's plan header and its result header name the same schema.
--release-repo joins the other release selectors the apply accepts in order to
refuse them by name.
The storage commands spelled the consent flag --allow-destructive, which
exists nowhere else: `schemabot apply` has always taken --allow-unsafe, and the
CLI's own output calls the changes destructive while naming that flag. Two
spellings for one concept meant an operator moving between the two commands got
kong's unknown-flag error on the one they had just used.

The convergence takes the flag. The plan no longer does: it runs nothing, so it
has no consent to take, and the normal plan/apply flow offers no preview of an
apply's --allow-unsafe either — the plan discloses the destructive statements
and names the flag, and seeing them as statements that will run means running
the apply and reading its preview. The apply's own preview is that path and
still sets the field, and a plan can report those statements as running with no
flag at all when the target's standing storage policy allows them.

Internal names are untouched. The config key
storage.allow_destructive_schema_changes and the API option
WithAllowDestructiveSchemaChanges predate this work and already spell the
concept "destructive"; only the flag surface was inconsistent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… apply does

storage apply converged the safe remainder and reported the refused
destructive statements afterwards, so an operator learned about a DROP
against SchemaBot's own storage from a summary of what had already run.
The rest of the CLI decides that question first: apply shows the plan,
names the statements, and stops until --allow-unsafe says otherwise.

The gate sits in front of the confirmation, so --auto-approve does not
skip it -- consenting to a convergence is not consenting to destroy
state. It reads the same preview the attended path already read, which
is now read on both paths so the unattended one has something to gate
on.

A deployment whose storage policy already allows destructive changes has
permitted them, and the gate does not narrow it (AV-9). Where it does
stop a run it runs strictly less than the convergence would have: the
bootstrap refuses the same statements on its own. What changes is when
the operator finds out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…efused

A blocked convergence now prints the command line that grants consent,
the way a blocked schema change apply does. It is built from the flags
that addressed this target rather than fixed, because the suggestion has
to converge the same storage database the refusal is about -- dropping
the flags would name the storage of whichever server the CLI points at,
which during a rollback is a different database. A DSN is named rather
than repeated: it carries the storage credentials, and this is printed to
a terminal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A report carrying both a manual-remediation entry and a destructive
statement stopped on the destructive one, but a plan renders every
statement as gated while a manual entry is outstanding and prints no
refusal for the destructive ones. The run exited non-zero with nothing
on screen saying which refusal it hit.

The manual entry is the refusal to report: it gates the whole drift set,
so a destructive statement behind it is not yet reachable, and its own
message names it on both the attended and unattended paths.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The manual-remediation refusal sat inside the attended branch, so a
`storage apply -y` against a report carrying one never reached it. With a
destructive statement present too the destructive gate deferred to a refusal
that could not run, and the command exited non-zero with nothing on screen;
with manual entries alone it went on to converge, which is a different
contract from the attended path's "resolve these first".

Both refusals now run either way, and manual runs first because it gates the
whole drift set -- a destructive statement behind it is unreachable rather
than merely refused, so naming the flag that permits it would name the wrong
remedy. The unattended path prints the plan before the error, since the error
says the entries are listed above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The direct path is the reason these commands exist: the server is down —
possibly down because its own schema bootstrap is failing — so the operator
points the CLI at the storage database itself. Nothing exercised it. Every test
drove the API path through a fake server, so dropping the bootstrap options,
inferring the wrong dialect, or losing the report's attribution would have
stayed green and surfaced during the incident the path was added for.

So it runs against a real, empty storage database: a convergence creates the
whole schema from nothing, the plan that follows agrees it did, a plan against
an unconverged database reports the whole schema outstanding with its own exit
status, and an attended apply's preview names this build as the schema it is
about to converge to.

While here, direct()'s doc no longer claims a $SCHEMABOT_CONFIG_FILE fallback
that cannot fire on these commands: a direct connection is already chosen by
the time the config is read, so the env var is the fallback source for the
direct-only maintenance commands and is never read here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan's hint is gone with the function that built it, so the plan ends on
its summary line. What an operator does next is run the convergence from the
release's own binary before the roll, which is this command's sibling rather
than a paragraph under every plan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ve storage DDL

The guide and the configuration reference both named a flag spelled only in
this feature. The convergence takes --allow-unsafe, the same flag `schemabot
apply` takes for destructive changes.

The config key keeps its name: allow_destructive_schema_changes is shipped
configuration, and the surrounding prose already distinguishes the standing
policy from the flag that widens it for one invocation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
storage apply now refuses to run anything until a destructive statement
is permitted, so the guide's exit-status table no longer has an outcome
where one is left refused and the run still succeeded. The startup
bootstrap still skips and continues, and the guide says why the two
differ.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three console transcripts carried reason strings no code path emits. The
destructive refusals showed "DROP TABLE destroys data", a phrase that exists
only in a unit-test fixture; the real line is the engine linter's own message,
carried through untouched. The PostgreSQL manual-remediation example dropped
the column name and the remedy, which are the two facts an operator needs in
order to write the DDL by hand — so the guide made the CLI look less actionable
than it is.

An operator grepping the documented phrase out of the output, or out of a
--json report, matched nothing.

Also: PostgreSQL index names are unique per schema rather than per database,
the interactive transcript was missing one of the two blank lines the summary
and the prompt each print, and the release checklist said MySQL additive
changes need no action — true of a table or a column, but an index add runs as
a table copy inside the startup budget, which is exactly why the guide says to
pre-create it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ocal hosting

Two claims in the guide were stated for every dialect and are not.

The leftover engine tables a converged run clears are a MySQL thing: the
PostgreSQL bootstrap has none to clear, and a PostgreSQL operator taking the run
for the stated reason got a no-op. What it does do there is re-check the shape
of every storage table — each expected index present, valid, and unique where
the schema requires it — which a column diff does not cover, so that is what the
guide now says for PostgreSQL.

And --allow-unsafe does not widen the policy everywhere: a locally hosted server
never runs destructive storage statements and refuses the request that opts in
(AZ-6), so the flag's description carries the carve-out and the reason for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three `storage plan` renderings carried the hint the CLI no longer
prints, which told an operator to converge by letting a release's boot do it.
Converging ahead of the roll is what this guide's own pre-deploy sequence
covers, and it names the command rather than the boot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…w does

The destructive and manual examples showed a notice with no statement
above it. A storage plan renders the whole difference between the two
schemas and counts it, so the examples show that, and the prose says
the summary is the difference rather than a promise of what will run.
… wait

A boot gives up on converging storage after five minutes because a pod
converging is a pod not yet serving, and one holding the bootstrap advisory
lock keeps every other pod from serving too. `storage apply` ran the same
bootstrap and so inherited that budget along with the code, which took the
work out of a roll but not out of the ceiling: an index build too slow for a
boot was still too slow from a terminal, and still had to be run by hand.

The budget now follows the path instead of the code. It is an option threaded
through both bootstrappers' contexts and both lock waits, defaulting to the
boot budget so an unconsidered caller keeps a boot's guarantee, and seeded to
an hour in front of the operator path's options so a caller's own budget still
wins. `--timeout` lowers it for a run that should fail fast; a budget outside
the permitted range is refused rather than clamped, so no surface reports a
budget it did not get. On PostgreSQL the per-statement DDL budget is derived
from the effective budget rather than once from the boot constant.

The maximum is held at the default for now. A convergence cannot yet be
stopped once it starts, and the lock it holds is what a booting pod waits on,
so an hour is the longest a mistake can keep pods from coming up. Raising it
is the payoff of the follow-up that adds a stop.

Establishes AV-11: a storage convergence is bounded by what its caller can
afford to wait. Upholds AV-9 unchanged — the budget is not a schema source and
the destructive gate is untouched.
Converging SchemaBot's own storage changes that storage's shape and nothing
else in it. That purity is what lets one implementation serve both a boot
against a database with no schema at all — where a table to record into does
not exist yet — and a deliberate convergence against tables being rewritten
under live traffic, where a write about the convergence would land in
something mid-change.

Nothing held it there. The convergence entry points take a connection string
rather than a store, which is why no row is written today, but a later change
that wanted to audit a convergence would find nothing in its way until it
failed on an empty database.

Establishes AV-12, enforced on both dialects by converging a seeded database
and requiring the row count of every table in the live catalog to come through
unchanged. The census reads the catalog rather than a list, so a table added
to the embedded schema is covered without anyone extending the test.
… it expires

A convergence that runs out of time named the boot's five minutes whatever
budget it had, because the failure builder read the constant rather than the
run's own ceiling. An operator who gave a convergence an hour, watched it
expire, and was told it did not finish in five minutes goes looking for a
timeout that never fired — on the one path where elapsed time is the first
thing they check.
@aparajon
aparajon force-pushed the armand/storage-schema-guide branch from 590ae1f to fe55852 Compare September 17, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant