Bound concurrent CA-key signing, shedding on the OCSP path - #285
Conversation
bfc27f1 to
a61ec48
Compare
This comment has been minimized.
This comment has been minimized.
a61ec48 to
48bf166
Compare
Review Council disposition — all 8 findings accepted and acted onRebased onto Every finding was checked against the code before being accepted. All eight were real. Four new commits. 🟠 HIGH1. CRL re-sign's queueing path never exercised through 2. Fixed in One correction to the recommendation on the issuance side, because it cannot be done as written: two concurrent issuances cannot contend for a slot. Mutation-tested against the failures the findings name, not just convenient ones:
The third wiring spec (explicit 🟡 MEDIUM3. 🛡️ The recommended remedy is not available here: forcing a reconnect is impossible, because the socketpair fd is inherited once at spawn and consumed by What does bound it is the bound this PR adds, which I think the analysis missed: every 4. 📚 OpenBao guide claims no CA-side metric exists and 6. ⚙️ OpenBao guide claims the bound is unimplemented, two lines above the bullet documenting it — both accepted, both fixed. Worth naming rather than quietly correcting: both paragraphs are text this branch inherited when it rebased onto merged #265, and my own edit landed immediately beside them without touching either. The inserted bullet read as an addition when it was also a contradiction. 5. 🧭 PR description understates its own I also found a worse instance of the same defect the finding is about, which it did not flag: the description's opening section still said "this targets the world after #265 lands" and "on today's 7. 🧪 Leaf-issuance queueing lacks real-concurrency and cancellation coverage — accepted; covered by the same new file, with the 8. 🧪 New Prometheus metrics have no test coverage — accepted. Three specs following Full |
This comment has been minimized.
This comment has been minimized.
98eeaf3 to
c8a3c45
Compare
Review Council disposition — both findings accepted, and the MEDIUM is under-ratedRebased onto 🟠 HIGH — no handler-level test for the 503 shed branchAccepted. Confirmed: the only cover was The new spec fills the bound the way a real caller would rather than by reaching into the CA: one slot, a gated signer parked inside Demonstrated rather than asserted. A coverage finding normally claims a gap; this one can be measured. Mutating the handler to write an empty body leaves 🟡 MEDIUM — slot release not panic-safeAccepted, and this is worth more than MEDIUM. The finding left the consequence as an assumption, and it does not hold up as merely hygienic. There is no The pool is small by design, which sets the blast radius. Why it was written that way, since the reason matters more than the fix. I chose sequential release deliberately, to hold the slot for the minimum, on the stated belief that a panic mid-signature takes the process with it anyway. That belief was wrong about the runtime, not about the design. Recording it is the durable part: without it the next person to touch this primitive re-derives the same rationale and puts the bug back. It is now in Fixed with the closure-with-defer shape, not a function-scoped Three specs pin it, one per call site, each recovering the panic the way Mutations, all against a committed tree
Rebase note. This now sits on merged #289, which touches Full |
c8a3c45 to
79c20d2
Compare
voxpupuli#166, voxpupuli#168, voxpupuli#212 and voxpupuli#265 merged; voxpupuli#284, voxpupuli#285, voxpupuli#288 and voxpupuli#289 added. Eight entries. Derived independently rather than taken on report, and it agreed. Ordering from a pairwise conflict matrix rather than from file overlap: the only pair that actually conflicts is voxpupuli#266 x voxpupuli#282 on magefile.go, so they stay adjacent and everything else is order-independent. voxpupuli#285 and voxpupuli#289 touch collector.go and metrics.md in common but merge cleanly against each other, so pairing them would buy nothing. Also recorded that membership is the only thing needing an edit here. Heads move constantly and cost nothing because the entries are refs, not shas, so 'wait until PR X settles before refreshing' answers the wrong question - X settling changes no membership, and meanwhile the list can be omitting open PRs, which is the half that makes a green build misleading.
voxpupuli#283, voxpupuli#288 and voxpupuli#289 merged; voxpupuli#294 and voxpupuli#296 added. Seven entries, and main has moved to 2d11cc0. Ordering from a pairwise conflict matrix over all six open PRs. Two pairs actually collide: voxpupuli#266 x voxpupuli#282 on magefile.go, as always, and a new one - voxpupuli#285 x voxpupuli#294 on cmd/openvox-ca/config.go. Each pair sits adjacent so its conflict surfaces once; the other two branches are order-independent. Trial replays that as 2 of 7, matching the matrix. Worth noting voxpupuli#266 is now 56 behind main, by far the stalest entry.
🟢 Review Council: APPROVE
Reviewed at commit TL;DR: PR #285 passed review with all five personas approving; four findings - three MEDIUM, one LOW - remain unresolved but non-blocking. Findings: 🔴 0 Critical, 🟠 0 High, 🟡 3 Medium, 🔵 1 Low
🟡 MEDIUM (3)
🔵 LOW (1)
Produced by Review Council, an open-source multi-persona code reviewer. Spot a wrong call or want the source? File feedback or browse the repository. |
`c.mu` was, incidentally, the only thing holding CA-key use to one signature in flight process-wide: every `CAKey.Sign` site held it. Moving the OCSP signature out from under that lock — the point of #197 / PR #265, since signing under the CA's global write lock stalled every other caller — leaves nothing bounding it. `/ocsp` is unauthenticated (tierPublic), the only rate limiter in internal/api is CSR-only, and a cache miss signs, so an unauthenticated caller can drive as many concurrent signatures as it can open connections, against a signer shared with issuance. Add a shared bound: one token per permitted concurrent signature, taken by every CAKey.Sign site — issuance, CRL re-signing and the OCSP responder together, because there is one key and what needs bounding is the load on whatever holds it. The two halves behave differently under it, and that asymmetry is the substance rather than a detail. Issuance and CRL work *queues*: it is authenticated, and refusing a certificate a client asked for in order to protect an unauthenticated responder would be the wrong way round. The OCSP responder *sheds*, answering RFC 6960 `tryLater` after a bounded wait. Shedding is not a stylistic choice there. Letting `/ocsp` queue would convert unbounded signing into unbounded queueing, bounding nothing — and worse, it would reinstate the very stall #197 removed, by a different route: issuance acquires its slot while holding `c.mu`, so an anonymous flood owning every slot would block a `c.mu` holder and stall every reader behind it, including the revocation check on the authentication path. Two things stop that. OCSP never queues, so it cannot build a line an issuance has to join; and both acquisitions honour ctx, so a caller that has gone away stops waiting rather than holding `c.mu` on nobody's behalf. Remove either and the stall is back. Refusing is unusually cheap here, which is what makes it a real relief valve rather than a slower route to the same load: an RFC 6960 non-success response carries no signature at all, so a shed request costs no CA-key work. No new dependency — a buffered channel, and `x/crypto/ocsp` for `tryLater` was already in the module graph. The bound is deliberately not sized here. The right number is a property of the deployment's signer: an in-process software key, an isolated signer over IPC and an OpenBao Transit key have very different concurrency envelopes, and non-positive `SigningConcurrency` leaves the bound unset, which every helper reads as unbounded, so this commit changes no behaviour on its own; the command layer resolves the shipped default. On today's `main` the mechanism is dormant by construction: `AnswerOCSP` still signs under `c.mu`, which serialises it against issuance, so at most one slot is ever held and the bound is never contended. It becomes load-bearing when Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shed branch added by the signing bound cannot be reached from a spec: on the state this builds against, every signing slot is taken under `c.mu`, so nothing can occupy one through the public API and provoke a real refusal. Extract the error → RFC 6960 mapping so the part worth pinning is testable without driving the handler into a state it cannot reach. The three outcomes say materially different things to a verifier — `tryLater` invites a retry, `internalError` reports a server fault, and `malformedRequest` tells it not to bother — so which one a given failure produces is a contract, not an implementation detail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/development/locking.md records, in the gap this closes, that no metric covers concurrent signing at all — an operator cannot tell "this never happens" from "this happens throughout every CRL update". A bound nobody can observe being approached is only half an answer, so the series land with the bound rather than after it. Three, because one would not be readable on its own: - puppetca_ca_signing_in_flight — signatures in flight now. - puppetca_ca_signing_limit — the configured ceiling; 0 means unbounded. - puppetca_ca_signing_shed_total — OCSP responses refused with tryLater. In-flight alone cannot say whether 8 concurrent signatures is comfortable or is the ceiling, which is why the limit is published beside it rather than left to be inferred from configuration. The limit is emitted even when it is 0: "unbounded" is a legitimate configured value and precisely the state worth alerting on, and it is indistinguishable at a glance from a bound that is simply never reached. Emitted alongside the failure counters and ahead of the storage gather, for the same reason those are: they are in-process state, so an unreadable backend must not blind an operator to the responder shedding. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wires the CA signing bound to config file, PUPPET_CA_SIGNING_CONCURRENCY and
--ca-signing-concurrency, following the csr_rate_limit sentinel convention
exactly: -1 ("unset") takes the built-in default, an explicit 0 disables the
bound, and positive values pass through. Getting 0 wrong in either direction is
the expensive mistake — rewriting it to the default takes away an operator's
deliberate opt-out, and rewriting the sentinel to 0 would ship unbounded
signing to everyone who never set the key, which is the exposure the bound
exists to close.
The default is max(4, GOMAXPROCS). Scaling with the CPU count suits the two
backends where a signature is CPU-bound — a software key in process, and the
default isolated signer, where it is CPU-bound in the signer child — since past
that point extra concurrency buys latency and memory rather than throughput.
The floor keeps a single-CPU container from getting a bound of 1, which would
serialise issuance behind the OCSP responder.
It is deliberately a safe ceiling and not a tuned value, which is the whole
reason #265 declined to pick a number: the right size is a property of the
deployment's signer, and an in-process key, a PKCS#11 token and an OpenBao
Transit key have very different envelopes. What the default guarantees is only
that the number is finite. Deployments with a remote signer are told, in the
docs, to lower it to that signer's capacity — a thing openvox-ca has no way to
discover. The floor guards the default only; an explicit 1 is honoured, because
a remote signer's capacity is routinely below the local CPU count.
Refs #274
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the ca_signing_concurrency key to configuration.md (flag, env and file rows, plus a section on why the bound exists and how to choose a value), the three new series to metrics.md, and the lock-ordering invariant to docs/development/locking.md. The guidance is written per backend rather than as one number, because that is the substance of the issue. The default is sized for the isolated signer, where signing is CPU-bound in the signer child. An OpenBao Transit deployment is told to set it explicitly: the default derives from the CA host's CPU count, which has no relationship to what a Transit key — possibly shared with other consumers — can sustain. Both notes say the bound is per process, so N replicas permit N times the limit against one shared key. The isolated signer gets its own subsection in ca-key-security.md rather than a cross-reference, deliberately. It is the default deployment and the less bounded of the two backends, since RemoteSigner.Sign waits on the signer child's reply with no per-call deadline, and documenting the bounded backend while leaving the unbounded default to a link would be the wrong way round. metrics.md says plainly that a rising shed counter is not by itself a fault — it is the bound working, and on an unauthenticated flood it is the protection working — and gives the two readings that distinguish "the limit is too low" from "something is driving /ocsp harder than this responder is sized for". It also suggests alerting on a limit of 0, since unbounded is a legitimate configured value and looks identical at a glance to a bound that is never reached. No Known-gaps entry: on main that list has no #274 entry to amend. The one describing this exposure exists only on the #265 branch, whose driver has already linked the issue and dropped the block quote, and duplicating it here would collide in a file four open branches are editing. Striking it through falls due when #265 lands. The locking.md change here is confined to a single additive bullet under Lock ordering. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RemoteSigner.Sign was a bare rpc.Client.Call, which returns when a reply arrives or the connection breaks. Neither happens when the signer child is wedged rather than dead, so the frontend waited indefinitely with no per-call bound. That made the *default* deployment the less bounded of the two backends: openbao.Signer.Sign already bounds each call at roughly twice its login timeout, while key isolation — which is what `openvox-ca serve` does unless told otherwise — bounded nothing. It is also why a concurrency bound alone does not finish the job #274 opens. Capping how many callers may be signing at once does nothing if each of them can wait forever: the slots never come back, and a bound whose slots are never released is an outage rather than a limit. Use an asynchronous call and select on a timer. The Done channel is buffered to depth 1 deliberately — net/rpc discards a reply it cannot deliver, so an unbuffered channel would drop the answer to any call this stopped waiting for, and with capacity 1 the client's pending-call entry is always cleaned up even when nobody is listening. Two minutes, and the number is a backstop rather than a tuning. It has to clear the slowest thing the child can legitimately be doing, which is not a local signature: under ca_key_provider: openbao the child's own Sign is a network round trip bounded at about 2x LoginTimeout, ~20s by default. Two minutes leaves room for an operator who has raised that substantially while still turning an indefinite hang into an error. What this does NOT do, and the commit should not be read as claiming: net/rpc has no cancellation, so the deadline bounds *this caller's wait*, not the signer's work. An abandoned call leaves the child still signing and its reply is delivered and dropped. It stops callers accumulating against a wedged signer — which is what makes the concurrency bound's slots recoverable — and it does not reduce the load on the signer itself. Nothing bounds that today. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The existing specs occupy slots by hand. That states the mechanism — it refuses when full, it admits again when a slot returns, it distinguishes a shed from a cancelled caller — but it does not state that the mechanism engages on the path it exists for. Every one of them would still pass if nothing ever took a slot during an actual signature. This spec takes none. It wraps the CA key in a signer that parks inside Sign, issues one OCSP request to occupy the only slot by signing, and then issues a second while the first is demonstrably still in there. The second is refused with ErrSigningBusy; releasing the first returns the slot and a third succeeds. **It could not have been written before #197.** The OCSP responder used to sign while holding c.mu, which serialised any two requests before they could reach the signature — so racing the responder against itself would have passed with the bound deleted. Signing outside the lock is what makes concurrent signatures reachable, which is the exposure this bound answers and, now, the thing a spec can actually observe. Barrier placement is what gives it the power to fail, and it is the whole design of the spec: the second request is issued only after the first is observed *inside* Sign. Fire the two off together and they may simply complete in sequence, leaving a spec that passes either way. Verified by mutation — with the shed removed from acquireSigningSlotOrShed, this spec fails. Unknown serials rather than nonces to force the cache miss: an unknown is never cached, so every such request reaches the signature, and the request needs nothing signed to build. Also corrects two comments that the #265 rebase made false. Both claimed the shed branch was beyond a spec's reach because every slot is taken under c.mu — true when they were written, and no longer true of the responder. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, 2 HIGH: the bound's queueing half was tested as a primitive and through neither of the two call sites that use it. signbound_test.go calls acquireSigningSlot directly and signboundrace_test.go proves the OCSP *shed*, so nothing exercised a CRL re-sign or an issuance against a full bound — and nothing would have noticed a slot acquired and never released on those paths. That is the failure worth catching: with a small bound, one lost slot makes every future revocation and CRL update hang, silently and permanently. Four specs, driving ReissueCRL and Generate against an occupied bound: - each queues rather than failing, and returns its slot on success - each gives up when its context is cancelled while queued The cancellation halves are not incidental. A blocking acquire is taken while holding c.mu, so a caller that has gone away must stop waiting rather than hold c.mu on its behalf; that ctx is half of what makes the blocking acquire safe at all, and nothing proved it fired at these sites. Both assert on the call site's own error text, so they cannot pass by noticing the cancellation somewhere else on the way in, and the CRL one asserts crlUpdateFailures increments. What the specs deliberately do NOT do is race two issuances against each other. They cannot contend: issueLeafLocked runs under c.mu, so c.mu serialises them long before either reaches the bound. The contention that can really happen is cross-path — the OCSP responder signs without c.mu and can hold the only slot while an issuance waits — which is what occupying the bound reproduces, and which is the interaction the asymmetric design was built around. Verified by mutation, against the failures the finding names rather than only ones convenient to catch: dropping either releaseSigningSlot fails the matching "returns it on success" spec, and making the queueing acquire ignore ctx fails both cancellation specs. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, HIGH. resolveSigningConcurrency and the file/env layering were each covered in isolation, but nothing called applyCAConfig and checked the value arrived — the one line that makes the feature take effect at server start. The failure mode is total silence, which is why this is worth its own spec rather than being implied by the others: drop or misassign that line and ca.CA.SigningConcurrency stays at its Go zero value, which means *unbounded*. The entire bound would be inert, the exposure #274 describes would be wide open, and every existing test — including all of signbound_test.go, which constructs its own CAs — would still pass. This is the project's own convention for exactly this class of gap; config.go's crl_chain_file block carries an It("reaches the CA, which is the step whose absence is silent") for the same reason, and this follows it. Three cases, because the interesting ones are not the happy path: an explicit value arrives intact; the unset sentinel resolves to the built-in default on the way through rather than to zero; and an explicit 0 survives as unbounded, since that is an operator's deliberate opt-out and rewriting it would take that away. Verified by mutation: deleting the applyCAConfig assignment fails the first two. The third correctly survives it — 0 is also the zero value, so it cannot distinguish, and pretending otherwise would be a spec that passes for the wrong reason. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, MEDIUM: the new series were registered and emitted with nothing
asserting they appear or carry the right values.
Three cases, following collector_test.go's existing gather/findByLabels
pattern:
- a configured limit reports with nothing in flight and nothing shed
- the limit is still *emitted* as 0 when signing is unbounded, rather than
omitted — that is the state an operator alerts on to catch a CA that is not
bounding its signer at all, and an absent series cannot be alerted on
- a real signature held open reports in-flight 1, a second OCSP request
behind it increments the shed counter, and in-flight falls back to 0 once
the first completes
The third drives the metrics from actual contention rather than by poking
counters: the CA key is wrapped in a signer that parks inside Sign, so the
gauge is read while a signature genuinely holds the only slot. That also pins
the counter to the shed path — it is asserted at 0 in the first case, where
issuance and CRL work have queued through the same bound without shedding.
Refs #274
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings, two of them the same defect: prose left behind by the change that falsified it. **The OpenBao guide claimed the bound did not exist, twice.** One paragraph said a configurable bound "would land" in #274 "if one is added" — two lines above the new bullet telling operators to configure it. The "What to monitor" bullet further down said there is no CA-side metric for OCSP crowding out issuance and to watch OpenBao alone, which is now precisely what puppetca_ca_signing_in_flight, _limit and _shed_total measure. Both are text this branch inherited when it rebased onto merged #265, and its own edit landed immediately beside them without correcting either. That is worth naming rather than quietly fixing: the inserted bullet read as an addition when it was also a contradiction, and an operator reading the section top to bottom met both. **The RemoteSigner deadline leaves a pending-call entry behind.** Documented only as a discarded reply until now, which understated it. net/rpc removes an entry when a reply for that sequence arrives or when the connection tears down; against a signer that is wedged rather than dead — the fault the deadline exists for — neither happens. The reviewer's suggested remedy, forcing a reconnect, is not available here: the socketpair fd is inherited once at spawn and consumed by DialConn, so there is no second dial to flush the map with. What bounds the cost instead is the bound this PR adds. Every Sign passes through ca_signing_concurrency, so at most `limit` calls are outstanding and each holds its slot for the full timeout — entries accrue at limit/timeout rather than per request, single digits per minute at the shipped default, and they drain whenever a reply does arrive. A permanently wedged signer leaks on the order of a megabyte a day, during an outage someone is already fixing. Recorded rather than engineered around, and the trade said plainly: before the deadline the frontend accumulated stuck goroutines instead and never got its signing slots back, which is worse. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, MEDIUM. All three call sites released the slot with a plain sequential call after the signature, so a panic crossing that call leaked it. I chose that shape deliberately when writing the bound, to hold the slot for the minimum, and reasoned that a panic mid-signature would take the process with it anyway. That reasoning was wrong, and it is worth writing down because it is what makes this more than hygiene: **net/http's conn.serve recovers a handler panic**, logs it and drops that one connection. The process survives. Nothing in this repository calls recover() itself, so it is the standard library's recovery, not ours, that turns a crash into permanent capacity loss — and two of the three sites are reachable from an HTTP handler, the OCSP one from an unauthenticated request an anonymous caller shapes. The pool is small by design, which sets the consequence. ca_signing_concurrency defaults to max(4, GOMAXPROCS), but operators running an isolated signer or OpenBao Transit are explicitly told — in ca.go's field doc, in the flag help and in the docs — to lower it to that signer's real capacity, so 1 or 2 is an ordinary setting. There, one leaked slot wedges issuance and CRL re-signing, which queue and do so under c.mu, and sheds every OCSP request with `tryLater` until restart. That is a permanent denial of service inside the control added to bound one, so it is worth more than the MEDIUM it came in as. Fixed with the closure-with-defer shape rather than a function-scoped defer, because scope matters here: signCRLLocked writes to storage after signing, and a defer at function scope would hold a signing slot across that write — inflating occupancy far past the work the bound exists to meter. This is rule 4 of docs/development/locking.md applied to the new primitive, and signing.go's "a panic mid-sign still frees the lock rather than wedging the CA" is the same argument already made for c.mu. The specs pin it at all three sites with a signer stand-in that panics, and recover the panic themselves the way net/http would. Each asserts the slot came back *and* that the CA still works afterwards, since a leak's real signature is the second request failing rather than the first. Verified by mutation against a committed tree: restoring any one of the three sequential releases turns its spec red on the leak assertion. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, HIGH. The shed branch was executed by no test at all. The only cover was ocsperror_internal_test.go, which pins ocspErrorResponse's mapping in isolation — it says which constants belong together and executes none of the handler: not the 503 write, not the response body, not the log line. That is the wrong thing to leave untested here. What an unauthenticated caller receives when the CA-key bound is full *is* the security property this change exists to provide, so it should be asserted on the bytes that reach the wire. The spec fills the bound the way a real caller would rather than by reaching into the CA: one slot, a gated signer parked inside Sign holding it, and a second request POSTed through the mux. It asserts 503 with xocsp.TryLaterErrorResponse and, explicitly, that the body is neither MalformedRequest nor InternalError — the three say materially different things to a verifier, and only tryLater invites the retry that makes shedding survivable. It then releases the first signature and shows the responder serving normally again, since a refusal that did not recover would be an outage rather than a queue depth. Shaped after the sibling spec in ocsp_test.go, "answers a signer failure with 500 internalError, not 400 malformedRequest", which exists for the same reason one layer along. Verified by a mutation the isolated mapping test cannot catch: making the handler write an empty body leaves ocsperror_internal_test.go green and turns this one red. That is the coverage gap the finding named, demonstrated rather than asserted. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First production report on this bound, from a deployment running ca_key_provider: openbao with a Transit-held key: the default is CPU-shaped and the work under an external provider is not. Every signature there is a network round trip, so max(4, GOMAXPROCS) measures this host's cores and says nothing about what that Transit key — possibly shared with other consumers — can sustain. On a well-provisioned node it lands far above what the provider wants, and nothing tells the deployment that has happened. The guidance already existed: ca.go's field doc, the --ca-signing-concurrency help text and docs/configuration.md all say a remote signer should be sized explicitly. What was missing is that the *default* actively points the other way for exactly those deployments, silently. Deriving a different default when the provider is remote was the other option and is worse. It would invent a number for a capacity openvox-ca cannot discover, which is what #265 declined to do and what resolveSigningConcurrency's own doc disclaims — the default is a ceiling, not a tuning. So say it once, plainly, and leave the number to whoever can measure it. Warned only for a signer reached over the network, and only when the value was left unset. Not for the isolated signer, which is the default topology: signing is CPU-bound in the signer child there, so a CPU-derived ceiling is the right shape and lowering it is a tuning rather than a correction. Not when an operator set a value, including an explicit 0 — that is a deliberate opt-out of the bound and nagging about a decision already made is how a warning stops being read. Serve only; the offline commands share applyCAConfig but sign one certificate at a time, so the bound never binds there. The specs assert the quiet cases as well as the loud one, because firing on the wrong combination is the failure that matters: a warning on every start of the default topology would be noise, and noise is indistinguishable from a warning nobody reads. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four findings, none blocking, all worth taking. Two are reasoning defects in my own prose rather than in the code, which is the shape this branch has produced repeatedly. **The pending-map justification had a hole exactly where the bound is disabled.** The RemoteSigner deadline's comment argues the abandoned net/rpc entry is bounded because every Sign passes through ca_signing_concurrency, so entries accrue at limit/timeout rather than per request. But `ca_signing_concurrency: 0` is a documented, supported value meaning unbounded signing, and it removes that rate limit entirely — leaving entries bounded only by in-flight request concurrency, which on the unauthenticated /ocsp path the caller chooses. Named rather than left for a reader to find: it is a consequence an operator opts into rather than a defect, but the sentence read as an unconditional guarantee and the whole argument for leaving the leak in place rests on it. **Init's stated reason for sizing the bound first was simply false.** It claimed Init signs through the bound on the bootstrap path. It does not: the self-signature calls x509.CreateCertificate directly and the bootstrap CRL goes straight to storage, so neither takes a slot — verified, Init reaches neither signCRLLocked nor issueLeafLocked. The ordering is still right, for a different reason now stated: Init has several exits and only the first statement is on all of them, and the fast path returns as soon as an existing CA loads, which is what almost every start does. Sizing placed after it would leave the commonest deployment unbounded with nothing saying so. **The per-shed log was an amplifier on an unauthenticated path.** One WARN per refused request means an anonymous caller chooses how much this CA writes to disk, turning a request flood into a log flood — amplifying the load the bound exists to shed. Dropped to Debug, which costs nothing: every one of these is already counted by puppetca_ca_signing_shed_total, and the metric is what the docs tell operators to alert on. The line only adds which caller provoked it, a debugging question rather than a monitoring one. **And a cache hit bypassing the bound was untested.** That property is what keeps the bound from being felt in normal service: a cached response returns under the read lock before the bound is consulted, so ordinary verifier traffic is answered while the bound is saturated. Without it a full bound would shed every request rather than only those that would sign, and the shed rate would stop meaning what the metric claims. Verified by mutation — consulting the bound before the cache fails the new spec and leaves its two siblings green, so it discriminates the ordering rather than the bound. Refs #274 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
79c20d2 to
089152d
Compare
Disposition — all four taken, though the verdict was APPROVERebased onto 🛡️ The pending-map justification has a hole where the bound is disabledAccepted, and it is the one I would have most wanted caught. The Named in the comment rather than left for a reader to discover. It is a consequence an operator opts into rather than a defect in the opt-out — accepting unbounded concurrent signing is accepting unbounded pending entries, seen from the other side — but the sentence read as an unconditional guarantee, and the entire argument for leaving that leak in place rests on it. A justification with a hole exactly where a supported configuration puts one is worth stating, especially since I argued the panic finding up on the neighbouring ground that a leaked resource is permanent. 🧭 Init's stated reason was falseAccepted — and it was not merely imprecise, it was wrong. The comment claimed The ordering is still correct, for a reason now stated instead: ⚙️ One WARN per refused request is an amplifierAccepted. On an unauthenticated endpoint, one WARN per shed means an anonymous caller decides how much this CA writes to disk — a request flood becomes a log flood, amplifying the load the bound exists to shed. Dropped to Debug, which costs nothing that was being relied on: every shed is already counted by 🧪 A cache hit bypassing the bound was untestedAccepted. This is the property that keeps the bound from being felt in normal service: a cached response returns under the read lock before the bound is consulted, so ordinary verifier traffic — repeat queries for a handful of serials — is answered while the bound is saturated. Without it, a full bound would shed every request rather than only those that would actually sign, and the shed rate would stop meaning what the metric claims. Verified by mutation, and the mutation was chosen to discriminate the ordering rather than the bound: consulting the bound before the cache fails the new spec and leaves both its siblings green. Two of the four were defects in prose rather than code — a justification that did not hold under a supported setting, and a stated reason that was simply untrue. That is the same class as the stale claims this branch has already corrected three times, and it is the class I now check for after every ruling and rebase rather than case by case. Full |
Bounds concurrent CA-key signing, shared across certificate issuance, CRL
re-signing and the OCSP responder, and configurable via
ca_signing_concurrency. Closes #274.What this builds on
#265 has merged, and this is rebased onto it. That matters because it is
what makes the bound load-bearing rather than dormant:
AnswerOCSPnow signsoutside
c.mu, so concurrent OCSP requests really do become concurrentCA-key signatures.
c.muwas, incidentally, the only thing holding CA-key use to one signature inflight process-wide — every
CAKey.Signsite held it. #265 removed that forOCSP, which was the entire point of that change. What it left is unbounded:
/ocspis unauthenticated (tierPublic), the only rate limiter ininternal/apiis CSR-only, and a cache miss signs, so an unauthenticatedcaller can drive as many concurrent signatures as it can open connections
against a signer shared with issuance.
An earlier revision of this PR was written against pre-#265
main, where themechanism could not be contended and the concurrency behaviour could not be
tested. It now can be, and is — see
internal/ca/signboundrace_test.go, whichholds a real signature open and shows a genuinely concurrent second request
being shed.
Why a shared blocking semaphore would have been wrong
The obvious implementation reinstates the bug #265 exists to fix, by a
different route.
issueLeafLockedandsignCRLLockedacquire while holdingc.mu. An anonymous flood owning every slot would block that acquisition,stalling every
c.mureader behind it — including the revocation check on theauthentication path. That is #197's process-wide stall, arrived at from the
other side.
So the bound is asymmetric:
serialised against each other by
c.mu; refusing a certificate a clientasked for in order to protect an unauthenticated responder would be the wrong
way round.
tryLaterover HTTP 503after a bounded wait. Letting it queue would convert unbounded signing into
unbounded queueing and bound nothing.
Both acquisitions honour
ctx, so a caller that has gone away stops waitingrather than holding
c.muon nobody's behalf. Remove either property and thestall is back; the invariant is recorded in
docs/development/locking.mdunderLock ordering.
Shedding is unusually cheap here, which is what makes it a real relief valve
rather than a slower route to the same load: an RFC 6960 non-success response
carries no signature, so a refused request costs no CA-key work at all.
The default
max(4, GOMAXPROCS), following thecsr_rate_limitsentinel convention(
-1/unset → default,0→ unbounded, positive → literal) across file, envand flag.
#265 declined to pick a number because the right size is a property of the
deployment's signer, and that reasoning is preserved here: the default is a
safe ceiling, not a tuning. Its only job is to make the number finite.
Scaling with CPU count suits the two backends where a signature is CPU-bound —
a software key in process, and the default isolated signer, where it is
CPU-bound in the signer child. Operators with a remote signer are told, in the
docs, to lower it to that signer's capacity, which openvox-ca cannot discover.
What this does NOT protect
permit N × the limit against that key. Documented in both
metrics.mdandopenbao-transit.md.memory remain unbounded; only CA-key work is capped.
responder treats sustained
tryLateras a revocation-checking outage. Thisconverts an unbounded signer load into a bounded, observable, visible
degradation — that is the trade, and
configuration.mdsays so plainly.RemoteSignerdeadline bounds the caller's wait, not the signer'swork.
net/rpchas no cancellation, so an abandoned call leaves the signerchild still signing. It stops callers accumulating — which is what makes the
slots recoverable — but does not reduce load on the signer. Nothing bounds
that today.
reclaimed only if a reply eventually arrives. Against a permanently wedged
signer none does. The bound itself rate-limits this — at most
limitcallsoutstanding, each holding its slot for the full timeout, so entries accrue at
limit/timeoutrather than per request — which works out at roughly amegabyte a day during an outage someone is already fixing. Reconnecting is
not available as a remedy: the socketpair fd is inherited once at spawn.
Recorded in
docs/ca-key-security.mdrather than engineered around.Observability
docs/development/locking.mdnotes that no metric covers this at all, and thata bound nobody can watch being approached is only half an answer. Three series
land with it:
puppetca_ca_signing_in_flight,puppetca_ca_signing_limitandpuppetca_ca_signing_shed_total. The limit is published beside in-flightbecause in-flight alone cannot say whether 8 concurrent signatures is
comfortable or is the ceiling, and it is emitted even when
0— "unbounded" isa legitimate configured value and looks identical at a glance to a bound that
is simply never reached.
Dependencies: none added, no FIPS trace owed
The bound is a buffered channel of
struct{}.tryLaterusesxocsp.TryLaterErrorResponse, fromgolang.org/x/crypto/ocsp, already adirect dependency and already used by this handler for
MalformedRequestErrorResponseandInternalErrorErrorResponse. The deadlineuses
net/rpcandtimefrom the standard library.git diff origin/main...HEAD -- go.mod go.sumis empty. No module is added,removed or upgraded, so the boringcrypto contract is untouched — no new code
path reaches a crypto primitive that was not already reached.
x/syncwasconsidered for the semaphore and deliberately not used: it is currently an
indirect dependency, and promoting it to direct would have been a dependency
change to justify for something a channel does.
The
RemoteSignerdeadline, and why it is in scopeRemoteSigner.Signwas a barerpc.Client.Callwith no per-call deadline atall, against the default deployment — key isolation is what
openvox-ca servedoes unless told otherwise — whileopenbao.Signer.Signis alreadybounded at roughly 2x its login timeout.
It reads at first like a separate concern that wandered in, and that is worth
answering directly, because it is not. A signing slot can fail to come back
three ways, and this PR closes all three:
net/httprecovers ahandler panic, so the process survives and the slot would be gone for good);
They are one defect class: a slot that leaves the pool and does not return. On a
bound configured at 1 or 2 — which the docs recommend for a constrained remote
signer — any one of them alone wedges issuance permanently. A concurrency bound
whose slots are never released is an outage rather than a limit, so closing two
of the three routes to an identical failure is what could not be justified.
(2) and (3) are complementary rather than redundant: a panic returns the slot
immediately, a wedged signer only after the deadline expires.
The caveat, stated tightly because it should not be oversold:
net/rpchasno cancellation, so the deadline bounds this caller's wait, not the child's
work. The slot comes back; the signer stays wedged. That is why two minutes is
a backstop rather than a tuning — it has to clear the slowest legitimate
signature, which under
ca_key_provider: openbaois the child's own ~2xLoginTimeoutround trip (~20s by default). An operator who raisesLoginTimeoutpast a minute should know this ceiling exists.locking.mdchangesTwo, and the second was deferred at first but is now due and done:
c.mu→signing-slot invariant and why a blocking acquire under
c.muis safe onlybecause the OCSP path sheds and both acquires honour
ctx;Fixed:note, followingthe file's own
#202/#187convention.The strike-through was deliberately not in the first revision:
mainhad no#274 entry to amend at the time — it existed only on #265's branch — so writing
one would have collided in a file four open branches were editing. #265 has
since merged, which made it due, and it landed in the rebase. The retained
analysis below the struck claim names the two clauses in it that are now
history, so it cannot be misread as current state.