Skip to content

Bound concurrent CA-key signing, shedding on the OCSP path - #285

Open
bootc wants to merge 15 commits into
mainfrom
fix/ocsp-signing-concurrency-bound
Open

Bound concurrent CA-key signing, shedding on the OCSP path#285
bootc wants to merge 15 commits into
mainfrom
fix/ocsp-signing-concurrency-bound

Conversation

@bootc

@bootc bootc commented Aug 31, 2026

Copy link
Copy Markdown
Member

Bounds concurrent CA-key signing, shared across certificate issuance, CRL
re-signing and the OCSP responder, and configurable via
ca_signing_concurrency. Closes #274.

What this builds on

#265 has merged, and this is rebased onto it. That matters because it is
what makes the bound load-bearing rather than dormant: AnswerOCSP now signs
outside c.mu, so concurrent OCSP requests really do become concurrent
CA-key signatures.

c.mu was, incidentally, the only thing holding CA-key use to one signature in
flight process-wide — every CAKey.Sign site held it. #265 removed that for
OCSP, which was the entire point of that change. What it left is unbounded:
/ocsp is unauthenticated (tierPublic), the only rate limiter in
internal/api is CSR-only, and a cache miss signs, so an unauthenticated
caller can drive as many concurrent signatures as it can open connections
against a signer shared with issuance.

An earlier revision of this PR was written against pre-#265 main, where the
mechanism could not be contended and the concurrency behaviour could not be
tested. It now can be, and is — see internal/ca/signboundrace_test.go, which
holds a real signature open and shows a genuinely concurrent second request
being shed.

Why a shared blocking semaphore would have been wrong

The obvious implementation reinstates the bug #265 exists to fix, by a
different route. issueLeafLocked and signCRLLocked acquire while holding
c.mu
. An anonymous flood owning every slot would block that acquisition,
stalling every c.mu reader behind it — including the revocation check on the
authentication path. That is #197's process-wide stall, arrived at from the
other side.

So the bound is asymmetric:

  • Issuance and CRL re-signing queue. They are authenticated and already
    serialised against each other by c.mu; refusing a certificate a client
    asked for in order to protect an unauthenticated responder would be the wrong
    way round.
  • The OCSP responder sheds, answering RFC 6960 tryLater over HTTP 503
    after a bounded wait. Letting it queue would convert unbounded signing into
    unbounded queueing and bound nothing.

Both acquisitions honour ctx, so a caller that has gone away stops waiting
rather than holding c.mu on nobody's behalf. Remove either property and the
stall is back; the invariant is recorded in docs/development/locking.md under
Lock ordering.

Shedding is unusually cheap here, which is what makes it a real relief valve
rather than a slower route to the same load: an RFC 6960 non-success response
carries no signature, so a refused request costs no CA-key work at all.

The default

max(4, GOMAXPROCS), following the csr_rate_limit sentinel convention
(-1/unset → default, 0 → unbounded, positive → literal) across file, env
and flag.

#265 declined to pick a number because the right size is a property of the
deployment's signer, and that reasoning is preserved here: the default is a
safe ceiling, not a tuning. Its only job is to make the number finite.
Scaling with CPU count suits the two backends where a signature is CPU-bound —
a software key in process, and the default isolated signer, where it is
CPU-bound in the signer child. Operators with a remote signer are told, in the
docs, to lower it to that signer's capacity, which openvox-ca cannot discover.

What this does NOT protect

  • It is per process. N replicas against one shared OpenBao Transit key
    permit N × the limit against that key. Documented in both metrics.md and
    openbao-transit.md.
  • It does not stop the flood. TLS handshakes, goroutines and connection
    memory remain unbounded; only CA-key work is capped.
  • Shedding has a real cost. A verifier that hard-fails on an unavailable
    responder treats sustained tryLater as a revocation-checking outage. This
    converts an unbounded signer load into a bounded, observable, visible
    degradation — that is the trade, and configuration.md says so plainly.
  • A default set too high bounds nothing, and nothing here can detect that.
  • The RemoteSigner deadline bounds the caller's wait, not the signer's
    work.
    net/rpc has no cancellation, so an abandoned call leaves the signer
    child still signing. It stops callers accumulating — which is what makes the
    slots recoverable — but does not reduce load on the signer. Nothing bounds
    that today.
  • An abandoned call also leaves an entry in the RPC client's pending table,
    reclaimed only if a reply eventually arrives. Against a permanently wedged
    signer none does. The bound itself rate-limits this — at most limit calls
    outstanding, each holding its slot for the full timeout, so entries accrue at
    limit/timeout rather than per request — which works out at roughly a
    megabyte a day during an outage someone is already fixing. Reconnecting is
    not available as a remedy: the socketpair fd is inherited once at spawn.
    Recorded in docs/ca-key-security.md rather than engineered around.

Observability

docs/development/locking.md notes that no metric covers this at all, and that
a bound nobody can watch being approached is only half an answer. Three series
land with it: puppetca_ca_signing_in_flight, puppetca_ca_signing_limit and
puppetca_ca_signing_shed_total. The limit is published beside in-flight
because in-flight alone cannot say whether 8 concurrent signatures is
comfortable or is the ceiling, and it is emitted even when 0 — "unbounded" is
a legitimate configured value and looks identical at a glance to a bound that
is simply never reached.

Dependencies: none added, no FIPS trace owed

The bound is a buffered channel of struct{}. tryLater uses
xocsp.TryLaterErrorResponse, from golang.org/x/crypto/ocsp, already a
direct dependency and already used by this handler for
MalformedRequestErrorResponse and InternalErrorErrorResponse. The deadline
uses net/rpc and time from the standard library.

git diff origin/main...HEAD -- go.mod go.sum is empty. No module is added,
removed or upgraded, so the boringcrypto contract is untouched — no new code
path reaches a crypto primitive that was not already reached. x/sync was
considered for the semaphore and deliberately not used: it is currently an
indirect dependency, and promoting it to direct would have been a dependency
change to justify for something a channel does.

The RemoteSigner deadline, and why it is in scope

RemoteSigner.Sign was a bare rpc.Client.Call with no per-call deadline at
all
, against the default deployment — key isolation is what openvox-ca serve does unless told otherwise — while openbao.Signer.Sign is already
bounded at roughly 2x its login timeout.

It reads at first like a separate concern that wandered in, and that is worth
answering directly, because it is not. A signing slot can fail to come back
three ways, and this PR closes all three:

  1. the caller gives up — the ctx-aware acquire;
  2. the signature panics — the deferred release (net/http recovers a
    handler panic, so the process survives and the slot would be gone for good);
  3. the signer never answers — this deadline.

They are one defect class: a slot that leaves the pool and does not return. On a
bound configured at 1 or 2 — which the docs recommend for a constrained remote
signer — any one of them alone wedges issuance permanently. A concurrency bound
whose slots are never released is an outage rather than a limit, so closing two
of the three routes to an identical failure is what could not be justified.

(2) and (3) are complementary rather than redundant: a panic returns the slot
immediately, a wedged signer only after the deadline expires.

The caveat, stated tightly because it should not be oversold: net/rpc has
no cancellation, so the deadline bounds this caller's wait, not the child's
work
. The slot comes back; the signer stays wedged. That is why two minutes is
a backstop rather than a tuning — it has to clear the slowest legitimate
signature, which under ca_key_provider: openbao is the child's own ~2x
LoginTimeout round trip (~20s by default). An operator who raises
LoginTimeout past a minute should know this ceiling exists.

locking.md changes

Two, and the second was deferred at first but is now due and done:

The strike-through was deliberately not in the first revision: main had no
#274 entry to amend at the time — it existed only on #265's branch — so writing
one would have collided in a file four open branches were editing. #265 has
since merged, which made it due, and it landed in the rebase. The retained
analysis below the struck claim names the two clauses in it that are now
history, so it cannot be misread as current state.

@bootc bootc added this to the 1.0.0 milestone Aug 31, 2026
@bootc bootc added bug Something isn't working security labels Aug 31, 2026
@bootc bootc self-assigned this Aug 31, 2026
@bootc
bootc force-pushed the fix/ocsp-signing-concurrency-bound branch from bfc27f1 to a61ec48 Compare August 31, 2026 22:17
@bootc
bootc requested a review from trevor-vaughan August 31, 2026 22:32
@trevor-vaughan-ai

This comment has been minimized.

@bootc
bootc force-pushed the fix/ocsp-signing-concurrency-bound branch from a61ec48 to 48bf166 Compare September 1, 2026 21:54
@bootc

bootc commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Review Council disposition — all 8 findings accepted and acted on

Rebased onto 43c8dd407655 (conflict-free) and pushed as 48bf1661c47f.

Every finding was checked against the code before being accepted. All eight were real. Four new commits.

🟠 HIGH

1. CRL re-sign's queueing path never exercised through signCRLLocked — accepted. Confirmed: SigningConcurrency appeared only in the two new test files, never in any CRL test.

2. applyCAConfig's SigningConcurrency wiring untested — accepted, and the reviewer's framing of the failure mode is the reason it deserved its own spec: drop that line and the field stays at its zero value, which means unbounded — the bound is inert, #274 is wide open, and every other test still passes.

Fixed in test(ca): drive the queueing half through its production call sites and test(cmd): assert ca_signing_concurrency actually reaches the CA.

One correction to the recommendation on the issuance side, because it cannot be done as written: two concurrent issuances cannot contend for a slot. issueLeafLocked runs under c.mu, so c.mu serialises them long before either reaches the bound — a spec racing two Sign/Generate calls would pass with the bound deleted. The contention that can really happen is cross-path: OCSP signs without c.mu and can hold the only slot while an issuance waits. That is what the specs reproduce, and it is the interaction the asymmetric design exists for.

Mutation-tested against the failures the findings name, not just convenient ones:

Mutation Result
drop releaseSigningSlot in signCRLLocked CRL "returns it on success" fails
drop releaseSigningSlot in issueLeafLocked issuance "returns it on success" fails
make the queueing acquire ignore ctx both cancellation specs fail
delete the applyCAConfig assignment 2 of 3 wiring specs fail

The third wiring spec (explicit 0) correctly survives that last mutation — 0 is also the zero value, so it cannot distinguish. Left as-is rather than contrived into failing.

🟡 MEDIUM

3. 🛡️ RemoteSigner timeout leaks its net/rpc pending-call entry — accepted, verified against the stdlib: client.pending[seq] is deleted only when a reply for that sequence arrives or the connection tears down, so a wedged-but-open connection retains it. The regression framing is right too — before the deadline the caller never returned to make a second call.

The recommended remedy is not available here: forcing a reconnect is impossible, because the socketpair fd is inherited once at spawn and consumed by DialConn. There is no second dial.

What does bound it is the bound this PR adds, which I think the analysis missed: every Sign passes through ca_signing_concurrency, so at most limit calls are outstanding and each holds its slot for the full timeout. Entries accrue at limit/timeout, not per request — single digits per minute at the shipped default, a few hundred bytes each, and they drain whenever a reply does arrive. A permanently wedged signer costs on the order of a megabyte a day, during an outage someone is already fixing. Documented with that arithmetic in signer.go and docs/ca-key-security.md rather than engineered around; the "document the residual cost explicitly if accepted as-is" branch of the recommendation.

4. 📚 OpenBao guide claims no CA-side metric exists and 6. ⚙️ OpenBao guide claims the bound is unimplemented, two lines above the bullet documenting it — both accepted, both fixed.

Worth naming rather than quietly correcting: both paragraphs are text this branch inherited when it rebased onto merged #265, and my own edit landed immediately beside them without touching either. The inserted bullet read as an addition when it was also a contradiction.

5. 🧭 PR description understates its own locking.md changes — accepted; the description was written before the rebase and never updated after the strike-through landed. Corrected.

I also found a worse instance of the same defect the finding is about, which it did not flag: the description's opening section still said "this targets the world after #265 lands" and "on today's main the mechanism is dormant". #265 has merged, so both were false. Both sections rewritten.

7. 🧪 Leaf-issuance queueing lacks real-concurrency and cancellation coverage — accepted; covered by the same new file, with the c.mu correction above.

8. 🧪 New Prometheus metrics have no test coverage — accepted. Three specs following collector_test.go's gather/findByLabels pattern, including the case the code comment calls out: the limit is still emitted as 0 when unbounded, since an absent series cannot be alerted on. The shed counter is driven by real contention — a gated key holds a signature open while a second OCSP request is refused — rather than by poking the counter.


Full go test ./..., go vet, gofmt and the push hook's race suite all green locally before pushing.

@trevor-vaughan-ai

This comment has been minimized.

@bootc
bootc force-pushed the fix/ocsp-signing-concurrency-bound branch from 98eeaf3 to c8a3c45 Compare September 2, 2026 18:42
@bootc

bootc commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

Review Council disposition — both findings accepted, and the MEDIUM is under-rated

Rebased onto 12eda0397a1a (picking up merged #289) and pushed as c8a3c4535601. Both findings were verified against the code before being accepted; both were real.

🟠 HIGH — no handler-level test for the 503 shed branch

Accepted. Confirmed: the only cover was ocsperror_internal_test.go, which pins ocspErrorResponse in isolation. It says which constants belong together and executes none of the handler — not the 503 write, not the response body, not the log line.

The new spec fills the bound the way a real caller would rather than by reaching into the CA: one slot, a gated signer parked inside Sign holding it, a second request POSTed through the real mux. It asserts 503 with TryLaterErrorResponse and, explicitly, that the body is neither MalformedRequest nor InternalError — the three say materially different things to a verifier, and only tryLater invites the retry that makes shedding survivable. It then releases the first signature and shows the responder serving normally again, since a refusal that did not recover would be an outage rather than a queue depth.

Demonstrated rather than asserted. A coverage finding normally claims a gap; this one can be measured. Mutating the handler to write an empty body leaves ocsperror_internal_test.go green and turns the new spec red — so the isolated mapping test structurally cannot catch handler regressions. That is the gap stated as a fact about what the existing suite can discriminate.

🟡 MEDIUM — slot release not panic-safe

Accepted, and this is worth more than MEDIUM. The finding left the consequence as an assumption, and it does not hold up as merely hygienic.

There is no recover() anywhere in this repository's non-test code — but net/http's conn.serve recovers a handler panic (server.go:1936), logs it, and drops that one connection. The process survives. So on the two HTTP-reachable sites the slot is not reclaimed by a restart that never happens: it is leaked permanently, in a live server.

The pool is small by design, which sets the blast radius. ca_signing_concurrency defaults to max(4, GOMAXPROCS), but ca.go's field doc, the --ca-signing-concurrency help text and the docs all tell operators with an isolated signer or OpenBao Transit to lower it to the backend's real capacity. A configured 1 or 2 is an ordinary deployment, not a pathological one. There, one panic permanently wedges issuance and CRL re-signing — both queue, and under c.mu — and sheds every OCSP request with tryLater until restart. That is a permanent denial of service inside the control added to bound a denial of service, reached through /ocsp, which is unauthenticated.

Why it was written that way, since the reason matters more than the fix. I chose sequential release deliberately, to hold the slot for the minimum, on the stated belief that a panic mid-signature takes the process with it anyway. That belief was wrong about the runtime, not about the design. Recording it is the durable part: without it the next person to touch this primitive re-derives the same rationale and puts the bug back. It is now in releaseSigningSlot's doc comment as an invariant every future call site has to keep.

Fixed with the closure-with-defer shape, not a function-scoped defer, and that distinction is load-bearing rather than stylistic. signCRLLocked writes to storage after signing; a function-scoped defer would hold a signing slot across that write, so occupancy would measure "sign plus persist" rather than "sign" — inflating it well past the work the bound exists to meter, and on a small configured concurrency that is the difference between shedding correctly and shedding spuriously. This is rule 4 of docs/development/locking.md applied to a new primitive, alongside signing.go's existing "a panic mid-sign still frees the lock rather than wedging the CA".

Three specs pin it, one per call site, each recovering the panic the way net/http would. Each asserts the slot came back and that the CA still works afterwards, since a leak's real signature is the second request failing rather than the first.

Mutations, all against a committed tree

Mutation Result
sequential release in ocsp.go "the slot leaked: this CA can never sign again"
sequential release in crl.go "the slot leaked: every future revocation would hang"
sequential release in signing.go "the slot leaked: issuance would queue for ever"
handler writes an empty body mapping spec green, mux spec red

Rebase note. This now sits on merged #289, which touches internal/metrics/collector.go, docs/metrics.md and docs/configuration.md — all files this PR also edits. The rebase was conflict-free, and I checked that it reverted nothing: zero deletions in all three files, and #289's own change is present (the sweep's per-entry re-sign and its "ran out of budget" path are absent from this tree, as they should be).

Full go test ./..., go vet, gofmt and the push hook's race suite green before pushing.

@bootc
bootc force-pushed the fix/ocsp-signing-concurrency-bound branch from c8a3c45 to 79c20d2 Compare September 2, 2026 22:12
bootc added a commit to bootc/openvox-ca that referenced this pull request Sep 2, 2026
voxpupuli#166, voxpupuli#168, voxpupuli#212 and voxpupuli#265 merged; voxpupuli#284, voxpupuli#285, voxpupuli#288 and voxpupuli#289 added. Eight
entries. Derived independently rather than taken on report, and it agreed.

Ordering from a pairwise conflict matrix rather than from file overlap: the
only pair that actually conflicts is voxpupuli#266 x voxpupuli#282 on magefile.go, so they stay
adjacent and everything else is order-independent. voxpupuli#285 and voxpupuli#289 touch
collector.go and metrics.md in common but merge cleanly against each other,
so pairing them would buy nothing.

Also recorded that membership is the only thing needing an edit here. Heads
move constantly and cost nothing because the entries are refs, not shas, so
'wait until PR X settles before refreshing' answers the wrong question - X
settling changes no membership, and meanwhile the list can be omitting open
PRs, which is the half that makes a green build misleading.
bootc added a commit to bootc/openvox-ca that referenced this pull request Sep 2, 2026
voxpupuli#283, voxpupuli#288 and voxpupuli#289 merged; voxpupuli#294 and voxpupuli#296 added. Seven entries, and main has
moved to 2d11cc0.

Ordering from a pairwise conflict matrix over all six open PRs. Two pairs
actually collide: voxpupuli#266 x voxpupuli#282 on magefile.go, as always, and a new one -
voxpupuli#285 x voxpupuli#294 on cmd/openvox-ca/config.go. Each pair sits adjacent so its
conflict surfaces once; the other two branches are order-independent.

Trial replays that as 2 of 7, matching the matrix. Worth noting voxpupuli#266 is now
56 behind main, by far the stalest entry.
@trevor-vaughan-ai

Copy link
Copy Markdown

🟢 Review Council: APPROVE

Automated LLM review, not a human sign-off. Findings are machine-generated, may contain errors, and are advisory input to human judgment.

Models used:

  • claude-opus-5
  • claude-sonnet-5

Reviewed at commit 79c20d2 (standard effort).

TL;DR: PR #285 passed review with all five personas approving; four findings - three MEDIUM, one LOW - remain unresolved but non-blocking.

Findings: 🔴 0 Critical, 🟠 0 High, 🟡 3 Medium, 🔵 1 Low

Reviewer Verdict Findings
🛡️ Adversary (code) ✅ Approve 2 MEDIUM
📚 Curator (code) ✅ Approve none
🧭 Guard (code) ✅ Approve 1 LOW
⚙️ Operator (code) ✅ Approve none
🧪 Tester (code) ✅ Approve 1 MEDIUM
🟡 MEDIUM (3)
  • 🛡️ Shed path logs one WARN per refused request, letting an unauthenticated caller drive log volume (internal/api/ocsp_handler.go:85)

    slog.Warn("OCSP response shed: CA signing concurrency limit reached",
    

    💡 Recommendation: Route the shed WARN through a threshold/window tracker in the shape of destructiveOpTracker (keyed on clientIP), or log only a periodic summary, so the per-event detail an operator wants during diagnosis cannot be turned into an unbounded log stream by the caller the bound is refusing. Leave puppetca_ca_signing_shed_total as the lossless signal.
    Constraint: OWASP A09:2021 (Security Logging and Monitoring Failures); CWE-779 (Logging of Excessive Data)

    💬 Full reviewer analysis

    The new 503 branch emits an unconditional WARN line, including the client IP, for every shed OCSP request. /ocsp is tierPublic and unauthenticated, and the shed branch is by construction only reachable while the CA is under exactly the concurrent-signature load this bound exists to absorb - so the one moment the control engages is the moment the process starts writing one log record per refused request, at whatever rate the caller can sustain concurrent connections (each shed costs the attacker only the 1s ocspSigningWait per connection). The signal is already carried losslessly by puppetca_ca_signing_shed_total, which the comment two lines above names as the thing to alert on. Searches run: grep -rn "shed\|Shed" --include=*.go internal/api/ (excluding tests) returns only this call site and its comment - there is no sampling or threshold wrapper around it; grep -rn "Sample\|logEvery\|rate.Limiter\|throttl" --include=*.go internal/api/ internal/ca/ (excluding tests) returns nothing. The repository already has the pattern this wants: destructiveOpTracker at internal/api/ratelimit.go:107 is a fixed-window per-identity counter documented as "When a single identity exceeds the threshold within the window, a warning is logged for operational awareness" (NIST 800-53 AU-6). Noted honestly: the pre-existing default branch ("OCSP request error") has the same per-request property for malformed input, so this is a new instance of an existing shape rather than a new class - which is why it is MEDIUM and not higher.

  • 🛡️ net/rpc pending-map leak is justified by a bound that ca_signing_concurrency: 0 removes (internal/signer/signer.go:330)

    The signing bound rate-limits it. Every Sign goes through
    

    💡 Recommendation: Qualify both the comment at internal/signer/signer.go:330-335 and docs/ca-key-security.md:99-103 to state that the accrual bound holds only while ca_signing_concurrency > 0, and that disabling the bound makes the pending-map growth per-request against a wedged signer. If that is considered unacceptable rather than merely undocumented, refuse to leave the isolated-signer path unbounded (reject or clamp ca_signing_concurrency: 0 when ExternalSigner is in use) rather than documenting it.
    Constraint: OWASP A06/CWE-400 (Uncontrolled Resource Consumption); persona rule: verify compliance/bound claims rather than accepting the author's reading

    💬 Full reviewer analysis

    The new two-minute deadline abandons the wait but not the call, leaving an entry in rpc.Client's pending map that is only reclaimed if a reply arrives. The comment justifies that trade with an unqualified claim - "Every Sign goes through ca_signing_concurrency, so at most limit calls can be outstanding at a time ... Entries therefore accrue at limit/timeout, not per request" - and docs/ca-key-security.md:101 repeats it as "bounded at ca_signing_concurrency entries per two minutes, because every signature passes through that bound first." That is false in a configuration this same PR documents as legitimate. ca_signing_concurrency: 0 is described in cmd/openvox-ca/config.go and docs/configuration.md as "0 disables the bound (unbounded signing)", and internal/ca/signbound.go:112-114 makes acquireSigningSlotOrShed return nil immediately when signSlots is nil. With the bound disabled and an isolated signer that is wedged rather than dead - the exact fault the deadline exists for - every cache-missing /ocsp request from an unauthenticated caller starts a Sign that abandons a permanent pending-map entry, so accrual is per request with no CA-side ceiling, not limit/timeout. I traced the path: internal/ca/ocsp.go:367 acquireSigningSlotOrShed -> internal/ca/signbound.go:112-114 (nil signSlots, no-op) -> ocsp.CreateResponse -> RemoteSigner.Sign. This is not a regression (before the deadline each such request leaked a permanently blocked goroutine as well), but the documented escape hatch silently voids the only thing the comment offers as the leak's bound, and an operator reading "bounded at ca_signing_concurrency entries" at a value of 0 gets the opposite of the truth.

  • 🧪 No test proves an OCSP cache hit bypasses a full signing bound (internal/ca/ocsp.go:271)

    	if !hasNonce {
    		if entry, ok := c.ocspCache[serialHex]; ok && time.Now().Before(entry.expiresAt) {
    			c.mu.RUnlock()
    			return OCSPAnswer{DER: bytes.Clone(entry.der), MaxAge: time.Until(entry.expiresAt)}, nil
    		}
    	}
    

    💡 Recommendation: Add a spec (e.g. alongside signboundrace_test.go or ocspshed_test.go) that: primes the OCSP cache for a serial (issue + one successful non-nonce AnswerOCSP call, or a direct cache hit through the handler), occupies the only signing slot, then asserts a second request for the same cached serial still returns 200/Good rather than tryLater - pinning that the cache short-circuit runs before the bound is consulted.
    Constraint: Coverage completeness - boundary-sensitive code the changeset introduced (severity.md: 'Missing edge case coverage for boundary-sensitive code the changeset introduced')

    💬 Full reviewer analysis

    This cache-hit branch returns before acquireSigningSlotOrShed is ever reached (that call is added later in the same function, at what is now line 367), so a cached response should be served even while the signing bound is fully occupied by a concurrent signature. That is a real, load-bearing property: the docs (docs/ca-key-security.md, docs/openbao-transit.md) tell operators the in-process cache is part of what keeps signing load down, and it only helps if cache hits are actually free of the bound. Checked against the three gates before assigning severity: (1) not a pre-existing gap - there was no bound to interact with before this changeset, so 'cache hit under a full bound' is a scenario the diff itself creates; (2) not covered elsewhere - I searched the new signing-bound specs (signbound_test.go, signboundcallsites_test.go, signboundpanic_test.go, signboundrace_test.go, ocspshed_test.go, ocsperror_internal_test.go, signingbound_test.go) and the pre-existing internal/ca/ocsp_test.go (which does have 'serves the cached response on a second call' and 'generates a fresh response (bypasses cache) when a nonce is present', but neither sets SigningConcurrency or occupies a slot) - none combine a populated cache entry with a fully occupied signSlots channel; (3) consequence is real but bounded - a future refactor that moved the slot acquisition ahead of the cache check would silently turn cache hits into 503 tryLater under load, an availability regression on the exact path operators are told to rely on, but it would not corrupt a certificate status or leak key material, so this stays at MEDIUM rather than HIGH.

🔵 LOW (1)
  • 🧭 Init's stated reason for sizing the bound first is not true of its own bootstrap signatures (internal/ca/init.go:101)

    // signs on the bootstrap path, so this has to precede that rather than
    	// merely precede serving.
    	c.initSigningBound()
    

    💡 Recommendation: Reword to state what is actually required - the bound is sized before the CA can serve - and record that Init's bootstrap self-sign and initial-CRL signatures deliberately run outside it (one-shot, before any consumer exists), or route them through acquireSigningSlot so the comment's claim holds.
    Constraint: Structural Coherence

    💬 Full reviewer analysis

    The comment justifies calling initSigningBound at the top of Init on the grounds that Init's bootstrap signing needs the bound already sized. It does not: no bootstrap signature consults signSlots. grep -rn "acquireSigningSlot\|acquireSigningSlotOrShed\|releaseSigningSlot" internal/ --include="*.go" (excluding _test.go) returns acquisitions at exactly three sites - internal/ca/signing.go:562 (issueLeafLocked), internal/ca/crl.go:78 (signCRLLocked) and internal/ca/ocsp.go:367 (AnswerOCSP) - and none in init.go. Init's own signatures are the self-signed CA certificate at internal/ca/init.go:536 (x509.CreateCertificate(rand.Reader, template, template, key.Public(), key)) and the two newEmptyCRL(c.CACert, c.CAKey, ...) calls at internal/ca/init.go:337 and :583, which reach crl.go:220 directly; none of the six signCRLLocked callers (cleanup.go:157, revoke.go:392, crl.go:286, crl.go:422, supersede.go:754) is reachable from Init. The ordering is harmless, but the rationale invites a maintainer to believe bootstrap signatures are metered by the bound and counted in puppetca_ca_signing_in_flight, which they are not - every other statement of the bound's scope in this changeset correctly names only issuance, CRL re-signing and the OCSP responder (internal/ca/ca.go SigningConcurrency doc, docs/metrics.md, docs/configuration.md).


Produced by Review Council, an open-source multi-persona code reviewer. Spot a wrong call or want the source? File feedback or browse the repository.

bootc and others added 12 commits September 4, 2026 16:58
`c.mu` was, incidentally, the only thing holding CA-key use to one signature
in flight process-wide: every `CAKey.Sign` site held it. Moving the OCSP
signature out from under that lock — the point of #197 / PR #265, since
signing under the CA's global write lock stalled every other caller — leaves
nothing bounding it. `/ocsp` is unauthenticated (tierPublic), the only rate
limiter in internal/api is CSR-only, and a cache miss signs, so an
unauthenticated caller can drive as many concurrent signatures as it can open
connections, against a signer shared with issuance.

Add a shared bound: one token per permitted concurrent signature, taken by
every CAKey.Sign site — issuance, CRL re-signing and the OCSP responder
together, because there is one key and what needs bounding is the load on
whatever holds it.

The two halves behave differently under it, and that asymmetry is the
substance rather than a detail. Issuance and CRL work *queues*: it is
authenticated, and refusing a certificate a client asked for in order to
protect an unauthenticated responder would be the wrong way round. The OCSP
responder *sheds*, answering RFC 6960 `tryLater` after a bounded wait.

Shedding is not a stylistic choice there. Letting `/ocsp` queue would convert
unbounded signing into unbounded queueing, bounding nothing — and worse, it
would reinstate the very stall #197 removed, by a different route: issuance
acquires its slot while holding `c.mu`, so an anonymous flood owning every
slot would block a `c.mu` holder and stall every reader behind it, including
the revocation check on the authentication path. Two things stop that. OCSP
never queues, so it cannot build a line an issuance has to join; and both
acquisitions honour ctx, so a caller that has gone away stops waiting rather
than holding `c.mu` on nobody's behalf. Remove either and the stall is back.

Refusing is unusually cheap here, which is what makes it a real relief valve
rather than a slower route to the same load: an RFC 6960 non-success response
carries no signature at all, so a shed request costs no CA-key work.

No new dependency — a buffered channel, and `x/crypto/ocsp` for `tryLater` was
already in the module graph.

The bound is deliberately not sized here. The right number is a property of
the deployment's signer: an in-process software key, an isolated signer over
IPC and an OpenBao Transit key have very different concurrency envelopes, and
non-positive `SigningConcurrency` leaves the bound unset, which every helper
reads as unbounded, so this commit changes no behaviour on its own; the
command layer resolves the shipped default.

On today's `main` the mechanism is dormant by construction: `AnswerOCSP` still
signs under `c.mu`, which serialises it against issuance, so at most one slot
is ever held and the bound is never contended. It becomes load-bearing when

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shed branch added by the signing bound cannot be reached from a spec: on
the state this builds against, every signing slot is taken under `c.mu`, so
nothing can occupy one through the public API and provoke a real refusal.

Extract the error → RFC 6960 mapping so the part worth pinning is testable
without driving the handler into a state it cannot reach. The three outcomes
say materially different things to a verifier — `tryLater` invites a retry,
`internalError` reports a server fault, and `malformedRequest` tells it not to
bother — so which one a given failure produces is a contract, not an
implementation detail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/development/locking.md records, in the gap this closes, that no metric
covers concurrent signing at all — an operator cannot tell "this never happens"
from "this happens throughout every CRL update". A bound nobody can observe
being approached is only half an answer, so the series land with the bound
rather than after it.

Three, because one would not be readable on its own:

  - puppetca_ca_signing_in_flight — signatures in flight now.
  - puppetca_ca_signing_limit — the configured ceiling; 0 means unbounded.
  - puppetca_ca_signing_shed_total — OCSP responses refused with tryLater.

In-flight alone cannot say whether 8 concurrent signatures is comfortable or is
the ceiling, which is why the limit is published beside it rather than left to
be inferred from configuration. The limit is emitted even when it is 0:
"unbounded" is a legitimate configured value and precisely the state worth
alerting on, and it is indistinguishable at a glance from a bound that is
simply never reached.

Emitted alongside the failure counters and ahead of the storage gather, for the
same reason those are: they are in-process state, so an unreadable backend must
not blind an operator to the responder shedding.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wires the CA signing bound to config file, PUPPET_CA_SIGNING_CONCURRENCY and
--ca-signing-concurrency, following the csr_rate_limit sentinel convention
exactly: -1 ("unset") takes the built-in default, an explicit 0 disables the
bound, and positive values pass through. Getting 0 wrong in either direction is
the expensive mistake — rewriting it to the default takes away an operator's
deliberate opt-out, and rewriting the sentinel to 0 would ship unbounded
signing to everyone who never set the key, which is the exposure the bound
exists to close.

The default is max(4, GOMAXPROCS). Scaling with the CPU count suits the two
backends where a signature is CPU-bound — a software key in process, and the
default isolated signer, where it is CPU-bound in the signer child — since past
that point extra concurrency buys latency and memory rather than throughput.
The floor keeps a single-CPU container from getting a bound of 1, which would
serialise issuance behind the OCSP responder.

It is deliberately a safe ceiling and not a tuned value, which is the whole
reason #265 declined to pick a number: the right size is a property of the
deployment's signer, and an in-process key, a PKCS#11 token and an OpenBao
Transit key have very different envelopes. What the default guarantees is only
that the number is finite. Deployments with a remote signer are told, in the
docs, to lower it to that signer's capacity — a thing openvox-ca has no way to
discover. The floor guards the default only; an explicit 1 is honoured, because
a remote signer's capacity is routinely below the local CPU count.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the ca_signing_concurrency key to configuration.md (flag, env and file
rows, plus a section on why the bound exists and how to choose a value), the
three new series to metrics.md, and the lock-ordering invariant to
docs/development/locking.md.

The guidance is written per backend rather than as one number, because that is
the substance of the issue. The default is sized for the isolated signer, where
signing is CPU-bound in the signer child. An OpenBao Transit deployment is told
to set it explicitly: the default derives from the CA host's CPU count, which
has no relationship to what a Transit key — possibly shared with other
consumers — can sustain. Both notes say the bound is per process, so N replicas
permit N times the limit against one shared key.

The isolated signer gets its own subsection in ca-key-security.md rather than a
cross-reference, deliberately. It is the default deployment and the less
bounded of the two backends, since RemoteSigner.Sign waits on the signer
child's reply with no per-call deadline, and documenting the bounded backend
while leaving the unbounded default to a link would be the wrong way round.

metrics.md says plainly that a rising shed counter is not by itself a fault —
it is the bound working, and on an unauthenticated flood it is the protection
working — and gives the two readings that distinguish "the limit is too low"
from "something is driving /ocsp harder than this responder is sized for". It
also suggests alerting on a limit of 0, since unbounded is a legitimate
configured value and looks identical at a glance to a bound that is never
reached.

No Known-gaps entry: on main that list has no #274 entry to amend. The one
describing this exposure exists only on the #265 branch, whose driver has
already linked the issue and dropped the block quote, and duplicating it here
would collide in a file four open branches are editing. Striking it through
falls due when #265 lands. The locking.md change here is confined to a single
additive bullet under Lock ordering.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RemoteSigner.Sign was a bare rpc.Client.Call, which returns when a reply
arrives or the connection breaks. Neither happens when the signer child is
wedged rather than dead, so the frontend waited indefinitely with no per-call
bound. That made the *default* deployment the less bounded of the two backends:
openbao.Signer.Sign already bounds each call at roughly twice its login
timeout, while key isolation — which is what `openvox-ca serve` does unless
told otherwise — bounded nothing.

It is also why a concurrency bound alone does not finish the job #274 opens.
Capping how many callers may be signing at once does nothing if each of them
can wait forever: the slots never come back, and a bound whose slots are never
released is an outage rather than a limit.

Use an asynchronous call and select on a timer. The Done channel is buffered to
depth 1 deliberately — net/rpc discards a reply it cannot deliver, so an
unbuffered channel would drop the answer to any call this stopped waiting for,
and with capacity 1 the client's pending-call entry is always cleaned up even
when nobody is listening.

Two minutes, and the number is a backstop rather than a tuning. It has to clear
the slowest thing the child can legitimately be doing, which is not a local
signature: under ca_key_provider: openbao the child's own Sign is a network
round trip bounded at about 2x LoginTimeout, ~20s by default. Two minutes
leaves room for an operator who has raised that substantially while still
turning an indefinite hang into an error.

What this does NOT do, and the commit should not be read as claiming: net/rpc
has no cancellation, so the deadline bounds *this caller's wait*, not the
signer's work. An abandoned call leaves the child still signing and its reply is
delivered and dropped. It stops callers accumulating against a wedged signer —
which is what makes the concurrency bound's slots recoverable — and it does not
reduce the load on the signer itself. Nothing bounds that today.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The existing specs occupy slots by hand. That states the mechanism — it
refuses when full, it admits again when a slot returns, it distinguishes a
shed from a cancelled caller — but it does not state that the mechanism
engages on the path it exists for. Every one of them would still pass if
nothing ever took a slot during an actual signature.

This spec takes none. It wraps the CA key in a signer that parks inside Sign,
issues one OCSP request to occupy the only slot by signing, and then issues a
second while the first is demonstrably still in there. The second is refused
with ErrSigningBusy; releasing the first returns the slot and a third succeeds.

**It could not have been written before #197.** The OCSP responder used to sign
while holding c.mu, which serialised any two requests before they could reach
the signature — so racing the responder against itself would have passed with
the bound deleted. Signing outside the lock is what makes concurrent signatures
reachable, which is the exposure this bound answers and, now, the thing a spec
can actually observe.

Barrier placement is what gives it the power to fail, and it is the whole
design of the spec: the second request is issued only after the first is
observed *inside* Sign. Fire the two off together and they may simply complete
in sequence, leaving a spec that passes either way. Verified by mutation —
with the shed removed from acquireSigningSlotOrShed, this spec fails.

Unknown serials rather than nonces to force the cache miss: an unknown is never
cached, so every such request reaches the signature, and the request needs
nothing signed to build.

Also corrects two comments that the #265 rebase made false. Both claimed the
shed branch was beyond a spec's reach because every slot is taken under c.mu —
true when they were written, and no longer true of the responder.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, 2 HIGH: the bound's queueing half was tested as a primitive and
through neither of the two call sites that use it. signbound_test.go calls
acquireSigningSlot directly and signboundrace_test.go proves the OCSP *shed*,
so nothing exercised a CRL re-sign or an issuance against a full bound — and
nothing would have noticed a slot acquired and never released on those paths.
That is the failure worth catching: with a small bound, one lost slot makes
every future revocation and CRL update hang, silently and permanently.

Four specs, driving ReissueCRL and Generate against an occupied bound:

  - each queues rather than failing, and returns its slot on success
  - each gives up when its context is cancelled while queued

The cancellation halves are not incidental. A blocking acquire is taken while
holding c.mu, so a caller that has gone away must stop waiting rather than hold
c.mu on its behalf; that ctx is half of what makes the blocking acquire safe at
all, and nothing proved it fired at these sites. Both assert on the call site's
own error text, so they cannot pass by noticing the cancellation somewhere else
on the way in, and the CRL one asserts crlUpdateFailures increments.

What the specs deliberately do NOT do is race two issuances against each other.
They cannot contend: issueLeafLocked runs under c.mu, so c.mu serialises them
long before either reaches the bound. The contention that can really happen is
cross-path — the OCSP responder signs without c.mu and can hold the only slot
while an issuance waits — which is what occupying the bound reproduces, and
which is the interaction the asymmetric design was built around.

Verified by mutation, against the failures the finding names rather than only
ones convenient to catch: dropping either releaseSigningSlot fails the matching
"returns it on success" spec, and making the queueing acquire ignore ctx fails
both cancellation specs.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, HIGH. resolveSigningConcurrency and the file/env layering were
each covered in isolation, but nothing called applyCAConfig and checked the
value arrived — the one line that makes the feature take effect at server
start.

The failure mode is total silence, which is why this is worth its own spec
rather than being implied by the others: drop or misassign that line and
ca.CA.SigningConcurrency stays at its Go zero value, which means *unbounded*.
The entire bound would be inert, the exposure #274 describes would be wide
open, and every existing test — including all of signbound_test.go, which
constructs its own CAs — would still pass.

This is the project's own convention for exactly this class of gap; config.go's
crl_chain_file block carries an It("reaches the CA, which is the step whose
absence is silent") for the same reason, and this follows it.

Three cases, because the interesting ones are not the happy path: an explicit
value arrives intact; the unset sentinel resolves to the built-in default on
the way through rather than to zero; and an explicit 0 survives as unbounded,
since that is an operator's deliberate opt-out and rewriting it would take that
away.

Verified by mutation: deleting the applyCAConfig assignment fails the first two.
The third correctly survives it — 0 is also the zero value, so it cannot
distinguish, and pretending otherwise would be a spec that passes for the wrong
reason.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, MEDIUM: the new series were registered and emitted with nothing
asserting they appear or carry the right values.

Three cases, following collector_test.go's existing gather/findByLabels
pattern:

  - a configured limit reports with nothing in flight and nothing shed
  - the limit is still *emitted* as 0 when signing is unbounded, rather than
    omitted — that is the state an operator alerts on to catch a CA that is not
    bounding its signer at all, and an absent series cannot be alerted on
  - a real signature held open reports in-flight 1, a second OCSP request
    behind it increments the shed counter, and in-flight falls back to 0 once
    the first completes

The third drives the metrics from actual contention rather than by poking
counters: the CA key is wrapped in a signer that parks inside Sign, so the
gauge is read while a signature genuinely holds the only slot. That also pins
the counter to the shed path — it is asserted at 0 in the first case, where
issuance and CRL work have queued through the same bound without shedding.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings, two of them the same defect: prose left behind by the change
that falsified it.

**The OpenBao guide claimed the bound did not exist, twice.** One paragraph
said a configurable bound "would land" in #274 "if one is added" — two lines
above the new bullet telling operators to configure it. The "What to monitor"
bullet further down said there is no CA-side metric for OCSP crowding out
issuance and to watch OpenBao alone, which is now precisely what
puppetca_ca_signing_in_flight, _limit and _shed_total measure. Both are text
this branch inherited when it rebased onto merged #265, and its own edit landed
immediately beside them without correcting either. That is worth naming rather
than quietly fixing: the inserted bullet read as an addition when it was also a
contradiction, and an operator reading the section top to bottom met both.

**The RemoteSigner deadline leaves a pending-call entry behind.** Documented
only as a discarded reply until now, which understated it. net/rpc removes an
entry when a reply for that sequence arrives or when the connection tears down;
against a signer that is wedged rather than dead — the fault the deadline
exists for — neither happens.

The reviewer's suggested remedy, forcing a reconnect, is not available here:
the socketpair fd is inherited once at spawn and consumed by DialConn, so there
is no second dial to flush the map with. What bounds the cost instead is the
bound this PR adds. Every Sign passes through ca_signing_concurrency, so at
most `limit` calls are outstanding and each holds its slot for the full
timeout — entries accrue at limit/timeout rather than per request, single
digits per minute at the shipped default, and they drain whenever a reply does
arrive. A permanently wedged signer leaks on the order of a megabyte a day,
during an outage someone is already fixing.

Recorded rather than engineered around, and the trade said plainly: before the
deadline the frontend accumulated stuck goroutines instead and never got its
signing slots back, which is worse.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review Council, MEDIUM. All three call sites released the slot with a plain
sequential call after the signature, so a panic crossing that call leaked it.

I chose that shape deliberately when writing the bound, to hold the slot for
the minimum, and reasoned that a panic mid-signature would take the process
with it anyway. That reasoning was wrong, and it is worth writing down because
it is what makes this more than hygiene: **net/http's conn.serve recovers a
handler panic**, logs it and drops that one connection. The process survives.
Nothing in this repository calls recover() itself, so it is the standard
library's recovery, not ours, that turns a crash into permanent capacity loss —
and two of the three sites are reachable from an HTTP handler, the OCSP one
from an unauthenticated request an anonymous caller shapes.

The pool is small by design, which sets the consequence. ca_signing_concurrency
defaults to max(4, GOMAXPROCS), but operators running an isolated signer or
OpenBao Transit are explicitly told — in ca.go's field doc, in the flag help and
in the docs — to lower it to that signer's real capacity, so 1 or 2 is an
ordinary setting. There, one leaked slot wedges issuance and CRL re-signing,
which queue and do so under c.mu, and sheds every OCSP request with `tryLater`
until restart. That is a permanent denial of service inside the control added
to bound one, so it is worth more than the MEDIUM it came in as.

Fixed with the closure-with-defer shape rather than a function-scoped defer,
because scope matters here: signCRLLocked writes to storage after signing, and
a defer at function scope would hold a signing slot across that write —
inflating occupancy far past the work the bound exists to meter. This is rule 4
of docs/development/locking.md applied to the new primitive, and signing.go's
"a panic mid-sign still frees the lock rather than wedging the CA" is the same
argument already made for c.mu.

The specs pin it at all three sites with a signer stand-in that panics, and
recover the panic themselves the way net/http would. Each asserts the slot came
back *and* that the CA still works afterwards, since a leak's real signature is
the second request failing rather than the first. Verified by mutation against
a committed tree: restoring any one of the three sequential releases turns its
spec red on the leak assertion.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bootc and others added 3 commits September 4, 2026 16:58
Review Council, HIGH. The shed branch was executed by no test at all. The only
cover was ocsperror_internal_test.go, which pins ocspErrorResponse's mapping in
isolation — it says which constants belong together and executes none of the
handler: not the 503 write, not the response body, not the log line.

That is the wrong thing to leave untested here. What an unauthenticated caller
receives when the CA-key bound is full *is* the security property this change
exists to provide, so it should be asserted on the bytes that reach the wire.

The spec fills the bound the way a real caller would rather than by reaching
into the CA: one slot, a gated signer parked inside Sign holding it, and a
second request POSTed through the mux. It asserts 503 with
xocsp.TryLaterErrorResponse and, explicitly, that the body is neither
MalformedRequest nor InternalError — the three say materially different things
to a verifier, and only tryLater invites the retry that makes shedding
survivable. It then releases the first signature and shows the responder
serving normally again, since a refusal that did not recover would be an outage
rather than a queue depth.

Shaped after the sibling spec in ocsp_test.go, "answers a signer failure with
500 internalError, not 400 malformedRequest", which exists for the same reason
one layer along.

Verified by a mutation the isolated mapping test cannot catch: making the
handler write an empty body leaves ocsperror_internal_test.go green and turns
this one red. That is the coverage gap the finding named, demonstrated rather
than asserted.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First production report on this bound, from a deployment running
ca_key_provider: openbao with a Transit-held key: the default is CPU-shaped and
the work under an external provider is not. Every signature there is a network
round trip, so max(4, GOMAXPROCS) measures this host's cores and says nothing
about what that Transit key — possibly shared with other consumers — can
sustain. On a well-provisioned node it lands far above what the provider wants,
and nothing tells the deployment that has happened.

The guidance already existed: ca.go's field doc, the --ca-signing-concurrency
help text and docs/configuration.md all say a remote signer should be sized
explicitly. What was missing is that the *default* actively points the other way
for exactly those deployments, silently.

Deriving a different default when the provider is remote was the other option
and is worse. It would invent a number for a capacity openvox-ca cannot
discover, which is what #265 declined to do and what resolveSigningConcurrency's
own doc disclaims — the default is a ceiling, not a tuning. So say it once,
plainly, and leave the number to whoever can measure it.

Warned only for a signer reached over the network, and only when the value was
left unset. Not for the isolated signer, which is the default topology: signing
is CPU-bound in the signer child there, so a CPU-derived ceiling is the right
shape and lowering it is a tuning rather than a correction. Not when an operator
set a value, including an explicit 0 — that is a deliberate opt-out of the bound
and nagging about a decision already made is how a warning stops being read.
Serve only; the offline commands share applyCAConfig but sign one certificate at
a time, so the bound never binds there.

The specs assert the quiet cases as well as the loud one, because firing on the
wrong combination is the failure that matters: a warning on every start of the
default topology would be noise, and noise is indistinguishable from a warning
nobody reads.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four findings, none blocking, all worth taking. Two are reasoning defects in my
own prose rather than in the code, which is the shape this branch has produced
repeatedly.

**The pending-map justification had a hole exactly where the bound is
disabled.** The RemoteSigner deadline's comment argues the abandoned net/rpc
entry is bounded because every Sign passes through ca_signing_concurrency, so
entries accrue at limit/timeout rather than per request. But
`ca_signing_concurrency: 0` is a documented, supported value meaning unbounded
signing, and it removes that rate limit entirely — leaving entries bounded only
by in-flight request concurrency, which on the unauthenticated /ocsp path the
caller chooses. Named rather than left for a reader to find: it is a consequence
an operator opts into rather than a defect, but the sentence read as an
unconditional guarantee and the whole argument for leaving the leak in place
rests on it.

**Init's stated reason for sizing the bound first was simply false.** It claimed
Init signs through the bound on the bootstrap path. It does not: the
self-signature calls x509.CreateCertificate directly and the bootstrap CRL goes
straight to storage, so neither takes a slot — verified, Init reaches neither
signCRLLocked nor issueLeafLocked. The ordering is still right, for a different
reason now stated: Init has several exits and only the first statement is on all
of them, and the fast path returns as soon as an existing CA loads, which is
what almost every start does. Sizing placed after it would leave the commonest
deployment unbounded with nothing saying so.

**The per-shed log was an amplifier on an unauthenticated path.** One WARN per
refused request means an anonymous caller chooses how much this CA writes to
disk, turning a request flood into a log flood — amplifying the load the bound
exists to shed. Dropped to Debug, which costs nothing: every one of these is
already counted by puppetca_ca_signing_shed_total, and the metric is what the
docs tell operators to alert on. The line only adds which caller provoked it,
a debugging question rather than a monitoring one.

**And a cache hit bypassing the bound was untested.** That property is what
keeps the bound from being felt in normal service: a cached response returns
under the read lock before the bound is consulted, so ordinary verifier traffic
is answered while the bound is saturated. Without it a full bound would shed
every request rather than only those that would sign, and the shed rate would
stop meaning what the metric claims. Verified by mutation — consulting the bound
before the cache fails the new spec and leaves its two siblings green, so it
discriminates the ordering rather than the bound.

Refs #274

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bootc
bootc force-pushed the fix/ocsp-signing-concurrency-bound branch from 79c20d2 to 089152d Compare September 4, 2026 16:02
@bootc

bootc commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Disposition — all four taken, though the verdict was APPROVE

Rebased onto 2e57bc739df6 and pushed as 089152d1de4e. None of these blocked, and all four were real.

🛡️ The pending-map justification has a hole where the bound is disabled

Accepted, and it is the one I would have most wanted caught. The RemoteSigner deadline's comment argues the abandoned net/rpc entry is bounded because every Sign passes through ca_signing_concurrency — entries accruing at limit/timeout rather than per request. But ca_signing_concurrency: 0 is a documented, supported value meaning unbounded signing, and it takes that rate limit with it. Entries are then bounded only by in-flight request concurrency, which on the unauthenticated /ocsp path is chosen by the caller.

Named in the comment rather than left for a reader to discover. It is a consequence an operator opts into rather than a defect in the opt-out — accepting unbounded concurrent signing is accepting unbounded pending entries, seen from the other side — but the sentence read as an unconditional guarantee, and the entire argument for leaving that leak in place rests on it. A justification with a hole exactly where a supported configuration puts one is worth stating, especially since I argued the panic finding up on the neighbouring ground that a leaked resource is permanent.

🧭 Init's stated reason was false

Accepted — and it was not merely imprecise, it was wrong. The comment claimed Init signs through the bound on the bootstrap path. It does not: the self-signature calls x509.CreateCertificate directly (init.go:536) and the bootstrap CRL goes straight to storage. Verified: Init reaches neither signCRLLocked nor issueLeafLocked.

The ordering is still correct, for a reason now stated instead: Init has several exits and only the first statement is on all of them. The fast path returns as soon as an existing CA loads — what almost every start does — so a sizing placed after it would leave the commonest deployment unbounded with nothing saying so. The bootstrap bypassing the bound is correct rather than an omission, since startup is single-threaded and has nothing to contend with.

⚙️ One WARN per refused request is an amplifier

Accepted. On an unauthenticated endpoint, one WARN per shed means an anonymous caller decides how much this CA writes to disk — a request flood becomes a log flood, amplifying the load the bound exists to shed.

Dropped to Debug, which costs nothing that was being relied on: every shed is already counted by puppetca_ca_signing_shed_total, and that metric is what the docs tell operators to alert on. The line only adds which caller provoked it — a debugging question, so it belongs at the level you turn on to ask it.

🧪 A cache hit bypassing the bound was untested

Accepted. This is the property that keeps the bound from being felt in normal service: a cached response returns under the read lock before the bound is consulted, so ordinary verifier traffic — repeat queries for a handful of serials — is answered while the bound is saturated. Without it, a full bound would shed every request rather than only those that would actually sign, and the shed rate would stop meaning what the metric claims.

Verified by mutation, and the mutation was chosen to discriminate the ordering rather than the bound: consulting the bound before the cache fails the new spec and leaves both its siblings green.


Two of the four were defects in prose rather than code — a justification that did not hold under a supported setting, and a stated reason that was simply untrue. That is the same class as the stale claims this branch has already corrected three times, and it is the class I now check for after every ruling and rebase rather than case by case.

Full go test ./..., go vet, gofmt and the push hook's race suite green before pushing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working security

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OCSP signing is unbounded on the CA key once it leaves the global write lock

2 participants