Add Redis / Valkey storage backend with optional Sentinel - #12
Merged
Conversation
Abstract the storage layer behind a Backend interface with a logical key namespace (KeyCACert, KeyCAKey, KeyCRL, KeySerial, KeyInventory, KeyInventoryHMAC, KeyHMACKey, KeyCAPubKey, and csr/cert prefixes). Add FilesystemBackend implementing the interface, preserving the existing Puppet CA layout and atomic-write / HMAC semantics. StorageService becomes a facade over Backend, exposing content-oriented methods (GetCACert/SaveCACert/HasCACert, GetCSR/SaveCSR/..., etc.) while keeping legacy *Path() methods for callers that still need filesystem paths via an optional PathProvider interface. Per-subject generated private keys continue to live on local disk through a separate localPrivateKeyDir, independent of the configured backend. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace direct os.ReadFile/os.WriteFile/os.Stat calls against Storage.*Path() with the new content-oriented methods (GetCACert/SaveCACert/HasCACert, GetCAKey/SaveCAKey, GetSerial, TouchInventory, CRLModTime, etc.) so callers no longer assume a filesystem-backed store. The *Path() accessors are still used for log messages and for resolving the optional CA-key passphrase path, which remain filesystem-scoped. Tests in the storage package retain a handful of filesystem-specific assertions (permissions, mtime) that validate the FilesystemBackend directly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement a Backend that stores CA cert/key, CRL, serial, inventory (and its HMAC), CSRs, and signed certificates in an etcd v3 cluster. Per-subject generated private keys continue to live on local disk regardless of the configured backend. Values are wrapped with an 8-byte big-endian unix-nano mtime prefix so ModTime is served from the same key without a second round-trip, keeping If-Modified-Since semantics on GET /crl working. AppendLine uses an etcd Txn guarded on the key's ModRevision with bounded retry, so inventory appends stay atomic across multiple puppet-ca processes sharing a cluster. A BackendSpec + NewServiceFromSpec helper selects the backend at startup; puppet-ca (frontend, signer, and single-process modes) builds the spec from new storage_backend / etcd_* config fields and matching PUPPET_CA_* env vars. Unit tests for the pure helpers (key translation, blob encoding) run by default. An etcd_integration build tag gates round-trip tests that spin up an in-process embedded etcd, including concurrent cross-client AppendLine to prove the Txn-based append holds under contention. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduce an OverlayBackend that wraps any Backend and redirects a fixed set of logical keys to explicit local filesystem paths. Surface it through BackendSpec as CACertFile / CAKeyFile, wired up via new ca_cert_file / ca_key_file config keys, matching PUPPET_CA_CA_*_FILE env vars, and --ca-cert-file / --ca-key-file CLI flags. Combined with the etcd backend this lets operators keep the CA material on a mounted secret volume while CSRs, signed certs, CRL, inventory and serial all live in the shared cluster. Also add --storage-backend, --etcd-endpoints, --etcd-key-prefix CLI flags so the backend choice doesn't require a config file, and document everything in docs/storage-backends.md (key layout, config examples for both backends, security considerations for the CA key in etcd, and pointers for adding further backends). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The key-isolation frontend read the CA cert with os.ReadFile(filepath.Join(absCADir, "ca_crt.pem")), bypassing the storage service and the overlay backend. When the CA cert is mounted via an overlay (e.g. a Kubernetes secret at /run/secrets/puppet-ca-ca/tls.crt), no file exists under absCADir and the frontend failed with ENOENT immediately after a successful PSK handshake with the signer. Defer the handshake and cert read until after the storage service is constructed, then read via store.GetCACert so the overlay is honoured. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Before this change, the etcd backend only had cross-node atomicity for
inventory AppendLine (via a ModRevision-guarded Txn). Everything else
that did a read-modify-write — CRL rotation during Revoke, CSR submission
with autosign, and first-run bootstrap — was protected only by a
process-local sync.RWMutex, so two replicas could race: losing a
revocation when both read the same CRL and wrote their own, issuing two
certs for the same subject, or generating two different CAs from a
simultaneous bootstrap.
Introduce an optional Locker capability on Backend, implemented on the
etcd backend via concurrency.Mutex over a lease-backed session, and
consumed through a new StorageService.WithLock helper that falls back to
a process-local named mutex when the backend does not (or cannot) provide
one. The filesystem backend keeps its single-process semantics
unchanged; OverlayBackend delegates to its base so CA-cert/key overrides
still participate in cluster-wide coordination.
Wire the CA layer through WithLock:
- bootstrap: Init loads without a lock on the fast path; on the slow
path it acquires a bootstrap lock, re-checks, and either loads the
winner's CA or generates a new one.
- crl: Revoke serialises the read-modify-write of the CRL so
concurrent revocations from different replicas don't clobber each
other.
- subject:<name>: Sign, SignWithTTL, and SaveRequest serialise on a
per-subject lock so two replicas can't both pass eviction and
produce duplicate certificates for the same identity.
concurrency.Mutex is not safe for re-entry by multiple goroutines
sharing one session, so the etcd Locker wraps every distributed lock in
a process-local per-name mutex first; unit tests cover the serialisation
invariants under both local fallback and cross-session contention.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Init's fast path runs finishLoadExisting() whenever loadCA() succeeds, which it does even when cert+key are served from a local-file overlay against a freshly-provisioned remote backend. In that setup the CRL, inventory, and serial counter live in the backend and don't exist yet, so loadCRLCache() failed with fs.ErrNotExist and startup aborted — bootstrapCA() never ran because loadCA() had already succeeded. Detect the missing CRL in finishLoadExisting(), take the bootstrap lock, re-check, and then generate an empty CRL signed by the loaded CA key, TouchInventory, and WriteSerial if absent. Skip seeding in frontend-only mode (ExternalSigner != nil) since the signer process owns bootstrapping. Covered by two new specs in internal/ca/ca_test.go: one asserts the supporting state is created on first Init with an overlay-style cert/key, the other asserts Init is idempotent across repeated calls. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a third storage backend alongside filesystem and etcd, backed by a
Redis- (or Valkey-) compatible server. The backend targets clusters that
already run Redis/Valkey as HA infrastructure and want to reuse it for CA
state rather than stand up etcd.
Connectivity supports both direct (redis_addrs) and Sentinel-managed
failover (redis_sentinel_master_name + redis_sentinel_addrs) via
go-redis's FailoverClient, with ACL auth and TLS for both the primary and
the Sentinels.
Storage layout mirrors the etcd backend's shape: values carry an 8-byte
big-endian UnixNano mtime prefix so ModTime is answered from the same
round-trip as the value; atomic cross-replica inventory appends are
performed by a server-side Lua script that reads, strips the old mtime,
appends, and writes back in one step.
Distributed locks — surfaced through the existing Locker capability and
StorageService.WithLock — use the standard Redis recipe: SET NX PX with a
per-acquisition random token, a background heartbeat that extends the
TTL via a token-checking Lua script, and an Unlock that runs the
token-matching delete script so a stale caller cannot release a lock
another holder has since acquired. A per-name process-local mutex wraps
the distributed lock the same way the etcd backend wraps concurrency.Mutex,
because SET NX is not re-entrant from a single client either.
Under Sentinel the replication is asynchronous, so an in-flight failover
can narrow the lock guarantee; this is documented with a pointer to the
etcd backend for operators needing strict linearizability.
Tests:
- 10 unit tests against in-process miniredis (CRUD, ModTime, List,
concurrent AppendLine across two backends, end-to-end via
StorageService, cross-replica lock mutual exclusion / serialisation /
distinct-names / cross-backend, and stale-token-safe Unlock after a
fast-forwarded TTL expiry).
- An opt-in integration suite behind -tags=redis_integration driven by
PUPPET_CA_TEST_REDIS_ADDR that exercises the same behaviours against a
real Redis / Valkey.
Config is exposed via YAML, PUPPET_CA_REDIS_* env vars, and CLI flags
(--storage-backend redis|valkey, --redis-addrs, --redis-sentinel-*,
--redis-key-prefix). docs/storage-backends.md gains a full section with
the key layout, coordination semantics, direct and Sentinel
configurations, and an updated backend comparison table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #11 (distributed locking). Adds Redis (and wire-compatible Valkey) as a third storage backend, sharing the cross-node coordination story with etcd.
Summary
internal/storage/redis.go): stores CA cert/key, CRL, serial, inventory (and its HMAC), CSRs, and signed certs in Redis/Valkey. Keys use the same 8-byte big-endian unix-nano mtime prefix as etcd soModTimeis served without a second round-trip, preservingIf-Modified-SinceonGET /crl.ListusesSCANiteration, notKEYS.AppendLineruns a server-side Lua script that reads the current value, strips the mtime prefix, appends the new line, and rewrites with a fresh mtime in one atomic step — no WATCH/MULTI/EXEC ping-pong needed.Locker. Implemented withSET NX PX+ a per-acquisition random token + a background heartbeat that extends the TTL while the holder is alive + a token-matching Lua unlock so a stale unlock after TTL expiry can't release another holder's lock. BecauseSET NXis not re-entrant from one client, every distributed lock is wrapped in a per-name process-localsync.Mutexfirst — same pattern the etcdLockeruses forconcurrency.Mutexsession re-entry.redis.NewClientor HA failover viaredis.NewFailoverClientwhen a Sentinel master name and sentinel addresses are configured. Both are driven throughredis.UniversalClient, so the backend code path is single.--storage-backend=redis(aliasvalkey),--redis-addrs,--redis-sentinel-master-name,--redis-sentinel-addrs,--redis-key-prefixCLI flags; matchingredis_*YAML config keys; full set ofPUPPET_CA_REDIS_*env vars (addrs, sentinel, auth, DB, key prefix, dial/request timeouts, lock TTL, TLS CA/cert/key files).docs/storage-backends.mdgains a full Redis/Valkey section: key layout, coordination semantics, direct and Sentinel configuration examples, and an updated three-column backend comparison table with a row on cross-node lock guarantees.Notes
--storage-backend=redis(or=valkey) is set.