Conversation
The buffered-push-data notice in isHealthyConn fired unconditionally. With client-side caching every tracked write parks an invalidate frame on idle pooled conns, so at default log level the line floods the log on nearly every pool Get under write-heavy workloads. It documents the normal healthy path, not a problem - gate it behind LogLevelDebug like the equivalent notices in maintnotifications.
…nvalidation batching
Builds on the CSC refresh-on-invalidate + base miss-coalescing PR. Adds:
- Coalescing MODES for the reader-miss path, selected by config:
* "workers" (default): a small pinned worker pool fetches reserved misses.
* "fullduplex": one held connection with a writer + reader goroutine pair
pipelines the reserved misses (concurrent, replies streamed back). Ordering
is preserved: the miss-coalescer's reservation dedups concurrent misses of
the same key to a single fetch, and each caller blocks on its own request's
completion — the caching client is blocking per goroutine, so no goroutine
ever sees its own reads reordered; full-duplex only overlaps independent
goroutines' fetches on the wire.
- Invalidation batching: coalesce invalidation-driven cache deletes within a
configurable window instead of one delete per push.
Config (no environment variables): new AutoPipeline-free Options fields —
ClientSideCacheCoalesceMisses (enable), ClientSideCacheCoalesceMode
("workers"/"fullduplex"), ClientSideCacheCoalesceWorkers, and
ClientSideCacheInvalidationBatchWindow. Prototype-only read-path telemetry
(READPATH_LOG + LocalCache stat counters) removed; no stats surface added.
Measured on a 50ms-RTT WAN proxy under invalidation churn, the coalescing fixes
turn published v9.22.0's miss-stampede collapse (p99 up to ~1.1s, throughput
floored) into ~5x throughput at ~1 RTT p99, matching rueidis; the full-duplex
mode gives the tighter mid-range tail at fewer connections. See
AP_CSC_TWOCLIENT_VS_RUEIDIS.md.
Depends on: feature/csc-refresh-and-miss-coalescing.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb8681e82b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
- lint (the only red CI check): drop the redundant `chan struct{}` type from the
recycle declaration (ST1023) and //nolint:unused the deliberately-off, measured
cscRefreshCooldown knob.
- Reject the "pinned" PROTOTYPE engine from the public ClientSideCacheCoalesceMode
option: it holds a connection with no idle invalidation drain and can serve
stale values (cursor HIGH). It now falls back to "workers" and is reachable
only via an internal benchmark hook (cscForcePinned).
- Pass the new CSC miss-coalescing / invalidation-batching knobs through
UniversalOptions.Simple() (ClientSideCacheRefreshOnInvalidate, CoalesceMisses,
CoalesceMode, CoalesceWorkers, InvalidationBatchWindow) so UniversalClient
users can enable them (codex P2).
- Give the invalidation batcher a stop path: it is stopped and cleared when the
last user releases the binding (releaseLocked), and ensureBatcher refuses to
start one for an already-released binding, so its goroutine no longer lives
past the binding re-arming its timer forever; a later re-acquire starts fresh.
- Full-duplex session: acquire the pool connection only after the first miss
arrives, and release it if CSC serving was disabled meanwhile, so an idle
session no longer starves a small pool (PoolSize:1) until PoolTimeout.
Adds unit tests for the pinned-mode rejection and the UniversalOptions passthrough.
💡 Codex Reviewgo-redis/csc_miss_coalesce_modes.go Lines 362 to 364 in 2f6a9e5  Close the socket when stopping full-duplex sessions
In full-duplex CSC, go-redis/csc_miss_coalesce_modes.go Lines 166 to 168 in 2f6a9e5 Apply the limiter to each full-duplex miss
With go-redis/csc_miss_coalesce_modes.go Lines 373 to 374 in 2f6a9e5 Join the supervisor before re-pooling the connection
After a clean full-duplex recycle, ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
Pull request overview
This PR extends the client-side caching (CSC) subsystem by moving miss-coalescing and invalidation-batching controls into Options/UniversalOptions, adding a full-duplex miss-coalescing engine, and introducing windowed background batching for invalidation-driven deletes.
Changes:
- Add new CSC configuration knobs to
Optionsand propagate them throughUniversalOptions.Simple(). - Introduce selectable miss-coalescing modes (
workersdefault,fullduplex), including a new full-duplex session engine. - Add optional windowed invalidation batching to offload cache deletes from the push-notification read path.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| universal.go | Adds new CSC config fields to UniversalOptions and propagates them into Options via Simple(). |
| options.go | Adds new CSC config fields to Options and documents coalescing/batching behavior. |
| csc_refresh_support.go | Removes env-var support helpers now that options drive behavior. |
| csc_refresh_on_invalidate.go | Removes env-var gates/knobs and switches refresh publishing to tracked main-pool connections. |
| csc_miss_coalesce.go | Refactors miss coalescer to be options-driven and to support multiple engines with shared helpers/stats. |
| csc_miss_coalesce_modes.go | Adds the pinned prototype and full-duplex miss-coalescing engines and related session lifecycle logic. |
| csc_miss_coalesce_modes_test.go | Adds coverage for full-duplex idle push draining correctness (positive + negative control). |
| csc_inval_batch.go | Introduces the windowed background invalidation batcher implementation. |
| csc_integration.go | Integrates invalidation batching into the push invalidation handler and threads the window from Options. |
| csc_coalesce_options_test.go | Adds tests for public mode selection behavior and UniversalOptions.Simple() propagation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f6a9e55ef
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…; doc - Full-duplex readOne now applies the read reply and lets fulfillCached gate only the cache publish on the captured conn id/generation -- matching the workers/pinned engines -- instead of failing the caller with ErrClosed and losing a good reply on a mid-flight id/gen change. - peekAndProcessPushNotifications also drains when the reader has buffered bytes (HasBufferedData), not only when the socket is readable (MaybeHasData), so a buffered invalidation is processed on the idle tick. - Workers-mode doc corrected: a tracked connection is acquired/released per batch, not held across batches.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5a231afa5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (2)
csc_miss_coalesce_modes.go:14
- The file header labels these engines as "PROTOTYPE", but the fullduplex engine is selectable via the public ClientSideCacheCoalesceMode option and described as feature-complete in this PR. This comment is misleading for maintainers/users reading the code.
// Alternate miss-coalescer engines (PROTOTYPE, benchmark comparison).
csc_miss_coalesce_modes.go:134
- The comment says a clean recycle continues immediately, but the code always waits for cscModeBackoff after every session end. Either skip the backoff on clean recycle, or update the comment so behavior and documentation match.
// Session ended on a connection error or a clean recycle. On error, back
// off briefly so a persistent dial failure does not hot-spin; a clean
// recycle continues immediately.
FTHybridCmd.readReply used ReadSlice, which reads a RESP3 map header's length as an element count and consumes only half its frames, leaving the connection desynced. Peek the type and read a map with ReadReply, like the other FT.* parsers.
- A coalesced miss re-runs uncached when CSC serving is disabled after the miss was reserved (RESP3 downgrade / CLIENT TRACKING loss during a conn re-init), via an internal retry-uncached sentinel that processCached catches, instead of surfacing a spurious pool.ErrClosed for a valid cacheable read. - A caller whose context cancels mid-fetch no longer races the coalescer: a CAS interlock (claimAbandon/claimApply) hands the Cmder to exactly one of the caller or the applying worker, so the worker never writes a Cmder the caller has taken back. The reply is still classified and published to the shared cache, and a cancelled Get returns the context error deterministically (matching the non-coalesced path). - A fetch after coalescer shutdown returns pool.ErrClosed instead of hanging on a post-drain enqueue race. Adds TestClassifyCachedReply, TestCSCMissReqClaimInterlock, TestCSCMissCoalesceAbandonedFetchNoRace, TestFullDuplexDisabledMidMissRetriesUncached.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (2)
csc_miss_coalesce_modes.go:216
- sessErr is an atomic.Value that is only Store()'d on I/O failure, but errored()/reasonErr() call Load() unconditionally. atomic.Value panics on Load before the first Store, so the graceful-recycle path can crash the client. Initialize sessErr with a non-nil typed holder (or switch to atomic.Pointer) so Load is always safe.
errored := func() bool { _, ok := sessErr.Load().(error); return ok }
reasonErr := func() error {
if e, ok := sessErr.Load().(error); ok {
return e
}
csc_miss_coalesce.go:25
- The header comment still says miss coalescing is "env-gated", but the gating was moved to Options.ClientSideCacheCoalesceMisses. This is now misleading for readers and users.
// Reader-miss coalescing (PROTOTYPE, env-gated).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5cae5e11b8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ange - A full cache flush (FLUSHDB/FLUSHALL) cleared the cache but left the invalidation batcher's queued per-key deletes, which then fired and evicted entries repopulated after the flush (an extra miss within the window). Drop the batcher's pending queue on flush. - A running batcher's window is fixed at creation, so a second client binding to the same shared handler with a stricter window kept the old cadence and its staleness bound did not hold. setInvalBatchWindow now drops the running batcher on a window change so the next invalidation starts a fresh one with the new window.
setup-go's "1.26.x" resolved to go1.26.5, which govulncheck flags for two standard-library vulnerabilities fixed in go1.26.6: GO-2026-6090 (crypto/tls) and GO-2026-5972 (encoding/asn1). Track the latest stable toolchain so future security patches are picked up automatically instead of pinning a patch.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Suppressed comments (4)
csc_inval_batch.go:60
- The buffered
dropChsignal does not order ahead of the timer case and cannot stop anapplyalready in progress. Around a window boundary,runcan selectt.Cand apply the pre-flush batch aftercache.Flush()(and after a reader repopulates the key), evicting the fresh entry despite this method's guarantee. Coordinate flush/drop with an epoch or mutex and wait for/neutralize any in-progress apply so all pre-FLUSH deletes are definitively superseded.
// Signal run() to clear its in-progress batch; the cap-1 buffer means a signal
// is never lost even if run() is not currently selecting.
select {
case b.dropCh <- struct{}{}:
default:
csc_miss_coalesce.go:336
getConncallsLimiter.Allowonly once for the whole batch, while every request inbatchis a separate client operation. This lets batches bypass per-operation rate/circuit limits and reports only one aggregate result, contrary to theLimitercontract inoptions.go:37-45. Check and report the limiter per request, excluding denied requests from the wire batch; use_getConnfor the underlying batch connection so it does not consume an extra limiter operation.
cn, err := c.getConn(ctx)
csc_miss_coalesce_modes.go:168
- This acquires one limiter permit via
getConnand holds it for the entire full-duplex session, which can execute thousands of independent operations for up to 30 seconds. A concurrency limiter may therefore reject unrelated commands while this permit sits idle, and a rate/circuit limiter never sees individual miss results. ApplyAllow/ReportResultper request and acquire the session connection through_getConninstead.
getCtx, getCancel := opCtx()
cn, err := c.getConn(getCtx)
getCancel()
csc_miss_coalesce_modes.go:158
- Waiting for the first miss does not prevent idle starvation after that miss completes: the session keeps the connection until the 30-second recycle timer. With the valid
PoolSize: 1configuration, a miss succeeds but the caller's next non-cacheable command (for exampleSETorPING) cannot acquire the sole pool turn and times out. Return the session connection once its in-flight queue drains and no miss is queued, or reject/fallback from full-duplex mode when the pool cannot reserve another connection.
// Do not hold a pool connection while idle: wait for the first miss BEFORE
// acquiring. An eagerly-held session connection would, at a small pool
// (PoolSize:1), starve non-cacheable commands (PING/SET/uncached reads) until
// PoolTimeout while the session sat waiting for work. The pulled miss is
// written first by the writer below.
Round-9 review fix (cursor r3811858390). The round-8 D fix let a custom push processor drain on HasBufferedData. But the buffered bytes can be a coalesced reply, not a push. A custom processor invoked then could consume the reply, cache it under the wrong key, and desync the stream. The built-in processor peeks the frame type and consumes only push frames, but the NotificationProcessor interface does not promise that. pushDrainWithin now peeks the frame type for a custom processor and returns without a read when the frame is not a push. The peek is bounded by the closure's hard read deadline.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd9392a25a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Round-10 review fix (codex r3812520847). The round-9 fix peeked the frame type before it handed a buffered frame to a custom push processor, but it returned nil on a peek error. PeekReplyType can partially consume a fragmented RESP3 attribute (DiscardNext) before it errors, which desyncs the stream. Swallowing the error kept the desynced connection, so later fragments could be read as a command reply and cached under the wrong key. The peek error now propagates, so the session fails and drops the connection, as the built-in buffered path does. A cleanly-peeked non-push frame still returns nil (a real reply left for the caller's read).
| // cache. Same policy as drainPushNotifications: fail the | ||
| // session so the connection is closed and removed. | ||
| internal.Logger.Printf(sctx, "csc: miss-coalesce push drain: %v", e) | ||
| return e |
There was a problem hiding this comment.
Coalescer push drain swallows desync
High Severity
The full-duplex miss reader still drains pushes with unbuffered ProcessPendingNotifications. That path breaks out of peek errors instead of returning them, so a mid-frame timeout or attribute discard can leave the stream misaligned. readOne then treats the leftover bytes as the miss reply and applyAndSettle can publish them under the wrong cache key. The idle-tick and refresh paths were already switched to propagate this class of failure.
Reviewed by Cursor Bugbot for commit 4c8c54d. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c8c54d7a9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // Options.init resolves the dial knobs to nonzero defaults. A custom | ||
| // DialerRetryBackoff returning delays beyond DialerRetryTimeout is not | ||
| // observable here; such configs should raise PoolTimeout to match. | ||
| if db := time.Duration(opt.DialerRetries)*(opt.DialTimeout+opt.DialerRetryTimeout) + opt.DialTimeout; db > d { |
There was a problem hiding this comment.
When DialerRetryBackoff returns a delay larger than DialerRetryTimeout—for example, 10 seconds while PoolTimeout remains at its six-second default—this synthetic acquisition deadline expires during the callback-selected backoff, so dialConn exits before completing the configured DialerRetries even when the caller has no deadline. The custom callback is explicitly part of the pool's retry policy; avoid an outer budget that ignores it, such as by using a stop-cancellable context while leaving pool waiting to PoolTimeout, or by accounting for the actual callback delays.
AGENTS.md reference: AGENTS.md:L172-L180
Useful? React with 👍 / 👎.
| for i, r := range buf { | ||
| select { | ||
| case inflight <- r: |
There was a problem hiding this comment.
When an opportunistic batch contains multiple large cacheable commands and the server or a RESP proxy writes each reply before reading the next request, every request is added to inflight only after WithWriter has written and flushed the entire batch. The reader therefore has no request to consume while the writer can block behind a full peer output buffer and the peer waits for that output to drain, causing healthy misses to fail at the write timeout. Make each request visible to the reader before its bytes can be flushed, with coordination for partial write failures, or otherwise avoid writing the whole batch before enabling reply consumption.
Useful? React with 👍 / 👎.
Round-11 review fix (cursor r3813795126). The full-duplex miss reader (readOne) drained pushes with the unbuffered ProcessPendingNotifications for both processor kinds -- a third drain site the earlier desync fix did not reach. The built-in processor there swallowed a mid-frame DiscardNext error on a fragmented attribute, and a custom processor got no frame-type peek, so either could leave the stream misaligned; readOne then read the shifted bytes as the reply and cached them under the wrong key. Both drain paths now share drainPushFrames: the built-in uses the Buffered variant (mid-frame errors propagate) and a custom processor is handed only a confirmed push frame (peek first, propagate the peek error). The helper sets no read deadline, so readOne's following ReadRawReply is unaffected.
| if err := c.pushProcessor.ProcessPendingNotifications(ctx, handlerCtx, rd); err != nil { | ||
| internal.Logger.Printf(ctx, "csc: refresh push drain: %v", err) | ||
| return err | ||
| } |
There was a problem hiding this comment.
Refresh drain swallows mid-frame errors
Medium Severity
The refresh reply reader calls ProcessPendingNotifications instead of drainPushFrames. The non-buffered processor treats a PeekReplyType failure as end-of-pushes and returns nil, even when DiscardNext already consumed part of a RESP3 attribute. Refresh then ReadRawReplys on a desynchronized stream and can publish a push fragment under the wrong cache key, despite the new abort-on-desync comment.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a51876c. Configure here.
Two review fixes. Pre-command push-drain desync (round-13 audit): _process drains pending pushes before executing a command, then reads the reply on the same conn and, on the CSC-miss capture path, caches it. A fragmented RESP3 frame straddling the drain's short hard cap left the reader mid-frame; the error was logged and ignored, so the reply read consumed the residue and could cache it under the wrong key. _process now closes the conn and retries on a fresh one instead of ignoring the error (peekAndProcessPushNotifications already propagated mid-frame errors via drainPushFrames). Refresh reader drain (cursor r3813795126 / r3815669446): the refresh reply reader still drained pushes with the unbuffered ProcessPendingNotifications, a fourth drain site that could swallow a mid-frame desync before ReadRawReply and publish a push fragment under the wrong cache key. It now routes through the shared drainPushFrames helper, like the idle-tick and miss-reader paths.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fa85cda36
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if mc != nil { | ||
| mc.stopWorkers() |
There was a problem hiding this comment.
When refresh-on-invalidate is enabled and the caller drops the client without calling Close, this cleanup stops the miss coalescer and invalidation drainer but never signals cscRefreshHandle.stop. Consequently runCSCRefresher remains blocked on its ticker/queue forever, retaining the base client, cache, and connection pools even after the wrapper is collected. Capture the refresh handle and signal it idempotently from this cleanup as well.
Useful? React with 👍 / 👎.
Round-14 review fix (codex r3817196217). When refresh-on-invalidate is enabled and the caller drops the client without Close, the runtime cleanup stopped the miss coalescer and the invalidation drainer but never signaled the refresh worker. runCSCRefresher then stayed parked on its ticker/queue, holding the baseClient, cache, and pools -- a goroutine and memory leak that defeated the drop-without-Close safety net. cscRegisterCleanups now captures the refresh handle and signals it, idempotent and non-blocking (a new cscRevalidateHandle.signalStop guarded by sync.Once, so Close and the cleanup cannot double-close). stopCSCRefresher uses the same signal.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1168befcf8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if rh != nil { | ||
| rh.signalStop() | ||
| } |
There was a problem hiding this comment.
When two refresh-enabled clients share a cache and push processor and the newest client is garbage-collected without Close, this cleanup stops its refresher but leaves its queue as invalidateHandler.refresh; the drainer's later release() does not clear the stack while another user remains. Invalidations are therefore offered to an unconsumed queue instead of restoring the older live client's queue, silently disabling its refresh-on-invalidate behavior. Fresh evidence beyond the Close-order fix is this GC cleanup path: capture the queue/handler and remove the binding before signaling the worker.
Useful? React with 👍 / 👎.
| // cancellable session ctx under ContextTimeoutEnabled=false, and with | ||
| // per-op timeouts disabled a blocked read/write could then never be | ||
| // interrupted (the supervisor's conn-close is the backstop). | ||
| werr := cn.WithWriter(sctx, c.opt.WriteTimeout, func(wr *proto.Writer) error { |
There was a problem hiding this comment.
When both ReadTimeout and WriteTimeout are configured as -2 (normalized to -1) and ConnMaxIdleTime is at most the 30-second session recycle age, these WithWriter/WithReader calls skip Conn.deadline, which is also the only per-I/O update of usedAt. Under continuous miss traffic the connection is active for the whole session but retains its acquisition timestamp, so the first pool Get after recycling classifies it as idle-expired and closes it, forcing a reconnect every session. Update the usage timestamp explicitly while serving deadline-free batches.
Useful? React with 👍 / 👎.
| if processor, ok := c.pushProcessor.(*push.Processor); ok { | ||
| return processor.ProcessPendingNotificationsBuffered(ctx, handlerCtx, rd) |
There was a problem hiding this comment.
When the built-in processor consumes a push while a coalesced or refresh reply is still pending, ProcessPendingNotificationsBuffered returns as soon as that push leaves no bytes in the reader buffer; it does not wait until a non-push frame has actually been peeked. If another invalidation or maintenance push arrives before the command reply, the immediately following ReadRawReply consumes that push as the request's result, and subsequent in-flight replies become shifted across cache keys. Fresh evidence beyond the fragmented-attribute fix is this clean frame-boundary gap: use a reply-reader mode that continues until it has observed the non-push reply, while retaining buffered-only behavior for idle probes.
Useful? React with 👍 / 👎.
Adds RefreshFailed to CSCRefreshStats -- the refresh-failure counter the HLD lists (go-redis section 4.1). A refresh round trip that errors increments it; those keys stay evicted and a later read repopulates them, so a rising count is the signal that refresh-on-invalidate is degrading to plain eviction. Counted per errored batch, not per key.
The CSC miss and refresh readers drained push notifications with the Buffered variant, which stops the instant the reader buffer empties. If a second invalidation was still on the socket ahead of the command reply, ReadRawReply read that push as the reply and cached it under the wrong key -- a one-frame shift that then cascaded to later replies. The readers now drain in BLOCKING mode: they block on the socket and skip push frames until a non-push frame (the reply) is next, the same non-buffered discipline the full-duplex reader already uses. PeekReplyType is attribute-aware, so a fragmented RESP3 attribute needs no separate buffered scan. A swallowed boundary-peek TIMEOUT is caught by the reader's shared read deadline (ReadRawReply hits the same expired deadline and fails the session); a swallowed non-timeout peek error would need a malformed attribute mid-push (a server protocol bug), so this path is no weaker than the full-duplex reader. Probe and idle paths keep the Buffered variant so they never block. A socket-pair regression test pins the frame order. Also in this change: - GC cleanup unbinds the dropped client's refresh queue from a shared invalidate handler (clearRefreshQueue), so a surviving sibling's refresh-on-invalidate keeps working instead of feeding a stopped queue. A refresh queue is only created inside attachSharedTrackingCSC, which also builds the drain handle, so the handle is always present when the queue is; a Conn() clone bails before the refresher starts. - Record usedAt on deadline-free (negative-timeout) reads and writes, so a long full-duplex session under ReadTimeout/WriteTimeout=-2 is not misjudged as idle-expired by the pool and needlessly reconnected.
| if t != proto.RespPush { | ||
| return nil | ||
| } | ||
| return c.pushProcessor.ProcessPendingNotifications(ctx, handlerCtx, rd) |
There was a problem hiding this comment.
Custom idle drain fatally closes
Medium Severity
On the idle probe path (blocking=false), a custom push processor still goes through ProcessPendingNotifications, which the helper itself documents as blocking until a non-push frame. After an isolated invalidation in the reader buffer, that extra read hits pushDrainWithin's hard deadline. The full-duplex idle tick treats any error as fatal and tears down a healthy session, failing in-flight misses.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 6ad25b6. Configure here.
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ad25b66c2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // cancellable session ctx under ContextTimeoutEnabled=false, and with | ||
| // per-op timeouts disabled a blocked read/write could then never be | ||
| // interrupted (the supervisor's conn-close is the backstop). | ||
| werr := cn.WithWriter(sctx, c.opt.WriteTimeout, func(wr *proto.Writer) error { |
There was a problem hiding this comment.
When sustained coalesced misses overlap a MIGRATING/FAILING_OVER push, this writer calls WithWriter concurrently with the reader's handler calling Conn.SetRelaxedTimeout. That setter publishes the counter and read/write timeout fields through separate atomics, while getEffectiveWriteTimeout reads the write field alone, so an interleaving can still observe the old zero value and arm the normal short write deadline after maintenance has begun. A command written in that window can therefore fail with the spurious timeout that maintenance notifications are intended to prevent; coordinate the timeout update with duplex I/O or publish/read it as one consistent state.
AGENTS.md reference: AGENTS.md:L136-L145
Useful? React with 👍 / 👎.
ofekshenawa
left a comment
ofekshenawa
left a comment
There was a problem hiding this comment.
Overall, the direction makes sense and the coalescing approach looks useful. I left a few inline comments on issues and agreed with several existing comments where the same concerns were already covered:
The reader should be able to process replies before the full batch finishes writing
Maintenance timeout state needs to be updated atomically
A few broader questions about the design:
-
Have we benchmarked mixed workloads with small
GETs and large replies likeHGETALLorJSON.GET? Since replies are processed in order on one session, one large reply can block many small requests behind it. -
The limits are request count based, but request and reply sizes are unbounded. Should there also be a byte based limit to bound memory usage and connection occupancy?
-
One session error can fail a large number of requests together, and these requests may not go through the normal retry path. Is that intentional?
-
Canceled requests may still be sent to warm the cache. What prevents a cancellation storm from filling the queue and session with work that no caller is waiting for?
-
Should
ClientSideCacheInvalidationBatchWindowbe validated againstMaxStaleness?
| } | ||
| cache.Flush() | ||
| case []interface{}: | ||
| // Offload path: enqueue keys to the windowed background batcher instead of |
There was a problem hiding this comment.
When the invalidation queue is full, enqueue calls apply inline and may block on applyMu. On the full duplex connection, this also blocks miss replies. Can overflow stay asynchronous, maybe by signaling the worker to flush the cache?
There was a problem hiding this comment.
Fixed in 9001bd8. On a full invalidation queue, enqueue no longer applies the delete inline on the reader — it parks the key on a spill buffer and nudges the worker, which drains spill through the same seen/pending dedup as the fast path. So a burst collapses to one delete per unique key and no cache work (and no applyMu) runs on the full-duplex reply reader.
Correctness is unchanged: spilled items carry their enqueue-time epoch and are applied under applyMu with the same epoch check as the ch path. No hard cap — distinct invalidations are bounded by the tracked keyset, duplicate keys collapse, and a co-occurring full Flush() skips the whole backlog via the epoch bump; the worker is now panic-safe so it always catches up. Added a spilled counter so the overflow is observable instead of silent, plus tests for overflow-spills-not-inline and dedup-collapse.
| return false, nil | ||
| } | ||
|
|
||
| // The hard-deadline reads below (probe and drain) leave their deadline armed. |
There was a problem hiding this comment.
WithReaderHardDeadline already clears the deadline. This defer can set a relaxed deadline again, which later breaks reads when ReadTimeout < 0. Can we remove this defer or clear the deadline directly?
There was a problem hiding this comment.
Fixed in 9001bd8 — removed the defer. You're right that WithReaderHardDeadline already clears the read deadline on exit (SetReadDeadline(time.Time{})), so the cleanup was redundant; and as written it actively re-armed a relaxed deadline under an active maintenance relaxation: WithReader(ctx, 0) runs getEffectiveReadTimeout, which returns the relaxed value even for a 0 timeout, so it set now+relaxed on the conn — which a ReadTimeout<0 conn then never clears on its next read, causing the spurious timeout you flagged. Dropping the defer relies on WithReaderHardDeadline's own clear.
| return nil | ||
| }); err != nil { | ||
| return err | ||
| // settleErr cancels the reservation and fails one waiting caller. It emits the |
There was a problem hiding this comment.
A canceled request records its cancellation, then the background fetch may record another error using context.Background(). Can abandoned work use a separate metric or skip the command error callback?
There was a problem hiding this comment.
The undercount is fixed in ed5d210: a cancelled coalesced miss now records a cancellation metric in fetch's ctx.Done branch, attributed to the caller's ctx (nil conn), matching processWithRetry.
On the second metric: that's intentional, not a double-count of the same event. When a caller cancels (e.g. a context timeout) the coalescer still completes the in-flight fetch on the real tracked connection to warm the cache. That fetch talks to the server, so any error it returns (WRONGTYPE, a connection/IO failure, etc.) is a genuine server-side outcome we want in the metrics regardless of whether a caller is still waiting. The two records describe two distinct facts:
- the caller's cancellation (context deadline/cancel), attributed to the caller's ctx;
- the background fetch's own server outcome, attributed to
context.Background()+ the serving conn.
So on a timeout/cancellation there may be an additional server-error metric when the background fetch independently fails. We treat that as meaningful signal — a real server error occurred — not noise. Dropping it would hide genuine server failures whenever miss coalescing is enabled, so we keep both.
- spill invalidation overflow off the full-duplex reply reader; the worker drains it through the same seen/pending dedup, so a burst collapses to one delete per key and no cache work runs on the reader - drop the drain deadline cleanup that re-armed a relaxed timeout (WithReaderHardDeadline already restores the deadline) - cap a coalesced write batch by bytes, not just count - warn when ClientSideCacheInvalidationBatchWindow exceeds MaxStaleness Refs #3965
|
Thanks for the review. Addressing the broader questions: Byte-based limit — added. A coalesced write batch is now capped by serialized payload size (
One session error fails many requests, outside the normal retry path — intentional, and it does not actually bypass retry. Replies are matched positionally on the held connection, so any mid-stream desync is unrecoverable — the session must fail whole and drop the conn. But the failed requests re-run: they settle with an internal retry-uncached sentinel that Cancellation storm warming the cache — cache-warming on a cancelled fetch is intentional (the fetch is not wasted) and bounded: the miss queue is capped ( Mixed workloads (small GETs behind a large HGETALL/JSON.GET) — a real property of a single ordered session (positional replies), and a deliberate trade for per-caller ordering without request IDs. Mitigations already in place: latency-first (a lone miss flushes immediately, so light load never batches), and N independent full-duplex sessions ( On the two existing comments you seconded — the reader/writer overlap granularity (reply reading before the full batch finishes writing) and the maintenance-timeout atomicity — I'd like to take both in the feature->master pass rather than widen this PR further. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9001bd8153
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // the item carries its enqueue-time epoch and the worker applies it under | ||
| // applyMu with the same epoch check as the ch path. | ||
| b.spillMu.Lock() | ||
| b.spill = append(b.spill, it) |
There was a problem hiding this comment.
When invalidations arrive faster than the worker can apply them—for example, while it is blocked on a cache shard lock—once the 8,192-item channel fills, every subsequent push is appended to this unbounded slice. Deduplication happens only after takeSpill, so even repeated invalidations for one hot key can grow memory indefinitely and potentially exhaust the process. Deduplicate or cap the spill at insertion time rather than relying on the worker eventually catching up.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 8 total unresolved issues (including 6 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9001bd8. Configure here.
| errorType, statusCode, isInternal := classifyCommandError(applyErr) | ||
| errorCallback(context.Background(), errorType, req.servedBy, statusCode, isInternal, 0) | ||
| } | ||
| } |
There was a problem hiding this comment.
Abandoned fetch double-counts errors
Medium Severity
A cancelled coalesced miss already records a cancellation via the caller's context, but the background settle path still emits a second command-error metric through context.Background() in applyAndSettle and settleErr. Abandoned work therefore inflates error/cancellation rates whenever miss coalescing is enabled.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9001bd8. Configure here.
| if h.batcher != nil { | ||
| h.batcher.stop() | ||
| h.batcher = nil | ||
| } |
There was a problem hiding this comment.
Batcher stop feeds dying refresh queue
Medium Severity
When clearRefreshQueue unbinds the active refresh owner and restores a sibling, it stops the invalidation batcher whose creation-time refresh snapshot still points at the closing queue. The stop-drain then offers hot entries to that dying queue instead of the restored sibling, so shared-cache refresh-on-invalidate can silently drop those refreshes.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9001bd8. Configure here.
0f20382
into
feature/csc-refresh-and-miss-coalescing
Two codex findings on #3989. P1 (csc_refresh_on_invalidate.go:655) — Close's refresher stop-drain used to refetch every buffered invalidated key over the network, bailing only after the first FAILED chunk (#3965 F2). Against a healthy server nothing fails, so that bound never triggered: a full backlog could cost thousands of round trips, one per chunk once the reply-byte budget shrinks each chunk to a single target. Those round trips were always wasted — stopCSCRefresherAndCoalescer's teardown already documented (csc_integration.go:1305-1316, pre-existing) that invalidateAllCoverage evicts every entry this connection would have published immediately after this flush returns, on every teardown path (Close and both self-disable exits share the one deferred cleanup that runs stopCSCRefresherAndCoalescer before invalidateAllCoverage). This finishes that conclusion: flush() now skips the refetch entirely when stopping, abandoning the buffered targets (they stay evicted; a reader repopulates them, same outcome an errored refetch already produced). The now-dead per-chunk stopping branches, the flush return value (never a live signal to any caller once the skip is unconditional), and the single-pass drain's window-cap-only shortfall are all folded in as part of the same change: the h.stop case now loops until q.ch is fully drained rather than leaving up to ~3584 buffered targets pinned in the channel, since c.cscRefreshQueue is never nilled on stop. RefreshFailed is intentionally left untouched by the skip: its doc scopes it to errored round trips, and an abandoned-by-choice target never attempted one. Test: TestCSCRefresherStopDrainSkipsRefetchEntirely, replacing TestCSCRefresherStopDrainBailsOnFirstFailure. Verified against the true pre-fix baseline (not just the changed line): reverting to the pre-existing code makes the erroring pooler's Get run once before the old bail-on-first-failure logic kicks in, which the test catches. P2 (csc_integration.go:709) — attachSharedTrackingCSC starts the drainer before startCSCRefresher/startCSCMissCoalescer publish their queue/handle. In principle the drainer's self-disable tick (reacting to an async conn init's disableCSCServing) could observe cscActive false and run stopCSCRefresherAndCoalescer, consuming workersStopOnce, before either start function publishes — leaving that worker with no remaining path to be stopped. This interleaving is not reachable today: cscMinDrainInterval floors the first tick at 1ms against synchronous, microsecond-scale construction, and nothing external holds a reference to call Close mid-construction. That is a timing-dependent invariant, not a structural one, so replace it with an explicit mechanism instead of only documenting it: cscDrainHandle gains workersMu/workersTornDown. stopCSCRefresherAndCoalescer's teardown body sets workersTornDown=true as its first action under the lock; each start function checks the same flag under the same lock immediately before publishing and declines to start (launching nothing) if teardown already ran. startBackgroundDrainer needs no matching guard: it is what allocates cscDrainHandle, so no teardown path can observe a non-nil handle to race against until after it returns. Rejected fixing this by reordering attachSharedTrackingCSC to start the workers before the drainer, per codex's literal suggestion: that would leave the workers unstoppable by Close if anything panicked between the reordered start calls and the drainer's construction, since stopBackgroundDrainer no-ops on a nil cscDrainHandle. The current order's panic-safety property is worth keeping; the lock-based guard closes the race without touching it. Tests: TestCSCConstructionRaceDeclinesToStartAfterTeardown (drives workersTornDown directly, independent of cscActive; verified to fail if either start function's guard is reverted), TestCSCConstructionWinsRaceStartsNormally (control), TestCSCStopCSCRefresherAndCoalescerSetsWorkersTornDown (plumbing).
* feat(csc): refresh-on-invalidate and reader-miss coalescing
Two additions over the shared-tracking client-side cache, both inert
unless CSC is enabled.
Refresh-on-invalidate re-fetches recently-read keys as soon as their
invalidation push arrives, instead of leaving them to be reloaded by the
next reader that misses. Invalidated cache keys that were valid and read
within the recency horizon are collected under the shard lock as they are
deleted, batched over a fixed window from the first collect, and flushed
when the window expires, when a size cap is hit, or on demand the moment a
reader misses a key still sitting in the window. Refetches reserve each
cache key before touching the network and release any reservation they do
not publish, so a reader never blocks on an orphaned in-progress entry.
Reads run on a background pipelined connection. Enabled via
Options.ClientSideCacheRefreshOnInvalidate; window and demand behaviour
tunable through GOREDIS_CSC_REFRESH_WINDOW_MS and GOREDIS_CSC_REFRESH_DEMAND.
Reader-miss coalescing pipelines concurrent cache-miss reads onto a single
CSC-tracked main-pool connection. Each miss is sent as the caller's own
per-key command (not rewritten to MGET, so it is cluster-safe); every reply
is applied to its caller's command and published to the cache under the
connection's captured tracking generation, so entries stay invalidatable.
initConn issues CLIENT TRACKING ON for every pooled connection while CSC is
active, so the main pool is tracked; using it explicitly (rather than the
pipeline pool) keeps the path correct even if a later change makes a
dedicated pipeline pool deliberately untracked, since publishing on an
untracked connection would serve stale until TTL. Enabled via
GOREDIS_CSC_COALESCE_MISSES; worker count via GOREDIS_CSC_COALESCE_WORKERS
(default 8).
Adds LocalCache.deleteByRedisKeyCollectingHot / LRUClock / InvalidationStats
and per-cache invalidation counters to support the above, with unit coverage
of the recency filter and an integration test asserting a refreshed entry is
itself invalidatable.
* feat(csc): full-duplex reader-miss coalescing + invalidation batching (#3965)
* docs: point security reports at Redis VDP (#3949)
* fix(pool): log buffered push data at debug level (#3948)
The buffered-push-data notice in isHealthyConn fired unconditionally.
With client-side caching every tracked write parks an invalidate frame
on idle pooled conns, so at default log level the line floods the log
on nearly every pool Get under write-heavy workloads. It documents the
normal healthy path, not a problem - gate it behind LogLevelDebug like
the equivalent notices in maintnotifications.
* feat(csc): reader-miss coalescing modes (workers + full-duplex) and invalidation batching
Builds on the CSC refresh-on-invalidate + base miss-coalescing PR. Adds:
- Coalescing MODES for the reader-miss path, selected by config:
* "workers" (default): a small pinned worker pool fetches reserved misses.
* "fullduplex": one held connection with a writer + reader goroutine pair
pipelines the reserved misses (concurrent, replies streamed back). Ordering
is preserved: the miss-coalescer's reservation dedups concurrent misses of
the same key to a single fetch, and each caller blocks on its own request's
completion — the caching client is blocking per goroutine, so no goroutine
ever sees its own reads reordered; full-duplex only overlaps independent
goroutines' fetches on the wire.
- Invalidation batching: coalesce invalidation-driven cache deletes within a
configurable window instead of one delete per push.
Config (no environment variables): new AutoPipeline-free Options fields —
ClientSideCacheCoalesceMisses (enable), ClientSideCacheCoalesceMode
("workers"/"fullduplex"), ClientSideCacheCoalesceWorkers, and
ClientSideCacheInvalidationBatchWindow. Prototype-only read-path telemetry
(READPATH_LOG + LocalCache stat counters) removed; no stats surface added.
Measured on a 50ms-RTT WAN proxy under invalidation churn, the coalescing fixes
turn published v9.22.0's miss-stampede collapse (p99 up to ~1.1s, throughput
floored) into ~5x throughput at ~1 RTT p99, matching rueidis; the full-duplex
mode gives the tighter mid-range tail at fewer connections. See
AP_CSC_TWOCLIENT_VS_RUEIDIS.md.
Depends on: feature/csc-refresh-and-miss-coalescing.
* fix(csc): address #3965 review and lint in coalescing modes
- lint (the only red CI check): drop the redundant `chan struct{}` type from the
recycle declaration (ST1023) and //nolint:unused the deliberately-off, measured
cscRefreshCooldown knob.
- Reject the "pinned" PROTOTYPE engine from the public ClientSideCacheCoalesceMode
option: it holds a connection with no idle invalidation drain and can serve
stale values (cursor HIGH). It now falls back to "workers" and is reachable
only via an internal benchmark hook (cscForcePinned).
- Pass the new CSC miss-coalescing / invalidation-batching knobs through
UniversalOptions.Simple() (ClientSideCacheRefreshOnInvalidate, CoalesceMisses,
CoalesceMode, CoalesceWorkers, InvalidationBatchWindow) so UniversalClient
users can enable them (codex P2).
- Give the invalidation batcher a stop path: it is stopped and cleared when the
last user releases the binding (releaseLocked), and ensureBatcher refuses to
start one for an already-released binding, so its goroutine no longer lives
past the binding re-arming its timer forever; a later re-acquire starts fresh.
- Full-duplex session: acquire the pool connection only after the first miss
arrives, and release it if CSC serving was disabled meanwhile, so an idle
session no longer starves a small pool (PoolSize:1) until PoolTimeout.
Adds unit tests for the pinned-mode rejection and the UniversalOptions passthrough.
* fix(csc): FD apply-and-gate on id/gen mismatch; drain buffered pushes; doc
- Full-duplex readOne now applies the read reply and lets fulfillCached gate
only the cache publish on the captured conn id/generation -- matching the
workers/pinned engines -- instead of failing the caller with ErrClosed and
losing a good reply on a mid-flight id/gen change.
- peekAndProcessPushNotifications also drains when the reader has buffered bytes
(HasBufferedData), not only when the socket is readable (MaybeHasData), so a
buffered invalidation is processed on the idle tick.
- Workers-mode doc corrected: a tracked connection is acquired/released per
batch, not held across batches.
* fix(search): read the full RESP3 map reply in FTHybridCmd (#3956)
FTHybridCmd.readReply used ReadSlice, which reads a RESP3 map header's
length as an element count and consumes only half its frames, leaving the
connection desynced. Peek the type and read a map with ReadReply, like the
other FT.* parsers.
* fix(csc): retry uncached on mid-miss disable; guard cancelled fetches
- A coalesced miss re-runs uncached when CSC serving is disabled after the miss
was reserved (RESP3 downgrade / CLIENT TRACKING loss during a conn re-init),
via an internal retry-uncached sentinel that processCached catches, instead of
surfacing a spurious pool.ErrClosed for a valid cacheable read.
- A caller whose context cancels mid-fetch no longer races the coalescer: a CAS
interlock (claimAbandon/claimApply) hands the Cmder to exactly one of the caller
or the applying worker, so the worker never writes a Cmder the caller has taken
back. The reply is still classified and published to the shared cache, and a
cancelled Get returns the context error deterministically (matching the
non-coalesced path).
- A fetch after coalescer shutdown returns pool.ErrClosed instead of hanging on a
post-drain enqueue race.
Adds TestClassifyCachedReply, TestCSCMissReqClaimInterlock,
TestCSCMissCoalesceAbandonedFetchNoRace, TestFullDuplexDisabledMidMissRetriesUncached.
* fix(csc): drop batched deletes on flush; rebuild batcher on window change
- A full cache flush (FLUSHDB/FLUSHALL) cleared the cache but left the
invalidation batcher's queued per-key deletes, which then fired and evicted
entries repopulated after the flush (an extra miss within the window). Drop the
batcher's pending queue on flush.
- A running batcher's window is fixed at creation, so a second client binding to
the same shared handler with a stricter window kept the old cadence and its
staleness bound did not hold. setInvalBatchWindow now drops the running batcher
on a window change so the next invalidation starts a fresh one with the new
window.
* ci(govulncheck): use stable Go to pick up security patches
setup-go's "1.26.x" resolved to go1.26.5, which govulncheck flags for two
standard-library vulnerabilities fixed in go1.26.6: GO-2026-6090 (crypto/tls)
and GO-2026-5972 (encoding/asn1). Track the latest stable toolchain so future
security patches are picked up automatically instead of pinning a patch.
* fix(csc): preserve queued deletes on rebuild; honor configured timeouts
- The invalidation batcher's stop path drains keys still buffered in its channel
into the final flush, so a window-change rebuild loses no queued deletes
(they would otherwise serve pre-invalidation values until TTL/MaxStaleness).
Adds TestInvalBatchStopAppliesQueuedDeletes.
- ensureBatcher reads the window under the handler lock (not the caller's
pre-lock snapshot) and returns nil at window 0, so a concurrent tighten or
disable cannot be undone by a batcher rebuilt with the stale cadence.
- setInvalBatchWindow folds windows strictest-wins across attached clients
(explicit 0/inline strictest, then smaller nonzero): a later, looser attach
can no longer lengthen batching past an earlier client's staleness bound.
The effective window resets when the last user releases.
- The miss-coalescer flush budget derives from the configured WriteTimeout +
ReadTimeout (plus a 5s pool-Get floor) instead of a fixed 5s, so clients with
deliberately long timeouts do not see only coalesced misses clipped early.
* fix(csc): batcher stop/release safety; FD idle release; flush budget
- enqueue/stop interlock: enqueue sends under stopMu.RLock and stop flips a
stopped flag under stopMu.Lock before closing stopCh, so a key sent by a
handler holding a stale batcher pointer lands before the close (applied by the
stop-drain) and a post-stop enqueue applies inline — no delete is parked in a
channel nothing drains. Adds TestInvalBatchEnqueueAfterStopAppliesInline.
- The batcher snapshots cache/refresh at creation and apply uses the snapshot,
so the release-time stop-drain still lands queued deletes on a shared cache
after releaseLocked has cleared the binding (a successor reusing that cache
would otherwise serve pre-invalidation values until TTL/MaxStaleness).
- Full-duplex coalescer sessions release their held connection after an idle
grace (cscFullDuplexSessionIdle, 1s) instead of holding it toward the 30s
recycle age: at PoolSize:1 a non-cacheable command otherwise blocks until
PoolTimeout. The next miss re-acquires; an idle coalescer holds zero
connections. Adds TestFullDuplexSessionReleasesIdleConn.
- batchBudget includes the configured PoolTimeout alongside Write/ReadTimeout
(5s floor for scheduling only), so a coalesced flush's pool Get waits the same
saturation budget as the normal path, including ReadTimeout: -1 setups.
* fix(csc): close coalescer lifecycle races; honor pool budget on acquire
- cscMissCoalescer is an atomic.Pointer: processCached loads it once (a cache
miss racing Client.Close could observe non-nil then call fetch on nil — a
panic and a data race); stop Swaps to nil so exactly one closer wins.
- Connection acquisition in all three engines goes through acquireCtx:
max(5s, PoolTimeout) so a configured pool budget above 5s is honored under
temporary saturation (matching the normal command path), and cancelled when
the coalescer stops so Close is not stalled behind a blocked pool Get. The
workers engine's per-batch Get is split out of the batch write/read budget
(batchBudget covers write+read only again).
- The runtime cleanup for a client dropped without Close also stops the
coalescer workers (idempotent stopWorkers), so a forgotten client no longer
leaks the worker goroutines and everything they retain (cache, pools).
- Document that ClientSideCacheCoalesceMisses requires the built-in LocalCache;
with a custom Cache implementation the option is ignored.
* fix(csc): coalescer latency, shutdown and clone fixes
- fullDuplexLoop backs off only after an error end; a clean idle/recycle end
starts the next session immediately instead of adding a flat 5ms to a miss
already queued.
- flushBatch no longer routes the engine's bounded batch ctx through
c.context(): with the default ContextTimeoutEnabled=false that stripped the
deadline, and with ReadTimeout disabled a non-replying server wedged the
worker in ReadRawReply and Close in wg.Wait.
- baseClient.clone carries the miss coalescer (atomic.Pointer Store of the
owner's Load), so WithTimeout clones keep coalescing instead of silently
falling back to per-caller fetches; lifecycle stays owner-only.
- Document that ClientSideCacheInvalidationBatchWindow requires the built-in
LocalCache (matching the miss-coalescing option).
* fix(csc): make FD session I/O interruptible; refresh rebuild; stat CAS
- The full-duplex session writer/reader pass the engine's cancellable session
ctx directly instead of through c.context(), which stripped it under the
default ContextTimeoutEnabled=false; and since no context can interrupt a
deadline-less socket read, the supervisor's stop path now bounds the graceful
drain — after mc.stop it waits up to the batch budget and then closes the held
connection. Close always unblocks: promptly on a healthy drain, within the
budget against a server that never replies.
- setRefreshQueue drops a running batcher when the refresh binding changes
(like setInvalBatchWindow), so a client attaching refresh-on-invalidate later
does not leave batched deletes permanently feeding a nil refresher.
- countBatch's max-batch statistic uses a CAS loop, so concurrent workers cannot
commit a smaller maximum over a larger one.
* fix(csc): join FD supervisor before release; drain queue in GC cleanup
- The FD session joins its supervisor goroutine (supDead) after closing
superDone and BEFORE releasing the connection: the deferred scancel makes
sctx.Done() ready at return, and a still-parked supervisor could pick it over
the equally-ready superDone and close a healthy connection already returned to
the pool — possibly under an unrelated command.
- The GC cleanup for a client dropped without Close drain-cancels the miss queue
after stopping the workers (non-blocking), so a ctx-cancelled caller's queued
request cannot strand an IN_PROGRESS reservation that would block another
client sharing the injected cache until StaleTimeout.
* fix(csc): clear the push-probe deadline for negative ReadTimeout
A negative ReadTimeout means every WithReader skips SetReadDeadline, so the 1ms
deadline peekAndProcessPushNotifications installs for its probe stayed on the
socket and poisoned the next reply read (a spurious i/o timeout on a held
full-duplex connection, or on a pooled conn reused right after a drain). Clear
the deadline after the probe for negative-ReadTimeout clients; timeout>=0
callers re-arm their own on the next read. Fixed at the source, covering the
FD coalescer tick, the autopipeline paths and the CSC hit-drain.
* refactor(csc): full-duplex is the only miss-coalescer engine
Misses are caller-blocking — a real request waits on every fetch — so the
latency-first engine is the right one, always: a lone miss is written
immediately, batching stays opportunistic (packs only what is already queued),
and new misses stream out while earlier replies are in flight (~1 RTT per
miss, no batch phase-lock, no pool Get on the hot path).
Remove the alternatives and their configuration surface:
- the "workers" engine (N pooled conns, half-duplex batches): its per-batch
round trip added a full RTT of tail latency to misses arriving mid-flight,
and its batching advantage is preserved by the FD writer's opportunistic
packing;
- the "pinned" benchmark prototype (no idle invalidation drain; could serve
stale after an invalidation while idle; never publicly selectable);
- Options.ClientSideCacheCoalesceMode and ClientSideCacheCoalesceWorkers (both
unreleased experimental fields) and their UniversalOptions mirrors:
ClientSideCacheCoalesceMisses now selects the one engine, no tuning needed.
CSCMissCoalesceStats.Mode becomes Active. Background invalidation traffic is
unaffected: it keeps the batching-first windowed batcher and refresh queue.
* fix(csc): held-conn probe safety and lifetime bounds
- checkForData no longer resets the connection deadline: the probe is a raw
MSG_PEEK|MSG_DONTWAIT syscall that never touches net.Conn deadlines, and on a
held full-duplex connection the reader's idle tick ran it CONCURRENTLY with
the writer inside WithWriter — stripping the armed write deadline and letting
a blocked write hang past its timeout. connCheck's own reset stays (it runs
on idle pool-owned conns).
- peekAndProcessPushNotifications falls back to a throttled timed drain
(TakeCscPeriodicReadPending, the CSC drainer's opaque-transport discipline)
when socket readiness cannot be inspected (Windows; dialer wrappers exposing
neither syscall.Conn nor NetConn), so a push addressed to the held connection
is consumed within the fallback interval instead of sitting until the next
reply.
- The full-duplex session's hold is bounded by the connection's REMAINING
absolute lifetime via a new pool.Conn.ExpiresAt() accessor (jitter included),
floored at 10ms — a near-expiry pooled connection is no longer held for a
fresh full recycle age past the expiry the pool's reaper enforces.
* fix(csc): scope idle-drain fallback to held conns; drainer deadline hygiene
- The opaque-transport timed-drain fallback moves out of the shared
peekAndProcessPushNotifications and into the full-duplex session tick, the
one place a connection has no alternative drain coverage. The blanket
fallback made every push path run throttled timed drains on uninspectable
transports (including mock connections), perturbing sequenced push handling
— the maintnotifications downgrade test caught it in CI. Pooled push paths
keep their drainer/health-check coverage. The timed drain itself is factored
into timedPushDrain, shared by both callers.
- drainPushNotifications explicitly clears the socket deadline after its
hard-deadline probe/drain for clients with a negative ReadTimeout (which
never re-arm): the SetDeadline reset removed from checkForData — because it
clobbered a concurrent writer's deadline on held full-duplex connections —
had been clearing this residue incidentally on the next drainer tick. This
is the explicit, correctly-scoped replacement.
* refactor(csc): drop the public coalescer stats API
Observability rides the client's normal telemetry — coalesced misses fire the
otel operation-duration and error callbacks like any other command path — so
CSCMissCoalesceStats (an unreleased experimental type) duplicated that surface
for no consumer. The engine's internal counters stay for the in-package tests,
which read them directly off the coalescer.
* refactor(csc): drop the public coalescer stats API
Observability rides the client's normal telemetry — coalesced misses fire the
otel operation-duration and error callbacks like any other command path — so
CSCMissCoalesceStats (an unreleased experimental type) duplicated that surface
for no consumer. The engine's internal counters stay for the in-package tests,
which read them directly off the coalescer.
Also: acquireCtx folds in the pool's full dial budget (DialerRetries x
(DialTimeout + DialerRetryTimeout) + DialTimeout) so a slow endpoint gets its
configured dial-retry sequence instead of being cancelled mid-sequence; and the
ClientSideCacheCoalesceMisses GoDoc documents pool sizing for the held
connection (PoolSize 1 + sustained misses starves non-cacheable commands).
Also compacts the review-round comments in the touched files (invariants and
contracts preserved; narration removed).
* fix(csc): bound the recycle-path drain like the stop path
Reaching the recycle age signaled the recycle and exited the supervisor while a
reader could stay blocked in a deadline-less read on a non-replying server —
with the supervisor gone nothing would ever close the conn, and a later Close
hung in wg.Wait. The recycle path now waits up to the batch budget for the
session to drain and then closes the connection, mirroring the stop path. Also
documents that a custom DialerRetryBackoff exceeding DialerRetryTimeout should
raise PoolTimeout (an opaque callback's total delay is not observable from
options).
* fix(csc): progress-based drain backstop; departial test flake
- The stop/recycle drain backstop force-closed the conn after one fixed
batch budget, which could cut a HEALTHY drain of a deep in-flight
pipeline (up to cscFullDuplexDepth replies, ~in-flight x RTT,
possibly under maintenance-relaxed timeouts). The supervisor now
samples the in-flight queue per budget interval: any consumed reply
extends the wait; only a zero-progress interval closes the conn.
- TestReleaseConnRemovesConnectionAfterPartialPushRead skips the benign
case where the drain probe times out before consuming any byte: the
frame is intact and re-pooling is safe; only a Put after bytes moved
into the reader is the desync bug the test pins.
* fix(pool): clear residual read deadline in checkForData
Removing the SetDeadline reset left a prior WithReader's read deadline
armed; once expired, rawConn.Read fails fast with i/o timeout before
the non-blocking peek runs, the CSC drainer treats the error as fatal
and removes an idle-but-healthy conn, evicting its cache coverage. The
full-duplex coalescer concentrates coverage on one held conn, so one
spurious removal wiped the whole cache (WAN: hit rate 25%, throughput
cut to a third). Clear only the READ deadline: the write deadline stays
untouched because checkForData runs concurrently with command writes on
a held full-duplex connection - which is what the old full SetDeadline
reset clobbered. Regression test pins the expired-deadline case.
* fix(csc): claim FD writes; progress-aware drain backstop
- The abandon interlock gated only the reply side, so the session
writer could serialize an abandoned caller's cmd args - a
use-after-return on mutable args (e.g. a []byte key), with the reply
publishable under the original cache key. The writer now claims each
request (PENDING->WRITING) around arg serialization, releases before
the flush, and drops abandoned requests with their reservation
cancelled; abandonOrWait yields through a WRITING claim, which spans
one in-memory batch encode.
- The drain backstop measures progress as completed reads (readsDone)
instead of len(inflight), which was blind to the reader's active
read, and its interval honors the connection's effective
(maintenance-relaxed) read timeout via pool.Conn.EffectiveReadTimeout
so a legitimately deep read is not cut short.
- The partial-push-read test completes the frame and requires a whole-
frame parse before skipping: an empty reader buffer alone does not
prove the probe consumed nothing.
* fix(csc): snapshot miss wire form at enqueue
Replace the WRITING ownership claim with a wire snapshot taken in
fetch, while the caller still owns cmd: the session writer writes only
engine-owned bytes and never reads cmd, so no ownership window spans
arg serialization (which could implicitly flush and block on the
socket), abandoning callers return immediately, and a post-abandon arg
mutation can neither reach the wire nor publish under the original
cache key. Abandoned fetches complete again, so their reservations
settle instead of stranding IN_PROGRESS behind the miss backlog.
Also: fetch's Close-race branch drains the queue after cancelling (a
request landing after the shutdown drain is not retained via a live
WithTimeout clone), and the recycle backstop honors explicitly
disabled read deadlines - it waits for the session instead of force-
closing on a zero-progress interval, escalating only once stop fires
so Close still terminates.
* fix(csc): harden FD session recycle and probe paths
- timedPushDrain reads under a hard deadline: WithReader let an active
maintenance-relaxed timeout replace the 1ms probe cap, parking the FD
session reader for the relaxed duration on a no-data probe while it
held the connection. WithReaderHardDeadline bypasses relaxation and
clears the deadline on exit (the negative-ReadTimeout cleanup block
is now redundant and removed).
- The supervisor also watches the recycle channel, so a
reader-triggered recycle (handoff/close-on-put) gets the same bounded
progress-based drain backstop as the age-triggered path - a stalled
in-flight reply can no longer postpone a requested handoff until
Close.
- The deadline-less recycle carve-out tests rt <= 0: Options.init
normalizes ReadTimeout -1 (indefinite) to 0, so the indefinite mode
was still force-closed after one budget interval.
* fix(csc): epoch inval drop; refresh ownership; close order
- Queued invalidations carry the batcher epoch they were enqueued
under; drop() bumps the epoch instead of draining, and apply skips
stale-epoch items (queue and in-progress batch alike, so the dropCh
signal is gone). The flush handler bumps before cache.Flush(), so a
per-key invalidation racing in from another tracked connection after
the flush keeps its delete - the drain-based drop could discard it
and leave a repopulated entry stale. Dedup is per key+epoch so a
post-flush re-arrival is not swallowed by the duplicate check.
- stopCSCRefresher clears the shared handler's refresh binding only
while it still points at the closing client's own queue: a sibling
client sharing the cache/processor keeps its refresher fed.
- Teardown deactivates cscActive before stopping the coalescer, and
fetch's stop paths settle with errCSCRetryUncached so a clone's miss
racing the teardown window re-runs uncached instead of failing with
ErrClosed on an open pool; the mid-apply branch returns the settle
result itself rather than discarding a successfully applied reply.
* fix(csc): serialize inval apply with drop; FD handler adapter
- apply/drop are serialized by a batcher mutex: apply snapshotting the
epoch once let a concurrent FLUSH land mid-batch and stale-epoch
deletes still run post-flush, evicting repopulations - the case the
epoch exists to prevent. drop() holding the mutex means an in-flight
batch finishes before the flush (harmless: the flush wipes it) and
every later apply sees the new epoch.
- Both FD-session push paths (the reader's per-reply drain and
timedPushDrain) hand handlers the nonblocking cscHandlerClient
adapter like the background drainer does: a custom push handler
calling Close() signaled-and-deferred instead of deadlocking on
mc.wg.Wait while the session reader is parked inside that handler.
* fix(csc): retry-uncached drains; refresh stack; weak close
- The shutdown drain and the GC cleanup settle queued misses with
errCSCRetryUncached, matching fetch's stop paths: a caller woken
during the teardown window re-runs its read on the still-open pool
instead of surfacing ErrClosed.
- The shared handler tracks refresh bindings in a stack: clearing the
ACTIVE binding restores the next-newest live sibling instead of
nil, so closing the newest owner no longer severs an older client's
still-running refresher (the identity check alone only protected the
reverse order).
- The push-handler adapter closes through the canonical *Client via a
weak back-pointer: Client.Close also stops the cached autopipeliners,
which baseClient.Close bypassed, leaving flush goroutines running
against closed pools. Weak so the wrapper stays collectible and the
drop-without-Close cleanup still fires.
* fix(csc): write-side backstop; probe gate; clone bypass
- The drain backstop honors the write side too: the interval covers the
connection's effective write timeout (pool.Conn.EffectiveWriteTimeout)
and the deadline-less carve-out fires when either read or write
deadlines are disabled - a deadline-free write blocked flushing a
large request shows no read progress by construction and must not be
cut by an age/handoff recycle.
- The opaque-transport speculative probe runs only with the built-in
push processor: a custom processor may surface the empty-probe
timeout per its contract, which would remove and redial a healthy
session on every idle probe.
- withTimeout clones whose timeouts diverge from the owner's bypass
miss coalescing: the shared engine reads the owner's options, so a
coalesced miss would honor the owner's deadlines, defeating
WithTimeout. Hits and uncached fetch caching are unchanged.
- releaseLocked clears refresh/refreshStack with the binding: a client
dropped without Close must not leave a dead queue for a successor to
inherit or restore.
- The drop-without-Close cleanup drain settles with errCSCRetryUncached
like every other stop path (the round-22 hunk for this block was a
silent patch no-op; applied for real).
* fix(csc): honor lifetime jitter in the FD recycle age
Capping the recycle age at the raw ConnMaxLifetime before consulting
ExpiresAt collapsed every positive-jitter connection back to the
unjittered lifetime, re-synchronizing session recycles across clients
started together - the herd ConnMaxLifetimeJitter exists to prevent.
Bound the age by the connection's actual absolute expiry only (jitter
included), subject to the independent 30s session cap.
* fix(csc): fail the FD session on push-processor errors
Log-and-continue past a surfaced push-processor error let the reader
proceed into ReadRawReply on a possibly desynchronized stream: a
processor that consumed part of a push before failing leaves the next
bytes mid-frame, and a fragment could be applied as a caller's reply
and published to the cache. Treat the error as session-fatal (close and
remove the connection), matching drainPushNotifications - the built-in
processor surfaces only mid-frame failures, and a custom processor's
contract cannot prove no bytes were consumed.
* feat(csc): native OTel attribution for coalesced misses
A coalesced miss returned from processCached without touching
processState, so the native recorder saw zero attempts and a nil
connection (empty server.address), and engine-failed requests bypassed
the error callback that processWithRetry emits. fetch now returns the
serving session connection (recorded by the reader before the done
settle, which is the happens-before edge) and processCached stamps it
into processState as one attempt; applyAndSettle and settleErr emit the
native error callback with parity to processWithRetry and the FD
autopipeline engine's failReqs - the retry-uncached sentinel is
excluded, since its command re-runs on the fully instrumented path.
Refs the deferred-observability thread on #3965.
* fix(csc): FD miss-coalescer review fixes
- Attribute write-side failures to the held session connection: the miss
coalescer sets req.servedBy for the whole batch before the write, so a
WithWriter error (or ctx cancel during the inflight handoff) settles with the
connection attribution the reply path records, not a nil conn / zero attempts.
- Tolerate fragmented push frames on held sessions: once readiness is
established, the drain uses the longer cscDrainHardReadCap budget (as the
background drainer does) instead of the 1ms probe cap, which could time out
mid-frame over TLS and fatally close a healthy session. The 1ms cap is
reserved for speculative no-data probes (opaque transports).
- Route a WithTimeout clone's CSC push-handler Close through the owner wrapper:
the clone's cscClientWeak now points at the canonical owner, so closeCanonical
calls owner.Close() (full CSC + autopipeliner teardown) instead of
baseClient.Close closing shared pools while the owner's drainer runs.
* fix(csc): harden refresh-on-invalidate push drain
Two fixes to the refresh-on-invalidate reply reader, matching the
miss-coalescer / background-drainer policy:
- Propagate push-processor errors instead of logging and continuing. A
surfaced error means bytes may have been consumed mid-frame; reading the next
reply on a desynchronized stream could publish a push fragment under the
wrong cache key. Returning the error aborts the batch so withConn retires the
connection.
- Use the nonblocking cscHandlerClient adapter for the drain. This reader is
part of the refresher's waitgroup, so a custom push handler calling Close()
on the raw client would deadlock shutdown (Close waits on the refresher
goroutine parked in the handler).
* fix(maintnotifications): skip endpoint DNS detect when mode is disabled (#3969)
* Update options.go
* Update options_test.go
* Update options.go
* fix flaky test
* fix(maintnotifications): skip endpoint DNS detect when mode is disabled
* perf(proto): add zero-copy semantics to scan, remove redundant data c… (#3972)
* perf(proto): add zero-copy semantics to scan, remove redundant data conversions in reader
* chore: unpin govulncheck version
* chore: require govulncheck to use latest go version
* chore: pin govulncheck and resolve go-version to stable
---------
Co-authored-by: Nedyalko Dyakov <1547186+ndyakov@users.noreply.github.com>
* fix(csc): record cancel metric on coalesced miss
A caller that cancels its context while waiting on a coalesced miss returned
early without recording an error metric: the serving session still completes
the background fetch (applyAndSettle success or settleErr), so neither records
THIS caller's cancellation, undercounting the cancellation rate whenever miss
coalescing is enabled. Report it here, attributed to the caller's ctx, matching
processWithRetry.
The conn is passed as nil deliberately: req.servedBy is written by the session
and only safe to read after the req.done receive (the happens-before edge),
which has not happened on the cancellation branch, and is nil anyway when the
cancel beats the serve. The metric recorder treats a nil conn as "no peer
attributes".
* fix(csc): retry-uncached on coalescer stop-acquire
When getConn fails because acquireCtx was cancelled by mc.stop during coalescer
teardown, the session settled the pulled miss and drained the queue with the raw
context.Canceled. processCached only retries uncached on the errCSCRetryUncached
sentinel, so in-flight coalesced reads surfaced a spurious cancellation even
though the caller's context was still valid, instead of re-running on the
still-open pool like the other stop paths.
acquireCtx derives from context.Background() and is cancelled only by mc.stop
(its sole non-deadline cancel source), so a context.Canceled here uniquely means
teardown: settle first and the queue drain with errCSCRetryUncached and report
stopped so the loop exits without backing off. Genuine acquire failures (dial
error, deadline) keep the raw error and the backoff path.
* fix(csc): detect coalescer stop by signal, not error
The full-duplex miss coalescer mapped a getConn cancellation to
errCSCRetryUncached by testing errors.Is(err, context.Canceled). But getConn runs
the dialer, credentials callbacks, and init hooks, any of which can return
context.Canceled from their OWN context while the coalescer is not stopping.
Treating the error value as proof that mc.stop fired permanently exited the sole
full-duplex worker; the current miss retried uncached, but later misses still
enqueued through the still-live coalescer with no worker to settle them, hanging
deadline-free callers and leaking IN_PROGRESS reservations.
Detect teardown by checking mc.stop directly instead. mc.stop is closed only by
stopWorkers, and close(mc.stop) is ordered before acquireCtx cancels its context
(before getConn returns), so a real teardown is always visible. On stop, settle
retry-uncached and exit; on any other acquire failure — including an unrelated
context.Canceled — fail the batch and back off so the worker stays alive. This
supersedes the errors.Is check added in the previous round.
* docs(csc): note per-session Limiter for miss coalescing
ClientSideCacheCoalesceMisses holds one connection per coalescer session and
serves many misses on it, so Options.Limiter is admitted (Allow/ReportResult)
once per session — per held connection — not per coalesced miss, unlike the
plain per-command path. This is inherent to the held-connection model; a
per-miss Allow would defeat the coalescing and re-admit a connection already
held. Document it on the option so callers that need strict per-command
admission or circuit-breaking know not to enable miss coalescing.
* chore(deps): bump rojopolis/spellcheck-github-actions (#3967)
Bumps [rojopolis/spellcheck-github-actions](https://github.com/rojopolis/spellcheck-github-actions) from 0.63.0 to 0.66.0.
- [Release notes](https://github.com/rojopolis/spellcheck-github-actions/releases)
- [Changelog](https://github.com/rojopolis/spellcheck-github-actions/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rojopolis/spellcheck-github-actions/compare/0.63.0...0.66.0)
---
updated-dependencies:
- dependency-name: rojopolis/spellcheck-github-actions
dependency-version: 0.66.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(csc): propagate drain errors on fragmented attributes
The held-session push drain (pushDrainWithin) dispatched the built-in processor
through ProcessPendingNotifications, which swallows an error that occurs AFTER
partially consuming a frame — e.g. a RESP3 attribute whose remainder arrives
slower than the hard read cap. A swallowed mid-frame error left the connection
desynchronized, so later fragments could be misread as command replies and cached
under the wrong key, or fail unrelated misses.
Use ProcessPendingNotificationsBuffered for the built-in processor, exactly as
drainPushNotifications already does, so a post-consumption error is propagated and
the caller retires the desynced session. Custom processors keep the unbuffered
call (their contract is "invoked only when notifications are known present").
* fix(csc): limit bare-socket push drain to built-in processor
Round-8 review fix (codex r3809020181).
peekAndProcessPushNotifications fired on MaybeHasData (raw socket
readiness). Over TLS, maybeHasData unwraps the transport. It then
returns true for a post-handshake control record that has zero RESP
bytes. (conn_check.go's connCheck refuses to unwrap TLS for this reason.
maybeHasData does unwrap it.) A custom NotificationProcessor then ran
ProcessPendingNotifications under the 50ms hard cap. The empty read
timed out. The FD session reader treats that timeout as fatal. It closed
a healthy connection, and its in-flight misses failed. The built-in
buffered processor accepts the empty read.
Now the bare-socket-readiness path runs only for the built-in processor.
A custom processor drains only when the reader already holds decoded
bytes (HasBufferedData), which means any TLS is already unwrapped.
Tradeoff: a push notification that arrives as bare socket readiness on a
custom and idle session waits for the next reply read, the session
recycle, or the MaxStaleness backstop. The connection stays open. This
mirrors the built-in-only guard on the opaque-transport probe in the FD
session tick.
* fix(csc): peek frame type before a custom push drain
Round-9 review fix (cursor r3811858390).
The round-8 D fix let a custom push processor drain on HasBufferedData. But
the buffered bytes can be a coalesced reply, not a push. A custom processor
invoked then could consume the reply, cache it under the wrong key, and
desync the stream. The built-in processor peeks the frame type and consumes
only push frames, but the NotificationProcessor interface does not promise
that. pushDrainWithin now peeks the frame type for a custom processor and
returns without a read when the frame is not a push. The peek is bounded by
the closure's hard read deadline.
* fix(csc): propagate custom push-drain peek errors
Round-10 review fix (codex r3812520847).
The round-9 fix peeked the frame type before it handed a buffered frame to a
custom push processor, but it returned nil on a peek error. PeekReplyType can
partially consume a fragmented RESP3 attribute (DiscardNext) before it errors,
which desyncs the stream. Swallowing the error kept the desynced connection,
so later fragments could be read as a command reply and cached under the wrong
key. The peek error now propagates, so the session fails and drops the
connection, as the built-in buffered path does. A cleanly-peeked non-push
frame still returns nil (a real reply left for the caller's read).
* fix(csc): drain miss-reader pushes through the shared safe helper
Round-11 review fix (cursor r3813795126).
The full-duplex miss reader (readOne) drained pushes with the unbuffered
ProcessPendingNotifications for both processor kinds -- a third drain site the
earlier desync fix did not reach. The built-in processor there swallowed a
mid-frame DiscardNext error on a fragmented attribute, and a custom processor
got no frame-type peek, so either could leave the stream misaligned; readOne
then read the shifted bytes as the reply and cached them under the wrong key.
Both drain paths now share drainPushFrames: the built-in uses the Buffered
variant (mid-frame errors propagate) and a custom processor is handed only a
confirmed push frame (peek first, propagate the peek error). The helper sets no
read deadline, so readOne's following ReadRawReply is unaffected.
* fix(csc): retire conn on pre-command drain desync; refresh drain helper
Two review fixes.
Pre-command push-drain desync (round-13 audit): _process drains pending
pushes before executing a command, then reads the reply on the same conn and,
on the CSC-miss capture path, caches it. A fragmented RESP3 frame straddling
the drain's short hard cap left the reader mid-frame; the error was logged and
ignored, so the reply read consumed the residue and could cache it under the
wrong key. _process now closes the conn and retries on a fresh one instead of
ignoring the error (peekAndProcessPushNotifications already propagated
mid-frame errors via drainPushFrames).
Refresh reader drain (cursor r3813795126 / r3815669446): the refresh reply
reader still drained pushes with the unbuffered ProcessPendingNotifications,
a fourth drain site that could swallow a mid-frame desync before ReadRawReply
and publish a push fragment under the wrong cache key. It now routes through
the shared drainPushFrames helper, like the idle-tick and miss-reader paths.
* fix(csc): stop the refresh worker on GC cleanup
Round-14 review fix (codex r3817196217).
When refresh-on-invalidate is enabled and the caller drops the client without
Close, the runtime cleanup stopped the miss coalescer and the invalidation
drainer but never signaled the refresh worker. runCSCRefresher then stayed
parked on its ticker/queue, holding the baseClient, cache, and pools -- a
goroutine and memory leak that defeated the drop-without-Close safety net.
cscRegisterCleanups now captures the refresh handle and signals it, idempotent
and non-blocking (a new cscRevalidateHandle.signalStop guarded by sync.Once, so
Close and the cleanup cannot double-close). stopCSCRefresher uses the same
signal.
* feat(csc): add refresh-failure metric
Adds RefreshFailed to CSCRefreshStats -- the refresh-failure counter the HLD
lists (go-redis section 4.1). A refresh round trip that errors increments it;
those keys stay evicted and a later read repopulates them, so a rising count is
the signal that refresh-on-invalidate is degrading to plain eviction. Counted
per errored batch, not per key.
* fix(csc): drain pending pushes before reading the reply
The CSC miss and refresh readers drained push notifications with the
Buffered variant, which stops the instant the reader buffer empties. If a
second invalidation was still on the socket ahead of the command reply,
ReadRawReply read that push as the reply and cached it under the wrong
key -- a one-frame shift that then cascaded to later replies. The readers
now drain in BLOCKING mode: they block on the socket and skip push frames
until a non-push frame (the reply) is next, the same non-buffered
discipline the full-duplex reader already uses. PeekReplyType is
attribute-aware, so a fragmented RESP3 attribute needs no separate
buffered scan. A swallowed boundary-peek TIMEOUT is caught by the reader's
shared read deadline (ReadRawReply hits the same expired deadline and
fails the session); a swallowed non-timeout peek error would need a
malformed attribute mid-push (a server protocol bug), so this path is no
weaker than the full-duplex reader. Probe and idle paths keep the Buffered
variant so they never block. A socket-pair regression test pins the frame
order.
Also in this change:
- GC cleanup unbinds the dropped client's refresh queue from a shared
invalidate handler (clearRefreshQueue), so a surviving sibling's
refresh-on-invalidate keeps working instead of feeding a stopped queue.
A refresh queue is only created inside attachSharedTrackingCSC, which
also builds the drain handle, so the handle is always present when the
queue is; a Conn() clone bails before the refresher starts.
- Record usedAt on deadline-free (negative-timeout) reads and writes, so
a long full-duplex session under ReadTimeout/WriteTimeout=-2 is not
misjudged as idle-expired by the pool and needlessly reconnected.
* fix(csc): harden miss-coalescing per review
- spill invalidation overflow off the full-duplex reply reader; the worker
drains it through the same seen/pending dedup, so a burst collapses to one
delete per key and no cache work runs on the reader
- drop the drain deadline cleanup that re-armed a relaxed timeout
(WithReaderHardDeadline already restores the deadline)
- cap a coalesced write batch by bytes, not just count
- warn when ClientSideCacheInvalidationBatchWindow exceeds MaxStaleness
Refs #3965
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: sadd amr3e <saddamr3e@gmail.com>
Co-authored-by: Albert Huynh <huynhalb@gmail.com>
Co-authored-by: Vladislav Kotsev <vladygk8@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(csc): address review findings on refresh, batcher, and spill
Refresh-on-invalidate:
- pass the refresher's internal 5s ctx directly to WithReader/WithWriter instead
of c.context(ctx); c.context strips it to Background when ContextTimeoutEnabled
is false, so with read/write timeouts disabled a stalled Redis hung the
refresher and wedged Close (cursor/codex).
- recover() the flush batch and defer its cancel(), so a panic can't kill the
refresher (silent permanent degradation) or leak the 5s timer.
- skip Reserve's (0,true) 'fetch uncached' targets: they wasted an MGET slot and
registered server-side tracking for a never-cached key.
Invalidation batcher:
- hold the handler lock across drop()+Flush() and read the batcher/cache fresh
under it, so a concurrent batcher rebuild can't leave stale-epoch deletes
applying after the flush and evicting post-flush repopulations (cursor).
- hard-cap the overflow spill and schedule one full Flush past the cap: the
invalidatable keyset is NOT bounded by cache size (the server tracks keys past
local LRU eviction), so an unbounded spill could OOM (codex). Adds a test.
Other:
- FTHybridCmd: populate rawVal on the RESP2 path for RawResult parity (Copilot).
- docs: mark ClientSideCacheRefreshOnInvalidate experimental + LocalCache-only
(Copilot); correct the HasBufferedData 'complete frame' comment; note the
Effective*Timeout post-expiry side effect; reconcile the refresh header's
stale-read claim; drop a stale symbol reference.
* fix(csc): follow-up review fixes (spill race, stats, refresh, acquire)
- inval batcher: set flushReq BEFORE releasing stopMu at the spill cap, so a
concurrent stop() cannot close stopCh and let the worker exit without seeing
the request (lost flush, stale entries).
- inval batcher: count a dedup-dropped duplicate invalidation as invalidation +
noop, so InvalidationStats reflects incoming pushes (the duplicate ratio) with
the window batcher on, matching the inline path.
- csc integration: unbind the refresh queue in the drainer's exit defer, so a
SELF-DISABLE exit (custom-processor damping / RESP3 downgrade) restores a
sibling's binding instead of silently breaking the survivor's refresh.
- miss coalescer: derive the connection-acquire budget from a custom
DialerRetryBackoff (sum the per-attempt delays), so a backoff longer than
DialerRetryTimeout no longer cancels Get mid-retry-sequence.
Addresses codex re-review findings on #3989.
* fix(csc): address codex re-review (relaxed-timeout race, acquire, spill stats)
- pool relaxed timeout: the counter is a relax/unrelax NOTIFICATION balance, not a
per-I/O count. getEffective{Read,Write}Timeout no longer decrements it per call;
a passed deadline triggers a single CAS-coordinated reset (expireRelaxedTimeout),
so a full-duplex connection's concurrent reader/writer/drain-backstop can't drive
it negative and wedge later relaxations, and a lost unrelax notification is
recovered at the deadline. ClearRelaxedTimeout is clamped at zero. Adds a
concurrent-expiry regression test.
- miss coalescer acquire budget: revert the DialerRetryBackoff summing — calling a
randomized/stateful backoff out-of-band consumes different jitter samples than
the pool's actual retries, so the estimate can still be wrong. Use the
deterministic flat estimate; mc.stop + PoolTimeout bound it for custom backoffs.
- inval batcher: count invalidations superseded by the spill-cap full-Flush
(enqueue discard + worker pending reset) so CSCRefreshStats doesn't undercount an
invalidation flood.
Addresses the codex re-review on #3989.
* fix(pool): retire one relaxed holder on expiry
The relaxed-timeout deadline safety net full-cleared the whole window on
expiry. On a full-duplex conn the reader, writer, and drain backstop all
observe the expired deadline concurrently; a CAS on the shared deadline
already gates that to one action, but a full clear also wiped a DIFFERENT
relaxation sharing the conn -- a notification SetRelaxedTimeout outstanding
alongside an expired handoff deadline lost its window (the :566 race).
Retire exactly one holder instead: expireRelaxedTimeout decrements the
counter by one (clamped, clears the timeout values only when the last
holder leaves). To keep overlapping handoffs from leaking a permanently
relaxed conn, SetRelaxedTimeoutWithDeadline now takes a holder slot only
for the FIRST deadline-scoped window and re-arms by replacing the deadline
in place (Swap), so the eventual expiry retires the single holder and the
window clears.
Add deterministic regression tests: a notification window survives an
expired handoff deadline, and overlapping handoffs clear without a leak.
Addresses the codex conn.go:566 finding on #3989.
* fix(csc): split invalidation and deletion stats
InvalidationStats conflated two quantities and undercounted whenever the
window batcher deduped, spilled, or fell back to a full Flush, which drove
a recurring "another uncounted path" review cycle. Split them:
- Invalidations counts keys named in INCOMING pushes, tallied once at the
handler choke point before dedup/batching.
- Deletions / DeletionsNoop count APPLIED deletes at the two cache choke
points (DeleteByRedisKey and deleteByRedisKeyCollectingHot), so the count
is accurate whether refresh-on-invalidate is on or off.
Invalidations - Deletions now measures dedup plus the flood full-Flush
fallback directly, so the batcher no longer needs ad-hoc per-drop counting
(removed addInvalidationStats and its three call sites). Both are exposed
on CSCRefreshStats.
Also snapshot the refresh recency horizon at ENQUEUE into
cscInvalItem.sinceToken and use it in apply, so a batch-window delay cannot
advance the live horizon and chill a key that was hot when the invalidation
arrived (it would degrade to plain eviction instead of a proactive
refresh). Value-passed, no added allocation.
Addresses the cursor csc_inval_batch.go batch-chill finding and the codex
csc_inval_batch.go stats-undercount finding on #3989.
* fix(pool): serialize relaxed-timeout mutations
SetRelaxedTimeoutWithDeadline stored its new timeout values before swapping the
deadline, so an expiry observed concurrently (a full-duplex reader/writer/drain
backstop) could CAS the old deadline to zero and clearRelaxedTimeout the fresh
values between the store and the swap -- leaving a window with a future deadline
but zeroed timeouts (active but ineffective; commands fall back to the shorter
normal timeout during a migration and can fail spuriously). Reordering alone only
moves the race.
Add relaxedMu to serialize all four mutators (Set, SetWithDeadline, Clear,
expire) so the timeout/deadline/counter atomics are always updated or cleared as
one unit. expireRelaxedTimeout re-validates the deadline under the lock (a
generation check): an expiry the caller observed lock-free is dropped if a newer
window replaced it. The read path (getEffective*/HasRelaxedTimeout) stays
lock-free and takes the lock only via expire on the rare observed expiry, so the
per-I/O hot path is unchanged. The former CAS retry loops collapse to plain
load/store under the lock.
Addresses the codex relaxed-timeout rearming-vs-expiry finding on #3989.
* fix(csc): repoint batcher to survivor before stop
The invalidation batcher captured its refresh binding at creation. When the
active refresh owner closed, clearRefreshQueue restored the surviving sibling on
h.refresh and stopped the batcher, but the stop-drain still offered evicted-hot
keys through the closing owner's queue -- whose drainer had already gone, so the
offers were dropped and the survivor never refetched them (they stayed evicted
until a reader missed). setRefreshQueue had the same staleness on a rebind.
Make cscInvalBatcher.refresh an atomic pointer and repoint it at the current
binding (survivor / new owner, nil if none) BEFORE stop() in both
set/clearRefreshQueue, so the stop-drain feeds the live refresher. An in-flight
apply already holding the old pointer feeds its current batch to the old queue --
a bounded, benign residual (offers to a stopped queue are dropped, never panic:
the queue channel is never closed).
Addresses the cursor "batcher drain feeds closed refresher" finding on #3989.
* fix(pool): relaxed timeout as one atomic snapshot
Ofek's review asked for the relaxed-timeout window to be published as one value
so a reader never sees it half-updated. Replace the mutex plus four atomics with
a single atomic.Pointer to an immutable relaxedState snapshot. Set,
SetWithDeadline, Clear, and expire install a new snapshot with a
compare-and-swap; the read path (getEffective* and HasRelaxedTimeout) does one
lock-free Load. This removes the writer-only mutex and the torn read a reader
could see between the counter bump and the timeout store.
It also fixes getEffective* returning the normal timeout on the call that
observes an expired deadline while a notification holder is still active. The
method now re-reads the snapshot after it retires the deadline holder and returns
the surviving relaxed timeout.
Addresses the codex relaxed-timeout thread and Ofek's snapshot request on #3989.
* fix(csc): bound coalesced writes, harden miss retry
Reader-miss coalescing and refresh both write a whole batch before they read the
replies, and the reader waits until the write finishes. A batch larger than the
connection write buffer (default 32 KiB) makes bufio flush in the middle of the
batch, which can deadlock the write and the read on large payloads. The batch
byte cap was 1 MiB. Bound each batch to the write buffer:
- The miss coalescer caps a batch at min(write buffer, 1 MiB). An over-budget
request is carried in the writer's own state and sent as the first request of
the next batch, not put back on the shared queue. So a shutdown cannot strand
it, and it also does not widen a pre-existing reservation-strand window on the
drop-without-Close path. The first request always goes.
- The refresh chunker applies the same byte bound next to its count cap.
Also route a coalescer session or transport failure through the normal retry
path. settleErr tags such errors as cscSessionError; processCached re-runs the
read with MaxRetries and backoff instead of a raw io.EOF to the caller. A
reply-level result (redis.Nil, WRONGTYPE, LOADING) stays untagged and is returned
as-is.
Addresses the Ofek review threads on #3989: write-before-read batches, the
refresh batch, the byte-cap overshoot, and the coalesced-miss retry path.
* fix(csc): drop ineffectual pending clear
The grabInto carry change made the pending = nil at the top of the writer loop a
dead store: grabInto now reassigns pending (with the next carry) before it is read
again. Drop the clear. No behavior change. Fixes the golangci-lint ineffassign
that CI flagged.
* fix(csc): read coalesced replies while writing the batch
The miss-coalescer wrote the whole batch and only then handed the requests to the
reader, so the reader was idle for the entire write. A batch larger than the
transport send capacity could deadlock: the server blocks writing early replies
that nobody reads while the writer blocks flushing later requests. Bounding the
batch to the write buffer did not fix this — the write buffer does not bound the
socket send buffer (a custom Dialer or a large WriteBufferSize breaks the
assumption), and with per-op timeouts disabled it can wedge.
Hand the batch to the reader BEFORE writing it (as the autopipeline full-duplex
engine already does), so the reader drains reply k while request k+1 is still
being written and relieves the server's send backpressure. The send cannot
deadlock the writer: a batch is at most cscMissBatchMax (128) and inflight is
cscFullDuplexDepth (4096) deep, the writer is sequential so every prior batch was
already flushed, and every reader-exit path either cancels the session context or
follows the writer's own exit. The byte cap remains only as a batch-size bound,
not as the deadlock fix.
Addresses the Ofek and codex/cursor write-before-read findings on #3989.
* fix(csc): stats without refresh, bound refresh write
Two refresh-path fixes from the review:
CSCRefreshStats returned an all-zero struct when the refresh queue was absent, so
with ClientSideCacheRefreshOnInvalidate off (the default) but invalidation
batching on it hid the cache-level Invalidations/Deletions/DeletionsNoop the
handler still records. Read those cache counters independently of the queue; gate
only the queue-specific counters on the queue.
The refresh writes a whole chunk before it reads the replies, so a chunk that the
server cannot drain (transport backpressure) blocks the flush. With per-op write
timeouts disabled the flush is deadline-less and the 5s batch context does NOT
interrupt it, so the refresher goroutine wedges — worse than the review reported.
Bound the write with a positive deadline (shared with the batch context via one
constant) so a stalled flush fails the refresh and the entries degrade to plain
eviction (self-healing). Full read/write concurrency in the refresher is a
possible follow-up, not done here.
Addresses the codex refresh-stats and refresh write-before-read findings on #3989.
* fix(csc): honor recycle/stop on the carry path
grabInto can defer one over-budget request as a carry, held in the writer's
pending slot and written as the first request of the next batch. While pending is
non-nil the writer took the carry branch and skipped the select that watches
recycle and mc.stop, so a producer whose commands are persistently over the byte
cap could keep the held connection busy and starve a maintenance handoff or a
lifetime recycle indefinitely (its OnPut hooks would never run).
Check recycle/stop/session-cancel (non-blocking) between batches even when a carry
is present, and settle the carry on bail — it lives only in the writer's state, so
the teardown drain, which covers inflight, would otherwise leak it and hang its
caller. Settle with the retry-uncached sentinel on recycle/stop so the caller
re-runs on the pool.
Addresses the codex carry-vs-recycle finding on #3989 (a follow-on from the carry
introduced last round).
* fix(csc): bound the refresh read too
Last round bounded the refresh WRITE so a deadline-less flush could not wedge the
refresher. The following WithReader still used the raw ReadTimeout: with per-op
timeouts disabled (options.go maps -2 to -1, which WithReader treats as no
deadline) a stalled reply or push drain parks refreshInvalidatedBatch forever, and
stopCSCRefresher waits on that goroutine, so Client.Close never returns.
Bound the read with a positive deadline (still capped by the 5s batch ctx, shared
via cscRefreshBatchTimeout), symmetric with the write, so a stall fails the refresh
and the entries degrade to plain eviction instead of hanging Close.
Addresses the cursor refresh-read-hangs-Close finding on #3989.
* fix(csc): per-chunk deadline for refresh round trips
The refresh split a window into chunks (by count and by write-buffer bytes) but
ran them all under one cscRefreshBatchTimeout context. When a window held more
than one chunk and their AGGREGATE latency exceeded the budget, later chunks
failed immediately even though each round trip was within the intended per-round-
trip timeout — four 2s chunks under a 5s budget would refresh only the first two
and leave the rest evicted.
Create the bounded context PER refreshInvalidatedBatch call so the documented
per-round-trip timeout applies to each chunk independently. The recover guard
stays at the loop level.
Addresses the codex per-chunk-deadline finding on #3989.
* fix(csc): flush snapshot cache on FLUSH invalidation
The FLUSHDB/FLUSHALL invalidation branch re-read the live h.cache under a second
RLock and flushed that, while per-key deletes use the cache pointer snapshotted at
handler entry. A last-user releaseLocked (drainer teardown / GC cleanup) can nil
h.cache between the entry RUnlock and the flush RLock, so the guarded
'if h.cache != nil' skipped the wipe entirely. On an injected shared cache
(cscOwnsCache == false) the teardown flush is gated off, so post-FLUSH entries
kept serving until TTL / MaxStaleness.
Flush the entry snapshot instead, matching the per-key branches. The second RLock
stays — it guards the batcher epoch against a concurrent rebuild — but only the
batcher is read fresh under it now; the cache uses the snapshot. The nil guard is
dropped: the entry guard already returns on a nil snapshot.
Addresses the cursor 'FLUSH can skip shared cache wipe' finding on #3989.
* fix(csc): drain all pushes before a coalesced/refresh reply
drainPushFrames in blocking mode, for a CUSTOM PushNotificationProcessor, peeked and
processed once. A second invalidation still on the socket ahead of the reply was
then read by the caller's ReadRawReply as the command value — published to the
shared cache under the wrong key and shifting every later reply. Loop peek-then-
process until a non-push (the reply) is next, matching the built-in blocking
discipline; non-blocking probes keep the single-pass behavior. Covers the coalescer
and refresh reply readers.
* fix(csc): bound in-flight coalesced wire by bytes
Miss coalescing serializes each caller's command into req.wire before it can enter
the bounded queue; arbitrarily many callers can block on the send holding a full
wire copy, and large cacheable commands (a big MGET) have no small encoded size, so
a burst could exhaust memory — the queue's item cap does not bound bytes. Track
total in-flight serialized bytes; before serializing, reserve this command's
approximate size and shed over cscMissWireBudgetBytes (8 MiB) to the pooled path
(errCSCRetryUncached), which is bounded by pool turns. Reserving before serialization
means over-budget callers never allocate the wire; a single command larger than the
budget sheds too (the pooled path runs it). TestCSCMissCoalescerWireBudget pins it.
* fix(push): consume a push whose name can't be peeked
The built-in blocking drain, on a frame already confirmed as a push, broke out of
the loop when PeekPushNotificationName failed for a non-'too-long' reason (a
non-string / malformed name). Breaking LEFT the push at the buffer head, so a
reply-expected reader (CSC refresh / miss-coalescer / the single-command pre-read
drain) then read the push as the command value — the intermittent 'redis: nil' and
cross-command reply …
Full-duplex reader-miss coalescing + invalidation batching
Builds on the CSC refresh-on-invalidate + base reader-miss coalescing PR
(
feature/csc-refresh-and-miss-coalescing). Two additions, both config-driven.1. Full-duplex miss coalescing (reader-miss path)
Concurrent cache misses of the same key are already deduped to a single fetch
via a per-key reservation (one owner fetches; the rest block on it and share the
result). This PR adds the engine that dispatches the owners' fetches: a held
tracked connection with a writer + reader goroutine pair that pipelines
reserved misses — commands stream out while replies stream back.
Misses are caller-blocking (a real request waits on every fetch), so the
engine is latency-first:
what is already queued), never waited-for;
no batch phase-lock, no pool
Geton the hot path;next miss, so an idle coalescer holds zero connections;
held connection even while idle, and returns the connection to the pool for
handoff/lifetime/pool-hook processing (idle grace, recycle age bounded by the
connection's remaining
ConnMaxLifetime,ShouldHandoffchecks).Ordering is preserved. Each caller blocks on its own request's completion,
so a goroutine never issues its next command until this value is in hand — the
engine only overlaps independent callers' fetches on the wire.
Earlier engine variants ("workers": N pooled connections with half-duplex
batches; "pinned": a benchmark prototype) were removed during review: the
per-batch round trip added tail latency to exactly the path where a caller is
waiting, and the batching advantage is preserved by the writer's opportunistic
packing. One engine, no tuning surface.
2. Invalidation batching
Coalesce invalidation-driven cache deletes within a configurable window instead
of one delete per push frame — smooths bursty invalidation churn. Background
traffic is batching-first by design (nobody waits on it): windowed, deduped,
with the delete queue preserved across batcher rebuilds and dropped on
FLUSHDB/FLUSHALL(a full flush supersedes queued per-key deletes).Config (no environment variables)
New
Optionsfields (flatClientSideCache*, matchingClientSideCacheRefreshOnInvalidate), mirrored inUniversalOptions:ClientSideCacheCoalesceMisses bool— enables the coalescer (requires thebuilt-in
LocalCache; ignored for customCacheimplementations).ClientSideCacheInvalidationBatchWindow time.Duration— 0 (default) appliesinvalidations inline; a nonzero window batches them (set it no larger than the
cache
MaxStaleness). Shared-handler clients fold windows strictest-wins.Observability rides the client's normal telemetry: coalesced misses fire the
otel operation-duration and error callbacks like any other command path (no
separate stats API — the engine's internal counters are test-only).
Hardening (from review)
The review rounds hardened the engine's lifecycle and edge behavior, including:
single-flight token settlement on every path (no caller ever hangs, no
reservation leaks); a CAS ownership interlock so a ctx-cancelled caller's
Cmderis never written concurrently;
Closeinterruption of blocked acquisitions andsocket reads (bounded drain, then conn close); retry-uncached fallback when CSC
is disabled mid-miss; GC cleanup for clients dropped without
Close;WithTimeoutclones sharing the coalescer; and held-connection probe safety(no deadline clobbering of concurrent I/O; opaque-transport idle-drain fallback).
Note
High Risk
Changes hot paths for RESP3 client-side caching, push draining, and connection reuse; regressions can serve stale cache entries or corrupt pooled connection reply streams.
Overview
Adds config-driven client-side cache tuning via
ClientSideCacheCoalesceMissesandClientSideCacheInvalidationBatchWindowonOptionsandUniversalOptions(replacing env-gated prototypes). Miss coalescing uses a full-duplex held connection (writer + reader) to pipeline reserved misses with latency-first packing; invalidations can be batched off the read path by a background worker with epoch-aware flush handling.Shared invalidate handlers now fold batch windows strictest-wins, stack refresh queues so sibling clients keep refresh-on-invalidate, and tear down coalescer/refresher/batcher on
Closeand GC cleanup. Mid-teardown and cancel paths settle witherrCSCRetryUncachedinstead of spuriousErrClosed, with CAS-guardedCmderownership and wire snapshots for abandoned fetches. Push handling is tightened: blockingdrainPushFramesbefore reply reads, pre-command drains retire desynced conns, refresh refetches use the tracked main pool, and pool peek/checkForDatafixes reduce false unhealthy removals on held CSC connections.Also fixes FT.HYBRID RESP3 map parsing (no connection desync),
proto.Scan/ScanSlicecopy semantics and capacity reuse, skips maint endpoint auto-detect when notifications are disabled, addsSECURITY.mdand updated vulnerability reporting, and bumps CI spellcheck/govulncheck settings.Reviewed by Cursor Bugbot for commit 9001bd8. Bugbot is set up for automated code reviews on this repo. Configure here.