perf: client DFKV_RDMA_DEPTH=8 degrades unpipelined PUT (batch=1) ~20% vs depth=1; benefit is batch-path-conditional

[ketor]

Summary

Client-side DFKV_RDMA_DEPTH=8 degrades unpipelined PUT (batch=1) by ~20% vs DFKV_RDMA_DEPTH=1, reproducibly. The same setting helps pipelined-batch paths (+7% PUT / +13% GET at threads=1 batch=16), so the depth knob's benefit is path-conditional: a win for the connector's batched datapath, a regression for single-op round-trips.

Found during client-side follow-up validation of the v1.9.0 depth work (#105) on the hd04 production ring.

Environment

• Server: hd04 glm ring, v1.9.0, --store-engine file, server depth qd=8 (ring INFO), 2×400G (ib7s400p0)
• Client: dfkv_bench from the v1.9.0 release, RDMA (DFKV_RDMA=1 DFKV_RDMA_DEV=ib7s400p0)
• Same-node target unless noted (gpu1-0013 -> self 192.168.1.30:28101-style member), production node (background load present — see caveat)
• Op size 2.75 MB (size=2752512), n=2000

Data (interleaved d1/d8, 2 reps each — ordering cancels load drift)

batch=1, threads=8 (unpipelined round-trips) — the regression:

run	PUT GB/s	GET GB/s
d1 #1	6.59	17.76
d8 #1	5.10	17.84
d1 #2	6.25	17.34
d8 #2	5.22	18.14

PUT: consistent −20% at depth=8, both reps. GET flat. p50 call-lat identical (~1.5 ms both); the loss is in throughput/tail (max spikes 407→680 ms d1→d8 rep1).

threads=1, batch=16 (single-connection pipelined) — the intended win, for contrast:

run	PUT GB/s	GET GB/s	GET p50 ms
d1	2.84 / 2.82	3.82 / 3.77	11.3 / 11.5
d8	2.96 / 3.06	4.32 / 4.24	9.9 / 10.2

(+7% PUT, +12–13% GET.) Cross-node (0013→0030) same pattern: PUT +4–5% with p99 54.7→30.2 ms; GET flat.

Repro

B=/path/to/v1.9.0/bin/dfkv_bench
for d in 1 8 1 8; do
  DFKV_RDMA=1 DFKV_RDMA_DEV=ib7s400p0 DFKV_RDMA_DEPTH=$d \
    $B --members self=<node>:28101 2>&1 | grep -E '^(PUT|GET)'   # defaults: threads=8 batch=1
done

Caveat

Measured on a production node with background traffic; the interleaved ordering and rep-consistency make the delta credible, but please re-run on the quiet hd05 rig used for the v1.9.0 benchmarks to confirm magnitude.

Hypotheses / suggested directions

1. With batch=1 there is nothing to pipeline: each call is a single round trip, but a depth-8 connection still carries 8 recv slots' worth of pinned buffers/bookkeeping (≈2×8 MiB×depth per conn per side). Candidate costs: extra posted-recv management on the PUT path, MR/pinning pressure, cache footprint.
2. Consider clamping the effective per-call window to min(depth, actual batch) client-side so depth>1 is free when a call can't use it — or lazily arming recv slots beyond slot 0.
3. At minimum, document depth as a batched-path knob: recommend depth>1 only for pipelined batch workloads; single-op/latency-path clients should stay at 1.

Not a blocker for the connector (it uses batched ops and gets the win); filing so the regression on the unpipelined path is a known, deliberate trade-off rather than a surprise.