oci: keep bulk transfers off a single HTTP/2 connection

[tonicmuroq]

What

Go's default transport negotiates HTTP/2, so every concurrent chunk request to Artifact Registry multiplexes onto one TCP connection, and every DATA frame for every stream passes through that connection's single clientConnReadLoop goroutine. A 14 GiB pull moves ~917k frames at the default 16 KiB frame size, all through one goroutine.

That goroutine was the ceiling. On target test node:

• snapshot pull sat at 337 MiB/s, uncompressed push at 411 MiB/s
• neither vk nor the cocoon snapshot import subprocess saturated a core
• 8 parallel range GETs over separate connections measured 1649 MiB/s on the same host, against the same registry

So the bytes were available and the CPU was available; the connection was not.

How

bulkTransport() clones http.DefaultTransport, turns off HTTP/2 (ForceAttemptHTTP2 = false plus NextProtos: ["http/1.1"], since ALPN would negotiate h2 anyway), and raises MaxIdleConnsPerHost to 32. Over HTTP/1.1 each in-flight chunk gets its own connection and its own reader. The idle pool has to hold every in-flight chunk or HTTP/1.1 pays a fresh TLS handshake per chunk, which is what the raised limit is for.

Measured

	before	after
snapshot pull (14 GiB)	42.5s	35.0s
push, uncompressed (14 GiB)	34.9s	28.0s (512 MiB/s)
push, compressed (6.25 GiB)	29.5s	29.4s

Compressed push does not move because it is zstd-encode-bound, not upload-bound — which is a useful independent confirmation of where each direction's limit sits.

Verified

go test -race ./..., make lint (linux + darwin, 0 issues), make fmt-check.

---

Why not just tune HTTP/2 instead

Fair question, and the first thing to try — dropping to HTTP/1.1 looks like going backwards. It was tried and measured, and it does not work against this server.

Go's HTTP/2 slowness on bulk transfer is a known issue (golang/go#47840, golang-nuts thread) and the usual remedy is the frame size: that thread reports 8 Gbps → 38 Gbps by raising it to 256 KiB. net/http.HTTP2Config.MaxReadFrameSize exposes it, so the fix should be two lines.

It isn't, because SETTINGS_MAX_FRAME_SIZE advertises what we are willing to receive; the sender picks the frame size. Probed against us-central1-docker.pkg.dev from target test node, fetching 512 MiB of a real blob with GODEBUG=http2debug=2:

advertised MAX_FRAME_SIZE	largest DATA frame received	DATA frames for 512 MiB
16384 (default)	8192	67828
1048576	8192	68112

Our SETTINGS go out correctly (wrote SETTINGS ... MAX_FRAME_SIZE=1048576), and the frontend sends 8 KiB frames either way — half the default, and not something a client can change. The 8→38 Gbps result in that thread is Go server to Go client, where raising the client's setting makes the Go server send bigger frames.

End to end, with everything else in this series in place (14 GiB compressed pull, budget 2048):

transport	pull	peak RSS
HTTP/2, default frames	58.2s	—
HTTP/2, advertising 1 MiB frames	56.0s	5.54 GiB
HTTP/1.1	36.2s	3.74 GiB

Sharding across several HTTP/2 connections would divide the reader-goroutine bottleneck, but not the framing cost: 8 KiB frames are per byte, not per connection. For a transfer that is ~100% body with no headers worth compressing and no need for multiplexing, HTTP/1.1's framing is the cheaper one — one Content-Length body read straight off the socket. This is matching the transport to the workload rather than regressing it, and it is scoped to this registry client, not to vk's other HTTP traffic.