fix(ipc): bound the sizes MessageDecoder asks callers to allocate

[joseph-isaacs]

Rationale for this change

An IPC message declares the length of its flatbuffer header and of its body before either has been received, and PollRead::NeedMore asks the caller to size its read buffer from those declarations:

The inner value is the total number of bytes the buffer should contain […] Callers should: 1. Resize the buffer to this length.

Neither declaration was bounded. msg_length is a u32 read straight off the wire (decoder.rs, State::Length) and body_size is a u64 from the message header (State::Reading), so a short frame declaring a large body made both SyncMessageReader and AsyncMessageReader resize their buffers to that length before a single body byte had arrived. Resource use was decoupled from bytes actually delivered, which matters wherever a Vortex IPC stream can be fed by something other than a Vortex writer.

The repository already has the pattern this follows: footer/postscript.rs checks a declared metadata count against MAX_METADATA_SEGMENTS before using it to size a Vec/HashSet. This applies the same shape to the IPC decoder.

What changes are included in this PR?

• MessageLimits — a Copy struct with max_header_size / max_body_size, a Default, and an UNLIMITED constant.
• Enforcement in MessageDecoder::read_next, at both declaration sites, placed before the corresponding NeedMore return so no caller ever sizes a buffer from a rejected declaration.
• Checked in the decoder, not at the resize call sites. The decoder is what turns untrusted bytes into a size request, so bounding it there covers every reader — including BufMessageReader, which errors on NeedMore today, and any reader added later — rather than requiring each to remember the check.
• with_limits constructors on SyncMessageReader, AsyncMessageReader, and BufMessageReader.
• Four tests in decoder.rs: an oversized header declaration and an oversized body declaration are rejected; a body under the limit still returns NeedMore normally (so the check does not short-circuit the ordinary incomplete-read path); and the limits are configurable in both directions.

Choice of defaults

16 MiB header / 1 GiB body. Headers carry only flatbuffer metadata — encoding ids, row count, dtype — which stays in the kilobytes even for wide schemas. Bodies carry one message's serialized array buffers, which writers chunk far below 1 GiB. Both are well clear of anything MessageEncoder produces, so this should reject only declarations that could not have come from a well-formed stream. Happy to move either number if you have a workload closer to the line than I'd assume.

What APIs are changed? Are there any user-facing changes?

Additive, no breaking changes:

• New public MessageLimits exported from vortex_ipc::messages.
• New MessageDecoder::new(limits) and MessageDecoder::limits(). MessageDecoder::default() still exists and now carries the default limits.
• New with_limits constructors on the three readers; existing new constructors are unchanged and delegate to the defaults.

The one behavioural change: a stream declaring a header above 16 MiB or a body above 1 GiB now returns an error where it previously attempted the allocation. MessageLimits::UNLIMITED restores the prior behaviour for trusted local input.

Related sites not changed here

I looked for the same shape elsewhere — a length read from the stream driving an allocation before the corresponding bytes are known to exist — and found these, all left alone to keep this reviewable:

• encodings/zstd/src/array.rs (~L1055) — with_capacity_aligned from a summed declared uncompressed size. Note the sibling path in zstd_buffers.rs (~L264) does call validate_frame_content_size against the real zstd frame header first, so the guarded version already exists next door.
• encodings/pco/src/array.rs (~L611, ~L630) — with_capacity / reserve from metadata value counts.
• encodings/fsst/src/canonical.rs (~L56) — capacity from the summed uncompressed_lengths child.

Two adjacent things I noticed and did not touch:

• zstd and pco both set_len before filling and then hand out &mut [T] over the uninitialised tail. The FSST path already does this correctly with spare_capacity_mut.
• SyncMessageReader::next resizes to nbytes and then calls Read::read once, which may fill only part of the buffer — but the next read_next sees remaining() == nbytes and decodes the zero-padded tail as if it were data. AsyncMessageReader handles this properly with its Filling state. This looks like a real partial-read bug for non-file readers, separate from this change; happy to open an issue or fold in a fix if you'd prefer.

Checks

• cargo test -p vortex-ipc — 10 passed, including the 4 new tests and the existing round-trip / chunked / single-byte-chunk partial-read tests.
• cargo clippy -p vortex-ipc --all-targets --all-features — clean.
• cargo +nightly fmt — applied.

Not run: workspace-wide build/clippy/tests. The change is confined to vortex-ipc and is additive, but the new public type is exported, so a wider check before merge is worthwhile.

---

Generated by Claude Code