ext_proc: two FULL_DUPLEX_STREAMED filters in chain causes request body truncation

[nerdalert]

While validating #43175 for @nirrozenbaum to confirm the ext_proc chaining fix works with BBR + EPP on Istio 1.29.1 (gateway-api-inference-extension#2115), I found that the fix might not resolve the truncation bug when both ext_proc filters use FULL_DUPLEX_STREAMED mode — which is the default BBR + EPP configuration.

Description:

PR #43175 fixed the two ext_proc filter chain bug (#41654) for the case where one filter processes body in FULL_DUPLEX_STREAMED and the other processes headers only. However, the fix does not resolve body
truncation when both filters use FULL_DUPLEX_STREAMED mode. This is the default configuration for BBR + EPP in the Gateway API Inference Extension.

What should happen: all responses should return the full ~130KB streaming body. What actually happens: ~80% of responses are truncated (partial body or empty), returning HTTP 200 with anywhere from 0 to ~94KB instead of ~130KB. The failure rate is the same with and without the fix.

The root cause (general guessing here (if it isn't obvious to maintainers I'm happy to hack on a fix): PR #43175 updates observed_decode_end_stream_ during injectDecodedDataToFilterChain(), but this only helps when Filter-0's inject fires before Filter-1's commonContinue(). When both filters use FULL_DUPLEX_STREAMED, the ordering is a race. When commonContinue() wins, it reads stale observedEndStream() = true (set when client data arrived) and sends partial data upstream with end_of_stream=true.

The existing integration test (TwoExtProcFiltersInResponseProcessing) only covers the one-body-one-header configuration with controlled ordering. The two-FULL_DUPLEX_STREAMED configuration is not tested.

Repro steps:

Full deployment and reproduction: https://github.com/nerdalert/envoy-ext-proc-bug-repro

Minimal reproduction:

1. Deploy two ext_proc filters in chain, both with request_body_mode: FULL_DUPLEX_STREAMED and response_body_mode: FULL_DUPLEX_STREAMED
2. Send a request with body > 6.2KB and stream: true
3. Run 10 times — most responses will be truncated

# Generate 8KB request body
dd if=/dev/zero bs=1 count=8000 2>/dev/null | tr '\0' 'x' > /tmp/pad.txt
printf '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"system","content":"' > /tmp/big.json
cat /tmp/pad.txt >> /tmp/big.json
printf '"},{"role":"user","content":"Hi"}],"max_tokens":500,"stream":true}' >> /tmp/big.json

for i in $(seq 1 10); do
  curl -s -o /dev/null -w "size=%{size_download}\n" \
    -X POST "http://$GW_IP:$GW_PORT/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d @/tmp/big.json --max-time 30
done

Typical output (full response should be ~130KB):

size=134921
size=134961
size=16470
size=14551
size=5436
size=6196
size=2452
size=4073

Results at scale (100 concurrent requests, 8KB body, streaming):

┌─────────────────────────┬───────────────────────────┬────────────────────────────────────┐
│                         │ Envoy 1.36.3-dev (no fix) │ Envoy 1.37.1-dev (with fix #43175) │
├─────────────────────────┼───────────────────────────┼────────────────────────────────────┤
│ Full responses (>100KB) │ 18/100                    │ 11/100                             │
├─────────────────────────┼───────────────────────────┼────────────────────────────────────┤
│ Truncated (1-100KB)     │ 73/100                    │ 66/100                             │
├─────────────────────────┼───────────────────────────┼────────────────────────────────────┤
│ Empty (0 bytes)         │ 9/100                     │ 23/100                             │
└─────────────────────────┴───────────────────────────┴────────────────────────────────────┘

Admin and Stats Output:

N/A — the truncation produces no errors in stats. All requests return HTTP 200.

Config:

Both ext_proc filters configured via Istio (BBR via EnvoyFilter, EPP via inference extension per-route override):

# Filter 0 (BBR) — inserted by EnvoyFilter
processing_mode:
  request_header_mode: SEND
  response_header_mode: SEND
  request_body_mode: FULL_DUPLEX_STREAMED
  response_body_mode: FULL_DUPLEX_STREAMED
  request_trailer_mode: SEND
  response_trailer_mode: SEND

# Filter 1 (EPP) — configured via per-route override
processing_mode:
  request_header_mode: SEND
  response_header_mode: SEND
  request_body_mode: FULL_DUPLEX_STREAMED
  response_body_mode: FULL_DUPLEX_STREAMED
  request_trailer_mode: SEND
  response_trailer_mode: SEND

Full config and deployment: https://github.com/nerdalert/envoy-ext-proc-bug-repro

Logs:

Envoy ext_proc debug logs for a truncated request (response was 9983 bytes instead of ~130KB):

  # BBR receives 8150 bytes from client in two parts:
  Sending a body chunk of 8150 bytes, end_stream false   (to BBR)
  Sending a body chunk of 0 bytes, end_stream true        (to BBR)

  # BBR processes and injects back — EPP receives it as one chunk:
  Sending a body chunk of 8150 bytes, end_stream true     (to EPP)

  # EPP processes, commonContinue() fires with stale observedEndStream()=true
  Finish external processing call. Next state: 0
  Continuing processing

  # Backend receives truncated request, returns short streaming response:
  Sending a body chunk of 9983 bytes, end_stream false    (response, filter 0)
  Sending a body chunk of 0 bytes, end_stream true        (response, filter 0)

No warnings or errors are logged. The truncation is silent.

Call Stack:

N/A — no crash. The issue is a logic race in commonContinue() at filter_manager.cc:132:

doData(observedEndStream() && !had_trailers_before_data);

Environment:

• Envoy 1.37.1-dev (commit fca9ef95, includes fix PR Support two ext_proc filters in the chain 2nd attempt #43175)
• Istio 1.29.1
• BBR: bbr:main (k8s staging registry)
• EPP: epp:v1.4.0-rc.2 (registry.k8s.io)
• Kind cluster (v1.35.0)

Related:

• ext_proc: FULL DUPLEX STREAMED response body truncated when multiple filters defined #41654 — original bug report
• Support two ext_proc filters in the chain 2nd attempt #43175 — fix (covers one-body-one-header case only)
• gateway-api-inference-extension#2115

Tyvm!

[yanjunxiang-google]

@nerdalert Thanks for your finding! It looks like when the 2nd filter commonContinue() is called, the 1st ext_proc server body response of the 1st filter hadn't arrived Envoy yet. So, basically, the 2nd ext_proc server header response arrives earlier than the 1st ext_proc server body response. Is that the case?

[yanjunxiang-google]

@nerdalert Just for me to understand the root cause,, is your 2nd ext_proc server sending header response as soon as it get the header request? If it is, can you change both of your ext_proc servers to receive all body chunks before sending header response, followed by body responses, and see how it goes?

[nirrozenbaum]

EPP is the second ext proc in this context. it is configured to receive some body chunks before sending header response, unless it receives end of stream on the RequestHeaders (which means there's no body).
see the code here:
https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/8929609927f0a5a2bd33bf56966c19282ae4982a/pkg/epp/handlers/request.go#L37

more specifically in this func, when one defines reqCtx.reqHeadersResp (in L52) it will be sent.
otherwise if it's not end of stream, reqCtx.reqHeadersResp will be set in the function that handles RequestBody.
ref here - https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/handlers/server.go#L250

[yanjunxiang-google]

Thanks for your prompt response! Again, could you please try change your ext_proc server implementation from: "receive some body chunks before sending header response", into "receive all body chunks before sending header response"?

BTW, here are the integration tests cover the case that ext_proc server receives the full body before sending header response:

envoy/test/extensions/filters/http/ext_proc/ext_proc_full_duplex_integration_test.cc

Line 585 in d3c7c2c

TEST_P(ExtProcIntegrationTest, TwoExtProcFiltersBothDuplexInBothDirection) {

[nirrozenbaum]

Thanks for your prompt response! Again, could you please try change your ext_proc server implementation from: "receive some body chunks before sending header response", into "receive all body chunks before sending header response"?

BTW, here are the integration tests cover the case that ext_proc server receives the full body before sending header response:

envoy/test/extensions/filters/http/ext_proc/ext_proc_full_duplex_integration_test.cc

Line 585 in d3c7c2c

TEST_P(ExtProcIntegrationTest, TwoExtProcFiltersBothDuplexInBothDirection) {

that's actually the case here :). I should have been more accurate in my previous response.
epp waits for whole body (all chunks) before returning headers response and body response.

here's the code:

case *extProcPb.ProcessingRequest_RequestBody:
			loggerTrace.Info("Incoming body chunk", "EoS", v.RequestBody.EndOfStream)
			// In the stream case, we can receive multiple request bodies.
			body = append(body, v.RequestBody.Body...)

			// Message is buffered, we can read and decode.
			if v.RequestBody.EndOfStream {
				loggerTrace.Info("decoding")
				reqCtx.Request.RawBody = body

				// Body stream complete. Capture raw size for flow control.
				reqCtx.RequestSize = len(body)
				body = []byte{}

				reqCtx, err = s.director.HandleRequest(ctx, reqCtx)
				if err != nil {
					logger.V(1).Error(err, "Error handling request")
					break
				}

				reqCtx.reqHeaderResp = s.generateRequestHeaderResponse(ctx, reqCtx)
				reqCtx.reqBodyResp = envoy.GenerateRequestBodyResponses(reqCtx.Request.RawBody)

				metrics.RecordRequestCounter(reqCtx.IncomingModelName, reqCtx.TargetModelName)
				metrics.RecordRequestSizes(reqCtx.IncomingModelName, reqCtx.TargetModelName, reqCtx.RequestSize)
			}

ref here - https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/handlers/server.go#L250

[nerdalert]

Thanks for looking @yanjunxiang-google! Looking at the integration test the ordering is explicit. I think the test passes because it's sequential, not because the fix handles concurrent responses.

The fix works when ordering is guaranteed, but not when two independent gRPC servers race. In the deployment which gRPC response arrives at Envoy first is non-deterministic. Looks like both ext_proc servers already wait for the full body before sending any response (thanks Nir!).

[yanjunxiang-google]

emm, if both gRPC servers are buffering the full body (until the end-of-stream=true is received ) before sending response, then it's pretty sequential. When the 1st gRPC server is buffering, nothing will be sent to the 2nd gRPC server. When the 2nd gRPC server is buffering, nothing will be sent to the rest of the filter chain.

BTW, to enable the fix added by for #43175, you have to set a runtime guard: envoy.reloadable_features.ext_proc_inject_data_with_state_update to be true. Please confirm.

[nerdalert]

Thanks @yanjunxiang-google I double checked and it's on by default and enabled.

$ kubectl exec deploy/inference-gateway-istio -c istio-proxy -- \
  curl -s localhost:15000/runtime | grep -A 5 ext_proc
  "envoy.reloadable_features.ext_proc_inject_data_with_state_update": {
   "final_value": "true",
   "layer_values": [
    "",
    "true"
   ]

$ kubectl exec deploy/inference-gateway-istio -c istio-proxy -- /usr/local/bin/envoy --version

/usr/local/bin/envoy  version: fca9ef956bd748bedb43d4c381af3d498dcf0723/1.37.1-dev/Clean/RELEASE/BoringSSL


$ GW_PORT=$(kubectl get svc inference-gateway-istio -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}')
GW_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
ENDPOINT="http://$GW_IP:$GW_PORT/v1/chat/completions"

# Generate 8KB request body
dd if=/dev/zero bs=1 count=8000 2>/dev/null | tr '\0' 'x' > /tmp/pad.txt
printf '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"system","content":"' > /tmp/big.json
cat /tmp/pad.txt >> /tmp/big.json
printf '"},{"role":"user","content":"Hi"}],"max_tokens":500,"stream":true}' >> /tmp/big.json

# Sending some requests (full responses consistently return ~130-135KB)

$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=131468
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=2453
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=0
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=11576
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=10789
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=132770
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=8665
$ curl -s -o /dev/null -w "size=%{size_download}\n"   -X POST "$ENDPOINT"   -H "Content-Type: application/json"   -d @/tmp/big.json
size=135112

Updated the readme here with the feature enabled validation output check.

Thanks a bunch!