Don't send body in HEAD response when using PipeWriter.Advance before headers flushed #59725

BrennanConroy · 2025-01-06T02:01:58Z

Bug exposed/introduced by #8199

Reason this hasn't been found in the >5 years since the bug was introduced is that it requires using PipeWriter.GetMemory(...) + PipeWriter.Advance(...) before flushing the headers and using a HEAD request at the same time. This is very uncommon except for in .NET 9 where we made writing JSON responses use the PipeWriter which uses the GetMemory + Advance path.

The fix is to pass the canWriteBody parameter to WriteDataWrittenBeforeHeaders and only copy the bytes from previous GetMemory + Advance calls to the body if canWriteBody is true.

Edit: Removed Transfer-Encoding: chunked header from non-body responses.
One additional change we might want to consider here, is removing the Transfer-Encoding: chunked header from the HEAD response. We have TransferEncodingNotSetOnHeadResponse and ManuallySettingTransferEncodingThrowsForHeadResponse which show us explicitly not letting the transfer encoding exist on a HEAD response, however according to the RFC 7231 section 4.3.2

The server SHOULD send the same
header fields in response to a HEAD request as it would have sent if
the request had been a GET, except that the payload header fields
(Section 3.3) MAY be omitted.

~~And https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding~~

~~When present on a response to a HEAD request that has no body, it indicates the value that would have applied to the corresponding GET message.~~

~~Both of which indicate it is perfectly fine to include the Transfer-Encoding header.~~

Looking at the history of why we added this restriction to Kestrel, we find aspnet/KestrelHttpServer#952 which was about a browser request hanging when accessing an endpoint that responded with another non-body response in the form of a 304 and we just added the restriction to HEAD response as well since it is a non-body response.

… headers flushed

Copilot

Copilot reviewed 5 out of 6 changed files in this pull request and generated no comments.

Files not reviewed (1)

src/Servers/Kestrel/Core/src/Internal/Http/IHttpOutputProducer.cs: Evaluated as low risk

BrennanConroy · 2025-01-06T21:09:48Z

src/Servers/Kestrel/test/InMemory.FunctionalTests/ResponseTests.cs

@@ -1471,7 +1534,7 @@ public async Task HeadResponseBodyNotWrittenWithSyncWrite()
        await using (var server = new TestServer(async httpContext =>
        {
            httpContext.Response.ContentLength = 12;
-            await httpContext.Response.BodyWriter.WriteAsync(new Memory<byte>(Encoding.ASCII.GetBytes("hello, world"), 0, 12));
+            httpContext.Response.Body.Write(Encoding.ASCII.GetBytes("hello, world"), 0, 12);


#7110 removed the sync write which this test and HeadResponseBodyNotWrittenWithSyncWrite above were explicitly testing.

BrennanConroy · 2025-01-06T21:12:47Z

src/Servers/Kestrel/test/InMemory.FunctionalTests/ResponseTests.cs

            }
        }
    }

    [Fact]
-    public async Task HeadResponseBodyNotWrittenWithAsyncWrite()
+    public async Task HeadResponseHeadersWrittenWithAsyncWriteBeforeAppCompletes()


aspnet/KestrelHttpServer#1204 changed some of the HEAD response tests to only check the response headers are flushed and doesn't fully check that the body doesn't exist. So split the test into two for the flushed headers check, and the lack of body check.

mgravell

LGTM, interesting find!

BrennanConroy · 2025-01-14T21:41:23Z

src/Servers/Kestrel/test/InMemory.FunctionalTests/ResponseTests.cs

+
+    // Rough attempt at checking that a non-body response doesn't affect future body responses
+    [Fact]
+    public async Task GetRequestAfterHeadRequestWorks()


There was a bug in the PR where calling PipeWriter.WriteAsync went through a code path that didn't set _canWriteBody, and since we weren't resetting the value it could result in a normal request after a non-body request not writing the entire body. Fixed by both resetting the _canWriteBody value, and also changing how we set the _canWriteBody to work for all code paths.

src/Servers/Kestrel/Core/src/Internal/Http/Http1OutputProducer.cs

halter73 · 2025-01-16T22:10:08Z

Both of which indicate it is perfectly fine to include the Transfer-Encoding header.

I'm fine with changing the behavior here if we think it's better. Personally, unlike Content-Length, I don't see the value in sending the Transfer-Encoding header in response to a HEAD request, but it's more important to me that writing to the response Stream or BodyWriter behaves the same way.

halter73

Thanks for adding the ResponseBodyMode. It's a lot easier to follow in my opinion than the two bools. I think an Uninitialized state could make things even easier to follow, and switch statements over nested if/else blocks could improve things even more, but these are all style preferences. The change functionally looks solid.

halter73 · 2025-02-28T22:33:52Z

src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs

@@ -1161,9 +1159,9 @@ private HttpResponseHeaders CreateResponseHeaders(bool appCompleted)
        }

        // Set whether response can have body
-        _canWriteResponseBody = CanWriteResponseBody();
+        _responseBodyMode = CanWriteResponseBody() ? ResponseBodyMode.ContentLength : ResponseBodyMode.Disabled;


This is why I suggested an Unitialized state. It seems wrong to have the mode be ContentLength both here and after calling Reset() when we could ultimately end up auto-chunking.

src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs

halter73

❤️

Don't send body in HEAD response when using PipeWriter.Advance before…

1a38b96

… headers flushed

BrennanConroy added feature-kestrel area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions labels Jan 6, 2025

BrennanConroy requested review from halter73, JamesNK and mgravell as code owners January 6, 2025 02:01

BrennanConroy requested a review from Copilot January 6, 2025 02:02

Copilot AI reviewed Jan 6, 2025

View reviewed changes

test + fix

9bb37ca

BrennanConroy commented Jan 6, 2025

View reviewed changes

mgravell approved these changes Jan 7, 2025

View reviewed changes

dotnet-policy-service bot added the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Jan 14, 2025

fix state bug

96c885b

BrennanConroy removed the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Jan 14, 2025

BrennanConroy commented Jan 14, 2025

View reviewed changes

halter73 reviewed Jan 16, 2025

View reviewed changes

src/Servers/Kestrel/Core/src/Internal/Http/Http1OutputProducer.cs Outdated Show resolved Hide resolved

src/Servers/Kestrel/Core/src/Internal/Http/Http1OutputProducer.cs Outdated Show resolved Hide resolved

dotnet-policy-service bot added the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Jan 24, 2025

fb

000f75d

BrennanConroy removed the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Feb 3, 2025

dotnet-policy-service bot added the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Feb 11, 2025

halter73 approved these changes Feb 28, 2025

View reviewed changes

fb

fdf96e3

halter73 approved these changes Mar 6, 2025

View reviewed changes

BrennanConroy merged commit fb513a5 into main Mar 6, 2025
27 checks passed

BrennanConroy deleted the brecon/head branch March 6, 2025 18:56

BrennanConroy removed the pending-ci-rerun When assigned to a PR indicates that the CI checks should be rerun label Mar 6, 2025

dotnet-policy-service bot added this to the 10.0-preview3 milestone Mar 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't send body in HEAD response when using PipeWriter.Advance before headers flushed #59725

Don't send body in HEAD response when using PipeWriter.Advance before headers flushed #59725

BrennanConroy commented Jan 6, 2025 •

edited

Loading

Copilot AI left a comment

BrennanConroy Jan 6, 2025

BrennanConroy Jan 6, 2025

mgravell left a comment

BrennanConroy Jan 14, 2025 •

edited

Loading

halter73 commented Jan 16, 2025 •

edited

Loading

halter73 left a comment

halter73 Feb 28, 2025

halter73 left a comment

Don't send body in HEAD response when using PipeWriter.Advance before headers flushed #59725

Don't send body in HEAD response when using PipeWriter.Advance before headers flushed #59725

Conversation

BrennanConroy commented Jan 6, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

BrennanConroy Jan 6, 2025

Choose a reason for hiding this comment

BrennanConroy Jan 6, 2025

Choose a reason for hiding this comment

mgravell left a comment

Choose a reason for hiding this comment

BrennanConroy Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

halter73 commented Jan 16, 2025 • edited Loading

halter73 left a comment

Choose a reason for hiding this comment

halter73 Feb 28, 2025

Choose a reason for hiding this comment

halter73 left a comment

Choose a reason for hiding this comment

BrennanConroy commented Jan 6, 2025 •

edited

Loading

BrennanConroy Jan 14, 2025 •

edited

Loading

halter73 commented Jan 16, 2025 •

edited

Loading