Skip to content

Fix HTTP/2 PROTOCOL_ERROR by configuring connection pool lifetimes #570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Fix HTTP/2 PROTOCOL_ERROR by configuring connection pool lifetimes #570

wants to merge 2 commits into from

Conversation

Copy link
Contributor

Copilot AI commented Nov 21, 2025 •
edited
Loading

Long-lived HTTP/2 connections can become corrupted, causing PROTOCOL_ERROR (0x1) failures in gRPC calls. This manifests as HttpProtocolException: The HTTP/2 server sent invalid data on the connection during streaming operations like Topic reads/writes.

Per grpc-dotnet#2641, the fix is to configure connection pool timeouts on SocketsHttpHandler.

Changes

  • Added connection lifecycle settings to DriverConfig

    • PooledConnectionIdleTimeout (default: 60s) - closes idle connections
    • PooledConnectionLifetime (default: 300s) - maximum connection age

  • Applied settings in GrpcChannelFactory.CreateChannel()

    var httpHandler = new SocketsHttpHandler
{
    // ... existing settings ...
    PooledConnectionIdleTimeout = _config.PooledConnectionIdleTimeout,
    PooledConnectionLifetime = _config.PooledConnectionLifetime
};

  • Exposed via connection string properties

    • PooledConnectionIdleTimeout=60;PooledConnectionLifetime=300

Values are configurable but defaults should handle most cases. Connection refresh happens transparently without disrupting active streams.

Original prompt

This section details on the original issue you should resolve

<issue_title>bug: HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)</issue_title>
<issue_description>```
[0] Error: Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)

</issue_description>

## Comments on the Issue (you are @copilot in this section)

<comments>
<comment_new><author>@KirillKurdyukov</author><body>
SLO Topic has detected this problem:

info: Grpc.Net.Client.Internal.GrpcCall[17]
Error reading message.
System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span1 buffer, Boolean partOfSyncRead) at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken) info: Grpc.Net.Client.Internal.GrpcCall[3] Call failed with gRPC error status. Status code: 'Internal', Message: 'Error reading next message. HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)'. System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span1 buffer, Boolean partOfSyncRead)
at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken) at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken) at Grpc.Net.Client.Internal.HttpContentClientStreamReader2.MoveNextCore(CancellationToken cancellationToken)
warn: Ydb.Sdk.Driver[0]
gRPC error [Status(StatusCode="Internal", Detail="Error reading next message. HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")] on channel http://localhost:2137
warn: Ydb.Sdk.Pool.EndpointPool[0]
Endpoint http://localhost:2137 was pessimized. New pessimization ratio: 1 / 5
fail: Ydb.Sdk.Services.Topic.Writer.Writer[0]
WriterSession[producer-7|aea3c619-e3a68a20-469023ba-c714dde0_0] have error on processing writeAck
Ydb.Sdk.Driver+TransportException: Transport exception: Status(StatusCode="Internal", Detail="Error reading next message. HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")
---> Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error reading next message. HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")

<comment_new><author>@KirillKurdyukov</author><body>
https://github.com/grpc/grpc-dotnet/issues/2641</body></comment_new>
<comment_new><author>@KirillKurdyukov</author><body>
https://github.com/ydb-platform/ydb/issues/20813</body></comment_new>
</comments>

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Fix HTTP/2 error code 'PROTOCOL_ERROR' in gRPC calls Fix HTTP/2 PROTOCOL_ERROR by configuring connection pool lifetimes Nov 21, 2025
Copilot AI requested a review from rekby November 21, 2025 11:17
Copilot finished work on behalf of rekby November 21, 2025 11:17
@KirillKurdyukov KirillKurdyukov marked this pull request as ready for review November 23, 2025 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rekby rekby Awaiting requested review from rekby

At least 1 approving review is required to merge this pull request.

Assignees

@rekby rekby

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug: HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)

2 participants

@rekby