Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http2: add session tracking and graceful server shutdown of http2 server #57586

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

pandeykushagra51
Copy link

@pandeykushagra51 pandeykushagra51 commented Mar 21, 2025

This change adds proper tracking of HTTP/2 server sessions to ensure they are gracefully closed when the server is shut down. It implements:

  • A new kSessions symbol for tracking active sessions
  • Adding/removing sessions from a SafeSet in the server
  • A closeAllSessions helper function to properly close all active sessions
  • Updates to Http2Server and Http2SecureServer close methods

Breaking Change: any client trying to create new requests on existing connections will not be able to do so once server close is initiated

Fixes: #57611
Refs: https://datatracker.ietf.org/doc/html/rfc7540#section-9.1
Refs: https://nodejs.org/api/http.html#serverclosecallback

More Details:

Purpose

This PR implements proper tracking and graceful shutdown of HTTP/2 sessions when a server is closed. It addresses the gap between the documented behavior of server.close() and its actual implementation for HTTP/2 servers. Also if server have fired close, we should allow it to close as early as possible to free up system resources quickly. This change ensure that once server wants to close, goaway frame will be sent to every open session(connection) so that client get to know that server started shutdown process and client should not send any new request on existing connection.

Current Issues

  • Currently, HTTP/2 servers are at the mercy of clients to close connections.
  • There is no mechanism to notify clients that the server wants to shut down.
  • Misbehaving/Non-compliant clients can keep connections open indefinitely, preventing proper server shutdown.
  • Clients can continue to initiate new requests on existing connections even when the server is trying to close.
  • Start of server shutdown can starve indefinitely, leading to processes that cannot terminate cleanly and causing resource leaks.

Implementation Details

  • Added a kSessions symbol and SafeSet to track active HTTP/2 sessions.
  • Implemented session registration when sessions are created.
  • Created a closeAllSessions helper function to initiate graceful shutdown of all tracked sessions
  • Updated Http2Server and Http2SecureServer close methods to use this functionality

Behavior Clarification

According to the Node.js documentation for server.close():

server.close stops the server from accepting new connections and closes all connections connected to this server which are not sending a request or waiting for a response.

This PR ensures that HTTP/2 servers follow this behavior by:
Closing existing HTTP/2 sessions gracefully, which means:

  • When session.close() is called, a GOAWAY frame is sent to notify clients that the server wants to close the session
  • No new streams (requests) can be initiated on existing connections
  • Existing streams can continue writing/reading data until completion
  • In-flight requests will be allowed to complete
  • Sessions will be terminated after all active streams complete

This implementation aligns with the HTTP/2 protocol specification for connection management as described in RFC 7540 Section 9.1.

API Impact

This is technically a breaking change, as clients attempting to create new requests on existing HTTP/2 connections will be unable to do so once server.close() is called. However, this behavior now correctly matches the documented behavior for HTTP servers.

Testing

This PR includes tests to verify that HTTP/2 sessions are properly tracked and gracefully shut down when the server is closed.

Newly added test files:

  • test-http2-request-after-server-close.js: Verifies that after server.close() is called, clients cannot initiate new requests while existing requests are allowed to complete. Confirms the server properly terminates once all connections are closed.
  • test-http2-server-close-client-behavior.js: Validates that idle connections are immediately closed when server.close() is called, while connections with active streams continue processing until completion. Ensures the server terminates after all connections are properly closed.

Modified test files:

  • test-http2-compat-serverresponse-statusmessage-property-set.js: After server close behavior is updated, this test started to fail. The failure happened because the request from client is under pendingAck state so the session is not yet initiated, as server close is fired during this time and hence client didn't received response which leads to test failure. Now the test is updated to fire server close once the request is complete. As this unit test, test that client should receive http2 status message response, so it is good to close the server only after the request/response cycle is completed. This line tells that pending stream will be cancelled once session close is emitted.
  • test-http2-compat-serverresponse-statusmessage-property.js: Same as above.
  • test-http2-capture-rejection.js: Server close should be fired after server have sent push promise frame.

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/http2
  • @nodejs/net

@nodejs-github-bot nodejs-github-bot added http2 Issues or PRs related to the http2 subsystem. needs-ci PRs that need a full CI run. labels Mar 21, 2025
@pandeykushagra51
Copy link
Author

This strategy is adopted in other server like Springboot (Tomcat) and golang as well for gracefull server close

@pandeykushagra51
Copy link
Author

pandeykushagra51 commented Mar 22, 2025

Please review the changes, I will be very happy to work on any suggestion and any scope of improvement.

@pandeykushagra51 pandeykushagra51 changed the title http2: add session tracking and graceful shutdown of http2 server http2: add session tracking and graceful server shutdown of http2 server Mar 23, 2025
@mcollina mcollina requested review from atlowChemi and jasnell March 27, 2025 16:41
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mcollina mcollina requested a review from pimterry March 27, 2025 16:42
@mcollina mcollina added the semver-major PRs that contain breaking changes and should be released in the next major version. label Mar 27, 2025
Copy link

codecov bot commented Mar 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.21%. Comparing base (af75d04) to head (90acf08).
Report is 108 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #57586      +/-   ##
==========================================
- Coverage   90.23%   90.21%   -0.02%     
==========================================
  Files         630      630              
  Lines      185055   185547     +492     
  Branches    36221    36392     +171     
==========================================
+ Hits       166984   167397     +413     
+ Misses      11043    11030      -13     
- Partials     7028     7120      +92     
Files with missing lines Coverage Δ
lib/internal/http2/core.js 95.67% <100.00%> (+0.06%) ⬆️

... and 80 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lpinca
Copy link
Member

lpinca commented Mar 27, 2025

  • A new kSessions symbol for tracking active sessions
  • Adding/removing sessions from a SafeSet in the server
  • A closeAllSessions helper function to properly close all active sessions
  • Updates to Http2Server and Http2SecureServer close methods

I just took a quick look, but can't all of this be done in userland? What is the advantage of doing this in core and forcing session tracking for everyone?

@pandeykushagra51
Copy link
Author

  • A new kSessions symbol for tracking active sessions
  • Adding/removing sessions from a SafeSet in the server
  • A closeAllSessions helper function to properly close all active sessions
  • Updates to Http2Server and Http2SecureServer close methods

I just took a quick look, but can't all of this be done in userland? What is the advantage of doing this in core and forcing session tracking for everyone?

No doubt that this can be done on user side as well but this feature is surely going to provide convenience to users.
In my opinion, once user call server.close(), it should be responsibility of nodejs to close the server and handle all internal things. It will be a bit unconventional to ask user to first close the server and then manually intercept all session and close session from there. The user should be free once they call server.close() and should rely on server that it can handle everything else.
Regarding tracking for everyone, we follow this approach in net and http module as well where we store data related to connection for each net server.
here is reference:

node/lib/net.js

Line 1800 in 0a91e98

this._connections = 0;

this[kConnections] ||= new ConnectionsList();

Also this strategy is widely adopted in other server like go and springboot (tomacat).

@pandeykushagra51
Copy link
Author

It will be really helpful if someone can tell what could be possible reason for test failure? I have tried running them locally (on mac-os) and all test test passed as expected.

@pandeykushagra51
Copy link
Author

  • A new kSessions symbol for tracking active sessions
  • Adding/removing sessions from a SafeSet in the server
  • A closeAllSessions helper function to properly close all active sessions
  • Updates to Http2Server and Http2SecureServer close methods

I just took a quick look, but can't all of this be done in userland? What is the advantage of doing this in core and forcing session tracking for everyone?

Also I have seen an issue #55459 which is under triaged state, that can also be done easily once this PR is merged.
As this PR add session tracking, we would only need to iterate the connection list and just destroy the session.

@pandeykushagra51
Copy link
Author

update regarding failing test cases:

test-http2-server-close-client-behavior (macOS): The test failed because the server should ideally close once all sessions are terminated. On my local macOS machine, it passes as expected, but it fails on CI—likely due to resource constraints. The failure appears to be caused by a slight delay of around 80ms in the actual server shutdown. This indicates test flakiness rather than an issue with the server itself. To address this, I will update the test to allow some flexibility, ensuring it still passes even if the server takes an additional 200-300ms to close. This accounts for internal processes like the garbage collector, which may introduce slight delays.

test-http2-server-http1-client.js(ubuntu-24.04, ubuntu-24.04-arm): Although these tests passed locally but not sure why failing on CI, will investigate further and keep posted here

@lpinca
Copy link
Member

lpinca commented Mar 28, 2025

In my opinion, once user call server.close(), it should be responsibility of nodejs to close the server and handle all internal things. It will be a bit unconventional to ask user to first close the server and then manually intercept all session and close session from there.

That's not how it works in Node.js. The close method of net.Server and tls.Server does not close active connections by design. It is user responsibility to do that.

The close method of http.Server also worked like this. In version 19.0.0, the behavior was changed to automatically close idle connections, but active connections (those that are sending a request or receiving a response) are kept open. I would be ok to align http2 to this behavior, but is there a way to find idle sessions?

Changing the close method behavior is a breaking change, and for consistency with other servers, it should not be done in my opinion. As written in the previous comment, the desired behavior can easily be achieved in userland (with the exact same code, no core changes are needed).

As for testing, if possible, do not use timers as they make the tests flaky.

@pandeykushagra51
Copy link
Author

pandeykushagra51 commented Mar 28, 2025

I would be ok to align http2 to this behavior, but is there a way to find idle sessions?

In my opinion, any session which are not having any open request/stream will be referred as idle session. Please let me know your thoughts on this.

That's not how it works in Node.js. The close method of net.Server and tls.Server does not close active connections by design. It is user responsibility to do that.

Confirming the behavior again, in current changes, active sessions are not closed.

Copy link
Member

@pimterry pimterry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @pandeykushagra51!

HTTP servers server.close() nowadays does close idle connections, and personally I'd be happy for HTTP/2 servers to match that behaviour, although as a breaking change. The basic approach seems sensible to me (although note the race condition I've commented on here).

AFAICT it does seem like this PR does actually match that close-idle behaviour already (it calls session.close(), which apparently allows all pending streams to complete before closing the session) so this isn't unreasonable. That said, we do definitely need a test which calls server.close() while a request is still pending and then checks it completes OK and closes afterwards.

The current tests here need some substantial changes though I think. Most importantly: we shouldn't be using setTimeout and timing checks anywhere in here. That will make these tests very flaky (as shown by CI here) and much slower than they need to be. There's also quite complicated, it seems like they're testing a few different things and it's not exactly clear what's going on.

We should avoid depending on specific timing (by instead waiting for events, and then triggering the next step after each previous event, etc), and these tests should be simplified and/or broken into separate tests too I think. With those changes, that will probably fix the CI test failures here. You might find it useful to take a look at the HTTP & net server close tests for some examples.

@@ -3205,6 +3227,7 @@ class Http2SecureServer extends TLSServer {
if (this[kOptions].allowHTTP1 === true) {
httpServerPreClose(this);
}
closeAllSessions(this);
ReflectApply(TLSServer.prototype.close, this, arguments);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same race condition as below. We might also need to do this before httpServerPreClose as well (needs investigation) and we should check this works for any HTTP/1 connections here too.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @pimterry could you please review again

@pandeykushagra51
Copy link
Author

Thanks for the contribution @pandeykushagra51!

HTTP servers server.close() nowadays does close idle connections, and personally I'd be happy for HTTP/2 servers to match that behaviour, although as a breaking change. The basic approach seems sensible to me (although note the race condition I've commented on here).

AFAICT it does seem like this PR does actually match that close-idle behaviour already (it calls session.close(), which apparently allows all pending streams to complete before closing the session) so this isn't unreasonable. That said, we do definitely need a test which calls server.close() while a request is still pending and then checks it completes OK and closes afterwards.

The current tests here need some substantial changes though I think. Most importantly: we shouldn't be using setTimeout and timing checks anywhere in here. That will make these tests very flaky (as shown by CI here) and much slower than they need to be. There's also quite complicated, it seems like they're testing a few different things and it's not exactly clear what's going on.

We should avoid depending on specific timing (by instead waiting for events, and then triggering the next step after each previous event, etc), and these tests should be simplified and/or broken into separate tests too I think. With those changes, that will probably fix the CI test failures here. You might find it useful to take a look at the HTTP & net server close tests for some examples.

thanks a lot @pimterry for your input, I will b working on these and will update the PR as requested

@lpinca
Copy link
Member

lpinca commented Mar 28, 2025

In my opinion, any session which are not having any open request/stream will be referred as idle session. Please let me know your thoughts on this.

It makes sense.

@lpinca
Copy link
Member

lpinca commented Mar 28, 2025

@pandeykushagra51 can you please rebase this against main, apply the new changes and force push here instead? In this way we have all the context/comments in the same place.

@pandeykushagra51
Copy link
Author

@pandeykushagra51 can you please rebase this against main, apply the new changes and force push here instead? In this way we have all the context/comments in the same place.

done the changes, thanks for informing 🙂

This change adds proper tracking of HTTP / 2 server sessions
to ensure they are gracefully closed when the server is
shut down.It implements:

- A new kSessions symbol for tracking active sessions
- Adding/removing sessions from a SafeSet in the server
- A closeAllSessions helper function to close active sessions
- Updates to Http2Server and Http2SecureServer close methods

Breaking Change: any client trying to create new requests
on existing connections will not be able to do so once
server close is initiated

Refs: https://datatracker.ietf.org/doc/html/rfc7540\#section-9.1
Refs: https://nodejs.org/api/http.html\#serverclosecallback
1. Fix server shutdown race condition
   - Stop listening for new connections before closing existing ones
   - Ensure server.close() properly completes in all scenarios

2. Improve HTTP/2 tests
   - Replace setTimeout with event-based flow control
   - Simplify test logic for better readability
   - Add clear state tracking for event ordering
   - Improve assertions to verify correct shutdown sequence

This eliminates a race condition where new sessions could connect
between the time existing sessions are closed and the server stops
listening, potentially preventing the server from fully shutting down.
Comment on lines 56 to 58
// When the request completes
req.on('end', common.mustCall());

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// When the request completes
req.on('end', common.mustCall());

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 61 to 62
// Should receive all data
assert.strictEqual(receivedData, 64 * 1024 * 16);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Should receive all data
assert.strictEqual(receivedData, 64 * 1024 * 16);
// Should receive all data
assert.strictEqual(req.readableEnded, true);
assert.strictEqual(receivedData, 64 * 1024 * 16);
assert.strictEqual(req.writableFinished, true);

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

});

stream.on('close', common.mustCall(() => {
assert.strictEqual(stream.writableFinished, true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert.strictEqual(stream.writableFinished, true);
assert.strictEqual(stream.readableFinished, true);
assert.strictEqual(stream.writableFinished, true);

Copy link
Author

@pandeykushagra51 pandeykushagra51 Apr 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just found from nodejs docs that readable do not have readableFinished property but have readableEnded so replaced with that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was a typo.


// Initiate the server close before client data is sent, this will
// test if the server properly waits for the stream to finish
server.close(common.mustCall());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
server.close(common.mustCall());
server.close();

Feel free to keep the callback, it is not wrong, but in my opinion it is not useful for the purpose of the test, it is only additional overhead.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sense, done the changes

Comment on lines +21 to +23
stream.on('close', common.mustCall(() => {
assert.strictEqual(stream.writableFinished, true);
}));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
stream.on('close', common.mustCall(() => {
assert.strictEqual(stream.writableFinished, true);
}));

Feel free to keep it, but I don't think it is useful as the server is closed after the request ends (the 'end' event is emitted on the client).

Copy link
Author

@pandeykushagra51 pandeykushagra51 Apr 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a but different opinion, I would like to keep it

Copy link
Author

@pandeykushagra51 pandeykushagra51 Apr 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should ensure that server side stream have closed before server close

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server does not close if there are open streams, no?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO if there are open stream we can’t call it graceful closure

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the point, if the stream is not closed via server.close(), the process does not exit. If the process exits then it means that the stream and the server were closed. stream.writableFinished is already tested on the client side. Anyway, it does not matter.

Copy link
Author

@pandeykushagra51 pandeykushagra51 Apr 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should still remove that?

// Make an initial request
const req = client.request({ ':path': '/' });

req.on('response', common.mustCall());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
req.on('response', common.mustCall());

I don't think this is useful for the purpose of the test.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done the changes

assert.strictEqual(data, 'hello');
// Close the server as client is idle now
setImmediate(() => {
server.close(common.mustCall());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
server.close(common.mustCall());
server.close();

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@pandeykushagra51 pandeykushagra51 requested a review from lpinca April 7, 2025 18:20
@pimterry pimterry added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 8, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 8, 2025
@nodejs-github-bot
Copy link
Collaborator

@pandeykushagra51
Copy link
Author

@mcollina @ronag this PR is having two approval and no any pending requested changes.
Could you please let me know what things need to be followed from my end to get this PR merged.
Also is there anything to do with failing checks?

@nodejs-github-bot
Copy link
Collaborator

@pimterry
Copy link
Member

pimterry commented Apr 8, 2025

@pandeykushagra51 We should wait for @lpinca to re-review, but once that's sorted then there's nothing else required from your side, we'll do the steps to wrap it up and merge it.

The CI tests can be a little flaky, but I don't think there's anything related to this change that's failing there, so nothing to worry about. We'll re-run those as required and sort that out, and then this PR will be landed.

@lpinca
Copy link
Member

lpinca commented Apr 8, 2025

My comments/requests have been addressed. I do not approve because I think it does not belong to core. The argument for consistency with httpServer.close() is valid. I'm neutral.

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with a nit

if (sessions.size > 0) {
sessions.forEach((session) => {
session.close();
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please use a for...of loop?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah sure

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 9, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 9, 2025
@nodejs-github-bot
Copy link
Collaborator

@pandeykushagra51
Copy link
Author

hey @lpinca @pimterry @mcollina I found an issue with graceful http2 session closure and implemented fix at #57808 . I will be very happy to optimise and improve it further in case of any scope and will be eagerly waiting for your feedback,

Sorry for putting it here but I think this is a potential bug (may be security related) so thought of informing here to fast forward process.

Thankyou

@pimterry pimterry added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Apr 11, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 11, 2025
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. http2 Issues or PRs related to the http2 subsystem. needs-ci PRs that need a full CI run. semver-major PRs that contain breaking changes and should be released in the next major version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

http2 server wait infinitely to close
6 participants