Hyper returns status 431 instead of 414 if url overflows max_buf_size

[jiahaoliang]

Version
Hyper 0.14.30

Platform
ALL

Description

In my user case I use hyper as server and config it with a fixed http1_max_buf_size

hyper::Server::builder(incoming)
        .http1_max_buf_size(64 * 1024)
        ..serve(hyper::service::make_service_fn(... )

If the incomming url is greater than 64k, hyper returns 431 Request Header Fields Too Large instead of 414 URL TOO LONG. And since hyper will close connection afterwards, my service_fn is unable to catch this exception.(Or maybe it's possible but I just don't know how to)

I inspected the code briefly and found out the problem is here: here

if url overflows max_buf_size hyper just returns Error::new_too_large() and parsed as 431 Request Header Fields Too Large

I also inspect the parse_headers() implememtation, it seems that hyper must consume the whole buf to parse both request line and headers correctly. If the buf overflows, the current implemementation is unable to tell the difference between url too long or header too large.

I am wondering if this is a bug or any special consideration for the current implemementation/

[code sample that causes the bug]

I expected to see this happen: 414 URL TOO LONG

Instead, this happened: 431 Request Header Fields Too Large

[seanmonstar]

That's true. So far, I didn't see a great need to differentiate between the two errors. The 431 is the more general error, since the URL can be considered part of the request headers (in HTTP/1, it's part of the message head, and HTTP/2. it's in the headers frame as :path).

Is there much to gain from returning 414 instead?

[jiahaoliang]

@seanmonstar Thank you for taking the time to explain your perspective.

For HTTP/2, your point makes perfect sense. For HTTP/1.1, however, RFC 6585 §5 explicitly ties the 431 status to oversized header fields:

The 431 status code indicates that the server is unwilling to process the request because its header fields are too large.

RFC 7230 §3 further differentiates the Start Line (containing the URL) from Header Fields as separate sub sections. This makes me wonder if using 414 (URI Too Long) might better align with the spec’s structure when the URL exceeds limits, reserving 431 strictly for header-related issues.

On a related note, for applications using Hyper, I’m also curious if there’s a recommended approach for gracefully handling such library errors and converting them into user-friendly responses—without resorting to setting an overly large http1_max_buf_size.

Would love to hear your insights!

Thanks again for your time and expertise! 🙌

[seanmonstar]

Thanks for the links! I am aware of the details in the RFCs, my feeling still stands: they're essentially the same thing.

Is there much to gain from returning a separate status code? Are there workflows that cannot exist because of what hyper currently does? I'm interested in real world usage too.

[jiahaoliang]

@seanmonstar Sure, Let me share a simplified example to illustrate the ambiguity.

As memtioned before, my server config max_buf to prevent unexpected oom:

hyper::Server::builder(incoming)
        .http1_max_buf_size(64 * 1024)
        ..serve(hyper::service::make_service_fn(... )

My server provides an API acceptting batch query from clients
GET /rest/demo/resources?query=someLongID1,someLongID2,someLongID3,someLongID4,....

I set a max limit of IDs as 1000 IDs. My service_fn parsed the URL and verify if it is more than 1000. If the limit is hit, it returns

414 URL TOO LONG
{
    "error": "URL is too long"
    "reason": "Only allow a maximum of 1000 query items"
}

Senario 1:
Client sends 1001 IDs, service_fn is able to get the URL and successfully return 414 as expected.

Senario 2:
Client sends 1500 IDs, service_fn is unable to get the URL and hyper just return 431 without further explanantion.

Senario 3:
Client sends 999 IDs, and a header value that happens to make URL+header hit the http1_max_buf_size. The service_fn is unable to get the URL and hyper just return 431 without further explanantion.

This ambiguity means clients can’t act meaningfully on 431 responses. With 414, they know to shorten their URL; with 431, they might waste time checking headers instead.

[jiahaoliang]

@seanmonstar Hi there! Just wanted to follow up on this thread when you have a moment. I’d love to hear your thoughts on whether differentiating 414/431 is feasible or if there’s a recommended pattern to handle these cases.

No rush at all, of course. Just eager to resolve the ambiguity for API clients. If there’s anything I can clarify or test further, let me know—happy to help.

Thanks again for your time and for maintaining Hyper!

[seanmonstar]

It's possible hyper could add some more code to better determine if the Start-Line never finished, when seeing a TooLarge error. Is that sufficient?

Still, though, while I understand very well the wording in the RFCs, I believe in practice the two different status codes don't provide much benefit. In either case, the "head" was too big. I find it fairly unlikely that a client is going to know the difference and would dynamically reduce the query parameters versus the header fields. 🤷

[jiahaoliang]

@seanmonstar thank you for the insight!

It's possible hyper could add some more code to better determine if the Start-Line never finished, when seeing a TooLarge error. Is that sufficient?

I agree that differentiating the ErrorKind like you said is sufficient for most of my use case.

I’ve also been brainstorming whether Hyper could expose a way for applications to catch parsing errors and customize the response—for instance, via a callback. This would let developers return specific status codes (414, 431) or attach detailed error messages.

Admittedly, this later part is more of a feature request, and I’m unsure how feasible it is within Hyper’s architecture. If you think this aligns with Hyper’s design goals, I’d be thrilled to contribute a PR if you could kindly provide with some guidance!

[shitcodebykaushik]

This behavior might not be a bug but rather a design choice based on how hyper handles HTTP request parsing. HTTP/1.1, as specified in RFC 7230, does not make a clear distinction between the URL and the headers in terms of buffer sizes. The HTTP/1 spec treats the request line and headers as a single block of data, so when the size exceeds the buffer limit, the error returned is more likely to be associated with headers as a whole (431).

However, it does seem that distinguishing between these two error conditions (URL too long vs. header too large) would be useful, especially for debugging or handling these cases more gracefully in your application.

Custom handling is one of the solution . from my pov

match hyper::Server::builder(incoming)
    .http1_max_buf_size(64 * 1024)
    .serve(hyper::service::make_service_fn(...)) 
{
    Ok(_) => { /* normal handling */ },
    Err(e) => {
        if let Some(hyper::Error::TooLarge(_)) = e.downcast_ref::<hyper::Error>() {
            // Handle URL too long or header too large
            return Err(hyper::Error::from(hyper::http::Error::HeaderOverflow));
        } else {
            // Handle other types of errors
        }
    }
}

Rest would love to know from your side ,