"HTTP server returned compressed partial data. That should not happen. CheerpJ cannot run."

[eliasen]

When a CheerpJ application tries to access a text file from the server, it returns the error in the console "HTTP server returned compressed partial data. That should not happen. CheerpJ cannot run."

This error message and "defensive" message are found in cheerpOS.js with the following code:

else if(resp.headers.get("Content-Encoding") !== null)
{
	// Be defensive against broken servers that mix up compression and partial content,
	// that is forbidden since it's ambiguous if the partial content is over the compressed
	// or uncompressed data
	ddlOnError(downloader, "HTTP server returned compressed partial data. That should not happen. CheerpJ cannot run.", /*fatal*/true);
}

The comment, error message and "defensive" check are incorrect in several ways:

1. You requested partial content by sending the HTTP header Range: bytes=0-131071. If you didn't want partial content, or don't know how to handle it correctly, simply don't send this header!
2. You requested compressed content by sending the HTTP header Accept-Encoding: gzip, deflate, br, zstd, identity . If you didn't want compressed content, don't send this header! It is not acceptable for users to turn off compression on their webservers to work around this bug.
3. Compression and partial content is in no way ambiguous nor forbidden by RFC 9110. Compression is done first. Then the bytes returned for a Range are respective to the compressed content. It is not ambiguous. The comment, defensive check, and error is written with an incorrect understanding of HTTP.
4. The server actually told you that it sent the complete file. According to RFC 9110, you must check the returned HTTP headers Content-Range: bytes 0-20693/20694. This told you that the compressed response contains the entire file you requested (you requested 131072 bytes) and the error message/check is totally wrong in disregarding it. You should have understood that the file was returned in its entirety and continued processing it. Section 15.3.7 of RFC 9110 says "A client MUST inspect a 206 response's Content-Type and Content-Range field(s) to determine what parts are enclosed and whether additional requests are needed."
5. "That should not happen" is incorrect. It should happen (because you explicitly requested it.)
6. Servers that understand and follow HTTP correctly are not "broken."

The fix is easy: if you don't know how to handle byte ranges correctly, just request the whole file! It will be easier and faster and less abusive to servers than requesting multiple byte ranges in mutliple requests.

This bug makes CheerpJ unable to even download a text file from a server if the server (reasonably) has compression turned on. Even if the server correctly returned you the entire file!

Please see:
RFC 9110

[alexp-sssup]

Hi, first of all I need to ask you to refrain from using this condescending tone when asking for technical support. CheerpJ is a sophisticated tool maintained by a team of expert engineers. Issues might not be as simple as you might think.

Moving on to the problem at hand: HTTP servers attempting both compression and ranges are quite uncommon in our experience. Can you clarify what server are you using?

CheerpJ uses byte ranges extensively and supports them correctly in the most common case of compression not being applied. The "simple" solution you recommend would make CheerpJ extremely inefficient on its main purpose: running large scale Java applications which are often composed of hundreds of megabytes of JAR files. By taking advantage of byte ranges CheerpJ can download only the subset of the files which are actually required at runtime, with extremely positive impacts on download time and size.

Specifically about point (2), you are incorrect, CheerpJ does not request compression. The browser does it automatically and outside the control of the application. As a matter of fact, if it were possible to disable compression, CheerpJ would do so.

Although you are right that in this specific case the content is complete, your argument would not apply in the general case of compressed files not fitting into a single download chunk.

Compression is done first. Then the bytes returned for a Range are respective to the compressed content.

I am also unsure about how this could ever work in practice and would appreciate a precise source. A partial compressed stream could lack appropriate gzip (for example) headers and as such the browser would not be able to decode it.

I believe you are right in asserting that CheerpJ should be more robust to servers behaving in unexpected ways. The proper solution would be to re-download the content as a plain full request in these dubious cases. This is already in our plans, but not an immediate priority.

Your idea of avoiding the redundant download if it the content-range proves the file is complete could also be considered as an optimization with this extended strategy.

The recommended immediate workaround is to disable HTTP compression for the resources you need to use as files from CheerpJ.

A common solution is to keep all these assets in a separate sub-directory to simplify server rules while still applying compression to plain HTTP assets. Another option is to manually download the resources using fetch and make them available to CheerpJ using the cheerpOSAddStringFile API (https://cheerpj.com/docs/reference/cheerpOSAddStringFile)

Consider joining our Discord for further questions and support: https://discord.leaningtech.com

[eliasen]

Moving on to the problem at hand: HTTP servers attempting both compression and ranges are quite uncommon in our experience. Can you clarify what server are you using?

The server is Apache (2.4.63). It is is the first-or second-most popular webserver in the world depending on how you count it. (See Netcraft Survey) Apache is known to focus on implementing standards correctly.

I am also unsure about how this could ever work in practice and would appreciate a precise source.

The most recent precise source for the HTTP protocol is RFC 9110 (linked in original.) In short, section 14.1.2 Byte Ranges says "If the representation data has a content coding applied" (i.e. compression) ", each byte range is calculated with respect to the encoded sequence of bytes, not the sequence of underlying bytes that would be obtained after decoding." So this is clear about how byte ranges and content-encoding headers interact.

As a check for backwards compatilbility, this behavior is not new and can be traced back to RFC 2068 or RFC 2616 from the year 1999 which in section 14.35.1 states "Byte range specifications in HTTP apply to the sequence of bytes in the entity-body (not necessarily the same as the message-body).". In the language of RFC2616, the "entity-body" refers to the data after content-encoding (i.e. compression) has been applied. (See section 7.2.1 of RFC 2616)

In either case, the modern standard and the standards dating back to 1997 are all in agreement on the correct behavior: content is compressed first according to the Content-Encoding header and then the byte range from a Range header is applied to the bytes of the the compressed content. The authors of the HTTP spec have always clearly specified that compression and range headers are expected to interact, have clearly defined the order of operations, have showed how byte-ranges are supposed to apply to compressed content, showing it is in fact expected and well-defined behavior (and not forbidden, ambiguous, broken, nor dubious, and is in fact expected and well-defined behavior.)

A partial compressed stream could lack appropriate gzip (for example) headers and as such the browser would not be able to decode it.

That is correct. A client implementation again, in the words of RFC 9110, "A client MUST inspect a 206 response's Content-Type and Content-Range field(s) to determine what parts are enclosed and whether additional requests are needed." it's possible that a simple check for a completed range in the "defensive" JavaScript check would make many files work correctly and would help follow the requirements set by the HTTP standard.

[eliasen]

By the way, this behavior is also confusing to the developer because it throws a FileNotFoundException in the Java application when the developer can see in the browser's Networking tab and on the web server that the file was delivered correctly to the client.