HTTP compression support for static responses

[ac000]

This pull-request introduces initial support for HTTP compression of static responses.

It does not include support for compressing applications responses, which will hopefully follow at some point. I thought it best to get the bulk of the work committed in the meantime while I continue took at compressing application responses.

This work introduces some new configuration options. Firstly there is a new settings.http.compression section under which you can specify one or two options

settings.http.compression.types

Is an array where you can specify a list of MIME types that should be considered suitable for compressing.

settings.http.compression.compressors

This is an array of objects describing what compressors to enable. While Unit itself can be compiled with support for one or more compression types, you can then enable one or more of these at run-time.

Each compressor object describes the compression scheme to enable, encoding, an optional level that says what compression level to use and an optional min_length that says what the minimum amount of data that should be compressed is.

An example configuration might look like

"settings": {                                                                   
    "http": {                                                                   
        "compression": {                                                        
            "types": [                                                          
                "text/*"                                                        
            ],                                                                  
            "compressors": [                                                    
                {                                                               
                    "encoding": "gzip",                                         
                    "level": 3,                                                 
                    "min_length": 2048                                          
                },                                                              
                {                                                               
                    "encoding": "deflate",                                      
                },                                                              
                {                                                               
                    "encoding": "zstd",                                         
                    "min_length": 2048                                          
                },                                                              
                {                                                               
                    "encoding": "br",                                           
                    "min_length": 256                                           
                }                                                               
            ]                                                                   
        }                                                                       
    }                                                                           
},

First of all we have a settings.http.compression.types settings which lists what MIME types should be considered for compression, in this case, text files.

Then we have settings.http.compression.compressors which is a list of compression schemes to enable. In this case we enable support for deflate, gzip, zstd and brotli.

For gzip we specify a compression level of 3, the others will just use that schemes default.

For gzip, zstd and brotli we specify minimum lengths for compression, anything under this size won't be compressed.
For deflate we don't set a minimum level so anything matching the requested MIME type of any size will be compressed.

The compressing of static share files is done by creating mmap(2)'s for both the requested file and file compressed, negating the need for temporary buffers. We are also able to properly set the Content-Length for the compressed files.

A note about the compressors. The compressors themselves nxt_{brotli,zlib,zstd}.c have been kept intentionally isolated from any knowledge of the Unit core. They are as small as possible to basically just take a buffer and compress it.

This means adding support for future compression schemes should be relatively trivial.

This pull-request is comprised of the following commits

• http: Add a mime_type member to nxt_http_response_t

This adds a new structure member mime_type to nxt_http_response_t which is used to store the MIME tpe of the response. This is used to determine if the response should be compressed or not.

• http: Add NXT_HTTP_NOT_ACCEPTABLE enum value

This adds a new enum value to represent the "Not Acceptable" status code. This is used to indicate the requested compression scheme wasn't available and we've been told not to send the response uncompressed.

• http: Add core http compression code

This adds the core compression code.

This doesn't include any of the actual compressors just the core infrastructure.

• http: Add zlib compression support
• http: Add support for zstd compression
• http: Add support for brotli compression

These commits add support for deflate, gzip, zstd and brotli compression.

• http: Wire up HTTP compression to the build system

This adds the needed configure scripts to enable support for the above. Any or all of the above compressors can be enabled.

• http: Wire up HTTP compression support to the config system

This wire the above into the config system.

• http: compress: Add a couple of helper functions

This adds a couple of helper functions that will be used to doing the actual compression.

• http: Compress static responses

This adds support for compressing static responses.

The following changes since commit 3b18ffe09370573f81220fda5e924124fcf8f0df:

  tests: Fix TLS tests with Python 3.13 (2025-03-27 23:39:32 +0000)

are available in the Git repository at:

  [email protected]:ac000/unit.git compr

for you to fetch changes up to d428c1c11ddea47e2f9eaffa660c0cd6e1d4cb57:

  http: Compress static responses (2025-04-07 22:15:18 +0100)

----------------------------------------------------------------
Andrew Clayton (10):
      http: Add a mime_type member to nxt_http_response_t
      http: Add NXT_HTTP_NOT_ACCEPTABLE enum value
      http: Add core http compression code
      http: Add zlib compression support
      http: Add support for zstd compression
      http: Add support for brotli compression
      http: Wire up HTTP compression to the build system
      http: Wire up HTTP compression support to the config system
      http: compress: Add a couple of helper functions
      http: Compress static responses

 auto/compression           |  96 +++++++
 auto/help                  |   4 +
 auto/options               |   8 +
 auto/sources               |  16 ++
 auto/summary               |   3 +
 configure                  |   7 +-
 src/nxt_brotli.c           |  78 ++++++
 src/nxt_conf_validation.c  |  93 +++++++
 src/nxt_http.h             |   2 +
 src/nxt_http_compression.c | 667 +++++++++++++++++++++++++++++++++++++++++++++
 src/nxt_http_compression.h | 103 +++++++
 src/nxt_http_static.c      |  31 +++
 src/nxt_router.c           |  10 +
 src/nxt_zlib.c             |  88 ++++++
 src/nxt_zstd.c             |  69 +++++
 15 files changed, 1273 insertions(+), 2 deletions(-)
 create mode 100644 auto/compression
 create mode 100644 src/nxt_brotli.c
 create mode 100644 src/nxt_http_compression.c
 create mode 100644 src/nxt_http_compression.h
 create mode 100644 src/nxt_zlib.c
 create mode 100644 src/nxt_zstd.c

[hongzhidao]

Great work!
Just wondering are you going to include some tests as well?

[ac000]

Great work! Just wondering are you going to include some tests as well?

Good question.

Certainly not as part of this PR, if I have time to figure it out in the future maybe....

[ac000]

Rebased with master

$ git range-diff d428c1c1...adaecb6f
 -:  -------- >  1:  f2e97bf3 rust: Update rust crates
 -:  -------- >  2:  0cbdcb15 pkg/contrib: Bump wasmtime to 31.0.0
 -:  -------- >  3:  326f42a5 Fully initialise nxt_port_msg_t msg structures
 -:  -------- >  4:  89149c06 Fully initialise the oob struct in nxt_socket_msg_oob_init()
 1:  6c0d2514 =  5:  45e40a8c http: Add a mime_type member to nxt_http_response_t
 2:  36729303 =  6:  415e2251 http: Add NXT_HTTP_NOT_ACCEPTABLE enum value
 3:  a4ac851d =  7:  94dffe47 http: Add core http compression code
 4:  311f76fc =  8:  47cbbbbf http: Add zlib compression support
 5:  4f0160f3 =  9:  511b2177 http: Add support for zstd compression
 6:  e8b1de47 = 10:  804ca81f http: Add support for brotli compression
 7:  7fcf61e7 = 11:  c9b2ecd2 http: Wire up HTTP compression to the build system
 8:  a6ed38b2 = 12:  85a00914 http: Wire up HTTP compression support to the config system
 9:  5867a5c2 = 13:  523d4226 http: compress: Add a couple of helper functions
10:  d428c1c1 = 14:  adaecb6f http: Compress static responses

[ac000]

Thanks Zhidao!