Compiling Ok in AlmaLinux 8.10, but not working

[decioaalves]

I managed to compile it ok on AlmaLinux 8.10, a dozen times, in different ways, with 7 different kernels, on three different hardware, and in none of them did I have any problems building the new version of this connector. However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.

But compiling the previous version of this connector, it works normally.

Is this problem happening only to me, or has anyone else noticed this problem?

[decioaalves]

In all times, I have recompiled libmodsecurity.so and nginx before compile ngx_http_modsecurity_module, and in all times the result is only head of the pages are supressed. No problems are shown in the logs.

[airween]

Hi @decioaalves,

... and in none of them did I have any problems building the new version of this connector.

Sorry, what is "this" version? Could you explain it exactly?

However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.

But compiling the previous version of this connector, it works normally.

Here too: what are the "previous" and "this" versions?

[decioaalves]

Hi @decioaalves,
Sorry, what is "this" version? Could you explain it exactly?

I mean the current version of the 'ModSecurity-nginx' module, as you get when you clone with git. After compile, it generates a 'ngx_http_modsecurity_module.so' library normally, with no warnings.

However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.
But compiling the previous version of this connector, it works normally.

Here too: what are the "previous" and "this" versions?

The 'this' version, is the current version. The connector was updated two months ago. The 'previous' version, was a version earlier version, from June, 2024, I was running.

I could to make a debug trace, and I can post it, if you want..

Thank you.

[airween]

The connector was updated two months ago.

You mean about PR #334? That was committed in 2 months ago (19th of Nov, 2024), but was merged few days ago.

To clarify and reproduce this issue we need:

• commit id of "this" version
• commit id of "previous" version
• relevant configs (both Nginx and ModSecurity)
• a curl example

We need to reproduce the issue.

[decioaalves]

Commit id of 'ModSecurity-nginx' module:

'Older version' (working good version):

git rev-parse HEAD
ef64996
v1.0.3-24-gef64996

'current version' (NOT working good version):

git rev-parse HEAD
fb678c5
v1.0.3-27-gfb678c5

System: AlmaLinux 8.10 (Cerulean Leopard)

Kernels tested:

/boot/kernel-4.18.0-553.34.1.el8_10.x86_64
/boot/kernel-4.18.0-553.33.1.el8_10.x86_64
/boot/kernel-4.18.0-553.32.1.el8_10.x86_64
/boot/kernel-4.18.0-553.30.1.el8_10.x86_64
/boot/kernel-4.18.0-553.27.1.el8_10.x86_64
/boot/kernel-4.18.0-553.el8_10.x86_64

ModSecurity installed pre-requisites

$ sudo dnf -y install pcre pcre-devel libxml2 libxml2-devel curl curl-devel openssl openssl-devel yajl yajl-devel
$ sudo dnf -y install gcc-c++ flex bison GeoIP-devel doxygen zlib-devel
$ sudo dnf -y install lmdb lmdb-devel ssdeep ssdeep-devel lua lua-devel

ModSecurity3 commit id installed in hardware #1:

git rev-parse HEAD
9e685bf86d45b21d1f9f6c4145c3979cc4da0196
v3.0.12-296-g9e685bf

ModSecurity3 commit id installed in machine #2 and #3:

git rev-parse HEAD
41fd21b0fbdbdaf202f830f700027767fbd6fba7
v3.0.12-277-g41fd21b

ModSecurity3 installation

$ sudo git clone --depth 100 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
$ cd ModSecurity
$ sudo git submodule init
$ sudo git submodule update

$ sudo libtoolize && aclocal && autoheader && autoconf && automake --add-missing
The last command is due to 'ltmain.sh no found' error, step recommended from: https://stackoverflow.com/questions/22603163/automake-error-ltmain-sh-not-found
$ sudo sh build.sh
$ sudo ./configure
$ sudo make -j /usr/bin/nproc
$ sudo make install
$ sudo ldconfig

Nginx must be customized, due to the need for compatibility with the CWP WebPanel that is installed
Nginx dependencies checked and installed from "Installing NGINX Dependencies" section from https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#installing-nginx-dependencies
Compilation process based in the tips "CentOS 7 Minimal (dynamic)" sequence steps from https://github.com/owasp-modsecurity/ModSecurity/wiki/Compilation-recipes-for-v3.x :

$ sudo dnf groupinstall 'Development Tools' -y
$ sudo dnf install gcc-c++ flex bison yajl yajl-devel curl-devel curl GeoIP-devel doxygen zlib-devel
$ sudo dnf install lmdb lmdb-devel libxml2 libxml2-devel ssdeep ssdeep-devel lua lua-devel
(changed) $ sudo git clone --depth 100 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
$ cd ModSecurity
$ sudo dnf submodule init
$ sudo dnf submodule update
(due to 'ltmain.sh no found' error, step recommended from: https://stackoverflow.com/questions/22603163/automake-error-ltmain-sh-not-found, added:)
(added) # libtoolize && aclocal && autoheader && autoconf && automake --add-missing
$ sudo ./build.sh
(changed) $ sudo ./configure --prefix=/usr/local/cwpsrv --sbin-path=/usr/local/cwpsrv/bin/cwpsrv --conf-path=/usr/local/cwpsrv/conf/cwpsrv.conf --pid-path=/usr/local/cwpsrv/var/run/nginx.pid --error-log-path=/usr/local/cwpsrv/logs/error_log --http-log-path=/usr/local/cwpsrv/logs/access_log --user=root --group=root --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_secure_link_module --with-http_ssl_module --add-module=src/misc/ngx_http_auth_pam_module --with-compat --add-dynamic-module=../ModSecurity-nginx
$ sudo make
$ sudo make install
(changed) $ sudo git clone https://github.com/SpiderLabs/ModSecurity-nginx
(changed) $ sudo wget http://nginx.org/download/nginx-1.15.10.tar.gz
(changed) $ sudo tar zxvf nginx-1.15.10.tar.gz
(changed) # cd nginx-1.15.10
(changed) $ sudo ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
$ sudo make modules
(changed) $ sudo cp objs/ngx_http_modsecurity_module.so /usr/local/cwpsrv/modules
(changed) $ sudo mkdir /usr/local/cwpsrv/conf/security
(changed) $ sudo cp ~/ModSecurity/unicode.mapping /usr/local/cwpsrv/conf/security/

Installed with fresh OWASP CRS 4.3.0, original steps, unchanged
Installation steps based in https://github.com/coreruleset/coreruleset/blob/main/INSTALL.md, just adapted base directoy, instead /etc/crs4/ to /usr/local/cwpsrv/conf/security/crs4/
Added step to install pam-devel to satisfy 'ngx_http_auth_pam_module':

$ sudo dnf install pam-devel

nginx -V

nginx version: nginx/1.15.10
built by gcc 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)
built with OpenSSL 1.1.1k FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/cwpsrv --sbin-path=/usr/local/cwpsrv/bin/cwpsrv --conf-path=/usr/local/cwpsrv/conf/cwpsrv.conf --pid-path=/usr/local/cwpsrv/var/run/nginx.pid --error-log-path=/usr/local/cwpsrv/logs/error_log --http-log-path=/usr/local/cwpsrv/logs/access_log --user=root --group=root --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_secure_link_module --with-http_ssl_module --add-module=src/misc/ngx_http_auth_pam_module --with-compat --add-dynamic-module=../ModSecurity-nginx

cwpsrv -V (original binary built from CWP Web Panel, that was tested too, and didn't work):

cwpsrv version: cwpsrv/1.15.10
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017 (running with OpenSSL 1.0.2o-fips 27 Mar 2018)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/cwpsrv --sbin-path=/usr/local/cwpsrv/bin/cwpsrv --conf-path=/usr/local/cwpsrv/conf/cwpsrv.conf --pid-path=/usr/local/cwpsrv/var/run/nginx.pid --error-log-path=/usr/local/cwpsrv/logs/error_log --http-log-path=/usr/local/cwpsrv/logs/access_log --user=root --group=root --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_secure_link_module --with-http_ssl_module --add-module=src/misc/ngx_http_auth_pam_module

changes to main configuration file from nginx/cwp: /usr/local/cwpsrv/conf/cwpsrv.conf :

worker_processes 2;
+ include /usr/local/cwpsrv/conf/mod-http-modsecurity.conf;
events {
(...)
+ modsecurity on;
+ modsecurity_rules_file /usr/local/cwpsrv/conf/security/conf/modsec_includes.conf;
location / {
(...)

Content from file /usr/local/cwpsrv/conf/mod-http-modsecurity.conf:

load_module modules/ngx_http_modsecurity_module.so;

Original installation from OWASP CRS 4.3.0:
Content from file /usr/local/cwpsrv/conf/security/conf/modsec_includes.conf:

Include /usr/local/cwpsrv/conf/security/crs4/modsecurity.conf (recommended original)
Include /usr/local/cwpsrv/conf/security/crs4/crs-setup.conf
Include /usr/local/cwpsrv/conf/security/crs4/plugins/-config.conf
Include /usr/local/cwpsrv/conf/security/crs4/plugins/-before.conf
Include /usr/local/cwpsrv/conf/security/crs4/rules/.conf
Include /usr/local/cwpsrv/conf/security/crs4/plugins/-after.conf

Synthoms:

-ModSecurity-nginx compiling and installing normally in version v1.0.3-27-gfb678c5, no errors nor warnings
-Enabling ModSecurity-nginx version v1.0.3-27-gfb678c5 does not work as expected, almost always gets partial pages, broken pages, rendered with strange characters, like binary characters. The problem with these broken pages is always at the beginning of the pages, about 30% a 40% of the beginning of the pages is suppressed, completely damaging their rendering.
-Disabling ModSecurity in configuration in nginx configuration (just setting "modsecurity off";" and/or commenting both "modsecurity" and "modsecurity_rules_file" diretives), the partial/broken pages error persist.
-Tested with originally built nginx version from CWP (cwpsrv) as dynamic module, but tested too with compiled sources from nginx original sources, version 1.15.10, but continues not working as expected.
-Almost always get partial/broken web pages, from php pages, with ModSecurity-nginx version v1.0.3-27-gfb678c5
-Note: PHP pages are from closed source control webpanel CWP, I have no access to sources.

Solution:

-When disabled ModSecurity-nginx, the pages come back to normal, working as expected (when is removed completely the directive "load_module modules/ngx_http_modsecurity_module.so;" from nginx configuration file /usr/local/cwpsrv/conf/mod-http-modsecurity.conf) unloading it from nginx.
-Another solution was downgrade to ModSecurity-nginx version v1.0.3-24-gef64996

(I'm having trouble making the request with curl, with a password protected page via POST method, as soon as I can, I'll post it here.)

[airween]

Hi @decioaalves,

thanks, I have these ones:

~/src/ModSecurity-nginx$ git describe
v1.0.3-27-gfb678c5
~/src/ModSecurity$ git describe
v3.0.12-296-g9e685bf8

so the newest commits from both repository.

I also installed Nginx, but during some tests yesterday I didn't get any problem. The curl request would be important, what's the request - and if you turn off the module, then please show us the response from the backend. If it contains any sensitive info, please send that to modsecurity@owasp.org.

Nginx must be customized, due to the need for compatibility with the CWP WebPanel that is installed

I see, but I need to know the configuration to reproduce the issue. There was a PR (#334) which probably can occur this issue, but without details I can't do anything. Under the mentioned PR you can see some comments, a customer who uses heavily this module confirms that this patch works as well. But of course, we need to find a solution - if we see what's the problem.

[DaveeHorvath]

Hi,
An issue similar to this one happened for me in a docker container based from debian:bullseye, nginx version 1.18 (installed from apt) using the configuration directly from https://github.com/coreruleset/coreruleset.git with the latest commit of Modsecurity. The requests for a fairly big file (~39kb) fails, as it made the body of the response scrambled as in similar size but a lot of parts moved over one another. The content size matches as well. I would happy to share the request, response and the original file, if that could help? The repo where the issue occured is https://github.com/DaveeHorvath/ascension, and the failing file was main.js. Any other information that would be needed?

[airween]

@DaveeHorvath,

thanks - do you also use the connector with last commit (fb678c5) with Bullseye?

The content size matches as well. I would happy to share the request, response and the original file, if that could help?

Yes, definitely.

The repo where the issue occured is ...

Sorry, I don't use Docker just very rarely, it would be huge help if you could provide the request (with payload) and an example response (that I can set as content).

And many thanks for these!

[decioaalves]

I was unable to create an example using curl, but I was able to generate two *.har files using Chrome, one with the correct page returned, and the other broken.
The sequence shown in these files represents the call to two pages, the first a simple call to the main page of the CWP system. After logging into this page, the following page represents the page in question being analyzed. For simplicity, I will call it the "second page".
The first file, 'request-good.har' represents these two calls working normally, with version v1.0.3-24-gef64996 of ModSecurit-nginx.
The second correct page is returned on line 956 ("text": ...).

So I then upgraded to version v1.0.3-27-gfb678c5, and generated the second file, 'request-broken.har', which shows these same HTTP calls, but results in the second page being broken, being returned at line 954.

Strangely, after doing a scan through the modsecurity audit logs, it shows the second page as if it had been returned to me in its entirety, rather than broken in half. But I had already tested it in two different browsers, Chrome, Firefox and links(text only), and the result is always the same.

request-good.har.zip
request-broken.har.zip

[airween]

@decioaalves,

thanks - I hope this will help me to investigate the issue. Let me check that in next few days.

Cc @Dr-Lazarus-V2 - if you able to check, please help me in this.

[promiteyka]

Have the same problem, this appeared after the commit fb678c5.
tried to work to work with nginx-1.26.2-1.el7.x86_64(EL) and nginx/stable,now 1.26.0-1 amd64(DEB)
Files more than 32kb are cut off at the front and become invalid

[makar112233]

Hi all. I breaked my brains with debugging same issue.
Please, help with solution.

[DaveeHorvath]

a temporary solution is to set use an older commit of the connector mentioned by @decioaalves, the link ef64996 should be a working version without this issue.

[airween]

Files more than 32kb are cut off at the front and become invalid

You mean if the response is bigger than 32 kB, then it will be cut off?

[promiteyka]

Files more than 32kb are cut off at the front and become invalid

You mean if the response is bigger than 32 kB, then it will be cut off?

We came to this through experiments. We have some static on the site, files larger than 32 kb open on the client without the beginning of the file. Although the size is shown the same as on the server.