Utility change: nginx + libmodsecurity3 infrastructure, with fail-fast as new CL argument

[Sebitosh]

Description

#11

Like the apache2 infrastructure, provides the necessary configuration files to run the test set using mrts.py on nginx + libmodsecurity3. I discovered that the existing test set fails on libmodsecurity 3.0.14 , so I changed the script to provide the --fail-fast option of go-ftw as an optional flag instead of a default behavior to watch every failed test.

Current output

I have left the small logging messages from nginx so that we understand what is happening with the local installation, mine gives this:

$ ./mrts/mrts.py -i config_infra/nginx_linux/ -r config_tests/ -e generated/rules/ -t generated/tests/regression/tests/
Generate rules and tests
Launch backend
Launch infrastructure
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2025/04/23 19:18:09 [notice] 158836#158836: ModSecurity-nginx v1.0.3 (rules loaded inline/local/remote: 0/104/0)
2025/04/23 19:18:09 [notice] 158836#158836: libmodsecurity3 version 3.0.14
Executing test set...
Error: failed 52 tests
💥💥💥 Failure: test set failed
Backend shutdown
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2025/04/23 19:18:09 [notice] 158852#158852: ModSecurity-nginx v1.0.3 (rules loaded inline/local/remote: 0/104/0)
2025/04/23 19:18:09 [notice] 158852#158852: libmodsecurity3 version 3.0.14
2025/04/23 19:18:09 [notice] 158852#158852: signal process started
Infrastructure shutdown
MRTS completed

The attempt at opening /var/log/nginx/error.log is because of the compiled path values, but I overwrite this in the start.py script to avoid the crash. Also I discovered that the modsecurity_rules_file line in the configuration does not use the given prefix on the nginx command line, so the start.py and stop.py are a bit adapted around that.

I was very confused as to why the tests failed, and after some research I found out that it is because no logs are happening for rules in phase 4. I confirmed this by rewriting the tests with only phases 1 to 3 and then they all pass (I am attaching files with the output for reference). I am unsure if this is behavior due to libmodsecurity3 or the configuration I provided. @airween do you know what is causing this ?

As for the apache2 infrastructure, the nginx server is run on port 80 and is a reverse proxy to albedo on port 8000. This implies that to run the script as default user, one must allow nginx to own port 80 as a non-root user.

Also, if using the provided go-ftw config file in the nginx_linux directory, the mrts.py utility must be run from the root of the project as demonstrated above.

The configuration assumes all the module binaries are located at /usr/lib/nginx/modules/ .

phase1-3-success.ftw.output.txt
phase1-4-failure.ftw.output.txt

[Sebitosh]

Fixed the phase 4 issue, found in the debug log that phase 4 did not process because the content header was missing. Now I use more_set_headers to set one if the response does not have any.

I do still have an issue:

While most runs run fine, some can report a failure. It looks like some logging does not happen when it has to log a lot of stuff. To mitigate this, I removed access logging and audit logging of the configuration. This reduces the number of failed runs but does not look like a complete fix. I am unsure if something can be done at the configuration level to help further solve this, or if this is a known issue of V3. To go further in relaxing the load to avoid this error we could use go-ftw's rate limiter. I tried to look at the configuration used in the CI pipeline for CRS but did not find something that helped them avoid this issue.

[airween]

Hi @Sebitosh,

I was very confused as to why the tests failed, and after some research I found out that it is because no logs are happening for rules in phase 4. I confirmed this by rewriting the tests with only phases 1 to 3 and then they all pass (I am attaching files with the output for reference). I am unsure if this is behavior due to libmodsecurity3 or the configuration I provided. @airween do you know what is causing this ?

I'm a bit confused about the naming convention. I tried the generated rule set with generated test cases, and this is what I see:

👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: ✔ passed in 3.786942ms (RTT 50.910581ms)
	running 2-2: ✔ passed in 4.346553ms (RTT 51.167352ms)
	running 2-3: ✔ passed in 3.909812ms (RTT 51.032199ms)
	running 2-4: ✔ passed in 4.124065ms (RTT 51.156972ms)
👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: 💥 2-1 failed in 3.578781ms (RTT 50.839066ms)
	running 2-2: 💥 2-2 failed in 4.591623ms (RTT 51.206085ms)
	running 2-3: 💥 2-3 failed in 4.698514ms (RTT 51.174826ms)
	running 2-4: 💥 2-4 failed in 4.652678ms (RTT 50.939284ms)
👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: ✔ passed in 4.979671ms (RTT 51.365253ms)

Actually I don't know what is the name of the used file, because there is no such file under generated folder, but there are tons of files with similar names, eg. MRTS_002_ARGS_A-GET_100000.yaml, ..., MRTS_002_ARGS_A-GET_100015.yaml. And all identifiers are the same - I mean in each section the id's are 2-1, ..., 2-4.
May be this is an issue in generator script, but I think first we have to fix this. I mean all tests must have a unique identifier.

[Sebitosh]

Hi @Sebitosh,

I was very confused as to why the tests failed, and after some research I found out that it is because no logs are happening for rules in phase 4. I confirmed this by rewriting the tests with only phases 1 to 3 and then they all pass (I am attaching files with the output for reference). I am unsure if this is behavior due to libmodsecurity3 or the configuration I provided. @airween do you know what is causing this ?

I'm a bit confused about the naming convention. I tried the generated rule set with generated test cases, and this is what I see:

👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: ✔ passed in 3.786942ms (RTT 50.910581ms)
	running 2-2: ✔ passed in 4.346553ms (RTT 51.167352ms)
	running 2-3: ✔ passed in 3.909812ms (RTT 51.032199ms)
	running 2-4: ✔ passed in 4.124065ms (RTT 51.156972ms)
👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: 💥 2-1 failed in 3.578781ms (RTT 50.839066ms)
	running 2-2: 💥 2-2 failed in 4.591623ms (RTT 51.206085ms)
	running 2-3: 💥 2-3 failed in 4.698514ms (RTT 51.174826ms)
	running 2-4: 💥 2-4 failed in 4.652678ms (RTT 50.939284ms)
👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: ✔ passed in 4.979671ms (RTT 51.365253ms)

Actually I don't know what is the name of the used file, because there is no such file under generated folder, but there are tons of files with similar names, eg. MRTS_002_ARGS_A-GET_100000.yaml, ..., MRTS_002_ARGS_A-GET_100015.yaml. And all identifiers are the same - I mean in each section the id's are 2-1, ..., 2-4. May be this is an issue in generator script, but I think first we have to fix this. I mean all tests must have a unique identifier.

It's an issue with the placement of the rule id in the generated test files. What happens is that go-ftw seems to use the field ruleid to remove the rule id from the file name (to go from MRTS_002_ARGS_A-GET_100000.yaml to MRTS_002_ARGS_A-GET.yaml) but parses the name from left to right and settles on the first number it finds to display running 2-1. The fix is to append the rule ID at the beginning of the generated test file and not at the end. I am opening a PR with that fix.

[Sebitosh]

Hi @Sebitosh,

I was very confused as to why the tests failed, and after some research I found out that it is because no logs are happening for rules in phase 4. I confirmed this by rewriting the tests with only phases 1 to 3 and then they all pass (I am attaching files with the output for reference). I am unsure if this is behavior due to libmodsecurity3 or the configuration I provided. @airween do you know what is causing this ?

I'm a bit confused about the naming convention. I tried the generated rule set with generated test cases, and this is what I see:

👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: ✔ passed in 3.786942ms (RTT 50.910581ms)
	running 2-2: ✔ passed in 4.346553ms (RTT 51.167352ms)
	running 2-3: ✔ passed in 3.909812ms (RTT 51.032199ms)
	running 2-4: ✔ passed in 4.124065ms (RTT 51.156972ms)
👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: 💥 2-1 failed in 3.578781ms (RTT 50.839066ms)
	running 2-2: 💥 2-2 failed in 4.591623ms (RTT 51.206085ms)
	running 2-3: 💥 2-3 failed in 4.698514ms (RTT 51.174826ms)
	running 2-4: 💥 2-4 failed in 4.652678ms (RTT 50.939284ms)
👉 executing tests in file MRTS_002_ARGS_A-GET.yaml
	running 2-1: ✔ passed in 4.979671ms (RTT 51.365253ms)

Actually I don't know what is the name of the used file, because there is no such file under generated folder, but there are tons of files with similar names, eg. MRTS_002_ARGS_A-GET_100000.yaml, ..., MRTS_002_ARGS_A-GET_100015.yaml. And all identifiers are the same - I mean in each section the id's are 2-1, ..., 2-4. May be this is an issue in generator script, but I think first we have to fix this. I mean all tests must have a unique identifier.

It's an issue with the placement of the rule id in the generated test files. What happens is that go-ftw seems to use the field ruleid to remove the rule id from the file name (to go from MRTS_002_ARGS_A-GET_100000.yaml to MRTS_002_ARGS_A-GET.yaml) but parses the name from left to right and settles on the first number it finds to display running 2-1. The fix is to append the rule ID at the beginning of the generated test file and not at the end. I am opening a PR with that fix.

Of course we could also add some kind of counter to the generated test files such that it displays something like that:

👉 executing tests in file MRTS_002_ARGS_A-GET--1.yaml
	running 100000-1: ✔ passed in 4.03606ms (RTT 51.456201ms)
	running 100000-2: ✔ passed in 2.100709ms (RTT 50.672271ms)
	running 100000-3: ✔ passed in 3.572981ms (RTT 51.673479ms)
	running 100000-4: ✔ passed in 2.85232ms (RTT 50.599715ms)
👉 executing tests in file MRTS_002_ARGS_A-GET--2.yaml
	running 100001-1: ✔ passed in 2.102875ms (RTT 50.515938ms)
	running 100001-2: ✔ passed in 2.188394ms (RTT 50.675838ms)
	running 100001-3: ✔ passed in 2.013156ms (RTT 50.583605ms)
	running 100001-4: ✔ passed in 1.750534ms (RTT 50.572695ms)

If this is more what you mean

[airween]

I saw your PR #32, looks good to me, I'll merge that soon.

Regarding to issues in phase:4:

Fixed the phase 4 issue, found in the debug log that phase 4 did not process because the content header was missing.

Yes, I'm glad you could catch that problem. Btw. in these cases you can send a request directly based on got-ftw request (which can be inspected with --debug flag - in this case go-ftw sends the requests and shows you many detailed information).

So, here is the request:

curl -X POST \
  -H "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
  -H "Connection: close" \
  -H "Host: localhost" \
  -H "User-Agent: OWASP MRTS test agent" \
  "http://localhost/reflect?foo=attack"

* Host localhost was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]...
* Connected to localhost (::1) port 80
* using HTTP/1.x
> POST /reflect?foo=attack HTTP/1.1
> Host: localhost
> Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
> Connection: close
> User-Agent: OWASP MRTS test agent
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Date: Wed, 07 May 2025 19:53:56 GMT
< Content-Length: 0
< Connection: close
< 
* shutting down connection #0

The "problem" is that if you configure Nginx as a proxy and use Albedo as back end, Albedo does not sends Content-Type header in response.

You can get the possible URI-s:

curl http://localhost:8000/capabilities | jq .

(if your Albedo listens on port 8000).

So the solution is using /reflect URI and add a JSON request to your POST request with a body key:

curl -X POST -d '{"body":"test"}' \
  -H "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
  -H "Connection: close" \
  -H "Host: localhost" \
  -H "User-Agent: OWASP MRTS test agent" \
  -H "Content-Type: application/json" \
  "http://localhost/reflect?foo=attack"
* Host localhost was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]...
* Connected to localhost (::1) port 80
* using HTTP/1.x
> POST /reflect?foo=attack HTTP/1.1
> Host: localhost
> Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
> Connection: close
> User-Agent: OWASP MRTS test agent
> Content-Length: 15
> Content-Type: application/json
> 
* upload completely sent off: 15 bytes
< HTTP/1.1 200 OK
< Date: Wed, 07 May 2025 20:16:06 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 4
< Connection: close
< 
< 
test

and with this request, the expected rule ID appears in the log:

... ModSecurity: Warning. Matched "Operator `Contains' with parameter `attack' against variable `ARGS:foo' (Value: `attack' ) [file "/home/airween/src/MRTS/generated/rules/MRTS_002_ARGS_A-GET.conf"] [line "28"] [id "100003"] [rev ""] [msg "ARGS:foo was caught in phase:4"] ...

Probably we should add this feature to our generate script: if the request is a POST and the phase is greater than 2, then the test should contains a Content-Type: application/json header and a request body (a JSON) with body key.

Or we can open a feature request in Albedo.

[Sebitosh]

Probably we should add this feature to our generate script: if the request is a POST and the phase is greater than 2, then the test should contains a Content-Type: application/json header and a request body (a JSON) with body key.

The issue I see here is: let's take the tests from CONF_002_TARGET_ARGS_A-GET.yaml, among the generated tests is a test for a phase 4 rule triggered by a GET request. In that case there is no way using /reflect to include that header, as it only accepts POST requests, yet we still want to check if ARGS works for GET requests in phase 4. Also what happens if the test uses a POST request to trigger a rule in phase 4 but also contains a body for the test case, like tests generated by CONF_002_TARGET_ARGS_B-POST.yaml ? My intuition is that relying on the test request to assure the presence of the Content-Type header in the response gets complicated, and might create cases where a test writer is writing tests that does things he does not anticipate.

Or we can open a feature request in Albedo.

This could be a solution, but 1) Albedo devs might not want the inclusion of this header to be the default behavior on /, meaning we will have to adapt MRTS test files in some way to acomodate for it 2) It seems to me they already solved this problem somehow as they run their own regression tests on a docker with libmodsecurity3 that is able to test the presence of rules in phase 4. The question remains exactly how, and I am going through the docker's configuration files to find how they do it (still haven't exactly found their fix for this).

CRS's regression test workflow file: https://github.com/coreruleset/coreruleset/blob/main/.github/workflows/test.yml
CRS's regression test docker compose file: https://github.com/coreruleset/coreruleset/blob/main/tests/docker-compose.yml
CRS's modsecurity3 + nginx configuration for their container: https://github.com/coreruleset/modsecurity-crs-docker/tree/main/nginx/templates

My intuition is that what should happen is that the proxy should be somehow configured to add a Content-Type header if not already present. It seems to me the "easiest" solution and the reason I added this line (n°48) in sites-available/default:

more_set_headers 'Content-Type: text/plain';

I realize that it overrides any set Content-Type previously contained in the response generated by albedo, and we might want to find a different solution because of it. Nonetheless configuring nginx the right way looks to me like the solution that prevents having to change many things in the surrounding components for this "simple" issue.

[airween]

My intuition is that what should happen is that the proxy should be somehow configured to add a Content-Type header if not already present.

Okay, let me check this soon, and I'll merge this PR then. Thanks!

[airween]

Hi @Sebitosh,

do you plan to add anything here, or we can consider it's done?

[Sebitosh]

Hi @Sebitosh,

do you plan to add anything here, or we can consider it's done?

I think we can consider the PR for the infra done. Because of issue coreruleset/go-ftw#501 I might add changes to the generator script (using go-ftw's output option retry_once) or the mrts utility (using CL option --rate-limiter) to get better reliability on V3 later, but they can be merged separatly