Change encoding for SP character #663

mjkim610 · 2025-07-23T02:32:06Z

According to RFC 1866 and RFC 3986, the SP character can only be encoded as + in application/x-www-form-urlencoded values. But encoding it as %20 is always correct. Therefore, prefer to encode with %20.

Related nuclei issue: projectdiscovery/nuclei#6162

I tested it locally with a simple case, but much more thorough testing is required.

❯ go build ; ./nuclei -u http://127.0.0.1:8000/ -dast -t ../../../fuzzing-test.yaml -vv -debug

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.4.7

                projectdiscovery.io

[INF] Current nuclei version: v3.4.7 (latest)
[INF] Current nuclei-templates version: v10.2.5 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 75
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[fuzzing-test] Some fuzzing test (@mj) [low]
[DBG] mjkim - path - key: 1 , value: (select 1 from sleep(5)
[DBG] mjkim - key: 1 , value: (select%201%20from%20sleep(5)
[INF] [fuzzing-test] Dumped HTTP request for http://127.0.0.1:8000/(select%201%20from%20sleep(5)

GET /%28select%25201%2520from%2520sleep%285%29 HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: Mozilla/5.0 (ZZ; Linux i686; rv:123.0) Gecko/20100101 Firefox/123.0
Accept-Encoding: gzip

[DBG] [fuzzing-test] Dumped HTTP response http://127.0.0.1:8000/(select%201%20from%20sleep(5)

HTTP/1.0 404 File not found
Content-Length: 335
Connection: close
Content-Type: text/html;charset=utf-8
Date: Wed, 23 Jul 2025 02:28:35 GMT
Server: SimpleHTTP/0.6 Python/3.12.4

<!DOCTYPE HTML>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Error response</title>
    </head>
    <body>
        <h1>Error response</h1>
        <p>Error code: 404</p>
        <p>Message: File not found.</p>
        <p>Error code explanation: 404 - Nothing matches the given URI.</p>
    </body>
</html>
[INF] Scan completed in 4.32725ms. No results found.

mjkim610 · 2025-07-23T02:33:11Z

@Mzack9999 PTAL :)

mjkim610 · 2025-07-25T04:58:45Z

Hi @ehsandeep @dogancanbakir
PTAL or let me know if there's some process I must follow :)

According to RFC 1866 and RFC 3986, the `SP` character can only be encoded as `+` in application/x-www-form-urlencoded values. But encoding it as `%20` is always correct. Therefore, prefer to encode with `%20`.

mjkim610 · 2025-07-26T06:44:19Z

Updated test cases and verified TC passes on my machine.

❯ go test ./...
ok      github.com/projectdiscovery/utils/async 2.349s
ok      github.com/projectdiscovery/utils/auth/pdcp     0.264s
ok      github.com/projectdiscovery/utils/batcher       3.170s
?       github.com/projectdiscovery/utils/buffer        [no test files]
ok      github.com/projectdiscovery/utils/channelutil   0.747s
?       github.com/projectdiscovery/utils/conn/connpool [no test files]
?       github.com/projectdiscovery/utils/consts        [no test files]
ok      github.com/projectdiscovery/utils/context       7.175s
ok      github.com/projectdiscovery/utils/conversion    0.474s
ok      github.com/projectdiscovery/utils/crypto        1.394s
?       github.com/projectdiscovery/utils/crypto/jarm   [no test files]
ok      github.com/projectdiscovery/utils/dedupe        1.452s
ok      github.com/projectdiscovery/utils/dns   1.851s
ok      github.com/projectdiscovery/utils/env   1.620s
ok      github.com/projectdiscovery/utils/errkit        1.858s
ok      github.com/projectdiscovery/utils/errors        1.904s
ok      github.com/projectdiscovery/utils/exec  1.923s
ok      github.com/projectdiscovery/utils/file  22.131s
ok      github.com/projectdiscovery/utils/folder        1.648s
ok      github.com/projectdiscovery/utils/generic       1.645s
?       github.com/projectdiscovery/utils/global        [no test files]
ok      github.com/projectdiscovery/utils/healthcheck   2.262s
ok      github.com/projectdiscovery/utils/http  1.740s
ok      github.com/projectdiscovery/utils/io    1.949s
ok      github.com/projectdiscovery/utils/ip    2.287s
ok      github.com/projectdiscovery/utils/log   1.951s
ok      github.com/projectdiscovery/utils/maps  1.647s
?       github.com/projectdiscovery/utils/memguardian   [no test files]
ok      github.com/projectdiscovery/utils/memoize       11.330s
?       github.com/projectdiscovery/utils/memoize/cmd   [no test files]
?       github.com/projectdiscovery/utils/memoize/gen/generic   [no test files]
?       github.com/projectdiscovery/utils/memoize/simpleflight  [no test files]
?       github.com/projectdiscovery/utils/memoize/tests [no test files]
?       github.com/projectdiscovery/utils/ml    [no test files]
?       github.com/projectdiscovery/utils/ml/metrics    [no test files]
?       github.com/projectdiscovery/utils/ml/model_selection    [no test files]
ok      github.com/projectdiscovery/utils/ml/naive_bayes        0.892s
ok      github.com/projectdiscovery/utils/net   1.111s
?       github.com/projectdiscovery/utils/os    [no test files]
ok      github.com/projectdiscovery/utils/patterns      1.309s
ok      github.com/projectdiscovery/utils/permission    1.383s
ok      github.com/projectdiscovery/utils/ports 1.452s
?       github.com/projectdiscovery/utils/pprof [no test files]
?       github.com/projectdiscovery/utils/process       [no test files]
?       github.com/projectdiscovery/utils/proxy [no test files]
ok      github.com/projectdiscovery/utils/ptr   1.357s
?       github.com/projectdiscovery/utils/race  [no test files]
ok      github.com/projectdiscovery/utils/rand  1.451s
ok      github.com/projectdiscovery/utils/reader        18.569s
?       github.com/projectdiscovery/utils/reader/examples/keypress/buffered     [no test files]
?       github.com/projectdiscovery/utils/reader/examples/keypress/raw  [no test files]
?       github.com/projectdiscovery/utils/reader/rawmode        [no test files]
ok      github.com/projectdiscovery/utils/reflect       1.383s
?       github.com/projectdiscovery/utils/reflect/tests [no test files]
ok      github.com/projectdiscovery/utils/regexp        0.450s
?       github.com/projectdiscovery/utils/routing       [no test files]
ok      github.com/projectdiscovery/utils/scripts/versionbump   3.290s
ok      github.com/projectdiscovery/utils/slice 0.958s
ok      github.com/projectdiscovery/utils/strings       1.154s
ok      github.com/projectdiscovery/utils/structs       0.925s
ok      github.com/projectdiscovery/utils/sync  5.760s
?       github.com/projectdiscovery/utils/sync/semaphore        [no test files]
ok      github.com/projectdiscovery/utils/sync/sizedpool        4.467s
ok      github.com/projectdiscovery/utils/syscallutil   1.340s
ok      github.com/projectdiscovery/utils/sysutil       1.232s
ok      github.com/projectdiscovery/utils/time  (cached)
ok      github.com/projectdiscovery/utils/trace 4.343s
?       github.com/projectdiscovery/utils/unit  [no test files]
ok      github.com/projectdiscovery/utils/update        0.509s
ok      github.com/projectdiscovery/utils/url   (cached)

4FunAndProfit · 2025-07-26T07:31:50Z

Hello @mjkim610 thanks a lot ! It should indeed resolve a part of my issue too (#6342)

Yet, when text/plain Space we should let as real Space not? In the changes I cannot see that? I think the code may have forgot this case ?

4FunAndProfit · 2025-07-26T08:05:27Z

Maybe we might add an option like encoding_type ? To cover all the case (plain, url, www-form,...)?
But I Think nuclei should auto encode accordingly to the content-type expected if it has the information (through the openapi for example) ?
[Also here, i think you changed the default encoding to url encoding which is correct for queryParams but isn't for body in many cases (i indeed suspect theses codes in url folder to be used also in bodyparams etc?)]

tarunKoyalwar

hey @mjkim610 , thanks for the pr, but url utils in this repo are not meant to follow any RFC's and are implemented on case to case basis based on the requirement of the poc being written ( apart from that we have many other optimizations that are not part of any RFC https://github.com/projectdiscovery/utils/tree/main/url )

this ordereparams is used for url encoding throughout nuclei ( in request body / url etc ) , so any change here will most likely break existing nuclei-templates and integration tests ,

can you share any exploit / poc where you are able to exploit a particular vulnerability via %20 but not + , if so we can plan some changes where we can support that without breaking existing templates

4FunAndProfit · 2025-07-26T15:42:05Z

@tarunKoyalwar see for example projectdiscovery/nuclei#6342 for an example where encoding is not good and consider an XXE for example in this example
Can't we have an option like space_encoding which is set by default to + and that we can change? It will not break existing templates with that

4FunAndProfit · 2025-07-26T16:02:05Z

Or maybe just authorize unsafe: true in fuzzing? It will resolve a big part of it

mjkim610 · 2025-07-28T03:01:07Z

@tarunKoyalwar thanks for the feedback.

To answer your question regarding POCs that require %20 instead of +:

[FEATURE] Add an Option to disable url-encoding when in fuzzing path nuclei#6162
[BUG] Nuclei in fuzzing mode refuses unsafe and encodes in text/plain and should not (and encodes wrongly query) nuclei#6342

Regarding your concern about breaking integration tests:

For URLs, the decoded URL will be exactly the same with this change. So if any tests fail due to this change, then most likely the expected value of the TC should be updated.
For others, different data should be encoded differently. (E.g. text/plain should not be encoded) This is an issue independent of this PR and I think this should be handled as a separate issue/PR.

Regarding your concern about breaking existing templates:

I agree this would be a problem. Can you provide some examples so that I can get a better understanding?
The %20 encoding could be set as a new option, as @4FunAndProfit suggested. This means maintaining compatibility while giving an option for new templates.

mjkim610 · 2025-08-08T11:02:04Z

@tarunKoyalwar could you provide feedback? :)

I think the best course of action would be to set %20 encoding as a new option in the short term, then support all other encoding types in the long term.

Mzack9999 · 2025-08-15T15:20:12Z

@mjkim610 Unfortunately we can't change the default behavior for edge cases. I've added an environment variable SPACE_ENCODING that you can set to percent in order to have the wanted behavior:

SPACE_ENCODING=percent nuclei -t xxx ...

mjkim610 · 2025-08-16T00:51:19Z

@mjkim610 Unfortunately we can't change the default behavior for edge cases. I've added an environment variable SPACE_ENCODING that you can set to percent in order to have the wanted behavior:
SPACE_ENCODING=percent nuclei -t xxx ...

Thanks for providing a way to change space encoding type! Although I have some reservations about using env vars, I understand why, given that implementing it in other methods (e.g. flags) would require lots of planning and big changes.

I will keep an eye out for any updates regarding these encoding issues.

ehsandeep requested a review from tarunKoyalwar July 25, 2025 23:17

Change encoding for SP character

86b0da6

According to RFC 1866 and RFC 3986, the `SP` character can only be encoded as `+` in application/x-www-form-urlencoded values. But encoding it as `%20` is always correct. Therefore, prefer to encode with `%20`.

mjkim610 force-pushed the url-encoding branch from 4910e44 to 86b0da6 Compare July 26, 2025 06:41

mjkim610 mentioned this pull request Jul 26, 2025

[BUG] Nuclei in fuzzing mode refuses unsafe and encodes in text/plain and should not (and encodes wrongly query) projectdiscovery/nuclei#6342

Open

1 task

tarunKoyalwar requested changes Jul 26, 2025

View reviewed changes

reverting + using env var

ed5e234

Mzack9999 merged commit bc0b0d6 into projectdiscovery:main Aug 15, 2025
7 checks passed

mjkim610 deleted the url-encoding branch August 15, 2025 15:47

mjkim610 mentioned this pull request Aug 16, 2025

[FEATURE] Add an Option to disable url-encoding when in fuzzing path projectdiscovery/nuclei#6162

Closed

tarunKoyalwar mentioned this pull request Aug 23, 2025

Path-Based Fuzzing SQL fix projectdiscovery/nuclei#6400

Merged

Change encoding for SP character #663

Change encoding for SP character #663

Uh oh!

Conversation

mjkim610 commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjkim610 commented Jul 23, 2025

Uh oh!

mjkim610 commented Jul 25, 2025

Uh oh!

mjkim610 commented Jul 26, 2025

Uh oh!

4FunAndProfit commented Jul 26, 2025

Uh oh!

4FunAndProfit commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tarunKoyalwar left a comment

Choose a reason for hiding this comment

Uh oh!

4FunAndProfit commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

4FunAndProfit commented Jul 26, 2025

Uh oh!

mjkim610 commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjkim610 commented Aug 8, 2025

Uh oh!

Mzack9999 commented Aug 15, 2025

Uh oh!

Uh oh!

mjkim610 commented Aug 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mjkim610 commented Jul 23, 2025 •

edited

Loading

4FunAndProfit commented Jul 26, 2025 •

edited

Loading

4FunAndProfit commented Jul 26, 2025 •

edited

Loading

mjkim610 commented Jul 28, 2025 •

edited

Loading