mbedtls: http.get_text returns nothing

[tankf33der]

Describe the bug

Works in browser, curl and wget.

wget -qO- "https://nest.pijul.com/pijul/pijul/changes"

Reproduction Steps

import net.http

fn main() {
        res := http.get_text('https://nest.pijul.com/pijul/pijul/changes')
        println(res)
}

Expected Behavior

Return me long lines of HTML.

Current Behavior

Returns nothing.

Possible Solution

No response

Additional Information/Context

No response

V version

V 0.4.9 e9c2314

Environment details (OS name and version, etc.)

Manjaro Linux.

Note

You can use the 👍 reaction to increase the issue's priority for developers.

Please note that only the 👍 reaction to the issue itself counts as a vote.
Other reactions and those to comments will not be taken into account.

[huly-for-github]

Connected to Huly®: V_0.6-22135

[jorgeluismireles]

I tested several pages around your site and seems changes is too large

page	size
$ curl -so /dev/null https://nest.pijul.com/pijul/pijul/changes -w '%{size_download}'	694831
$ curl -so /dev/null https://nest.pijul.com/pijul/pijul -w '%{size_download}'	28598
$ curl -so /dev/null https://nest.pijul.com/pijul/pijul/tags -w '%{size_download}'	11639
$ curl -so /dev/null https://nest.pijul.com/pijul/pijul/discussions -w '%{size_download}'	23851

For the above pages V's http returned data except for changes like if size matters

[JalonSolov]

Simple enough theory to test...

import net.http

fn main() {
        res1 := http.get_text('https://nest.pijul.com/pijul/pijul/changes')
        println(res1.len)
        res2 := http.get_text('https://nest.pijul.com/pijul/pijul')
        println(res2.len)
        res3 := http.get_text('https://nest.pijul.com/pijul/pijul/tags')
        println(res3.len)
        res4 := http.get_text('https://nest.pijul.com/pijul/pijul/discussions')
        println(res4.len)
}

output is

0
28598
11639
23851

So yes, definitely size related.

[jorgeluismireles]

I have this program trying to inspect some callbacks for both URLs, a small one and a large one:

import net.http

fn progress(req &http.Request, chunk []u8, so_far u64) ! {
        println('* request       chunk:${chunk.len} read_so_far:${so_far}')
}

fn progress_body(req &http.Request, chunk []u8, so_far u64, expected_size u64, status_code int) ! {
        println('* request_body  chunk:${chunk.len} read_so_far:${so_far} exp:${expected_size} ${status_code}')
}

fn finish(req &http.Request, size u64) ! {
        println('* finish        size:${size}')
}

fn main() {
        for url in [
                'https://nest.pijul.com/pijul/pijul'
                'https://nest.pijul.com/pijul/pijul/changes'
        ] {
                println('url ${url}')
                config := http.FetchConfig{
                        url: url
                        on_progress:      progress
                        on_progress_body: progress_body
                        on_finish:        finish
                }
                res := http.fetch(config)!
                _ = res
        }
}

Result

url https://nest.pijul.com/pijul/pijul
* request       chunk:16384 read_so_far:16384
* request_body  chunk:16151 read_so_far:16151 exp:0 200
* request       chunk:12460 read_so_far:28844
* request_body  chunk:12460 read_so_far:28611 exp:0 200
* finish        size:28844
url https://nest.pijul.com/pijul/pijul/changes
* finish        size:0
* finish        size:0
* finish        size:0
* finish        size:0
* finish        size:0
V panic: result not set (response does not start with HTTP/, line: ``)
v hash: 81a2e7a
/tmp/v_1000/23719.01JM2NJDFWDK8CTYZCC4MBJ5MM.tmp.c:9137: at _v_panic: Backtrace
/tmp/v_1000/23719.01JM2NJDFWDK8CTYZCC4MBJ5MM.tmp.c:9103: by panic_result_not_set
/tmp/v_1000/23719.01JM2NJDFWDK8CTYZCC4MBJ5MM.tmp.c:19055: by main__main
/tmp/v_1000/23719.01JM2NJDFWDK8CTYZCC4MBJ5MM.tmp.c:19365: by main

Seems with the large https://nest.pijul.com/pijul/pijul/changes http module calls several times finish with size 0 as like a bug.

[changrui]

...\_vlang_proj_\v-test> v run .\http.get_text.v
694794
28598
11639
23851

Simple enough theory to test...

import net.http

fn main() {
res1 := http.get_text('https://nest.pijul.com/pijul/pijul/changes')
println(res1.len)
res2 := http.get_text('https://nest.pijul.com/pijul/pijul')
println(res2.len)
res3 := http.get_text('https://nest.pijul.com/pijul/pijul/tags')
println(res3.len)
res4 := http.get_text('https://nest.pijul.com/pijul/pijul/discussions')
println(res4.len)
}
output is

0
28598
11639
23851

So yes, definitely size related.

...\_vlang_proj_\v-test> v run .\http.get_text.v
694794
28598
11639
23851

[jorgeluismireles]

What is your platform changrui ? I also get first 0 with JalonSolov code. I am in Ubuntu.

[changrui]

...\_vlang_proj_\v-test> v run .\http.get_text.v
694794
28598
11639
23851

Simple enough theory to test...

import net.http

fn main() {
res1 := http.get_text('https://nest.pijul.com/pijul/pijul/changes')
println(res1.len)
res2 := http.get_text('https://nest.pijul.com/pijul/pijul')
println(res2.len)
res3 := http.get_text('https://nest.pijul.com/pijul/pijul/tags')
println(res3.len)
res4 := http.get_text('https://nest.pijul.com/pijul/pijul/discussions')
println(res4.len)
}
output is

0
28598
11639
23851

So yes, definitely size related.

...\_vlang_proj_\v-test> v run .\http.get_text.v
694794
28598
11639
23851

What is your platform changrui ? I also get first 0 with JalonSolov code. I am in Ubuntu.

Windows 11 Home

[changrui]

What is your platform changrui ? I also get first 0 with JalonSolov code. I am in Ubuntu.

...\_vlang_proj_\v-test> v run .\v-fetch.v
          url https://nest.pijul.com/pijul/pijul
          * request       chunk:28844 read_so_far:28844
          * finish        size:28844
          url https://nest.pijul.com/pijul/pijul/changes
          * request       chunk:695066 read_so_far:695066
          * finish        size:695066
          url https://nest.pijul.com/pijul/pijul/tags
          * request       chunk:11910 read_so_far:11910
          * finish        size:11910

[changrui]

@jorgeluismireles Using Ubuntu 24.04 on WLS2, I got the same result with you.

[felipensp]

Closing, it is related to #23719 missing HTTP/2 support.

[tankf33der]

I protest, remote web server supports both protocols.

[mpech@lambda src]$ curl -I --http1.1 https://nest.pijul.com/pijul/pijul/changes
HTTP/1.1 200 OK
strict-transport-security: max-age=31536000; includeSubDomains
content-type: text/html
vary: Cookie
cache-control: no-cache
last-modified: Mon, 04 Mar 2024 21:28:23 GMT
date: Sun, 16 Feb 2025 11:38:08 GMT

[mpech@lambda src]$ curl -I --http2 https://nest.pijul.com/pijul/pijul/changes
HTTP/2 200 
strict-transport-security: max-age=31536000; includeSubDomains
content-type: text/html
vary: Cookie
cache-control: no-cache
last-modified: Mon, 04 Mar 2024 21:28:23 GMT
date: Sun, 16 Feb 2025 11:38:15 GMT

[mpech@lambda src]$ 

[JalonSolov]

The remote server may support it, but V doesn't, currently, and there's already an open issue to add that support.

[tankf33der]

I see v do.

GET /pijul/pijul/changes HTTP/1.1
Host: nest.pijul.com
User-Agent: v.http
Connection: close

I don’t understand why you’re focusing on HTTP/2. I don’t see the reasoning behind it. Please show me how you arrived at this conclusion. Thank you.

[JalonSolov]

From your own post:

[mpech@lambda src]$ curl -I --http2 https://nest.pijul.com/pijul/pijul/changes
HTTP/2 200 

[felipensp]

Debugging with -d trace_ssl

[felipensp]

It seems to be about: MBEDTLS_ERR_SSL_TIMEOUT.

[JalonSolov]

Definitely mbed_tls related. I tried running my example again, only this time adding -d use_openssl, and this time it worked:

$ v -d use_openssl run http.get_text.v
694831
28598
11639
23851
$

[felipensp]

Good catch, @JalonSolov

[tankf33der]

I can make mbedtls work with any TLS site if I increase mbedtls_client_read_timeout_ms.
I have no idea how it even appeared here.
-d use_openssl works because the default timeout values work.

I found another server in Japan that my server in Germany can't communicate with in time, resulting in a timeout and empty result.
https://www.japan.go.jp/kizuna/2024/09/mother_to_mother_networks.html
And then it manages to download because Amazon CloudFront kicks in.

[JalonSolov]

One minor fix would be to increase the default timeouts in mbedtls to match the default timeouts in openssl.

[spytheman]

The net.mbedtls module has:

const mbedtls_client_read_timeout_ms = $d('mbedtls_client_read_timeout_ms', 550)

i.e. you can pass -d mbedtls_client_read_timeout_ms=7000 .

It is a worse workaround imho, compared to -d use_openssl, and has the negative that such programs will have slower http requests for sites that do not timeout (the better solution would be to debug what the underlying mbedtls/wrapper interaction issue is).

[spytheman]

With my current ISP, I can go as low as: -d mbedtls_client_read_timeout_ms=750. Values below that, cause the original sample to print '' .

[spytheman]

Tested with:

v -d mbedtls_client_read_timeout_ms=750 -o z_mbedtls z.v
v -d use_openssl -o z_openssl z.v

and later:

0[14:04:00]@m1: (master) /opt/v $ v time ./z_openssl |wc
> 1336.486 ms. Exit code:   0. Command: ./z_openssl
      15   52324  694825
0[14:04:04]@m1: (master) /opt/v $ v time ./z_mbedtls |wc
> 6246.024 ms. Exit code:   0. Command: ./z_mbedtls
      15   52324  694825

What is interesting to note, is that the OpenSSL version is faster, while doing the same work,
but that may be because it is using a prebuilt libssl, that was already optimized for my OS,
while the mbedtls one is compiled without optimizations (and the cryptography operations are math heavy).

Edit: I am wrong, optimisations does not seem to be directly related. Even with -prod, the openssl variant is faster:

0[14:04:15]@m1: (master) /opt/v $ v -d mbedtls_client_read_timeout_ms=700 -prod -cc clang -o z_mbedtls z.v
0[14:09:33]@m1: (master) /opt/v $ v time ./z_mbedtls |wc
> 5405.536 ms. Exit code:   0. Command: ./z_mbedtls
       1       0       1
0[14:10:10]@m1: (master) /opt/v $ v -d use_openssl -prod -o z_openssl z.v
0[14:10:37]@m1: (master) /opt/v $ v time ./z_openssl |wc
> 1729.593 ms. Exit code:   0. Command: ./z_openssl
      15   52324  694825
0[14:10:43]@m1: (master) /opt/v $ v time ./z_mbedtls |wc
> 5531.396 ms. Exit code:   0. Command: ./z_mbedtls
      15   52324  694825
0[14:10:53]@m1: (master) /opt/v $