Skip to content

lustrefs-exporter throws axum Error#136

Draft
naveenkumar262 wants to merge 1 commit into
mainfrom
nakumar/EX-11724
Draft

lustrefs-exporter throws axum Error#136
naveenkumar262 wants to merge 1 commit into
mainfrom
nakumar/EX-11724

Conversation

@naveenkumar262

Copy link
Copy Markdown

This PR fixes lustrefs-exporter crashing with Too many open files (os error 24) under Insight‑style flapping connections by bounding concurrent connections, adding timeouts, and ensuring long‑lived/idle sockets are cleaned up so the exporter stays stable under the same load

  1. Add a global connection cap (MAX_CONNECTIONS = 512) using a semaphore so active TCP connections are bounded.
  2. Add a 30s HTTP header read timeout in Hyper to drop partial/empty requests that never complete headers.
  3. Enable TCP keepalive (60s idle, 10s interval) via socket2 so dead peers are eventually cleaned up by the kernel.
  4. Add a 900s per‑connection max lifetime using tokio::time::timeout so long‑lived keep‑alive sockets cannot pin all slots forever.
  5. Replace the simple axum::serve call with a custom Hyper serve loop that applies cap, keepalive, header timeout, and lifetime per connection.
  6. Update dependencies (hyper 1, hyper-util, socket2) in Cargo.toml / Cargo.lock to support the new server behavior.

Before Fix:
image
image

After Fix:
I manually exit this with ctrl+c and sees after 30 minute it works well
image

@codecov

codecov Bot commented Apr 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.67%. Comparing base (f71b038) to head (b96fe11).

Files with missing lines Patch % Lines
lustrefs-exporter/src/main.rs 0.00% 48 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #136      +/-   ##
==========================================
- Coverage   94.40%   93.67%   -0.73%     
==========================================
  Files          44       44              
  Lines        5681     5725      +44     
  Branches     5681     5725      +44     
==========================================
  Hits         5363     5363              
- Misses        245      289      +44     
  Partials       73       73              
Flag Coverage Δ
2_14_0_ddn133 36.50% <ø> (ø)
2_14_0_ddn145 38.43% <ø> (ø)
all-tests 93.67% <0.00%> (-0.73%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Apr 24, 2026

Copy link
Copy Markdown

🐰 Bencher Report

Branchnakumar/EX-11724
Testbedci-runner
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)
Lower Boundary
nanoseconds (ns)
(Limit %)
Upper Boundary
nanoseconds (ns)
(Limit %)
parse_benchmarks/combine_performance📈 view plot
🚷 view threshold
93,637,000.00 ns
(-71.00%)Baseline: 322,897,500.00 ns
-927,576,599.95 ns
(-990.61%)
1,573,371,599.95 ns
(5.95%)
🐰 View full continuous benchmarking report in Bencher

@github-actions

Copy link
Copy Markdown

🐰 Bencher Report

Branchnakumar/EX-11724
Testbedci-runner

⚠️ WARNING: No Threshold found!

Without a Threshold, no Alerts will ever be generated.

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results
BenchmarkD1 Miss Ratemisses (%)D1mrmisses (reads) x 1e3D1mwmisses (writes) x 1e3DLmrmisses (reads)DLmwmisses (writes) x 1e3Drreads x 1e6Dwwrites x 1e6Estimated Cyclescycles x 1e6I1 Miss Ratemisses (%)I1mrmisses (reads) x 1e3ILmrmisses (reads)InstructionsBenchmark Result
instructions x 1e6
(Result Δ%)
Lower Boundary
instructions x 1e6
(Limit %)
Upper Boundary
instructions x 1e6
(Limit %)
L1 Hit Ratehits (%)L1 Hitshits x 1e6LL Hit Ratehits (%)LL Hitshits x 1e3LL Miss Ratemisses (%)LLd Miss Ratemisses (%)LLi Miss Ratemisses (%)RAM Hit Ratehits (%)RAM Hitshits x 1e3Total read+writereads/writes x 1e6
lustre_metrics::memory_benches::bench_encode_lustre_metrics with_setup:generate_records()📈 view plot
⚠️ NO THRESHOLD
0.93 %📈 view plot
⚠️ NO THRESHOLD
25.05 reads x 1e3📈 view plot
⚠️ NO THRESHOLD
9.13 writes x 1e3📈 view plot
⚠️ NO THRESHOLD
114.00 reads📈 view plot
⚠️ NO THRESHOLD
6.46 writes x 1e3📈 view plot
⚠️ NO THRESHOLD
2.47 x 1e6📈 view plot
⚠️ NO THRESHOLD
1.23 x 1e6📈 view plot
⚠️ NO THRESHOLD
14.81 x 1e6📈 view plot
⚠️ NO THRESHOLD
0.01 %📈 view plot
⚠️ NO THRESHOLD
1.05 reads x 1e3📈 view plot
⚠️ NO THRESHOLD
895.00 reads📈 view plot
🚷 view threshold
10.75 x 1e6
(-24.97%)Baseline: 14.33 x 1e6
1.62 x 1e6
(15.03%)
27.04 x 1e6
(39.76%)
📈 view plot
⚠️ NO THRESHOLD
99.76 %📈 view plot
⚠️ NO THRESHOLD
14.41 x 1e6📈 view plot
⚠️ NO THRESHOLD
0.19 %📈 view plot
⚠️ NO THRESHOLD
27.77 x 1e3📈 view plot
⚠️ NO THRESHOLD
0.05 %📈 view plot
⚠️ NO THRESHOLD
0.18 %📈 view plot
⚠️ NO THRESHOLD
0.01 %📈 view plot
⚠️ NO THRESHOLD
0.05 %📈 view plot
⚠️ NO THRESHOLD
7.47 x 1e3📈 view plot
⚠️ NO THRESHOLD
14.44 x 1e6
🐰 View full continuous benchmarking report in Bencher

@github-actions

github-actions Bot commented Apr 24, 2026

Copy link
Copy Markdown

🐰 Bencher Report

Branchnakumar/EX-11724
Testbedci-runner

⚠️ WARNING: No Threshold found!

Without a Threshold, no Alerts will ever be generated.

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results
Benchmarkavg_runtime_rss_mibMeasure (MiB)avg_runtime_virtual_mibMeasure (MiB)end_rss_mibMeasure (MiB)end_virtual_mibMeasure (MiB)memory_growth_mibMeasure (MiB)peak_over_start_rss_ratioMeasure (units)peak_over_start_virtual_ratioMeasure (units)peak_rss_mibBenchmark Result
Measure (MiB)
(Result Δ%)
Lower Boundary
Measure (MiB)
(Limit %)
Upper Boundary
Measure (MiB)
(Limit %)
peak_virtual_mibMeasure (MiB)start_rss_mibMeasure (MiB)start_virtual_mibMeasure (MiB)virtual_growth_mibMeasure (MiB)
scrape_allocations📈 view plot
⚠️ NO THRESHOLD
45.25 MiB📈 view plot
⚠️ NO THRESHOLD
878.50 MiB📈 view plot
⚠️ NO THRESHOLD
45.33 MiB📈 view plot
⚠️ NO THRESHOLD
878.56 MiB📈 view plot
⚠️ NO THRESHOLD
0.34 MiB📈 view plot
⚠️ NO THRESHOLD
1.03 units📈 view plot
⚠️ NO THRESHOLD
1.02 units📈 view plot
🚷 view threshold
46.62 MiB
(-47.62%)Baseline: 89.01 MiB
-162.65 MiB
(-348.86%)
340.68 MiB
(13.69%)
📈 view plot
⚠️ NO THRESHOLD
878.83 MiB📈 view plot
⚠️ NO THRESHOLD
44.99 MiB📈 view plot
⚠️ NO THRESHOLD
873.21 MiB📈 view plot
⚠️ NO THRESHOLD
5.35 MiB
🐰 View full continuous benchmarking report in Bencher

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant