EHT-1396: Expand lustre_connected_clients to track OST and MGT exports by ydande · Pull Request #126 · whamcloud/lustrefs-exporter

ydande · 2026-01-08T14:29:54Z

Overview

Extends the existing lustre_connected_clients metric to track client connections across all three Lustre target types: MDT (Metadata Targets), OST (Object Storage Targets), and MGT (Management Target).

Previous behavior: Only tracked MDT client connections
New behavior: Tracks MDT, OST, and MGT client connections with proper component labels

Changes Summary

Update client_count_parser to parse obdfilter.*OST*.exports.*.uuid for OST targets
Add support for parsing mgs.MGS.exports.*.uuid for MGT targets
Maintain existing MDT client counting functionality (mdt.*MDT*.exports.*.uuid)
Use component="ost" and component="mgt" labels to distinguish target types
Handle production UUID formats with both inline and multiline UUIDs
Add TargetVariant enum (Mdt, Ost, Mgt) for target type classification
Add comprehensive test coverage for all three target types
All 128 tests pass successfully (20 client count tests)

Dev Testing

Tested on Trevis:

All 8 nodes deployed successfully

Node1-4: HA Group 1 (MDT0000-0003, OST0000-0007, MGS)
Node5-8: HA Group 2 (MDT0004-0007, OST0008-0015)

All 3 component types working

MDT: Showing metadata clients (1 per MDT)
OST: Showing active I/O clients (1 on OST0009 and OST000b)
MGT: Showing all connected servers (8 total)

OST client tracking validated:

Files striped to OST0009 and OST000b
Those OSTs correctly show 1 client
Other OSTs correctly show 0 (no active I/O)

notion-workspace · 2026-01-08T14:29:59Z

GCP-47 - lustrefs-exporter - expand 'lustre_connected_clients' to track OST and MGT exports

github-actions · 2026-01-08T14:31:58Z

Bencher Report

Branch	ydande/EHT-1396
Testbed	ci-runner

⚠️ WARNING: No Threshold found!
Without a Threshold, no Alerts will ever be generated.
LL Hit Rate (hits (%))
DLmr (misses (reads))
DLmw (misses (writes))
I1 Miss Rate (misses (%))
Dw (writes)
ILmr (misses (reads))
Total read+write (reads/writes)
L1 Hits (hits)
LL Miss Rate (misses (%))
RAM Hit Rate (hits (%))
LLd Miss Rate (misses (%))
D1mw (misses (writes))
LLi Miss Rate (misses (%))
Dr (reads)
D1 Miss Rate (misses (%))
I1mr (misses (reads))
L1 Hit Rate (hits (%))
LL Hits (hits)
RAM Hits (hits)
D1mr (misses (reads))
Estimated Cycles (cycles)
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results

Benchmark	D1 Miss Rate	misses (%)	D1mr	misses (reads) x 1e3	D1mw	misses (writes) x 1e3	DLmr	misses (reads)	DLmw	misses (writes) x 1e3	Dr	reads x 1e6	Dw	writes x 1e6	Estimated Cycles	cycles x 1e6	I1 Miss Rate	misses (%)	I1mr	misses (reads) x 1e3	ILmr	misses (reads)	Instructions	Benchmark Result instructions x 1e6 (Result Δ%)	Lower Boundary instructions x 1e6 (Limit %)	Upper Boundary instructions x 1e6 (Limit %)	L1 Hit Rate	hits (%)	L1 Hits	hits x 1e6	LL Hit Rate	hits (%)	LL Hits	hits x 1e3	LL Miss Rate	misses (%)	LLd Miss Rate	misses (%)	LLi Miss Rate	misses (%)	RAM Hit Rate	hits (%)	RAM Hits	hits x 1e3	Total read+write	reads/writes x 1e6
lustre_metrics::memory_benches::bench_encode_lustre_metrics with_setup:generate_records()	📈 view plot ⚠️ NO THRESHOLD	0.91 %	📈 view plot ⚠️ NO THRESHOLD	24.92 reads x 1e3	📈 view plot ⚠️ NO THRESHOLD	9.00 writes x 1e3	📈 view plot ⚠️ NO THRESHOLD	120.00 reads	📈 view plot ⚠️ NO THRESHOLD	6.49 writes x 1e3	📈 view plot ⚠️ NO THRESHOLD	2.48 x 1e6	📈 view plot ⚠️ NO THRESHOLD	1.24 x 1e6	📈 view plot ⚠️ NO THRESHOLD	14.86 x 1e6	📈 view plot ⚠️ NO THRESHOLD	0.01 %	📈 view plot ⚠️ NO THRESHOLD	1.05 reads x 1e3	📈 view plot ⚠️ NO THRESHOLD	888.00 reads	📈 view plot 🚷 view threshold	10.78 x 1e6 (-16.35%) Baseline: 12.88 x 1e6	3.33 x 1e6 (30.88%)	22.44 x 1e6 (48.03%)	📈 view plot ⚠️ NO THRESHOLD	99.76 %	📈 view plot ⚠️ NO THRESHOLD	14.46 x 1e6	📈 view plot ⚠️ NO THRESHOLD	0.19 %	📈 view plot ⚠️ NO THRESHOLD	27.46 x 1e3	📈 view plot ⚠️ NO THRESHOLD	0.05 %	📈 view plot ⚠️ NO THRESHOLD	0.18 %	📈 view plot ⚠️ NO THRESHOLD	0.01 %	📈 view plot ⚠️ NO THRESHOLD	0.05 %	📈 view plot ⚠️ NO THRESHOLD	7.50 x 1e3	📈 view plot ⚠️ NO THRESHOLD	14.50 x 1e6

🐰 View full continuous benchmarking report in Bencher

github-actions · 2026-01-08T14:32:07Z

Bencher Report

Branch	ydande/EHT-1396
Testbed	ci-runner

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Lower Boundary nanoseconds (ns) (Limit %)	Upper Boundary nanoseconds (ns) (Limit %)
parse_benchmarks/combine_performance	📈 view plot 🚷 view threshold	128,760,000.00 ns (-42.04%) Baseline: 222,135,200.00 ns	-554,982,258.62 ns (-431.02%)	999,252,658.62 ns (12.89%)

🐰 View full continuous benchmarking report in Bencher

codecov · 2026-01-08T14:32:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.58%. Comparing base (1e1ff08) to head (231531c).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #126      +/-   ##
==========================================
+ Coverage   94.40%   94.58%   +0.18%     
==========================================
  Files          44       44              
  Lines        5681     5875     +194     
  Branches     5681     5875     +194     
==========================================
+ Hits         5363     5557     +194     
  Misses        245      245              
  Partials       73       73

Flag	Coverage Δ
2_14_0_ddn133	`35.46% <15.42%> (-1.04%)`	⬇️
2_14_0_ddn145	`37.30% <15.42%> (-1.14%)`	⬇️
all-tests	`94.58% <100.00%> (+0.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-01-08T14:35:17Z

Bencher Report

Branch	ydande/EHT-1396
Testbed	ci-runner

⚠️ WARNING: No Threshold found!
Without a Threshold, no Alerts will ever be generated.
peak_virtual_mib (Measure (MiB))
start_rss_mib (Measure (MiB))
avg_runtime_rss_mib (Measure (MiB))
end_rss_mib (Measure (MiB))
memory_growth_mib (Measure (MiB))
virtual_growth_mib (Measure (MiB))
peak_over_start_rss_ratio (Measure (units))
end_virtual_mib (Measure (MiB))
avg_runtime_virtual_mib (Measure (MiB))
peak_over_start_virtual_ratio (Measure (units))
start_virtual_mib (Measure (MiB))
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results

Benchmark	avg_runtime_rss_mib	Measure (MiB)	avg_runtime_virtual_mib	Measure (MiB)	end_rss_mib	Measure (MiB)	end_virtual_mib	Measure (MiB)	memory_growth_mib	Measure (MiB)	peak_over_start_rss_ratio	Measure (units)	peak_over_start_virtual_ratio	Measure (units)	peak_rss_mib	Benchmark Result Measure (MiB) (Result Δ%)	Lower Boundary Measure (MiB) (Limit %)	Upper Boundary Measure (MiB) (Limit %)	peak_virtual_mib	Measure (MiB)	start_rss_mib	Measure (MiB)	start_virtual_mib	Measure (MiB)	virtual_growth_mib	Measure (MiB)
scrape_allocations	📈 view plot ⚠️ NO THRESHOLD	43.60 MiB	📈 view plot ⚠️ NO THRESHOLD	812.48 MiB	📈 view plot ⚠️ NO THRESHOLD	43.67 MiB	📈 view plot ⚠️ NO THRESHOLD	812.55 MiB	📈 view plot ⚠️ NO THRESHOLD	0.33 MiB	📈 view plot ⚠️ NO THRESHOLD	1.03 units	📈 view plot ⚠️ NO THRESHOLD	1.02 units	📈 view plot 🚷 view threshold	45.29 MiB (-37.39%) Baseline: 72.34 MiB	-114.05 MiB (-251.80%)	258.72 MiB (17.51%)	📈 view plot ⚠️ NO THRESHOLD	812.79 MiB	📈 view plot ⚠️ NO THRESHOLD	43.34 MiB	📈 view plot ⚠️ NO THRESHOLD	807.79 MiB	📈 view plot ⚠️ NO THRESHOLD	4.76 MiB

🐰 View full continuous benchmarking report in Bencher

Extend the existing client count parser to support OST and MGT targets in addition to the existing MDT support. Changes: - Update client_count_parser to parse obdfilter.*OST*.exports.*.uuid - Add support for parsing mgs.MGS.exports.*.uuid - Maintain existing MDT client counting functionality - Use component="ost" and component="mgt" labels to distinguish targets - Handle production UUID formats with inline UUIDs in export names - Add TargetVariant enum (Mdt, Ost, Mgt) for target type handling - Add comprehensive test coverage for all three target types All 18 client count tests pass successfully.

After Git LFS installation, the regular snapshots were updated to remove BRW stats metrics that are no longer generated by the test fixtures. The OTEL snapshots need to be updated to match the regular snapshots (with the otel_scope_name label added). Updated snapshots: - lustrefs_exporter__routes__tests__jobstats_with_stderr_output.otelsnap - lustrefs_exporter__routes__tests__metrics_endpoint_is_idempotent.otelsnap

- Add documentation comments to target_kind, parse, interface_clients, mdt_interface, ost_interface, and mgt_interface functions - Change target_kind() to use match with guards instead of if-else - Update HELP text for lustre_connected_clients metric to explain MDT/OST/MGT differences - Remove Ord/PartialOrd from TargetVariant enum - Update all snapshot files with new HELP text

These snapshot files were no longer referenced by any tests. The tests have been migrated to use inline snapshots instead. This fixes the CI failure in the 'Stale Snapshots' check.

Commit 8d74656 deleted 81 snapshot files using 'cargo insta test --unreferenced=auto'. The previous commit cd54da5 restored 4 of them. This commit restores the remaining 77 snapshot files. All 128 tests now pass successfully.

@r

These 8 snapshot files are genuinely unreferenced because their tests were converted to use inline snapshots (using @r###...### syntax) in commit 7c48d6a. Tests affected (all using inline snapshots): - test_export_param - test_no_interface_clients - test_interface_clients - test_multiple_interface_clients - test_ost_interface - test_ost_interface_clients - test_mgt_interface - test_mgt_interface_clients All 128 tests still pass.

breuhan · 2026-02-17T12:21:36Z

Search for all occurences of .*.uuid in all files and update with your new cmd args. Otherwise you will loose a lot of valid stats. Look especially for json files.

breuhan

Snaps have a lot of removed lines

…nces - Add obdfilter.*.exports.*.uuid and mgs.MGS.exports.*.uuid to test commands - Remove 'Trevis-158' references from test names and comments - Rename tests to generic names (test_multiline_uuid_format, test_mixed_targets_with_multiline_uuid) - Update snapshots with new OST/MGT metrics (quota, health, memory, LDLM, disk I/O) - Update OTEL snapshots for compatibility All 129 tests passing. Fixes: EHT-1396

- Regenerate OTEL snapshots with otel_scope_name="lustre" label - Ensures OTEL format compatibility with prometheus-client fork

ydande · 2026-02-23T14:07:17Z

Search for all occurences of .*.uuid in all files and update with your new cmd args. Otherwise you will loose a lot of valid stats. Look especially for json files.

Thanks! I've updated all 7 JSON test files with the new OST/MGT UUID parameters. The snapshots now include all the additional metrics.

ydande · 2026-02-23T14:09:03Z

Snaps have a lot of removed lines

Thanks for catching this! I've regenerated all snapshots with the new UUID parameters.

ydande force-pushed the ydande/EHT-1396 branch from b70d7ad to f954301 Compare January 8, 2026 14:34

ydande requested review from ayush-ddn, ddn-ajagdhane, gadekar-nitesh and sagar-gharte January 9, 2026 04:37

ydande added 2 commits January 13, 2026 14:02

Apply rustfmt formatting to client_count_parser and types

c004055

ydande force-pushed the ydande/EHT-1396 branch from f954301 to c004055 Compare January 13, 2026 08:41

ydande added 8 commits January 19, 2026 23:13

Update jobstats snapshot after Git LFS installation

7a671ff

Remove unreferenced insta snapshot files

8d74656

These snapshot files were no longer referenced by any tests. The tests have been migrated to use inline snapshots instead. This fixes the CI failure in the 'Stale Snapshots' check.

Restore snapshot files that are still referenced by tests

cd54da5

Restore remaining 77 snapshot files incorrectly deleted

e47eccf

Commit 8d74656 deleted 81 snapshot files using 'cargo insta test --unreferenced=auto'. The previous commit cd54da5 restored 4 of them. This commit restores the remaining 77 snapshot files. All 128 tests now pass successfully.

Add test for Trevis-158 export format with client UUID on separate line

01e4c7c

ydande force-pushed the ydande/EHT-1396 branch from 4d3cfc0 to cf2d97b Compare February 15, 2026 13:15

ydande self-assigned this Feb 16, 2026

ydande marked this pull request as ready for review February 17, 2026 09:16

ydande requested review from breuhan and johnsonw February 17, 2026 12:02

breuhan requested changes Feb 17, 2026

View reviewed changes

ydande added 2 commits February 23, 2026 17:31

Fix OTEL snapshots to include otel_scope_name label

231531c

- Regenerate OTEL snapshots with otel_scope_name="lustre" label - Ensures OTEL format compatibility with prometheus-client fork

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

EHT-1396: Expand lustre_connected_clients to track OST and MGT exports#126

EHT-1396: Expand lustre_connected_clients to track OST and MGT exports#126
ydande wants to merge 12 commits into
mainfrom
ydande/EHT-1396

ydande commented Jan 8, 2026 •

edited

Loading

Uh oh!

notion-workspace Bot commented Jan 8, 2026

Uh oh!

github-actions Bot commented Jan 8, 2026 •

edited

Loading

⚠️ WARNING: No Threshold found!

Uh oh!

github-actions Bot commented Jan 8, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jan 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jan 8, 2026 •

edited

Loading

⚠️ WARNING: No Threshold found!

Uh oh!

breuhan commented Feb 17, 2026

Uh oh!

breuhan left a comment

Uh oh!

ydande commented Feb 23, 2026

Uh oh!

ydande commented Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ydande commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Changes Summary

Dev Testing

Uh oh!

notion-workspace Bot commented Jan 8, 2026

Uh oh!

github-actions Bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

⚠️ WARNING: No Threshold found!

Uh oh!

github-actions Bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

codecov Bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

⚠️ WARNING: No Threshold found!

Uh oh!

breuhan commented Feb 17, 2026

Uh oh!

breuhan left a comment

Choose a reason for hiding this comment

Uh oh!

ydande commented Feb 23, 2026

Uh oh!

ydande commented Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ydande commented Jan 8, 2026 •

edited

Loading

github-actions Bot commented Jan 8, 2026 •

edited

Loading

github-actions Bot commented Jan 8, 2026 •

edited

Loading

codecov Bot commented Jan 8, 2026 •

edited

Loading

github-actions Bot commented Jan 8, 2026 •

edited

Loading