EX-10318: add metrics for osc.*.state.current_state - #137
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #137 +/- ##
==========================================
+ Coverage 94.68% 94.71% +0.03%
==========================================
Files 46 48 +2
Lines 6774 6931 +157
Branches 6774 6931 +157
==========================================
+ Hits 6414 6565 +151
- Misses 271 276 +5
- Partials 89 90 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Bencher Report| Project | lustrefs-exporter |
| Branch | flei-ddn/EX-10318 |
| Testbed | ci-runner |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result nanoseconds (ns) (Result Δ%) | Lower Boundary nanoseconds (ns) (Limit %) | Upper Boundary nanoseconds (ns) (Limit %) |
|---|---|---|---|---|
| parse_benchmarks/combine_performance | 📈 view plot 🚷 view threshold | 127,250,000.00 ns(-35.42%)Baseline: 197,029,806.45 ns | -523,513,940.35 ns (-411.41%) | 917,573,553.25 ns (13.87%) |
|
| Project | lustrefs-exporter |
| Branch | flei-ddn/EX-10318 |
| Testbed | ci-runner |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
- L1 Hits (hits)
- D1mw (misses (writes))
- Estimated Cycles (cycles)
- LL Hit Rate (hits (%))
- RAM Hits (hits)
- RAM Hit Rate (hits (%))
- ILmr (misses (reads))
- Total read+write (reads/writes)
- DLmw (misses (writes))
- LL Miss Rate (misses (%))
- L1 Hit Rate (hits (%))
- Dw (writes)
- DLmr (misses (reads))
- Dr (reads)
- I1mr (misses (reads))
- D1 Miss Rate (misses (%))
- LLi Miss Rate (misses (%))
- LL Hits (hits)
- D1mr (misses (reads))
- LLd Miss Rate (misses (%))
- I1 Miss Rate (misses (%))
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | D1 Miss Rate | misses (%) | D1mr | misses (reads) x 1e3 | D1mw | misses (writes) x 1e3 | DLmr | misses (reads) | DLmw | misses (writes) x 1e3 | Dr | reads x 1e6 | Dw | writes x 1e6 | Estimated Cycles | cycles x 1e6 | I1 Miss Rate | misses (%) | I1mr | misses (reads) x 1e3 | ILmr | misses (reads) | Instructions | Benchmark Result instructions x 1e6 (Result Δ%) | Lower Boundary instructions x 1e6 (Limit %) | Upper Boundary instructions x 1e6 (Limit %) | L1 Hit Rate | hits (%) | L1 Hits | hits x 1e6 | LL Hit Rate | hits (%) | LL Hits | hits x 1e3 | LL Miss Rate | misses (%) | LLd Miss Rate | misses (%) | LLi Miss Rate | misses (%) | RAM Hit Rate | hits (%) | RAM Hits | hits x 1e3 | Total read+write | reads/writes x 1e6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| lustre_metrics::memory_benches::bench_encode_lustre_metrics with_setup:generate_records() | 📈 view plot | 0.93 % | 📈 view plot | 25.55 reads x 1e3 | 📈 view plot | 9.14 writes x 1e3 | 📈 view plot | 134.00 reads | 📈 view plot | 6.38 writes x 1e3 | 📈 view plot | 2.49 x 1e6 | 📈 view plot | 1.23 x 1e6 | 📈 view plot | 14.85 x 1e6 | 📈 view plot | 0.01 % | 📈 view plot | 1.12 reads x 1e3 | 📈 view plot | 965.00 reads | 📈 view plot 🚷 view threshold | 10.76 x 1e6(-14.21%)Baseline: 12.54 x 1e6 | 3.48 x 1e6 (32.35%) | 21.60 x 1e6 (49.81%) | 📈 view plot | 99.75 % | 📈 view plot | 14.44 x 1e6 | 📈 view plot | 0.20 % | 📈 view plot | 28.32 x 1e3 | 📈 view plot | 0.05 % | 📈 view plot | 0.18 % | 📈 view plot | 0.01 % | 📈 view plot | 0.05 % | 📈 view plot | 7.48 x 1e3 | 📈 view plot | 14.48 x 1e6 |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
April 30, 2026 01:02
2bcc363 to
e236e35
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
2 times, most recently
from
April 30, 2026 02:41
6a4cd9b to
1916615
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
May 6, 2026 02:02
1916615 to
8ad7a76
Compare
palash3
requested changes
May 19, 2026
palash3
left a comment
palash3
left a comment
Contributor
There was a problem hiding this comment.
Please update implementation using combine.
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
May 20, 2026 06:01
8ad7a76 to
a9c77d4
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
June 3, 2026 00:25
a9c77d4 to
1d0e923
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
June 3, 2026 00:58
1d0e923 to
f07e6d0
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
June 11, 2026 04:59
f07e6d0 to
dd15e5c
Compare
Contributor
|
Tested by stopping the NICs. The exporter reported OSCs in CONNECTING/CLOSED states with a metric value of 0. |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
5 times, most recently
from
June 15, 2026 01:17
9c3b629 to
05c670d
Compare
Contributor
|
Tested it and its working as expected. Unmount one ost verify that OST is disconnected and export metrics as 0. verify from lustre collector. |
palash3
reviewed
Jun 16, 2026
palash3
left a comment
palash3
left a comment
Contributor
There was a problem hiding this comment.
some comments other wise looks good
Contributor
|
@spoutn1k Can you please review ? |
Tested the ChangesTest 1: Basic FunctionalityVerified that the new Pre Conditions[root@node1 scratch]# lctl get_param osc.*.state | grep "^osc\." | wc -l
16
[root@node1 scratch]# lctl get_param osc.*.state | grep "current_state:"
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULLDeployed New Binary[root@node1 scratch]# systemctl stop lustrefs_exporter
[root@node1 scratch]# cp /usr/bin/lustrefs-exporter /usr/bin/lustrefs-exporter.backup.$(date +%Y%m%d)
[root@node1 scratch]# cp lustrefs-exporter /usr/bin/lustrefs-exporter
cp: overwrite '/usr/bin/lustrefs-exporter'? yes
[root@node1 scratch]# systemctl start lustrefs_exporter
[root@node1 scratch]# systemctl status lustrefs_exporter
● lustrefs_exporter.service - Prometheus exporter for Lustre filesystem
Loaded: loaded (/usr/lib/systemd/system/lustrefs_exporter.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/lustrefs_exporter.service.d
└─90-exporter-port.conf
Active: active (running) since Tue 2026-06-16 06:25:05 UTC; 5s ago
Docs: https://github.com/whamcloud/lustrefs-exporter
Main PID: 1212940 (lustrefs_export)
Tasks: 9 (limit: 36497)
Memory: 1.7M (high: 2.6G max: 3.0G)
CGroup: /system.slice/lustrefs_exporter.service
└─1212940 /usr/bin/lustrefs_exporter
Jun 16 06:25:05 node1 systemd[1]: Started Prometheus exporter for Lustre filesystem.
Jun 16 06:25:05 node1 lustrefs_exporter[1212940]: 2026-06-16T06:25:05.546869Z INFO lustrefs_exporter: Listening on http://0.0.0.0:32221/metricsVerified Metrics Endpoint[root@node1 scratch]# curl -s localhost:32221/metrics | grep "lustre_osc_state"
# HELP lustre_osc_state Lustre OSC connection state.
# TYPE lustre_osc_state gauge
lustre_osc_state{controller="fs-OST0000-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0001-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0002-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0003-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0004-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0005-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0006-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0007-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0008-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0009-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000a-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000b-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000c-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000d-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000e-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000f-osc-MDT0000",current_state="FULL"} 1
[root@node1 scratch]# curl -s localhost:32221/metrics | grep "^lustre_osc_state{" | wc -l
16Result: Test 2: lustre_collector Parsing[root@node1 scratch]# ./lustre_collector | jq '.[] | select(.Controller.OscState) | .Controller.OscState' | head -20
{
"kind": "Osc",
"param": "state",
"controller": "fs-OST000f-osc-MDT0000",
"value": {
"current_state": "FULL"
}
}
{
"kind": "Osc",
"param": "state",
"controller": "fs-OST0001-osc-MDT0000",
"value": {
"current_state": "FULL"
}
}
{
"kind": "Osc",
"param": "state",
"controller": "fs-OST0004-osc-MDT0000",Result: ConclusionThe implementation is working correctly. |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
July 15, 2026 03:08
23aa335 to
7a19d49
Compare
flei-ddn
marked this pull request as ready for review
July 15, 2026 03:23
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
July 18, 2026 01:26
7a19d49 to
cfb759c
Compare
Contributor
Author
|
@utopiabound Would you please review this PR again. Thanks. |
utopiabound
requested changes
Jul 30, 2026
flei-ddn
marked this pull request as draft
July 31, 2026 01:24
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
2 times, most recently
from
July 31, 2026 01:33
3c85ea4 to
6dfc69a
Compare
flei-ddn
marked this pull request as ready for review
July 31, 2026 06:27
Contributor
Author
Rebased. |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
August 4, 2026 06:11
6dfc69a to
029bb54
Compare
Contributor
Author
|
Ping reviewers. |
Contributor
|
Codecov checks are failing |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
August 18, 2026 11:24
029bb54 to
c5d15a2
Compare
flei-ddn
marked this pull request as draft
August 18, 2026 11:29
Add metrics for osc.*.state.current_state For lustrefs-exporter, if current_state is FULL or IDLE, the metrics value is 1. Or the metrics value is 0 Signed-off-by: Feng Lei <flei@ddn.com>
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
August 18, 2026 11:36
c5d15a2 to
a33d10b
Compare
flei-ddn
marked this pull request as ready for review
August 18, 2026 11:51
Contributor
Author
Codecov checking is OK now. |
spoutn1k
approved these changes
Aug 20, 2026
Comment on lines
+138
to
+141
| fn test_params() { | ||
| let params = params(); | ||
| assert_eq!(params, vec!["osc.*.state"]); | ||
| } |
johnsonw
reviewed
Aug 20, 2026
| } | ||
| } | ||
|
|
||
| pub fn build_controller_stats(x: &ControllerStats, metrics: &mut ControllerMetrics) { |
johnsonw
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add metrics for osc.*.state.current_state
For lustrefs-exporter, if current_state is FULL or IDLE, the metrics value is 1. Or the metrics value is 0
Test on an SFA system
On the first node, umount 1 OST
Check output of
lctl,lustre_collectorandlustrefs_exporterimmediatelyWe can see:
lustrefs_exporter, the value oflustre_osc_state{controller="tesfs-OST0000-osc-MDT0000",current_state="DISCONN"}is02.state:2.jsonThe unmounted OST will be recovered soon automatically
The HA system will recover the OST automatically