EX-10318: add metrics for osc.*.state.current_state#137
Open
flei-ddn wants to merge 1 commit into
Open
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #137 +/- ##
==========================================
- Coverage 94.57% 94.06% -0.52%
==========================================
Files 46 46
Lines 6107 6246 +139
Branches 6107 6246 +139
==========================================
+ Hits 5776 5875 +99
- Misses 256 292 +36
- Partials 75 79 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Bencher Report| Branch | flei-ddn/EX-10318 |
| Testbed | ci-runner |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result nanoseconds (ns) (Result Δ%) | Lower Boundary nanoseconds (ns) (Limit %) | Upper Boundary nanoseconds (ns) (Limit %) |
|---|---|---|---|---|
| parse_benchmarks/combine_performance | 📈 view plot 🚷 view threshold | 135,990,000.00 ns(-45.36%)Baseline: 248,884,789.47 ns | -698,347,190.86 ns (-513.53%) | 1,196,116,769.81 ns (11.37%) |
|
| Branch | flei-ddn/EX-10318 |
| Testbed | ci-runner |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
- Total read+write (reads/writes)
- Dr (reads)
- Estimated Cycles (cycles)
- D1mw (misses (writes))
- RAM Hit Rate (hits (%))
- DLmr (misses (reads))
- LLi Miss Rate (misses (%))
- I1mr (misses (reads))
- LL Miss Rate (misses (%))
- I1 Miss Rate (misses (%))
- D1mr (misses (reads))
- DLmw (misses (writes))
- LL Hit Rate (hits (%))
- D1 Miss Rate (misses (%))
- L1 Hit Rate (hits (%))
- LL Hits (hits)
- ILmr (misses (reads))
- L1 Hits (hits)
- Dw (writes)
- LLd Miss Rate (misses (%))
- RAM Hits (hits)
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | D1 Miss Rate | misses (%) | D1mr | misses (reads) x 1e3 | D1mw | misses (writes) x 1e3 | DLmr | misses (reads) | DLmw | misses (writes) x 1e3 | Dr | reads x 1e6 | Dw | writes x 1e6 | Estimated Cycles | cycles x 1e6 | I1 Miss Rate | misses (%) | I1mr | misses (reads) x 1e3 | ILmr | misses (reads) | Instructions | Benchmark Result instructions x 1e6 (Result Δ%) | Lower Boundary instructions x 1e6 (Limit %) | Upper Boundary instructions x 1e6 (Limit %) | L1 Hit Rate | hits (%) | L1 Hits | hits x 1e6 | LL Hit Rate | hits (%) | LL Hits | hits x 1e3 | LL Miss Rate | misses (%) | LLd Miss Rate | misses (%) | LLi Miss Rate | misses (%) | RAM Hit Rate | hits (%) | RAM Hits | hits x 1e3 | Total read+write | reads/writes x 1e6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| lustre_metrics::memory_benches::bench_encode_lustre_metrics with_setup:generate_records() | 📈 view plot | 0.93 % | 📈 view plot | 25.14 reads x 1e3 | 📈 view plot | 9.15 writes x 1e3 | 📈 view plot | 122.00 reads | 📈 view plot | 6.66 writes x 1e3 | 📈 view plot | 2.47 x 1e6 | 📈 view plot | 1.23 x 1e6 | 📈 view plot | 14.83 x 1e6 | 📈 view plot | 0.01 % | 📈 view plot | 1.08 reads x 1e3 | 📈 view plot | 918.00 reads | 📈 view plot 🚷 view threshold | 10.75 x 1e6(-19.47%)Baseline: 13.35 x 1e6 | 2.42 x 1e6 (22.55%) | 24.28 x 1e6 (44.28%) | 📈 view plot | 99.76 % | 📈 view plot | 14.42 x 1e6 | 📈 view plot | 0.19 % | 📈 view plot | 27.66 x 1e3 | 📈 view plot | 0.05 % | 📈 view plot | 0.18 % | 📈 view plot | 0.01 % | 📈 view plot | 0.05 % | 📈 view plot | 7.70 x 1e3 | 📈 view plot | 14.45 x 1e6 |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
2bcc363 to
e236e35
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
2 times, most recently
from
6a4cd9b to
1916615
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
1916615 to
8ad7a76
Compare
palash3
requested changes
May 19, 2026
palash3
left a comment
palash3
left a comment
Contributor
There was a problem hiding this comment.
Please update implementation using combine.
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
8ad7a76 to
a9c77d4
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
a9c77d4 to
1d0e923
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
1d0e923 to
f07e6d0
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
f07e6d0 to
dd15e5c
Compare
Contributor
|
Tested by stopping the NICs. The exporter reported OSCs in CONNECTING/CLOSED states with a metric value of 0. |
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
5 times, most recently
from
9c3b629 to
05c670d
Compare
Contributor
|
Tested it and its working as expected. Unmount one ost verify that OST is disconnected and export metrics as 0. verify from lustre collector. |
palash3
reviewed
Jun 16, 2026
palash3
left a comment
palash3
left a comment
Contributor
There was a problem hiding this comment.
some comments other wise looks good
Contributor
|
@spoutn1k Can you please review ? |
Tested the ChangesTest 1: Basic FunctionalityVerified that the new Pre Conditions[root@node1 scratch]# lctl get_param osc.*.state | grep "^osc\." | wc -l
16
[root@node1 scratch]# lctl get_param osc.*.state | grep "current_state:"
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULL
current_state: FULLDeployed New Binary[root@node1 scratch]# systemctl stop lustrefs_exporter
[root@node1 scratch]# cp /usr/bin/lustrefs-exporter /usr/bin/lustrefs-exporter.backup.$(date +%Y%m%d)
[root@node1 scratch]# cp lustrefs-exporter /usr/bin/lustrefs-exporter
cp: overwrite '/usr/bin/lustrefs-exporter'? yes
[root@node1 scratch]# systemctl start lustrefs_exporter
[root@node1 scratch]# systemctl status lustrefs_exporter
● lustrefs_exporter.service - Prometheus exporter for Lustre filesystem
Loaded: loaded (/usr/lib/systemd/system/lustrefs_exporter.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/lustrefs_exporter.service.d
└─90-exporter-port.conf
Active: active (running) since Tue 2026-06-16 06:25:05 UTC; 5s ago
Docs: https://github.com/whamcloud/lustrefs-exporter
Main PID: 1212940 (lustrefs_export)
Tasks: 9 (limit: 36497)
Memory: 1.7M (high: 2.6G max: 3.0G)
CGroup: /system.slice/lustrefs_exporter.service
└─1212940 /usr/bin/lustrefs_exporter
Jun 16 06:25:05 node1 systemd[1]: Started Prometheus exporter for Lustre filesystem.
Jun 16 06:25:05 node1 lustrefs_exporter[1212940]: 2026-06-16T06:25:05.546869Z INFO lustrefs_exporter: Listening on http://0.0.0.0:32221/metricsVerified Metrics Endpoint[root@node1 scratch]# curl -s localhost:32221/metrics | grep "lustre_osc_state"
# HELP lustre_osc_state Lustre OSC connection state.
# TYPE lustre_osc_state gauge
lustre_osc_state{controller="fs-OST0000-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0001-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0002-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0003-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0004-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0005-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0006-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0007-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0008-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST0009-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000a-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000b-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000c-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000d-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000e-osc-MDT0000",current_state="FULL"} 1
lustre_osc_state{controller="fs-OST000f-osc-MDT0000",current_state="FULL"} 1
[root@node1 scratch]# curl -s localhost:32221/metrics | grep "^lustre_osc_state{" | wc -l
16Result: Test 2: lustre_collector Parsing[root@node1 scratch]# ./lustre_collector | jq '.[] | select(.Controller.OscState) | .Controller.OscState' | head -20
{
"kind": "Osc",
"param": "state",
"controller": "fs-OST000f-osc-MDT0000",
"value": {
"current_state": "FULL"
}
}
{
"kind": "Osc",
"param": "state",
"controller": "fs-OST0001-osc-MDT0000",
"value": {
"current_state": "FULL"
}
}
{
"kind": "Osc",
"param": "state",
"controller": "fs-OST0004-osc-MDT0000",Result: ConclusionThe implementation is working correctly. |
palash3
previously approved these changes
Jun 16, 2026
utopiabound
requested changes
Jun 16, 2026
utopiabound
left a comment
utopiabound
left a comment
Contributor
There was a problem hiding this comment.
Missing support for non-osc states, and named as if only osc.*.state exist
utopiabound
reviewed
Jun 16, 2026
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
2 times, most recently
from
0047851 to
881dc13
Compare
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
881dc13 to
9651fd3
Compare
spoutn1k
reviewed
Jun 18, 2026
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
9651fd3 to
76abead
Compare
spoutn1k
force-pushed
the
flei-ddn/EX-10318
branch
from
76abead to
22d4793
Compare
Add metrics for osc.*.state.current_state For lustrefs-exporter, if current_state is FULL or IDLE, the metrics value is 1. Or the metrics value is 0 Signed-off-by: Feng Lei <flei@ddn.com>
flei-ddn
force-pushed
the
flei-ddn/EX-10318
branch
from
22d4793 to
9242260
Compare
Author
|
@utopiabound @spoutn1k Please reivew this patch again. Thanks. |
Contributor
|
@spoutn1k Could you please re review ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add metrics for osc.*.state.current_state
For lustrefs-exporter, if current_state is FULL or IDLE, the metrics value is 1. Or the metrics value is 0
Test on an SFA system
On the first node, umount 1 OST
Check output of
lctl,lustre_collectorandlustrefs_exporterimmediatelyWe can see:
lustrefs_exporter, the value oflustre_osc_state{controller="tesfs-OST0000-osc-MDT0000",current_state="DISCONN"}is02.state:2.jsonThe unmounted OST will be recovered soon automatically
The HA system will recover the OST automatically