Skip to content

DLPX-96701 estat commands are failing with redefinition and forward declaration errors#115

Merged
tonynguien merged 1 commit intodevelopfrom
dlpx/pr/tonynguien/a3a2e6ed-777a-45b9-b83d-47f3b6106b60
Mar 10, 2026
Merged

DLPX-96701 estat commands are failing with redefinition and forward declaration errors#115
tonynguien merged 1 commit intodevelopfrom
dlpx/pr/tonynguien/a3a2e6ed-777a-45b9-b83d-47f3b6106b60

Conversation

@tonynguien
Copy link
Contributor

@tonynguien tonynguien commented Mar 6, 2026 •
edited
Loading

Background

This PR fixes minor issues in our estat and stbtrace BCC scripts.

Along with the new BCC build enabled by delphix/linux-pkg#389 and delphix/bcc#20, we should have profile-bpfcc and estat/stbtrace scripts working properly again.

Testing Done

Successful appliance build - https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/13435/

estat backend-io | iscsi | metaslab-alloc | nfs | nfs-by-client | zio | zio-queue | zpl | zvol

delphix@ip-10-110-198-210:~$ sudo estat backend-io -m 5
03/09/26 - 16:27:04 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                                   write
value range                 count ------------- Distribution -------------
[700, 800)                      3 |@
[800, 900)                     10 |@@@@
[900, 1000)                    39 |@@@@@@@@@@@@@@@@@
[1000, 2000)                   42 |@@@@@@@@@@@@@@@@@@
[3000, 4000)                    1 |@

                                       iops(/s)  avg latency(us)       stddev(us)  throughput(k/s)
write                                        19             1030            88913              249


                                       iops(/s)  throughput(k/s)
total                                        19              249

delphix@ip-10-110-198-210:~$ sudo estat iscsi -m 5    <-- No output as I don't have iSCSI configured
03/09/26 - 16:29:24 UTC

 Tracing enabled... Hit Ctrl-C to end.
 
delphix@ip-10-110-198-210:~$ sudo estat metaslab-alloc  -m 5
03/09/26 - 16:29:53 UTC

 Tracing enabled... Hit Ctrl-C to end.
03/09/26 - 16:29:58 UTC

^Cdelphix@ip-10-110-198-210:~$ sudo estat nfs-by-client -m 5
03/09/26 - 16:32:02 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                           v4/read/cached
value range                 count ------------- Distribution -------------
[20, 30)                        6 |@
[30, 40)                      333 |@@@@@@@@@@@@@@@@@@@
[40, 50)                      233 |@@@@@@@@@@@@@@
[50, 60)                       64 |@@@@
[60, 70)                       18 |@
[70, 80)                        8 |@
[80, 90)                       10 |@
[90, 100)                       4 |@
[100, 200)                     25 |@@
[200, 300)                      5 |@
[300, 400)                      5 |@
[400, 500)                      1 |@
[800, 900)                      1 |@

delphix@ip-10-110-198-210:~$ sudo estat nfs -m 5

03/09/26 - 16:27:42 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                          v4/read, cached
value range                 count ------------- Distribution -------------
[20, 30)                       18 |@@@
[30, 40)                      117 |@@@@@@@@@@@@@@@@@
[40, 50)                       42 |@@@@@@
[50, 60)                       12 |@@
[60, 70)                        5 |@
[70, 80)                        1 |@
[80, 90)                        2 |@
[90, 100)                       8 |@
[100, 200)                     67 |@@@@@@@@@@
[200, 300)                     11 |@@
[700, 800)                      1 |@

   microseconds                                        v4/read, uncached
value range                 count ------------- Distribution -------------
[500, 600)                      1 |@
[600, 700)                     78 |@@@@@
[700, 800)                    305 |@@@@@@@@@@@@@@@@@@@@
[800, 900)                    127 |@@@@@@@@@
[900, 1000)                    61 |@@@@
[1000, 2000)                   50 |@@@@
[2000, 3000)                    1 |@

                                       iops(/s)  avg latency(us)       stddev(us)  throughput(k/s)
v4/read, cached                              56               70             4056              454
v4/read, uncached                           124              812            26571              996


                                       iops(/s)  throughput(k/s)
total                                       181             1454


delphix@ip-10-110-198-210:~$ sudo estat zio -m 5
03/09/26 - 16:32:51 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                            write, asyncw
value range                 count ------------- Distribution -------------
[300, 400)                     18 |@
[400, 500)                    128 |@@@@@@@
[500, 600)                     98 |@@@@@
[600, 700)                     50 |@@@
[700, 800)                     26 |@@
[800, 900)                     16 |@
[900, 1000)                    12 |@
[1000, 2000)                  385 |@@@@@@@@@@@@@@@@@@@@
[2000, 3000)                   54 |@@@
[3000, 4000)                    2 |@
[4000, 5000)                    1 |@
[100000, 200000)                1 |@
[300000, 400000)                1 |@
[1000000, 2000000)                2 |@
[2000000, 3000000)                2 |@

^Cdelphix@ip-10-110-198-210:~$ sudo estat zio-queue -m 5
03/09/26 - 16:33:19 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                            write, asyncw
value range                 count ------------- Distribution -------------
[0, 10)                       281 |@@@
[10, 20)                        2 |@
[20, 30)                        4 |@
[30, 40)                        5 |@
[40, 50)                        8 |@
[50, 60)                        8 |@
[60, 70)                        7 |@
[70, 80)                        7 |@
[80, 90)                        6 |@
[90, 100)                      10 |@
[100, 200)                     71 |@
[200, 300)                     91 |@
[300, 400)                    100 |@
[400, 500)                     83 |@
[500, 600)                     76 |@
[600, 700)                     72 |@
[700, 800)                     84 |@
[800, 900)                    111 |@
[900, 1000)                   120 |@
[1000, 2000)                 1060 |@@@@@@@@@
[2000, 3000)                  998 |@@@@@@@@@
[3000, 4000)                  766 |@@@@@@@
[4000, 5000)                  529 |@@@@@
[5000, 6000)                  175 |@@
[6000, 7000)                   22 |@
[3000000, 4000000)               68 |@

delphix@ip-10-110-198-210:~$ sudo estat zpl -m 5
03/09/26 - 16:35:40 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                           zfs_read async
value range                 count ------------- Distribution -------------
[20, 30)                      144 |@@@@
[30, 40)                      242 |@@@@@@
[40, 50)                       46 |@@
[50, 60)                       14 |@
[60, 70)                        4 |@
[70, 80)                        4 |@
[80, 90)                        3 |@
[100, 200)                     10 |@
[200, 300)                      1 |@
[400, 500)                      1 |@
[500, 600)                     19 |@
[600, 700)                    234 |@@@@@@
[700, 800)                    597 |@@@@@@@@@@@@@@
[800, 900)                    209 |@@@@@
[900, 1000)                    88 |@@@
[1000, 2000)                   98 |@@@
[2000, 3000)                    8 |@
[3000, 4000)                    2 |@

   microseconds                                           zfs_write sync
value range                 count ------------- Distribution -------------
[800, 900)                      1 |@
[900, 1000)                     6 |@
[1000, 2000)                 1613 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[2000, 3000)                   46 |@@
[3000, 4000)                    9 |@
[4000, 5000)                    5 |@
[5000, 6000)                    3 |@
[9000, 10000)                   1 |@

                                       iops(/s)  avg latency(us)       stddev(us)  throughput(k/s)
zfs_read async                              344              600           158592             2756
zfs_write sync                              336             1322           174169             2694


                                       iops(/s)  throughput(k/s)
total                                       681             5454

delphix@ip-10-110-198-210:~$ sudo estat zvol -m 5
03/09/26 - 16:39:31 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                                     read
value range                 count ------------- Distribution -------------
[10, 20)                       61 |@
[20, 30)                      612 |@@@@@
[30, 40)                     1142 |@@@@@@@@@@
[40, 50)                      243 |@@
[50, 60)                      330 |@@@
[60, 70)                     1188 |@@@@@@@@@@
[70, 80)                      583 |@@@@@
[80, 90)                      237 |@@
[90, 100)                     114 |@
[100, 200)                    297 |@@@
[200, 300)                     88 |@
[300, 400)                     43 |@
[400, 500)                     27 |@
[500, 600)                     14 |@
[600, 700)                     14 |@
[700, 800)                      6 |@
[800, 900)                      7 |@
[900, 1000)                     2 |@
[1000, 2000)                   20 |@
[2000, 3000)                    3 |@
[3000, 4000)                    3 |@
[4000, 5000)                    2 |@
[5000, 6000)                    1 |@
[10000, 20000)                  1 |@

                                       iops(/s)  avg latency(us)       stddev(us)  throughput(k/s)
read                                       1002               85            79687             8022


                                       iops(/s)  throughput(k/s)
total                                      1008             8064

stbtrace io | iscsi | nfs | vfs | zio | zpl

delphix@ip-10-110-198-210:~$ sudo stbtrace io
{"t":"1773074520", "op":"read", "device":"nvme0n1", "error":"0", "count":"82", "avgLatency":"704960", "throughput":"671744", "latency":"{500000,5},{600000,39},{700000,22},{800000,2},{900000,3},{1000000,2},{2000000,8},{3000000,1}", "size":"{16383,82}"}

{"t":"1773074520", "op":"write", "device":"nvme0n1", "error":"0", "count":"124", "avgLatency":"1004553", "throughput":"1523712", "latency":"{900000,5},{1000000,68},{2000000,51}", "size":"{16383,124}"}

delphix@ip-10-110-198-210:~$ sudo stbtrace iscsi     <-- No output because I don't have iSCSI configured
^Cdelphix@ip-10-110-198-210:~$

^Cdelphix@ip-10-110-198-210:~$ sudo stbtrace nfs
{"t":"1773074638", "op":"write", "ver":"v4", "sync":"0", "cached":"-1", "client":"*", "count":"2", "avgLatency":"2237882", "throughput":"16384", "latency":"{2000000,1},{3000000,1}", "size":"{16383,2}"}

{"t":"1773074638", "op":"write", "ver":"v4", "sync":"1", "cached":"-1", "client":"*", "count":"159", "avgLatency":"1589633", "throughput":"1302528", "latency":"{2000000,147},{3000000,6},{4000000,3},{5000000,1},{8000000,1},{9000000,1}", "size":"{16383,159}"}

{"t":"1773074638", "op":"read", "ver":"v4", "sync":"-1", "cached":"0", "client":"*", "count":"166", "avgLatency":"1019408", "throughput":"1359872", "latency":"{700000,5},{800000,48},{900000,39},{1000000,21},{2000000,47},{3000000,5},{5000000,1}", "size":"{16383,166}"}

delphix@ip-10-110-198-210:~$ sudo stbtrace vfs
{"t":"1773074676", "op":"write", "sync":"0", "cached":"-1", "count":"6", "avgLatency":"19386", "throughput":"1016", "latency":"{10000,2},{20000,2},{40000,2}", "size":"{63,2},{255,2},{511,2}"}

{"t":"1773074677", "op":"write", "sync":"0", "cached":"-1", "count":"19", "avgLatency":"70682", "throughput":"163840", "latency":"{40000,3},{50000,8},{60000,1},{70000,2},{90000,2},{200000,2},{300000,1}", "size":"{16383,18},{32767,1}"}

{"t":"1773074677", "op":"write", "sync":"0", "cached":"-1", "count":"4", "avgLatency":"100806", "throughput":"550", "latency":"{30000,1},{50000,2},{300000,1}", "size":"{31,2},{255,1},{511,1}"}

delphix@ip-10-110-198-210:~$ sudo stbtrace zio
{"t":"1773074722", "op":"read", "count":"149", "avgLatency":"793323", "throughput":"1205248", "latency":"{600000,16},{700000,74},{800000,34},{900000,10},{1000000,1},{2000000,11},{3000000,1},{6000000,1},{7000000,1}", "size":"{1023,2},{16383,147}"}

{"t":"1773074722", "op":"write", "count":"160", "avgLatency":"1098208", "throughput":"1966080", "latency":"{1000000,45},{2000000,114},{4000000,1}", "size":"{16383,160}"}

{"t":"1773074723", "op":"read", "count":"118", "avgLatency":"790674", "throughput":"966656", "latency":"{600000,10},{700000,58},{800000,25},{900000,5},{1000000,4},{2000000,14},{3000000,2}", "size":"{16383,118}"}

{"t":"1773074723", "op":"write", "count":"122", "avgLatency":"1167812", "throughput":"1507328", "latency":"{1000000,31},{2000000,87},{3000000,4},{6000000,1}", "size":"{16383,124}"}

delphix@ip-10-110-198-210:~$ sudo stbtrace zpl

{"t":"1773074765", "op":"write", "sync":"0", "cached":"-1", "count":"9", "avgLatency":"281591", "throughput":"25592", "latency":"{10000,1},{20000,1},{30000,3},{60000,1},{100000,1},{2000000,2}", "size":"{63,2},{255,2},{511,2},{16383,3}"}

{"t":"1773074765", "op":"write", "sync":"1", "cached":"-1", "count":"67", "avgLatency":"1396422", "throughput":"548864", "latency":"{2000000,62},{3000000,5}", "size":"{16383,67}"}

{"t":"1773074765", "op":"read", "sync":"-1", "cached":"0", "count":"72", "avgLatency":"849620", "throughput":"589824", "latency":"{700000,10},{800000,32},{900000,15},{1000000,5},{2000000,9},{4000000,1}", "size":"{16383,72}"}

{"t":"1773074765", "op":"read", "sync":"-1", "cached":"1", "count":"6", "avgLatency":"1051229", "throughput":"49152", "latency":"{800000,4},{900000,1},{3000000,1}", "size":"{16383,6}"}

@tonynguien tonynguien force-pushed the dlpx/pr/tonynguien/a3a2e6ed-777a-45b9-b83d-47f3b6106b60 branch 4 times, most recently from c2a9c08 to 0849f10 Compare March 9, 2026 16:54
@tonynguien tonynguien marked this pull request as ready for review March 9, 2026 17:02
dbjwhs-perforce
dbjwhs-perforce approved these changes Mar 9, 2026
View reviewed changes
Copy link

@dbjwhs-perforce dbjwhs-perforce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just two nits

@tonynguien tonynguien force-pushed the dlpx/pr/tonynguien/a3a2e6ed-777a-45b9-b83d-47f3b6106b60 branch from 0849f10 to 6191c9f Compare March 9, 2026 20:22
@tonynguien tonynguien merged commit a45c6a8 into develop Mar 10, 2026
10 checks passed
@tonynguien tonynguien deleted the dlpx/pr/tonynguien/a3a2e6ed-777a-45b9-b83d-47f3b6106b60 branch March 10, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@prakashsurya prakashsurya prakashsurya approved these changes

@sebroy sebroy sebroy approved these changes

@dbjwhs-perforce dbjwhs-perforce dbjwhs-perforce approved these changes

@david-mendez1 david-mendez1 Awaiting requested review from david-mendez1

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tonynguien @prakashsurya @sebroy @dbjwhs-perforce