feat(app): Backend frame count metrics #3308

cratelyn · 2024-10-30T23:33:02Z

☁️ overview

this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC
response bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes
included in body frames.

this middleware allows operators to reason about how large or small the packets being served in a backend's
response bodies are.

a route-level middleware that instruments request bodies will be added in a follow-on PR.

📝 changes

an overview of changes made here:

the linkerd-http-prom has a new body_data submodule. it exposes request and response halves, to
be explicit about which body is being instrumented on a tower::Service.
the linkerd-http-prom crate now has a collection of new dependencies: bytes is added as a dependency
in order to inspect the data chunk when the inner body yields a new frame. futures-util and http-body
are added as dev-dependencies for the accompanying test coverage.
body metrics are affixed to the RouteBackendMetrics<L> structure, and registered at startup.

🔗 related

cratelyn

🍞 breadcrumbs for review...

linkerd/http/prom/src/body_data/body.rs

linkerd/http/prom/src/body_data/metrics.rs

linkerd/app/outbound/src/http/logical/policy/route/backend/metrics.rs

linkerd/http/prom/src/body_data/response.rs

### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added Signed-off-by: katelyn martin <[email protected]>

### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. Signed-off-by: katelyn martin <[email protected]>

cratelyn · 2024-11-10T23:10:52Z

this pr has been rebased onto main as of release v2.263.1.

linkerd/http/prom/src/body_data/metrics.rs

linkerd/http/prom/src/body_data/response.rs

linkerd/http/prom/src/body_data/metrics.rs

olix0r

I through together a quick example of what a histogram implementation here looks like: kate/record-body-data...ver/kate/record-body-data (It compiles, though I didn't check the test status).

This replaces the counter/total metrics pair: the frame_size counters give us the total number of frames processed, and the sum gives us the total number of bytes. Getting in the proper metrics in a first pass saves us a deprecation cycle.

I chose a fairly constrained set of buckets to give us orders of magnitude: 128, 1024, 10240 (and we'll get an +Inf).

I also tried to handle the wrapping case a little more gracefully.

linkerd/http/prom/src/body_data/metrics.rs

cratelyn · 2024-11-20T19:25:10Z

📝 NB: c874231 was lacking a DCO signoff, so i've added one on your behalf to satiate CI, @olix0r. that's now 225674b.

see; https://github.com/linkerd/linkerd2-proxy/runs/33282345859

this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC response bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this middleware allows operators to reason about how large or small the packets being served in a backend's response bodies are. a route-level middleware that instruments request bodies will be added in a follow-on PR. ### 📝 changes an overview of changes made here: * the `linkerd-http-prom` has a new `body_data` submodule. it exposes `request` and `response` halves, to be explicit about which body is being instrumented on a `tower::Service`. * the `linkerd-http-prom` crate now has a collection of new dependencies: `bytes` is added as a dependency in order to inspect the data chunk when the inner body yields a new frame. `futures-util` and `http-body` are added as dev-dependencies for the accompanying test coverage. * body metrics are affixed to the `RouteBackendMetrics<L>` structure, and registered at startup. Signed-off-by: katelyn martin <[email protected]>

Signed-off-by: katelyn martin <[email protected]>

continuing this theme of inlining, we inline the passthrough methods on `Body` as well. Signed-off-by: katelyn martin <[email protected]>

Signed-off-by: katelyn martin <[email protected]>

Signed-off-by: Oliver Gould <[email protected]>

see: * prometheus/client_rust#242 * prometheus/client_rust#241 for now, refactor this test so that it gates all use of the (proposed) `sum()` and `count()` accessors behind a temporary feature gate. Signed-off-by: katelyn martin <[email protected]>

cratelyn · 2024-11-20T19:38:47Z

this pr has been rebased onto main as of release v2.666.0.

cratelyn · 2024-11-20T19:52:58Z

I through together a quick example of what a histogram implementation here looks like: kate/record-body-data...ver/kate/record-body-data (It compiles, though I didn't check the test status).

This replaces the counter/total metrics pair: the frame_size counters give us the total number of frames processed, and the sum gives us the total number of bytes. Getting in the proper metrics in a first pass saves us a deprecation cycle.

I chose a fairly constrained set of buckets to give us orders of magnitude: 128, 1024, 10240 (and we'll get an +Inf).

I also tried to handle the wrapping case a little more gracefully.

changes from this branch have been incorporated here. i've added a DCO signoff, here.

olix0r

This looks great!

### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, and (b) the number of bytes included in body frames. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. Signed-off-by: katelyn martin <[email protected]>

### ⛅ overview this introduces a new tower middleware for Prometheus metrics, used for instrumenting HTTP and gRPC request bodies, and observing (a) the number of frames yielded by a body, (b) the number of bytes included in body frames, and (c) a distribution of the size of frames yielded. this builds upon the backend-level metrics added in #3308. this additionally uses the route label extractor, hoisted out of the retry middleware's Prometheus telemetry in #3337. ### 📝 changes * a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData` middleware is added, which complements the equivalent `linkerd_http_prom::body_data::response` middleware. * this is added to policy routes' metrics layer. see prometheus/client_rust#241 and prometheus/client_rust#242, which track upstream proposals to add accessors to `Histogram` that will allow us to make test assertions that metrics are working properly. for now, these are feature gated as also done in #3308. Signed-off-by: katelyn martin <[email protected]>

#3308 introduced tower middleware to the outbound proxy that records the size and number of response body's data frames. this commit introduces an equivalent layer to the inbound proxy's prometheus metrics. Signed-off-by: katelyn martin <[email protected]>

<#3308> introduced tower middleware to the outbound proxy that records the size and number of response body's data frames. this commit introduces an equivalent layer to the inbound proxy's prometheus metrics. NB: this is based upon <#4165> Signed-off-by: katelyn martin <[email protected]>

<#3308> introduced tower middleware to the outbound proxy that records the size and number of response body's data frames. this commit introduces an equivalent layer to the inbound proxy's prometheus metrics. NB: this is based upon <#4166> Signed-off-by: katelyn martin <[email protected]>

cratelyn self-assigned this Oct 30, 2024

cratelyn force-pushed the kate/record-body-data branch 5 times, most recently from 044cc73 to 41d2c9c Compare October 31, 2024 21:18

cratelyn changed the title ~~feat(app): work in progress body frame metrics~~ feat(app): Backend frame count metric Nov 1, 2024

cratelyn force-pushed the kate/record-body-data branch 2 times, most recently from f9d86cb to c08a6b5 Compare November 1, 2024 14:44

cratelyn marked this pull request as ready for review November 1, 2024 14:55

cratelyn requested a review from a team as a code owner November 1, 2024 14:55

cratelyn commented Nov 1, 2024

View reviewed changes

cratelyn changed the title ~~feat(app): Backend frame count metric~~ feat(app): Backend frame count metrics Nov 1, 2024

sfleen self-requested a review November 1, 2024 16:10

olix0r reviewed Nov 1, 2024

View reviewed changes

linkerd/http/prom/src/body_data/response.rs Outdated Show resolved Hide resolved

linkerd/http/prom/src/body_data/response.rs Show resolved Hide resolved

cratelyn mentioned this pull request Nov 4, 2024

refactor(app): Simplify backend metrics tower layer #3318

Merged

cratelyn requested a review from olix0r November 4, 2024 18:52

cratelyn mentioned this pull request Nov 7, 2024

refactor(app): Outline route label extractor #3337

Merged

cratelyn force-pushed the kate/record-body-data branch from cd2d1b9 to 3b1db24 Compare November 10, 2024 22:33

cratelyn mentioned this pull request Nov 10, 2024

feat(app): Route frame count metrics #3334

Merged

cratelyn removed their assignment Nov 10, 2024

DavidMcLaughlin assigned olix0r Nov 11, 2024

olix0r reviewed Nov 15, 2024

View reviewed changes

olix0r reviewed Nov 19, 2024

View reviewed changes

linkerd/http/prom/src/body_data/metrics.rs Outdated Show resolved Hide resolved

cratelyn force-pushed the kate/record-body-data branch from 52f55c6 to cc04760 Compare November 20, 2024 19:23

cratelyn and others added 8 commits November 20, 2024 14:37

review: Inline attribute to service passthrough

0da2152

Signed-off-by: katelyn martin <[email protected]>

review: Inline attribute to body passthrough

670143c

continuing this theme of inlining, we inline the passthrough methods on `Body` as well. Signed-off-by: katelyn martin <[email protected]>

review: Box <RecordBodyData as Service>::Future values

0d4a41b

Signed-off-by: katelyn martin <[email protected]>

review: rename get() to metrics()

5a1a978

Signed-off-by: katelyn martin <[email protected]>

review: simplify RecordBodyData<S> response

f147343

Signed-off-by: katelyn martin <[email protected]>

Update ResponseBody metrics to use a histogram

da6e10c

Signed-off-by: Oliver Gould <[email protected]>

cratelyn force-pushed the kate/record-body-data branch from 95e796f to c728e18 Compare November 20, 2024 19:38

cratelyn force-pushed the kate/record-body-data branch from c728e18 to 33f5f60 Compare November 20, 2024 19:47

cratelyn requested a review from olix0r November 20, 2024 19:53

olix0r approved these changes Nov 20, 2024

View reviewed changes

cratelyn merged commit 049e01b into main Nov 20, 2024
16 checks passed

cratelyn deleted the kate/record-body-data branch November 20, 2024 20:45

cratelyn mentioned this pull request Nov 21, 2024

Implement number of HTTP frames metric #3044

Closed

cratelyn mentioned this pull request Jan 9, 2025

The counter metrics about the number of sent/received events in streams linkerd/linkerd2#13115

Closed

cratelyn mentioned this pull request Sep 16, 2025

feat(app/inbound): response body frame size metrics #4165

Merged

feat(app): Backend frame count metrics #3308

feat(app): Backend frame count metrics #3308

Uh oh!

Conversation

cratelyn commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

☁️ overview

📝 changes

🔗 related

Uh oh!

cratelyn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cratelyn commented Nov 10, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

olix0r left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cratelyn commented Nov 20, 2024

Uh oh!

cratelyn commented Nov 20, 2024

Uh oh!

cratelyn commented Nov 20, 2024

Uh oh!

olix0r left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cratelyn commented Oct 30, 2024 •

edited

Loading