New metric for total time spent processing a request and response in skipper

[nishantssr]

Is your feature request related to a problem? Please describe.
we are currently working on our gateway for external partners to invoke our internal systems. We are in the process of defining our monitoring and alerts and were hoping on using the metrics you are exposing.
What we want to measure is the amount of time spent in skipper AND filters (in bold):
client invocation -> skipper[1] + filters[2] (request) - > backend service -> skipper[3] + filters[4] (response)

Currently we could use the following metrics:
skipper_proxy_total_duration_seconds_bucket will return [1] + [3]
skipper_filter_all_combined_request_duration_seconds_bucket will return [2]
skipper_filter_all_combined_response_duration_seconds_bucket will return [4]

Given that those metrics are histograms, we cannot add them to obtain the total number spent in skipper.

Describe the solution you would like
We would like to have a new metric exposed which will be the sum of
skipper_proxy_total_duration_seconds_bucket + skipper_filter_all_combined_request_duration_seconds_bucket + skipper_filter_all_combined_response_duration_seconds_bucket

Describe alternatives you've considered (optional)
For now, the only solution we have is to use separate metrics, which makes it difficult to define our alerts.
For instance, if our SLO of total time spent in the gateway is 100 ms, we would need to divide that time into those three different metrics

Would you like to work on it?
No

[szuecs]

We had a similar idea but filters highly depend on user choice. That's why we decided to have a proxy only metric that measures the overhead of the proxy and not the features applied by the users.
Likely the best choice for your users is to have proxy metric as core overhead and then filters separately such that your users can see what they want and should use for a given requirement.

As example: a user does some fault injection adding latency("200ms") and your SLO will be impaired and you have now to figure out why. So it's actually good for you to split your SLO.

Beware that summing pN latency percentiles is not correct for calculating a pN value.