feat: otel metrics

[fenos]

What kind of change does this PR introduce?

Feature

What is the current behavior?

Currently metrics are gathered using the prom-client and exposed to a metric endpoint

What is the new behavior?

• All metrics are now otel metrics and can be pushed to a 3rd party otel backend
• We still can optionally expose prometheus metrics in the /metrics endpoint using the otel prom exporter (ENABLE_PROMETHEUS_METRICS)
• All children spans will now include the tenant.ref attribute from a top level span
• Revamped grafana dashboard using otel metrics

Grafana dashboard

Running completely locally

[coveralls]

Pull Request Test Coverage Report for Build 21515862680

Details

• 673 of 1285 (52.37%) changed or added relevant lines in 33 files are covered.
• 2 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.4%) to 75.548%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/admin-app.ts	5	6	83.33%
src/http/routes/s3/commands/create-multipart-upload.ts	1	2	50.0%
src/internal/monitoring/metrics.ts	194	197	98.48%
src/internal/queue/queue.ts	3	6	50.0%
src/storage/backend/s3/adapter.ts	3	6	50.0%
src/internal/monitoring/logger.ts	9	13	69.23%
src/internal/queue/event.ts	1	5	20.0%
src/internal/database/pool.ts	47	55	85.45%
src/http/plugins/log-request.ts	17	28	60.71%
src/internal/monitoring/system/base-collector.ts	33	45	73.33%

Files with Coverage Reduction	New Missed Lines	%
src/admin-app.ts	1	89.19%
src/http/routes/s3/commands/upload-part.ts	1	65.68%

Totals
Change from base Build 21449071800:	-0.4%
Covered Lines:	25902
Relevant Lines:	34015

---

💛 - Coveralls

[ferhatelmas]

This was supposed to be multipart I guess according to description but instead duplicates query A

[ferhatelmas]

I think we need to capture returned function unregister here to pass shutdown below. Otherwise, it might prevent process from exiting

[fenos]

Thanks! I have double checked but this function is not returning anything.
I guess it registers it internally, and on stop, it will clean it up

[ferhatelmas]

It returns a cleanup function, I believe we need to call it before meter shutdown for clean exit because it will stop timer for metric reader but won't touch the timer inside custom instrument. Another approach could be making timer unblocking as in #819 (comment)