Skip to content

Commit d98c705

Browse files
mhausenblaschalincodebotensvrnmcartermp
authored
inits Deployment contrib (open-telemetry#2498)
Co-authored-by: Patrice Chalin <[email protected]> Co-authored-by: Alex Boten <[email protected]> Co-authored-by: Patrice Chalin <[email protected]> Co-authored-by: Severin Neumann <[email protected]> Co-authored-by: Phillip Carter <[email protected]> Co-authored-by: Juraci Paixão Kröhling <[email protected]>
1 parent 3d9fe51 commit d98c705

File tree

11 files changed

+1066
-33
lines changed

11 files changed

+1066
-33
lines changed

‎content/en/blog/2023/end-user-q-and-a-02.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ Uplight currently has a few different Collector configurations:
198198

199199
Doug’s ultimate goal is for any deployment in any environment to be able to
200200
easily send telemetry to an
201-
[OTel Collector gateway](/docs/collector/deployment/#gateway).
201+
[OTel Collector gateway](/docs/collector/deployment/gateway/).
202202

203203
Collectors at Uplight are typically run and maintained by the infrastructure
204204
team, unless individual teams decide to take ownership of their own Collectors.

‎content/en/docs/collector/deployment.md

-32
This file was deleted.

‎content/en/docs/collector/deployment/_index.md

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
title: Deployment
3+
description: Patterns you can apply to deploy the OpenTelemetry collector
4+
weight: 2
5+
---
6+
7+
The OpenTelemetry collector consists of a single binary which you can use in
8+
different ways, for different use cases. This section describes deployment
9+
patterns, their use cases along with pros and cons and best practices for
10+
collector configurations for cross-environment and multi-backend deployments.
11+
12+
## Resources
13+
14+
- KubeCon NA 2021 Talk on [OpenTelemetry Collector Deployment
15+
Patterns][y-patterns]
16+
- [Deployment Patterns][gh-patterns] accompanying the talk
17+
18+
[gh-patterns]:
19+
https://github.com/jpkrohling/opentelemetry-collector-deployment-patterns/
20+
[y-patterns]: https://www.youtube.com/watch?v=WhRrwSHDBFs

‎content/en/docs/collector/deployment/agent.md

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
---
2+
title: Agent
3+
description:
4+
Why and how to send signals to collectors and from there to backends
5+
weight: 2
6+
---
7+
8+
The agent collector deployment pattern consists of applications —
9+
[instrumented][instrumentation] with an OpenTelemetry SDK using [OpenTelemetry
10+
protocol (OTLP)][otlp] — or other collectors (using the OTLP exporter) that send
11+
telemetry signals to a [collector][collector] instance running with the
12+
application or on the same host as the application (such as a sidecar or a
13+
daemonset).
14+
15+
Each client-side SDK or downstream collector is configured with a collector
16+
location:
17+
18+
![Decentralized collector deployment concept](../../img/agent-sdk.svg)
19+
20+
1. In the app, the SDK is configured to send OTLP data to a collector.
21+
1. The collector is configured to send telemetry data to one or more backends.
22+
23+
## Example
24+
25+
A concrete example of the agent collector deployment pattern could look as
26+
follows: you manually instrument, say, a [Java application to export
27+
metrics][instrument-java-metrics] using the OpenTelemetry Java SDK. In the
28+
context of the app, you would set the `OTEL_METRICS_EXPORTER` to `otlp` (which
29+
is the default value) and configure the [OTLP exporter][otlp-exporter] with the
30+
address of your collector, for example (in Bash or `zsh` shell):
31+
32+
```
33+
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.example.com:4318
34+
```
35+
36+
The collector serving at `collector.example.com:4318` would then be configured
37+
like so:
38+
39+
<!-- prettier-ignore-start -->
40+
{{< tabpane lang=yaml persistLang=false >}}
41+
{{< tab Traces >}}
42+
receivers:
43+
otlp: # the OTLP receiver the app is sending traces to
44+
protocols:
45+
grpc:
46+
47+
processors:
48+
batch:
49+
50+
exporters:
51+
jaeger: # the Jaeger exporter, to ingest traces to backend
52+
endpoint: "https://jaeger.example.com:14250"
53+
insecure: true
54+
55+
service:
56+
pipelines:
57+
traces/dev:
58+
receivers: [otlp]
59+
processors: [batch]
60+
exporters: [jaeger]
61+
{{< /tab >}}
62+
{{< tab Metrics >}}
63+
receivers:
64+
otlp: # the OTLP receiver the app is sending metrics to
65+
protocols:
66+
grpc:
67+
68+
processors:
69+
batch:
70+
71+
exporters:
72+
prometheusremotewrite: # the PRW exporter, to ingest metrics to backend
73+
endpoint: "https://prw.example.com/v1/api/remote_write"
74+
75+
service:
76+
pipelines:
77+
metrics/prod:
78+
receivers: [otlp]
79+
processors: [batch]
80+
exporters: [prometheusremotewrite]
81+
82+
{{< /tab >}}
83+
{{< tab Logs >}}
84+
receivers:
85+
otlp: # the OTLP receiver the app is sending logs to
86+
protocols:
87+
grpc:
88+
89+
processors:
90+
batch:
91+
92+
exporters:
93+
file: # the File Exporter, to ingest logs to local file
94+
path: "./app42_example.log"
95+
rotation:
96+
97+
service:
98+
pipelines:
99+
logs/dev:
100+
receivers: [otlp]
101+
processors: [batch]
102+
exporters: [file]
103+
{{< /tab >}}
104+
{{< /tabpane>}}
105+
<!-- prettier-ignore-end -->
106+
107+
If you want to try it out for yourself, you can have a look at the end-to-end
108+
[Java][java-otlp-example] or [Python][py-otlp-example] examples.
109+
110+
## Tradeoffs
111+
112+
Pros:
113+
114+
- Simple to get started
115+
- Clear 1:1 mapping between application and collector
116+
117+
Cons:
118+
119+
- Scalability (human and load-wise)
120+
- Inflexible
121+
122+
[instrumentation]: /docs/instrumentation/
123+
[otlp]: /docs/reference/specification/protocol/
124+
[collector]: /docs/collector/
125+
[instrument-java-metrics]: /docs/instrumentation/java/manual/#metrics
126+
[otlp-exporter]: /docs/reference/specification/protocol/exporter/
127+
[java-otlp-example]:
128+
https://github.com/open-telemetry/opentelemetry-java-docs/tree/main/otlp
129+
[py-otlp-example]:
130+
https://opentelemetry-python.readthedocs.io/en/stable/examples/metrics/instruments/README.html
131+
[lb-exporter]:
132+
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter
133+
[spanmetrics-processor]:
134+
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanmetricsprocessor

‎content/en/docs/collector/deployment/gateway.md

+170
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
---
2+
title: Gateway
3+
description:
4+
Why and how to send signals to a single OTLP end-point and from there to
5+
backends
6+
weight: 3
7+
---
8+
9+
The gateway collector deployment pattern consists of applications (or other
10+
collectors) sending telemetry signals to a single OTLP endpoint provided by one
11+
or more collector instances running as a standalone service (for example, a
12+
deployment in Kubernetes), typically per cluster, per data center or per region.
13+
14+
In the general case you can use an out-of-the-box load balancer to distribute
15+
the load amongst the collectors:
16+
17+
![Gateway deployment concept](../../img/gateway-sdk.svg)
18+
19+
For use cases where the processing of the telemetry data processing has to
20+
happen in a specific collector, you would use a two-tiered setup with a
21+
collector that has a pipeline configured with the [Trace ID/Service-name aware
22+
load-balancing exporter][lb-exporter] in the first tier and the collectors
23+
handling the scale out in the second tier. For example, you will need to use the
24+
load-balancing exporter when using the [Tail Sampling
25+
processor][tailsample-processor] so that all spans for a given trace reach the
26+
same collector instance where the tail sampling policy is applied.
27+
28+
Let's have a look at such a case where we are using the load-balancing exporter:
29+
30+
![Gateway deployment with load-balancing exporter](../../img/gateway-lb-sdk.svg)
31+
32+
1. In the app, the SDK is configured to send OTLP data to a central location.
33+
1. A collector configured using the load-balancing exporter that distributes
34+
signals to a group of collectors.
35+
1. The collectors are configured to send telemetry data to one or more backends.
36+
37+
{{% alert title="Note" color="info" %}} Currently, the load-balancing exporter
38+
only supports pipelines of the `traces` type. {{% /alert %}}
39+
40+
## Example
41+
42+
For a concrete example of the centralized collector deployment pattern we first
43+
need to have a closer look at the load-balancing exporter. It has two main
44+
configuration fields:
45+
46+
- The `resolver`, which determines where to find the downstream collectors (or:
47+
backends). If you use the `static` sub-key here, you will have to manually
48+
enumerate the collector URLs. The other supported resolver is the DNS resolver
49+
which will periodically check for updates and resolve IP addresses. For this
50+
resolver type, the `hostname` sub-key specifies the hostname to query in order
51+
to obtain the list of IP addresses.
52+
- With the `routing_key` field you tell the load-balancing exporter to route
53+
spans to specific downstream collectors. If you set this field to `traceID`
54+
(default) then the Load-balancing exporter exports spans based on their
55+
`traceID`. Otherwise, if you use `service` as the value for `routing_key`, it
56+
exports spans based on their service name which is useful when using
57+
connectors like the [Span Metrics connector][spanmetrics-connector], so all
58+
spans of a service will be send to the same downstream collector for metric
59+
collection, guaranteeting accurate aggregations.
60+
61+
The first-tier collector servicing the OTLP endpoint would be configured as
62+
shown below:
63+
64+
<!-- prettier-ignore-start -->
65+
{{< tabpane lang=yaml persistLang=false >}}
66+
{{< tab Static >}}
67+
receivers:
68+
otlp:
69+
protocols:
70+
grpc:
71+
72+
exporters:
73+
loadbalancing:
74+
protocol:
75+
otlp:
76+
insecure: true
77+
resolver:
78+
static:
79+
hostnames:
80+
- collector-1.example.com:4317
81+
- collector-2.example.com:5317
82+
- collector-3.example.com
83+
84+
service:
85+
pipelines:
86+
traces:
87+
receivers: [otlp]
88+
exporters: [loadbalancing]
89+
{{< /tab >}}
90+
{{< tab DNS >}}
91+
receivers:
92+
otlp:
93+
protocols:
94+
grpc:
95+
96+
exporters:
97+
loadbalancing:
98+
protocol:
99+
otlp:
100+
insecure: true
101+
resolver:
102+
dns:
103+
hostname: collectors.example.com
104+
105+
service:
106+
pipelines:
107+
traces:
108+
receivers: [otlp]
109+
exporters: [loadbalancing]
110+
{{< /tab >}}
111+
{{< tab "DNS with service" >}}
112+
receivers:
113+
otlp:
114+
protocols:
115+
grpc:
116+
117+
exporters:
118+
loadbalancing:
119+
routing_key: "service"
120+
protocol:
121+
otlp:
122+
insecure: true
123+
resolver:
124+
dns:
125+
hostname: collectors.example.com
126+
port: 5317
127+
128+
service:
129+
pipelines:
130+
traces:
131+
receivers: [otlp]
132+
exporters: [loadbalancing]
133+
{{< /tab >}}
134+
{{< /tabpane>}}
135+
<!-- prettier-ignore-end -->
136+
137+
The load-balancing exporter emits metrics including
138+
`otelcol_loadbalancer_num_backends` and `otelcol_loadbalancer_backend_latency`
139+
that you can use for health and performance monitoring of the OTLP endpoint
140+
collector.
141+
142+
## Tradeoffs
143+
144+
Pros:
145+
146+
- Separation of concerns such as centrally managed credentials
147+
- Centralized policy management (for example, filtering certain logs or
148+
sampling)
149+
150+
Cons:
151+
152+
- It's one more thing to maintain and that can fail (complexity)
153+
- Added latency in case of cascaded collectors
154+
- Higher overall resource usage (costs)
155+
156+
[instrumentation]: /docs/instrumentation/
157+
[otlp]: /docs/reference/specification/protocol/
158+
[collector]: /docs/collector/
159+
[instrument-java-metrics]: /docs/instrumentation/java/manual/#metrics
160+
[otlp-exporter]: /docs/reference/specification/protocol/exporter/
161+
[java-otlp-example]:
162+
https://github.com/open-telemetry/opentelemetry-java-docs/tree/main/otlp
163+
[py-otlp-example]:
164+
https://opentelemetry-python.readthedocs.io/en/stable/examples/metrics/instruments/README.html
165+
[lb-exporter]:
166+
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter
167+
[tailsample-processor]:
168+
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor
169+
[spanmetrics-connector]:
170+
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector

0 commit comments

Comments
 (0)