From 0a790ff8f947976b9424d0827fe95f658a947670 Mon Sep 17 00:00:00 2001 From: ps48 Date: Thu, 25 Sep 2025 09:59:04 -0700 Subject: [PATCH 01/11] add documentation for unified OTLP source Signed-off-by: ps48 --- .../configuration/sources/otlp-source.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 _data-prepper/pipelines/configuration/sources/otlp-source.md diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md new file mode 100644 index 00000000000..7a60b109529 --- /dev/null +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -0,0 +1,253 @@ +--- +layout: default +title: OTLP source +parent: Sources +grand_parent: Pipelines +nav_order: 85 +--- + +# OTLP source + +The `otlp` source is a unified OpenTelemetry source that follows the [OpenTelemetry Protocol (OTLP) Specification](https://opentelemetry.io/docs/specs/otlp/) and can receive logs, metrics, and traces through a single endpoint. This source consolidates the functionality of the individual `otel_logs_source`, `otel_metrics_source`, and `otel_trace_source` sources, providing a streamlined approach to ingesting all OpenTelemetry telemetry signals. + +The OTLP source supports both the `OTLP/gRPC` and `OTLP/HTTP` protocols. For `OTLP/HTTP`, only Protobuf encoding is currently supported. This makes it compatible with a wide range of OpenTelemetry collectors and instrumentation libraries. +{: .note} + +## Configuration + +You can configure the `otlp` source with the following options. + +| Option | Type | Description | +| :--- | :--- | :--- | +| `port` | Integer | The port on which the OTLP source listens. Default value is `21893`. | +| `logs_path` | String | The path for sending unframed HTTP requests for logs. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| `metrics_path` | String | The path for sending unframed HTTP requests for metrics. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export`. | +| `traces_path` | String | The path for sending unframed HTTP requests for traces. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.trace.v1.TraceService/Export`. | +| `request_timeout` | Integer | The request timeout duration in milliseconds. Default value is `10000`. | +| `health_check_service` | Boolean | Enables a gRPC health check service under `grpc.health.v1.Health/Check`. When `unframed_requests` is also `true`, enables HTTP health check at `/health`. Default value is `false`. | +| `proto_reflection_service` | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. | +| `unframed_requests` | Boolean | Enables requests not framed using the gRPC wire protocol. Default value is `false`. | +| `thread_count` | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. | +| `max_connection_count` | Integer | The maximum allowed number of open connections. Default value is `500`. | +| `max_request_length` | String | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. | +| `compression` | String | The compression type applied on the client request payload. Valid values are `none` (no compression) or `gzip` (apply GZip decompression). Default value is `none`. | +| `output_format` | String | Specifies the decoded output format for all signals (logs, metrics, traces) if individual output format options are not set. Valid values are `otel` (OpenTelemetry format) or `opensearch` (OpenSearch format). Default value is `otel`. | +| `logs_output_format` | String | Specifies the decoded output format specifically for logs. Takes precedence over `output_format` for logs. Valid values are `otel` or `opensearch`. Default value is `otel`. | +| `metrics_output_format` | String | Specifies the decoded output format specifically for metrics. Takes precedence over `output_format` for metrics. Valid values are `otel` or `opensearch`. Default value is `otel`. | +| `traces_output_format` | String | Specifies the decoded output format specifically for traces. Takes precedence over `output_format` for traces. Valid values are `otel` or `opensearch`. Default value is `otel`. | + +If an individual output format (for example, `logs_output_format`) is set, it takes precedence over the generic `output_format` for that signal type. If neither is set, the default is `otel`. +{: .note} + +### SSL/TLS configuration + +You can configure SSL/TLS in the `otlp` source with the following options. + +| Option | Type | Description | +| :--- | :--- | :--- | +| `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | +| `sslKeyCertChainFile` | String | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| `sslKeyFile` | String | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| `useAcmCertForSSL` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| `acmCertificateArn` | String | Represents the ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `useAcmCertForSSL` is set to `true`. | +| `awsRegion` | String | Represents the AWS Region used by ACM or Amazon S3. Required if `useAcmCertForSSL` is set to `true` or if `sslKeyCertChainFile` and `sslKeyFile` are Amazon S3 paths. | + +### Authentication configuration + +By default, the OTLP source runs without authentication. You can configure authentication using the following options. + +You can explicitly disable authentication with: + +```yaml +source: + otlp: + authentication: + unauthenticated: +``` +{% include copy.html %} + +To enable HTTP Basic authentication: + +```yaml +source: + otlp: + authentication: + http_basic: + username: my-user + password: my_s3cr3t +``` +{% include copy.html %} + +This plugin uses pluggable authentication for gRPC servers. To provide custom authentication, create a plugin that implements [`GrpcAuthenticationProvider`](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/armeria-common/src/main/java/org/opensearch/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java). + +### Retry information + +Data Prepper replies with a `RetryInfo` object specifying how long to wait for the next request when backpressure builds up. The retry information implements exponential backoff with a configurable maximum delay. + +```yaml +source: + otlp: + retry_info: + min_delay: 100ms # defaults to 100ms + max_delay: 2s # defaults to 2s +``` +{% include copy.html %} + +## Usage + +### Basic configuration + +To get started with the `otlp` source, create a `pipeline.yaml` file with the following minimal configuration: + +```yaml +pipeline: + source: + otlp: + ssl: false + sink: + - stdout: +``` +{% include copy.html %} + +### Routing telemetry signals + +One of the key features of the OTLP source is the ability to route different telemetry signals (logs, metrics, traces) to different processors or sinks based on specific needs. The routing uses metadata-based routing with the `getEventType()` function. + +```yaml +otel-telemetry-pipeline: + source: + otlp: + ssl: false + route: + - logs: 'getEventType() == "LOG"' + - traces: 'getEventType() == "TRACE"' + - metrics: 'getEventType() == "METRIC"' + sink: + - pipeline: + name: "logs-pipeline" + routes: + - "logs" + - pipeline: + name: "traces-pipeline" + routes: + - "traces" + - pipeline: + name: "metrics-pipeline" + routes: + - "metrics" + +logs-pipeline: + source: + pipeline: + name: "otel-telemetry-pipeline" + processor: + # Add logs-specific processors here + sink: + - opensearch: + index: "logs-index" + +traces-pipeline: + source: + pipeline: + name: "otel-telemetry-pipeline" + processor: + # Add traces-specific processors here + sink: + - opensearch: + index: "traces-index" + +metrics-pipeline: + source: + pipeline: + name: "otel-telemetry-pipeline" + processor: + # Add metrics-specific processors here + sink: + - opensearch: + index: "metrics-index" +``` +{% include copy.html %} + +### Using OpenSearch output format + +To generate data in the OpenSearch format for all telemetry signals: + +```yaml +source: + otlp: + output_format: opensearch +``` +{% include copy.html %} + +To use different output formats for different signal types: + +```yaml +source: + otlp: + logs_output_format: opensearch + metrics_output_format: otel + traces_output_format: opensearch +``` +{% include copy.html %} + +### Configuring with SSL/TLS + +To enable SSL/TLS with local certificates: + +```yaml +source: + otlp: + ssl: true + sslKeyCertChainFile: "/path/to/certificate.crt" + sslKeyFile: "/path/to/private-key.key" +``` +{% include copy.html %} + +To use AWS Certificate Manager: + +```yaml +source: + otlp: + ssl: true + useAcmCertForSSL: true + acmCertificateArn: "arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012" + awsRegion: "us-east-1" +``` +{% include copy.html %} + +## Metrics + +The `otlp` source includes the following metrics to monitor its performance and health. + +### Counters + +| Metric | Description | +| :--- | :--- | +| `requestTimeouts` | Measures the total number of requests that time out. | +| `requestsReceived` | Measures the total number of requests received by the OTLP source. | +| `successRequests` | Measures the total number of requests successfully processed by the OTLP source. | +| `badRequests` | Measures the total number of requests with invalid format processed by the OTLP source. | +| `requestsTooLarge` | Measures the total number of requests that exceed the maximum allowed size. | +| `internalServerError` | Measures the total number of requests processed by the OTLP source with custom exception types. | + +### Timers + +| Metric | Description | +| :--- | :--- | +| `requestProcessDuration` | Measures the latency of requests processed by the OTLP source in seconds. | + +### Distribution summaries + +| Metric | Description | +| :--- | :--- | +| `payloadSize` | Measures the distribution of incoming request payload sizes in bytes. | + +## Migration from individual OTel sources + +If you're currently using separate `otel_logs_source`, `otel_metrics_source`, or `otel_trace_source` sources, you can migrate to the unified OTLP source by: + +1. Replacing all three sources with a single `otlp` source +2. Using the routing configuration shown above to direct different signal types to their appropriate pipelines +3. Adjusting the port numbers if needed (the OTLP source uses port `21893` by default) + +The OTLP source provides all the functionality of the individual sources while simplifying configuration and reducing resource usage. \ No newline at end of file From 238d4ed4f2ed895e83a7fd30096dc4195d824e4b Mon Sep 17 00:00:00 2001 From: ps48 Date: Thu, 25 Sep 2025 10:07:22 -0700 Subject: [PATCH 02/11] update getEventType function documentation Signed-off-by: ps48 --- _data-prepper/pipelines/get-eventtype.md | 68 ++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 3 deletions(-) diff --git a/_data-prepper/pipelines/get-eventtype.md b/_data-prepper/pipelines/get-eventtype.md index dfffa7e814d..6897ae81c51 100644 --- a/_data-prepper/pipelines/get-eventtype.md +++ b/_data-prepper/pipelines/get-eventtype.md @@ -8,13 +8,75 @@ nav_order: 45 # getEventType() -The `getEventType()` function returns the internal event type of the current event. +The `getEventType()` function returns the internal event type of the current event. This function is particularly useful when working with unified sources like the OTLP source that can receive multiple types of telemetry data. -The return value is one of the event types defined in the `EventType.java` file. For example, if the event is an OpenTelemetry trace event, the returned event type is `TRACE`. +## Syntax -Use this function to check the event type before performing conditional processing, as shown in the following example: +``` +getEventType() +``` + +## Return value + +The function returns a string representing the event type. The supported event types are: `LOG`, `TRACE`, `METRIC`, `DOCUMENT` + +## Usage + +Use this function to check the event type before performing conditional processing or routing. This is especially useful when you need to handle different types of telemetry data differently in your pipeline. + +### Basic example + +Check if an event is a trace event: ```json getEventType() == "TRACE" ``` {% include copy.html %} + +### Routing example with OTLP source + +Route different telemetry signals to different pipelines based on event type: + +```yaml +otel-telemetry-pipeline: + source: + otlp: + ssl: false + route: + - logs: 'getEventType() == "LOG"' + - traces: 'getEventType() == "TRACE"' + - metrics: 'getEventType() == "METRIC"' + sink: + - pipeline: + name: "logs-pipeline" + routes: + - "logs" + - pipeline: + name: "traces-pipeline" + routes: + - "traces" + - pipeline: + name: "metrics-pipeline" + routes: + - "metrics" +``` +{% include copy.html %} + +### Conditional processing example + +Process events differently based on their type: + +```yaml +processor: + - add_entries: + when: 'getEventType() == "LOG"' + entries: + - key: "log_processed" + value: true + - add_entries: + when: 'getEventType() == "METRIC"' + entries: + - key: "metric_type" + value: "otel" +``` +{% include copy.html %} From 92e6c18d257dbc5b3c7494dc609c6af2a7b8cd03 Mon Sep 17 00:00:00 2001 From: ps48 Date: Thu, 25 Sep 2025 10:26:36 -0700 Subject: [PATCH 03/11] Adding style updates Signed-off-by: ps48 --- .../pipelines/configuration/sources/otlp-source.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index 7a60b109529..d501f91425e 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -30,7 +30,7 @@ You can configure the `otlp` source with the following options. | `thread_count` | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. | | `max_connection_count` | Integer | The maximum allowed number of open connections. Default value is `500`. | | `max_request_length` | String | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. | -| `compression` | String | The compression type applied on the client request payload. Valid values are `none` (no compression) or `gzip` (apply GZip decompression). Default value is `none`. | +| `compression` | String | The compression type applied on the client request payload. Valid values are `none` (no compression) or `gzip` (apply `gzip` decompression). Default value is `none`. | | `output_format` | String | Specifies the decoded output format for all signals (logs, metrics, traces) if individual output format options are not set. Valid values are `otel` (OpenTelemetry format) or `opensearch` (OpenSearch format). Default value is `otel`. | | `logs_output_format` | String | Specifies the decoded output format specifically for logs. Takes precedence over `output_format` for logs. Valid values are `otel` or `opensearch`. Default value is `otel`. | | `metrics_output_format` | String | Specifies the decoded output format specifically for metrics. Takes precedence over `output_format` for metrics. Valid values are `otel` or `opensearch`. Default value is `otel`. | @@ -46,7 +46,7 @@ You can configure SSL/TLS in the `otlp` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | | `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | -| `sslKeyCertChainFile` | String | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| `sslKeyCertChainFile` | String | Represents the SSL certificate chain file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | | `sslKeyFile` | String | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | | `useAcmCertForSSL` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | | `acmCertificateArn` | String | Represents the ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `useAcmCertForSSL` is set to `true`. | @@ -95,6 +95,8 @@ source: ## Usage +The following examples demonstrate how to configure and use the OTLP source in various scenarios. + ### Basic configuration To get started with the `otlp` source, create a `pipeline.yaml` file with the following minimal configuration: From 756ea3198916c53d35d67501c36a051c33631be8 Mon Sep 17 00:00:00 2001 From: ps48 Date: Thu, 25 Sep 2025 10:29:17 -0700 Subject: [PATCH 04/11] update when condition Signed-off-by: ps48 --- _data-prepper/pipelines/get-eventtype.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/pipelines/get-eventtype.md b/_data-prepper/pipelines/get-eventtype.md index 6897ae81c51..41249d44770 100644 --- a/_data-prepper/pipelines/get-eventtype.md +++ b/_data-prepper/pipelines/get-eventtype.md @@ -69,14 +69,14 @@ Process events differently based on their type: ```yaml processor: - add_entries: - when: 'getEventType() == "LOG"' entries: - key: "log_processed" value: true + add_when: 'getEventType() == "LOG"' - add_entries: - when: 'getEventType() == "METRIC"' entries: - key: "metric_type" value: "otel" + add_when: 'getEventType() == "METRIC"' ``` {% include copy.html %} From a535376f5556b915203a4c91ec74d40c11f6cb06 Mon Sep 17 00:00:00 2001 From: ps48 Date: Thu, 25 Sep 2025 13:43:38 -0700 Subject: [PATCH 05/11] resolve comments Signed-off-by: ps48 --- .../configuration/sources/otlp-source.md | 84 ++++++++++--------- 1 file changed, 46 insertions(+), 38 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index d501f91425e..9f028ee43f1 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -23,7 +23,7 @@ You can configure the `otlp` source with the following options. | `logs_path` | String | The path for sending unframed HTTP requests for logs. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | | `metrics_path` | String | The path for sending unframed HTTP requests for metrics. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export`. | | `traces_path` | String | The path for sending unframed HTTP requests for traces. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.trace.v1.TraceService/Export`. | -| `request_timeout` | Integer | The request timeout duration in milliseconds. Default value is `10000`. | +| `request_timeout` | Duration | The request timeout duration. Default value is `10s`. | | `health_check_service` | Boolean | Enables a gRPC health check service under `grpc.health.v1.Health/Check`. When `unframed_requests` is also `true`, enables HTTP health check at `/health`. Default value is `false`. | | `proto_reflection_service` | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. | | `unframed_requests` | Boolean | Enables requests not framed using the gRPC wire protocol. Default value is `false`. | @@ -46,11 +46,12 @@ You can configure SSL/TLS in the `otlp` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | | `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | -| `sslKeyCertChainFile` | String | Represents the SSL certificate chain file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | -| `sslKeyFile` | String | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | -| `useAcmCertForSSL` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | -| `acmCertificateArn` | String | Represents the ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `useAcmCertForSSL` is set to `true`. | -| `awsRegion` | String | Represents the AWS Region used by ACM or Amazon S3. Required if `useAcmCertForSSL` is set to `true` or if `sslKeyCertChainFile` and `sslKeyFile` are Amazon S3 paths. | +| `ssl_certificate_file` | String | Represents the SSL certificate chain file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| `ssl_key_file` | String | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| `use_acm_cert_for_ssl` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| `acm_certificate_arn` | String | Represents the ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `use_acm_cert_for_ssl` is set to `true`. | +| `acm_private_key_password` | String | Represents the ACM private key password that decrypts the private key. If not provided, Data Prepper uses the private key unencrypted. | +| `aws_region` | String | Represents the AWS Region used by ACM or Amazon S3. Required if `use_acm_cert_for_ssl` is set to `true` or if `ssl_certificate_file` and `ssl_key_file` are Amazon S3 paths. | ### Authentication configuration @@ -116,57 +117,64 @@ pipeline: One of the key features of the OTLP source is the ability to route different telemetry signals (logs, metrics, traces) to different processors or sinks based on specific needs. The routing uses metadata-based routing with the `getEventType()` function. ```yaml -otel-telemetry-pipeline: +version: "2" +otel-telemetry: source: otlp: ssl: false route: - - logs: 'getEventType() == "LOG"' - traces: 'getEventType() == "TRACE"' + - logs: 'getEventType() == "LOG"' - metrics: 'getEventType() == "METRIC"' sink: - - pipeline: - name: "logs-pipeline" + - opensearch: routes: - - "logs" + - logs + hosts: [ "https://opensearch:9200" ] + index: logs-%{yyyy.MM.dd} + username: admin + password: yourStrongPassword123! + insecure: true - pipeline: - name: "traces-pipeline" + name: traces-raw routes: - - "traces" + - traces - pipeline: - name: "metrics-pipeline" + name: otel-metrics routes: - - "metrics" - -logs-pipeline: - source: - pipeline: - name: "otel-telemetry-pipeline" - processor: - # Add logs-specific processors here - sink: - - opensearch: - index: "logs-index" + - metrics -traces-pipeline: +traces-raw: source: pipeline: - name: "otel-telemetry-pipeline" + name: otel-telemetry processor: - # Add traces-specific processors here + - otel_trace_raw: sink: - opensearch: - index: "traces-index" + hosts: [ "https://opensearch:9200" ] + index_type: trace-analytics-raw + username: admin + password: yourStrongPassword123! + insecure: true -metrics-pipeline: +otel-metrics: source: pipeline: - name: "otel-telemetry-pipeline" + name: otel-telemetry processor: - # Add metrics-specific processors here + - otel_metrics: + calculate_histogram_buckets: true + calculate_exponential_histogram_buckets: true + exponential_histogram_max_allowed_scale: 10 + flatten_attributes: false sink: - opensearch: - index: "metrics-index" + hosts: [ "https://opensearch:9200" ] + index: metrics-otel-%{yyyy.MM.dd} + username: admin + password: yourStrongPassword123! + insecure: true ``` {% include copy.html %} @@ -200,8 +208,8 @@ To enable SSL/TLS with local certificates: source: otlp: ssl: true - sslKeyCertChainFile: "/path/to/certificate.crt" - sslKeyFile: "/path/to/private-key.key" + ssl_certificate_file: "/path/to/certificate.crt" + ssl_key_file: "/path/to/private-key.key" ``` {% include copy.html %} @@ -211,9 +219,9 @@ To use AWS Certificate Manager: source: otlp: ssl: true - useAcmCertForSSL: true - acmCertificateArn: "arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012" - awsRegion: "us-east-1" + use_acm_cert_for_ssl: true + acm_certificate_arn: "arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012" + aws_region: "us-east-1" ``` {% include copy.html %} From ecdafdc9bf08fbdc04af9448cb6ad994f18c04a6 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Mon, 29 Sep 2025 13:42:49 -0400 Subject: [PATCH 06/11] Doc review Signed-off-by: Fanit Kolchina --- .../configuration/sources/otlp-source.md | 106 ++++++++++-------- _data-prepper/pipelines/get-eventtype.md | 16 ++- 2 files changed, 69 insertions(+), 53 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index 9f028ee43f1..5e1a216af22 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -8,9 +8,9 @@ nav_order: 85 # OTLP source -The `otlp` source is a unified OpenTelemetry source that follows the [OpenTelemetry Protocol (OTLP) Specification](https://opentelemetry.io/docs/specs/otlp/) and can receive logs, metrics, and traces through a single endpoint. This source consolidates the functionality of the individual `otel_logs_source`, `otel_metrics_source`, and `otel_trace_source` sources, providing a streamlined approach to ingesting all OpenTelemetry telemetry signals. +The `otlp` source is a unified OpenTelemetry source that follows the [OpenTelemetry Protocol (OTLP) specification](https://opentelemetry.io/docs/specs/otlp/) and can receive logs, metrics, and traces through a single endpoint. This source consolidates the functionality of the individual `otel_logs_source`, `otel_metrics_source`, and `otel_trace_source` sources, providing a streamlined approach to ingesting all OpenTelemetry telemetry signals. -The OTLP source supports both the `OTLP/gRPC` and `OTLP/HTTP` protocols. For `OTLP/HTTP`, only Protobuf encoding is currently supported. This makes it compatible with a wide range of OpenTelemetry collectors and instrumentation libraries. +The OTLP source supports both the `OTLP/gRPC` and `OTLP/HTTP` protocols. For `OTLP/HTTP`, only Protobuf encoding is supported. This makes it compatible with a wide range of OpenTelemetry collectors and instrumentation libraries. {: .note} ## Configuration @@ -19,22 +19,23 @@ You can configure the `otlp` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | -| `port` | Integer | The port on which the OTLP source listens. Default value is `21893`. | -| `logs_path` | String | The path for sending unframed HTTP requests for logs. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | -| `metrics_path` | String | The path for sending unframed HTTP requests for metrics. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export`. | -| `traces_path` | String | The path for sending unframed HTTP requests for traces. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.trace.v1.TraceService/Export`. | -| `request_timeout` | Duration | The request timeout duration. Default value is `10s`. | -| `health_check_service` | Boolean | Enables a gRPC health check service under `grpc.health.v1.Health/Check`. When `unframed_requests` is also `true`, enables HTTP health check at `/health`. Default value is `false`. | -| `proto_reflection_service` | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. | -| `unframed_requests` | Boolean | Enables requests not framed using the gRPC wire protocol. Default value is `false`. | -| `thread_count` | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is `200`. | -| `max_connection_count` | Integer | The maximum allowed number of open connections. Default value is `500`. | -| `max_request_length` | String | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default value is `10mb`. | -| `compression` | String | The compression type applied on the client request payload. Valid values are `none` (no compression) or `gzip` (apply `gzip` decompression). Default value is `none`. | -| `output_format` | String | Specifies the decoded output format for all signals (logs, metrics, traces) if individual output format options are not set. Valid values are `otel` (OpenTelemetry format) or `opensearch` (OpenSearch format). Default value is `otel`. | -| `logs_output_format` | String | Specifies the decoded output format specifically for logs. Takes precedence over `output_format` for logs. Valid values are `otel` or `opensearch`. Default value is `otel`. | -| `metrics_output_format` | String | Specifies the decoded output format specifically for metrics. Takes precedence over `output_format` for metrics. Valid values are `otel` or `opensearch`. Default value is `otel`. | -| `traces_output_format` | String | Specifies the decoded output format specifically for traces. Takes precedence over `output_format` for traces. Valid values are `otel` or `opensearch`. Default value is `otel`. | +| `port` | Integer | The port on which the OTLP source listens. Default is `21893`. | +| `logs_path` | String | The path for sending unframed HTTP requests for logs. Must start with `/` and have a minimum length of 1. Default is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| `metrics_path` | String | The path for sending unframed HTTP requests for metrics. Must start with `/` and have a minimum length of 1. Default is `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export`. | +| `traces_path` | String | The path for sending unframed HTTP requests for traces. Must start with `/` and have a minimum length of 1. Default is `/opentelemetry.proto.collector.trace.v1.TraceService/Export`. | +| `request_timeout` | Duration | The request timeout duration. Default is `10s`. | +| `retry_info` | Object | Configures retry behavior. Supports `min_delay` (default `100ms`) and `max_delay` (default `2s`) parameters to control exponential backoff. See [Retry information](#retry-information).| +| `health_check_service` | Boolean | Enables a gRPC health check service under `grpc.health.v1.Health/Check`. When `unframed_requests` is `true`, enables HTTP health check at `/health`. Default is `false`. | +| `proto_reflection_service` | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default is `false`. | +| `unframed_requests` | Boolean | Enables requests not framed using the gRPC wire protocol. Default is `false`. | +| `thread_count` | Integer | The number of threads to keep in the scheduled thread pool. Default is `200`. | +| `max_connection_count` | Integer | The maximum allowed number of open connections. Default is `500`. | +| `max_request_length` | String | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default is `10mb`. | +| `compression` | String | The compression type applied on the client request payload. Valid values are `none` (no compression) or `gzip` (apply `gzip` decompression). Default is `none`. | +| `output_format` | String | Specifies the decoded output format for all signals (logs, metrics, and traces) if individual output format options are not set. Valid values are `otel` (OpenTelemetry format) and `opensearch` (OpenSearch format). Default is `otel`. | +| `logs_output_format` | String | Specifies the decoded output format specifically for logs. Takes precedence over `output_format` for logs. Valid values are `otel` and `opensearch`. Default is `otel`. | +| `metrics_output_format` | String | Specifies the decoded output format specifically for metrics. Takes precedence over `output_format` for metrics. Valid values are `otel` and `opensearch`. Default is `otel`. | +| `traces_output_format` | String | Specifies the decoded output format specifically for traces. Takes precedence over `output_format` for traces. Valid values are `otel` and `opensearch`. Default is `otel`. | If an individual output format (for example, `logs_output_format`) is set, it takes precedence over the generic `output_format` for that signal type. If neither is set, the default is `otel`. {: .note} @@ -45,19 +46,19 @@ You can configure SSL/TLS in the `otlp` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | -| `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file` | String | Represents the SSL certificate chain file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | -| `ssl_key_file` | String | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | -| `use_acm_cert_for_ssl` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | -| `acm_certificate_arn` | String | Represents the ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `use_acm_cert_for_ssl` is set to `true`. | -| `acm_private_key_password` | String | Represents the ACM private key password that decrypts the private key. If not provided, Data Prepper uses the private key unencrypted. | -| `aws_region` | String | Represents the AWS Region used by ACM or Amazon S3. Required if `use_acm_cert_for_ssl` is set to `true` or if `ssl_certificate_file` and `ssl_key_file` are Amazon S3 paths. | +| `ssl` | Boolean | Enables TLS/SSL. Default is `true`. | +| `ssl_certificate_file` | String | The SSL certificate chain file path or Amazon S3 path (for example, `s3:///`). Required if `ssl` is set to `true`. | +| `ssl_key_file` | String | The SSL key file path or Amazon S3 path (for example, `s3:///`). Required if `ssl` is set to `true`. | +| `use_acm_cert_for_ssl` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | +| `acm_certificate_arn` | String | The ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `use_acm_cert_for_ssl` is set to `true`. | +| `acm_private_key_password` | String | The ACM private key password that decrypts the private key. If not provided, Data Prepper uses the private key unencrypted. | +| `aws_region` | String | The AWS Region used by ACM or Amazon S3. Required if `use_acm_cert_for_ssl` is set to `true` or if `ssl_certificate_file` and `ssl_key_file` are Amazon S3 paths. | ### Authentication configuration By default, the OTLP source runs without authentication. You can configure authentication using the following options. -You can explicitly disable authentication with: +To explicitly disable authentication, specify the following settings: ```yaml source: @@ -67,7 +68,7 @@ source: ``` {% include copy.html %} -To enable HTTP Basic authentication: +To enable HTTP Basic authentication, specify the following settings: ```yaml source: @@ -83,7 +84,7 @@ This plugin uses pluggable authentication for gRPC servers. To provide custom au ### Retry information -Data Prepper replies with a `RetryInfo` object specifying how long to wait for the next request when backpressure builds up. The retry information implements exponential backoff with a configurable maximum delay. +You can set retry behavior using the `retry_info` setting, specifying how long to wait for the next request when backpressure occurs. The retry mechanism applies exponential backoff with a configurable maximum delay: ```yaml source: @@ -114,7 +115,7 @@ pipeline: ### Routing telemetry signals -One of the key features of the OTLP source is the ability to route different telemetry signals (logs, metrics, traces) to different processors or sinks based on specific needs. The routing uses metadata-based routing with the `getEventType()` function. +One of the key features of the OTLP source is the ability to route different telemetry signals (logs, metrics, and traces) to different processors or sinks based on your specific needs. Routing is determined by metadata using the getEventType() function: ```yaml version: "2" @@ -180,7 +181,7 @@ otel-metrics: ### Using OpenSearch output format -To generate data in the OpenSearch format for all telemetry signals: +To generate data in the OpenSearch format for all telemetry signals, specify the following settings: ```yaml source: @@ -189,7 +190,7 @@ source: ``` {% include copy.html %} -To use different output formats for different signal types: +To use different output formats for different signal types, specify the following settings: ```yaml source: @@ -202,7 +203,7 @@ source: ### Configuring with SSL/TLS -To enable SSL/TLS with local certificates: +To enable SSL/TLS with local certificates, specify the following settings: ```yaml source: @@ -213,7 +214,7 @@ source: ``` {% include copy.html %} -To use AWS Certificate Manager: +To use the AWS Certificate Manager, specify the following settings: ```yaml source: @@ -231,33 +232,44 @@ The `otlp` source includes the following metrics to monitor its performance and ### Counters +The following counters track request activity and errors in the OTLP source. + | Metric | Description | | :--- | :--- | -| `requestTimeouts` | Measures the total number of requests that time out. | -| `requestsReceived` | Measures the total number of requests received by the OTLP source. | -| `successRequests` | Measures the total number of requests successfully processed by the OTLP source. | -| `badRequests` | Measures the total number of requests with invalid format processed by the OTLP source. | -| `requestsTooLarge` | Measures the total number of requests that exceed the maximum allowed size. | -| `internalServerError` | Measures the total number of requests processed by the OTLP source with custom exception types. | +| `requestTimeouts` | The total number of requests that timed out. | +| `requestsReceived` | The total number of requests received by the OTLP source. | +| `successRequests` | The total number of requests successfully processed by the OTLP source. | +| `badRequests` | The total number of requests with invalid format processed by the OTLP source. | +| `requestsTooLarge` | The total number of requests that exceed the maximum allowed size. | +| `internalServerError` | The total number of requests processed by the OTLP source with custom exception types. | ### Timers +The following timers track request activity and errors in the OTLP source. + | Metric | Description | | :--- | :--- | -| `requestProcessDuration` | Measures the latency of requests processed by the OTLP source in seconds. | +| `requestProcessDuration` | The latency of requests processed by the OTLP source, in seconds. | ### Distribution summaries +The following distribution summaries track request activity and errors in the OTLP source. + | Metric | Description | | :--- | :--- | -| `payloadSize` | Measures the distribution of incoming request payload sizes in bytes. | +| `payloadSize` | The distribution of incoming request payload sizes, in bytes. | + +## Migrating from individual OTel sources -## Migration from individual OTel sources +If you're using separate `otel_logs_source`, `otel_metrics_source`, or `otel_trace_source` sources, you can migrate to the unified OTLP source by following these steps: -If you're currently using separate `otel_logs_source`, `otel_metrics_source`, or `otel_trace_source` sources, you can migrate to the unified OTLP source by: +1. Replace all three sources with a single `otlp` source. +2. Use [routing configuration](#routing-telemetry-signals) to direct different signal types to their appropriate pipelines. +3. Change the port numbers if needed (the OTLP source uses port `21893` by default). -1. Replacing all three sources with a single `otlp` source -2. Using the routing configuration shown above to direct different signal types to their appropriate pipelines -3. Adjusting the port numbers if needed (the OTLP source uses port `21893` by default) +## Related articles -The OTLP source provides all the functionality of the individual sources while simplifying configuration and reducing resource usage. \ No newline at end of file +- [OTel logs source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-logs-source/) +- [OTel metrics source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-metrics-source/) +- [OTel trace source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-trace-source/) +- [getEventType()]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/get-eventtype/) \ No newline at end of file diff --git a/_data-prepper/pipelines/get-eventtype.md b/_data-prepper/pipelines/get-eventtype.md index 41249d44770..add405379d4 100644 --- a/_data-prepper/pipelines/get-eventtype.md +++ b/_data-prepper/pipelines/get-eventtype.md @@ -3,22 +3,22 @@ layout: default title: getEventType() parent: Functions grand_parent: Pipelines -nav_order: 45 +nav_order: 12 --- # getEventType() -The `getEventType()` function returns the internal event type of the current event. This function is particularly useful when working with unified sources like the OTLP source that can receive multiple types of telemetry data. +The `getEventType()` function returns the internal event type of the current event. This function is particularly useful when working with unified sources that can receive multiple types of telemetry data, such as the [OTLP source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otlp-source/). ## Syntax -``` +```java getEventType() ``` ## Return value -The function returns a string representing the event type. The supported event types are: `LOG`, `TRACE`, `METRIC`, `DOCUMENT` +The function returns a string representing the event type. The supported event types are: `LOG`, `TRACE`, `METRIC`, and `DOCUMENT`. ## Usage @@ -35,7 +35,7 @@ getEventType() == "TRACE" ### Routing example with OTLP source -Route different telemetry signals to different pipelines based on event type: +To route different telemetry signals to different pipelines based on event type, use the `getEventType()` function to determine each event's type and route the different event types to different pipelines: ```yaml otel-telemetry-pipeline: @@ -64,7 +64,7 @@ otel-telemetry-pipeline: ### Conditional processing example -Process events differently based on their type: +To process events differently based on their type, use the `add_when` expression to conditionally add fields to each event: ```yaml processor: @@ -80,3 +80,7 @@ processor: add_when: 'getEventType() == "METRIC"' ``` {% include copy.html %} + +## Related articles + +- [OTLP source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otlp-source/) \ No newline at end of file From e3b777e06f27433a700503eebb0a1588bdbc148f Mon Sep 17 00:00:00 2001 From: ps48 Date: Tue, 30 Sep 2025 11:45:41 -0700 Subject: [PATCH 07/11] update migration section with example Signed-off-by: ps48 --- .../configuration/sources/otlp-source.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index 5e1a216af22..d46167b8ee8 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -267,6 +267,65 @@ If you're using separate `otel_logs_source`, `otel_metrics_source`, or `otel_tra 2. Use [routing configuration](#routing-telemetry-signals) to direct different signal types to their appropriate pipelines. 3. Change the port numbers if needed (the OTLP source uses port `21893` by default). +### Before: Using individual OTel sources + +#### Pipeline using OTel logs source +```yaml +logs-pipeline: + source: + otel_logs_source: + port: 21892 + sink: + - opensearch: + index: logs +``` +#### Pipeline using OTel metrics source +```yaml +metrics-pipeline: + source: + otel_metrics_source: + port: 21891 + sink: + - opensearch: + index: metrics +``` +#### Pipeline using OTel trace source +```yaml +traces-pipeline: + source: + otel_trace_source: + port: 21890 + sink: + - opensearch: + index: traces +``` + +### After: Using unified OTLP source + +```yaml +otlp-pipeline: + source: + otlp: + port: 21893 + route: + - logs: 'getEventType() == "LOG"' + - metrics: 'getEventType() == "METRIC"' + - traces: 'getEventType() == "TRACE"' + sink: + - opensearch: + routes: + - logs + index: logs + - opensearch: + routes: + - metrics + index: metrics + - opensearch: + routes: + - traces + index: traces +``` + ## Related articles - [OTel logs source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-logs-source/) From 213aca26a23e9794d5d59f41bab00c9dfa9c0c0f Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Wed, 1 Oct 2025 11:27:11 -0400 Subject: [PATCH 08/11] Apply suggestions from code review Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- .../pipelines/configuration/sources/otlp-source.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index d46167b8ee8..988f27b5afc 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -267,9 +267,12 @@ If you're using separate `otel_logs_source`, `otel_metrics_source`, or `otel_tra 2. Use [routing configuration](#routing-telemetry-signals) to direct different signal types to their appropriate pipelines. 3. Change the port numbers if needed (the OTLP source uses port `21893` by default). -### Before: Using individual OTel sources +### Migration example + +The following example demonstrates how to consolidate separate OTel logs, metrics, and traces sources into a single OTLP source with routing. + +Consider a setup where logs, metrics, and traces are configured separately: -#### Pipeline using OTel logs source ```yaml logs-pipeline: source: @@ -279,7 +282,6 @@ logs-pipeline: - opensearch: index: logs ``` -#### Pipeline using OTel metrics source ```yaml metrics-pipeline: source: @@ -289,7 +291,6 @@ metrics-pipeline: - opensearch: index: metrics ``` -#### Pipeline using OTel trace source ```yaml traces-pipeline: source: @@ -300,7 +301,7 @@ traces-pipeline: index: traces ``` -### After: Using unified OTLP source +You can consolidate logs, metrics, and traces into a single OTLP source as follows: ```yaml otlp-pipeline: From 1018e02007c8305dbe087d46097d3cdcd1e5a7f1 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Wed, 1 Oct 2025 11:27:45 -0400 Subject: [PATCH 09/11] Update _data-prepper/pipelines/configuration/sources/otlp-source.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _data-prepper/pipelines/configuration/sources/otlp-source.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index 988f27b5afc..cd4833be452 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -326,6 +326,7 @@ otlp-pipeline: - traces index: traces ``` +{% include copy.html %} ## Related articles From 87cfc93e676713ddde2df8eafee6230b0cdd496b Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Wed, 1 Oct 2025 11:28:24 -0400 Subject: [PATCH 10/11] Update _data-prepper/pipelines/configuration/sources/otlp-source.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _data-prepper/pipelines/configuration/sources/otlp-source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index cd4833be452..13afee90cf1 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -269,7 +269,7 @@ If you're using separate `otel_logs_source`, `otel_metrics_source`, or `otel_tra ### Migration example -The following example demonstrates how to consolidate separate OTel logs, metrics, and traces sources into a single OTLP source with routing. +The following example demonstrates how to consolidate separate OTel logs, metrics, and traces sources into a single OTLP source. Consider a setup where logs, metrics, and traces are configured separately: From 2f57dc063202d48f595b73cda8395116225b6c65 Mon Sep 17 00:00:00 2001 From: Nathan Bower Date: Wed, 1 Oct 2025 12:16:34 -0400 Subject: [PATCH 11/11] Apply suggestions from code review Signed-off-by: Nathan Bower --- .../configuration/sources/otlp-source.md | 52 +++++++++---------- _data-prepper/pipelines/get-eventtype.md | 6 +-- 2 files changed, 29 insertions(+), 29 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otlp-source.md b/_data-prepper/pipelines/configuration/sources/otlp-source.md index 13afee90cf1..f38a63307e1 100644 --- a/_data-prepper/pipelines/configuration/sources/otlp-source.md +++ b/_data-prepper/pipelines/configuration/sources/otlp-source.md @@ -10,7 +10,7 @@ nav_order: 85 The `otlp` source is a unified OpenTelemetry source that follows the [OpenTelemetry Protocol (OTLP) specification](https://opentelemetry.io/docs/specs/otlp/) and can receive logs, metrics, and traces through a single endpoint. This source consolidates the functionality of the individual `otel_logs_source`, `otel_metrics_source`, and `otel_trace_source` sources, providing a streamlined approach to ingesting all OpenTelemetry telemetry signals. -The OTLP source supports both the `OTLP/gRPC` and `OTLP/HTTP` protocols. For `OTLP/HTTP`, only Protobuf encoding is supported. This makes it compatible with a wide range of OpenTelemetry collectors and instrumentation libraries. +The `otlp` source supports both the `OTLP/gRPC` and `OTLP/HTTP` protocols. For `OTLP/HTTP`, only Protobuf encoding is supported. This makes it compatible with a wide range of OpenTelemetry collectors and instrumentation libraries. {: .note} ## Configuration @@ -19,7 +19,7 @@ You can configure the `otlp` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | -| `port` | Integer | The port on which the OTLP source listens. Default is `21893`. | +| `port` | Integer | The port on which the `otlp` source listens. Default is `21893`. | | `logs_path` | String | The path for sending unframed HTTP requests for logs. Must start with `/` and have a minimum length of 1. Default is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | | `metrics_path` | String | The path for sending unframed HTTP requests for metrics. Must start with `/` and have a minimum length of 1. Default is `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export`. | | `traces_path` | String | The path for sending unframed HTTP requests for traces. Must start with `/` and have a minimum length of 1. Default is `/opentelemetry.proto.collector.trace.v1.TraceService/Export`. | @@ -31,7 +31,7 @@ You can configure the `otlp` source with the following options. | `thread_count` | Integer | The number of threads to keep in the scheduled thread pool. Default is `200`. | | `max_connection_count` | Integer | The maximum allowed number of open connections. Default is `500`. | | `max_request_length` | String | The maximum number of bytes allowed in the payload of a single gRPC or HTTP request. Default is `10mb`. | -| `compression` | String | The compression type applied on the client request payload. Valid values are `none` (no compression) or `gzip` (apply `gzip` decompression). Default is `none`. | +| `compression` | String | The compression type applied to the client request payload. Valid values are `none` (no compression) or `gzip` (apply `gzip` decompression). Default is `none`. | | `output_format` | String | Specifies the decoded output format for all signals (logs, metrics, and traces) if individual output format options are not set. Valid values are `otel` (OpenTelemetry format) and `opensearch` (OpenSearch format). Default is `otel`. | | `logs_output_format` | String | Specifies the decoded output format specifically for logs. Takes precedence over `output_format` for logs. Valid values are `otel` and `opensearch`. Default is `otel`. | | `metrics_output_format` | String | Specifies the decoded output format specifically for metrics. Takes precedence over `output_format` for metrics. Valid values are `otel` and `opensearch`. Default is `otel`. | @@ -46,17 +46,17 @@ You can configure SSL/TLS in the `otlp` source with the following options. | Option | Type | Description | | :--- | :--- | :--- | -| `ssl` | Boolean | Enables TLS/SSL. Default is `true`. | -| `ssl_certificate_file` | String | The SSL certificate chain file path or Amazon S3 path (for example, `s3:///`). Required if `ssl` is set to `true`. | +| `ssl` | Boolean | Enables SSL/TLS. Default is `true`. | +| `ssl_certificate_file` | String | The SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path (for example, `s3:///`). Required if `ssl` is set to `true`. | | `ssl_key_file` | String | The SSL key file path or Amazon S3 path (for example, `s3:///`). Required if `ssl` is set to `true`. | -| `use_acm_cert_for_ssl` | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | +| `use_acm_cert_for_ssl` | Boolean | Enables SSL/TLS using a certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | | `acm_certificate_arn` | String | The ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `use_acm_cert_for_ssl` is set to `true`. | -| `acm_private_key_password` | String | The ACM private key password that decrypts the private key. If not provided, Data Prepper uses the private key unencrypted. | +| `acm_private_key_password` | String | The ACM private key password that decrypts the private key. If not provided, OpenSearch Data Prepper uses the private key unencrypted. | | `aws_region` | String | The AWS Region used by ACM or Amazon S3. Required if `use_acm_cert_for_ssl` is set to `true` or if `ssl_certificate_file` and `ssl_key_file` are Amazon S3 paths. | ### Authentication configuration -By default, the OTLP source runs without authentication. You can configure authentication using the following options. +By default, the `otlp` source runs without authentication. You can configure authentication using the following options. To explicitly disable authentication, specify the following settings: @@ -97,7 +97,7 @@ source: ## Usage -The following examples demonstrate how to configure and use the OTLP source in various scenarios. +The following examples demonstrate how to configure and use the `otlp` source in various scenarios. ### Basic configuration @@ -115,7 +115,7 @@ pipeline: ### Routing telemetry signals -One of the key features of the OTLP source is the ability to route different telemetry signals (logs, metrics, and traces) to different processors or sinks based on your specific needs. Routing is determined by metadata using the getEventType() function: +One of the key features of the `otlp` source is its ability to route different telemetry signals (logs, metrics, and traces) to different processors or sinks based on your specific needs. Routing is determined by metadata using the `getEventType()` function: ```yaml version: "2" @@ -214,7 +214,7 @@ source: ``` {% include copy.html %} -To use the AWS Certificate Manager, specify the following settings: +To use the ACM, specify the following settings: ```yaml source: @@ -228,48 +228,48 @@ source: ## Metrics -The `otlp` source includes the following metrics to monitor its performance and health. +The `otlp` source includes the following metrics for monitoring its performance and health. ### Counters -The following counters track request activity and errors in the OTLP source. +The following counters track request activity and errors in the `otlp` source. | Metric | Description | | :--- | :--- | | `requestTimeouts` | The total number of requests that timed out. | -| `requestsReceived` | The total number of requests received by the OTLP source. | -| `successRequests` | The total number of requests successfully processed by the OTLP source. | -| `badRequests` | The total number of requests with invalid format processed by the OTLP source. | +| `requestsReceived` | The total number of requests received by the `otlp` source. | +| `successRequests` | The total number of requests successfully processed by the `otlp` source. | +| `badRequests` | The total number of requests with an invalid format processed by the `otlp` source. | | `requestsTooLarge` | The total number of requests that exceed the maximum allowed size. | -| `internalServerError` | The total number of requests processed by the OTLP source with custom exception types. | +| `internalServerError` | The total number of requests processed by the `otlp` source with custom exception types. | ### Timers -The following timers track request activity and errors in the OTLP source. +The following timers track request activity and errors in the `otlp` source. | Metric | Description | | :--- | :--- | -| `requestProcessDuration` | The latency of requests processed by the OTLP source, in seconds. | +| `requestProcessDuration` | The latency of requests processed by the `otlp` source, in seconds. | ### Distribution summaries -The following distribution summaries track request activity and errors in the OTLP source. +The following distribution summaries track request activity and errors in the `otlp` source. | Metric | Description | | :--- | :--- | | `payloadSize` | The distribution of incoming request payload sizes, in bytes. | -## Migrating from individual OTel sources +## Migrating from individual OpenTelemetry sources -If you're using separate `otel_logs_source`, `otel_metrics_source`, or `otel_trace_source` sources, you can migrate to the unified OTLP source by following these steps: +If you're using separate `otel_logs_source`, `otel_metrics_source`, or `otel_trace_source` sources, you can migrate to the unified `otlp` source by following these steps: 1. Replace all three sources with a single `otlp` source. 2. Use [routing configuration](#routing-telemetry-signals) to direct different signal types to their appropriate pipelines. -3. Change the port numbers if needed (the OTLP source uses port `21893` by default). +3. Change the port numbers if needed (the `otlp` source uses port `21893` by default). ### Migration example -The following example demonstrates how to consolidate separate OTel logs, metrics, and traces sources into a single OTLP source. +The following example demonstrates how to consolidate separate OpenTelemetry log, metric, and trace sources into a single `otlp` source. Consider a setup where logs, metrics, and traces are configured separately: @@ -301,7 +301,7 @@ traces-pipeline: index: traces ``` -You can consolidate logs, metrics, and traces into a single OTLP source as follows: +You can consolidate logs, metrics, and traces into a single `otlp` source as follows: ```yaml otlp-pipeline: @@ -328,7 +328,7 @@ otlp-pipeline: ``` {% include copy.html %} -## Related articles +## Related pages - [OTel logs source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-logs-source/) - [OTel metrics source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otel-metrics-source/) diff --git a/_data-prepper/pipelines/get-eventtype.md b/_data-prepper/pipelines/get-eventtype.md index add405379d4..f0a9fbfdb55 100644 --- a/_data-prepper/pipelines/get-eventtype.md +++ b/_data-prepper/pipelines/get-eventtype.md @@ -18,7 +18,7 @@ getEventType() ## Return value -The function returns a string representing the event type. The supported event types are: `LOG`, `TRACE`, `METRIC`, and `DOCUMENT`. +The function returns a string representing the event type. The supported event types are `LOG`, `TRACE`, `METRIC`, and `DOCUMENT`. ## Usage @@ -26,7 +26,7 @@ Use this function to check the event type before performing conditional processi ### Basic example -Check if an event is a trace event: +Check whether an event is a trace event: ```json getEventType() == "TRACE" @@ -81,6 +81,6 @@ processor: ``` {% include copy.html %} -## Related articles +## Related pages - [OTLP source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/otlp-source/) \ No newline at end of file