Add documentation for unified OTLP source in Data prepper #11033

ps48 · 2025-09-25T17:00:28Z

Description

Add documentation for unified OTLP source in Data prepper

Issues Resolved

Closes #11029

Version

Data prepper 2.12

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: ps48 <[email protected]>

github-actions · 2025-09-25T17:00:40Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Signed-off-by: ps48 <[email protected]>

ps48 · 2025-09-25T17:39:26Z

Requested @dlvenable @shenkw1 for tech review

dlvenable

Thank you @ps48 for this contribution!

dlvenable · 2025-09-25T19:40:44Z

_data-prepper/pipelines/configuration/sources/otlp-source.md

+| Option | Type | Description |
+| :--- | :--- | :--- |
+| `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. |
+| `sslKeyCertChainFile` | String | Represents the SSL certificate chain file path or Amazon S3 path. For example, see the Amazon S3 path `s3://<bucketName>/<path>`. Required if `ssl` is set to `true`. |


We are using underscore for spaces in this plugin.

https://github.com/opensearch-project/data-prepper/blob/118c30320ca3e3b8c2d5096420089650472885aa/data-prepper-plugins/otlp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSourceConfig.java#L37-L42

Please update the configuration names accordingly.

I think the readme is out of date, right now updated the docs PR a535376. Will update readme separately in Data Prepper repo.

dlvenable · 2025-09-25T19:41:34Z

_data-prepper/pipelines/configuration/sources/otlp-source.md

+| `logs_path` | String | The path for sending unframed HTTP requests for logs. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. |
+| `metrics_path` | String | The path for sending unframed HTTP requests for metrics. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export`. |
+| `traces_path` | String | The path for sending unframed HTTP requests for traces. Must start with `/` and have a minimum length of 1. Default value is `/opentelemetry.proto.collector.trace.v1.TraceService/Export`. |
+| `request_timeout` | Integer | The request timeout duration in milliseconds. Default value is `10000`. |


This is a Duration type, not an integer. The default value is 10s.

yes updated it here: a535376

dlvenable · 2025-09-25T19:43:04Z

_data-prepper/pipelines/configuration/sources/otlp-source.md

+source:
+  otlp:
+    ssl: true
+    useAcmCertForSSL: true


Please correct to use underscores.

updated:a535376

dlvenable · 2025-09-25T19:45:19Z

_data-prepper/pipelines/configuration/sources/otlp-source.md

+One of the key features of the OTLP source is the ability to route different telemetry signals (logs, metrics, traces) to different processors or sinks based on specific needs. The routing uses metadata-based routing with the `getEventType()` function.
+
+```yaml
+otel-telemetry-pipeline:


I'd like to recommend using this as our default configuration to share. It reduces some of the pipelines and works for sending traces, metrics, and logs to OpenSearch.

https://github.com/dlvenable/data-prepper-samples/blob/a8e7b4350e0ac88b829869131ac736dbff6d3910/samples/otlp-pipeline/data-prepper/pipelines/pipeline.yaml

oh yes this is way simper, updated: a535376

dlvenable · 2025-09-25T19:46:19Z

_data-prepper/pipelines/configuration/sources/otlp-source.md

+source:
+  otlp:
+    ssl: true
+    sslKeyCertChainFile: "/path/to/certificate.crt"


Please correct to use underscores.

updated: a535376

Signed-off-by: ps48 <[email protected]>

dlvenable

Thanks @ps48 !

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws · 2025-09-29T17:45:10Z

Thank you, @ps48! Doc review complete. It would be very helpful to provide an example in the migration section. Could you add one that has existing configuration using logs, metrics, and traces in different sources and a new configuration using the otlp source?

Signed-off-by: ps48 <[email protected]>

ps48 · 2025-09-30T18:46:45Z

Hi @kolchfa-aws, I've the migration section with an example e3b777e

_data-prepper/pipelines/configuration/sources/otlp-source.md

Signed-off-by: kolchfa-aws <[email protected]>

_data-prepper/pipelines/configuration/sources/otlp-source.md

Signed-off-by: kolchfa-aws <[email protected]>

_data-prepper/pipelines/configuration/sources/otlp-source.md

Signed-off-by: kolchfa-aws <[email protected]>

natebower

Editorial review

_data-prepper/pipelines/configuration/sources/otlp-source.md

_data-prepper/pipelines/get-eventtype.md

Signed-off-by: Nathan Bower <[email protected]>

natebower

LGTM

* add documentation for unified OTLP source Signed-off-by: ps48 <[email protected]> * update getEventType function documentation Signed-off-by: ps48 <[email protected]> * Adding style updates Signed-off-by: ps48 <[email protected]> * update when condition Signed-off-by: ps48 <[email protected]> * resolve comments Signed-off-by: ps48 <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * update migration section with example Signed-off-by: ps48 <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Update _data-prepper/pipelines/configuration/sources/otlp-source.md Signed-off-by: kolchfa-aws <[email protected]> * Update _data-prepper/pipelines/configuration/sources/otlp-source.md Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Signed-off-by: Nathan Bower <[email protected]> --------- Signed-off-by: ps48 <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Nathan Bower <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit e493b73) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…11154)

add documentation for unified OTLP source

0a790ff

Signed-off-by: ps48 <[email protected]>

ps48 requested review from AMoo-Miki, dlvenable, epugh, kolchfa-aws, natebower and sumobrian as code owners September 25, 2025 17:00

github-actions bot assigned kolchfa-aws Sep 25, 2025

ps48 added 3 commits September 25, 2025 10:07

update getEventType function documentation

238d4ed

Signed-off-by: ps48 <[email protected]>

Adding style updates

92e6c18

Signed-off-by: ps48 <[email protected]>

update when condition

756ea31

Signed-off-by: ps48 <[email protected]>

dlvenable reviewed Sep 25, 2025

View reviewed changes

resolve comments

a535376

Signed-off-by: ps48 <[email protected]>

dlvenable approved these changes Sep 26, 2025

View reviewed changes

kolchfa-aws added Doc review PR: Doc review in progress backport 3.2 labels Sep 26, 2025

shenkw1 approved these changes Sep 27, 2025

View reviewed changes

Doc review

ecdafdc

Signed-off-by: Fanit Kolchina <[email protected]>

update migration section with example

e3b777e

Signed-off-by: ps48 <[email protected]>

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Outdated Show resolved Hide resolved

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Outdated Show resolved Hide resolved

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Outdated Show resolved Hide resolved

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Outdated Show resolved Hide resolved

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Outdated Show resolved Hide resolved

Apply suggestions from code review

213aca2

Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Show resolved Hide resolved

Update _data-prepper/pipelines/configuration/sources/otlp-source.md

1018e02

Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws reviewed Oct 1, 2025

View reviewed changes

_data-prepper/pipelines/configuration/sources/otlp-source.md Outdated Show resolved Hide resolved

Update _data-prepper/pipelines/configuration/sources/otlp-source.md

87cfc93

Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws added Editorial review PR: Editorial review in progress and removed Doc review PR: Doc review in progress labels Oct 1, 2025

kolchfa-aws assigned natebower Oct 1, 2025

natebower reviewed Oct 1, 2025

View reviewed changes

Apply suggestions from code review

2f57dc0

Signed-off-by: Nathan Bower <[email protected]>

natebower approved these changes Oct 1, 2025

View reviewed changes

natebower removed the Editorial review PR: Editorial review in progress label Oct 1, 2025

natebower merged commit e493b73 into opensearch-project:main Oct 1, 2025
6 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 1, 2025

[Backport 3.2] Add documentation for unified OTLP source in Data prepper #11154

Merged

github-actions bot pushed a commit that referenced this pull request Oct 1, 2025

Add documentation for unified OTLP source in Data prepper (#11033) (#…

d5f4852

…11154)

Add documentation for unified OTLP source in Data prepper #11033

Add documentation for unified OTLP source in Data prepper #11033

Uh oh!

Conversation

ps48 commented Sep 25, 2025

Description

Issues Resolved

Version

Checklist

Uh oh!

github-actions bot commented Sep 25, 2025

Uh oh!

ps48 commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented Sep 29, 2025

Uh oh!

ps48 commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ps48 commented Sep 25, 2025 •

edited

Loading