[DOCS-11765] Add OP Splunk HEC Distribution of OTel and DDOT #32704

maycmlee · 2025-11-07T21:50:52Z

What does this PR do? What is the motivation?

Merge instructions

Merge readiness:

Ready for merge

For Datadog employees:

Your branch name MUST follow the <name>/<description> convention and include the forward slash (/). Without this format, your pull request will not pass CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.

If your branch doesn't follow this format, rename it or create a new branch and PR.

[6/5/2025] Merge queue has been disabled on the documentation repo. If you have write access to the repo, the PR has been reviewed by a Documentation team member, and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #documentation channel in Slack.

Additional notes

github-actions · 2025-11-07T21:54:57Z

Preview links (active after the `build_preview` check completes)

Modified Files

mikhacim · 2025-11-10T16:06:42Z

content/en/observability_pipelines/sources/opentelemetry.md

+
+To send logs from DDOT:
+1. Deploy the DDOT Agent using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions.
+1. [Set up a pipeline][6] on Observabiity Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui).


*Observability

mikhacim · 2025-11-10T16:10:55Z

content/en/observability_pipelines/sources/opentelemetry.md


+## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines
+
+To send logs from DDOT:


To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT):

mikhacim · 2025-11-10T16:11:38Z

content/en/observability_pipelines/sources/opentelemetry.md

+## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines
+
+To send logs from DDOT:
+1. Deploy the DDOT Agent using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions.


Change from:
"Deploy the DDOT Agent"
To:
"Deploy the DDOT Collector"

content/en/observability_pipelines/sources/opentelemetry.md

mikhacim · 2025-11-10T16:21:51Z

content/en/observability_pipelines/sources/splunk_hec.md


 {{% observability_pipelines/log_source_configuration/splunk_hec %}}

+## Send logs from the Splunk Distributor of the OpenTelemetry collector to Observability Pipelines


Capitalize "Collector" here. Also, change "Distributor" to "Distribution"

mikhacim · 2025-11-10T16:22:49Z

content/en/observability_pipelines/sources/splunk_hec.md


+## Send logs from the Splunk Distributor of the OpenTelemetry collector to Observability Pipelines
+
+To send logs from the Splunk Distributor of the OpenTelemetry Collector:


change "Distributor" to "Distribution"

mikhacim · 2025-11-10T16:24:07Z

content/en/observability_pipelines/sources/datadog_agent.md


 Use Observability Pipelines' Datadog Agent source to receive logs from the Datadog Agent. Select and set up this source when you [set up a pipeline][1].

+**Note**: If you are using the Datadog Distribution of OpenTelemetry Collector, you must [use the OpenTelemetry source to send logs to Observability Pipelines][4].


add (DDOT) after "using the Datadog Distribution of OpenTelemetry Collector"

content/en/observability_pipelines/sources/datadog_agent.md

content/en/observability_pipelines/sources/opentelemetry.md

content/en/observability_pipelines/sources/splunk_hec.md

ckelner

Left a few comments, I think there's some things to address.

content/en/observability_pipelines/sources/opentelemetry.md

ckelner · 2025-11-10T21:46:34Z

content/en/observability_pipelines/sources/splunk_hec.md

+1. Install the Splunk OpenTelemetry Collector based on the your environment:
+    - [Kubernetes][2]
+    - [Linux][3]
+2. Configure the Splunk OpenTelemetry Collector:


I think we should swap steps two and three as seen here: https://github.com/DataDog/logs-psa-private/tree/main/POCs-Opps-hacks/splunk-otel-op#install-and-setup-op -- because you will need the IP Address / Load Balancer URL of OP before you can configure your Splunk OTel Collector.

Swapped the steps and added a note about firewalls.

ckelner · 2025-11-10T21:48:31Z

content/en/observability_pipelines/sources/splunk_hec.md

+    # Splunk HEC endpoint URL, if forwarding to Splunk Observability Cloud
+    # SPLUNK_HEC_URL=https://ingest.us0.signalfx.com/v1/log
+    # If you're forwarding to a Splunk Enterprise instance running on example.com, with HEC at port 8088:
+    SPLUNK_HEC_URL=http://0.0.0.0:8088/services/collector


We should replace this with placeholder text. 0.0.0.0 will never work here. It should be the actual addressable IP address or Load balancer URL for OP(s). So something like where we've done this elsewhere in the docs like <IP_ADDRESS_OR_LOAD_BALANCER_URL_FOR_OP> -- though that's a bit of a mouthful.

Thanks, I updated it to the <OPW_HOST> placeholder and used the explanation we use in the other parts of the doc. Let me know if that correct.

content/en/observability_pipelines/sources/opentelemetry.md

ckelner

More comments! Sorry!

ckelner · 2025-11-11T17:25:21Z

content/en/observability_pipelines/sources/opentelemetry.md

+    1. When you install the Worker, for the OpenTelemetry source environment variables:
+        1. Set your HTTP listener to `0.0.0.0:4318`.
+        1. Set your gRPC listener to `0.0.0.0:4317`.
+    1. After you installed the Worker and deployed the pipeline, update the OpenTelemetry Collector's `collector-config.yaml` to include an exporter that sends logs to Observability Pipelines. For example:


I'm not sure how to properly call this file is or if it is commonly named that. @krlv do you have any suggestion?

From this doc, it seems like it'd just be called config.yaml? https://opentelemetry.io/docs/collector/configuration/#location

Here: https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset?tab=helm#configure-the-opentelemetry-collector -- we call it otel-config.yaml 🤣 -- maybe we use that? (Should we also maybe link to that section of docs so they know what we are talking about?)

Works for me :D I updated and linked to that doc. Is that what you are thinking?
Also updated the help upgrade command with otel-config.yaml.

content/en/observability_pipelines/sources/opentelemetry.md

ckelner · 2025-11-11T17:27:00Z

content/en/observability_pipelines/sources/opentelemetry.md

+
+**Notes**:
+- These settings are used when setting up the Datadog Agent source in Observability Pipelines:
+    - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED:true`


We can probably just leave it DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED and DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL without the values I think, WDYT?

Works for me!

ckelner · 2025-11-11T17:28:50Z

content/en/observability_pipelines/sources/opentelemetry.md

+- These settings are used when setting up the Datadog Agent source in Observability Pipelines:
+    - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED:true`
+    - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL:"http://opw-observability-pipelines-worker.default.svc.cluster.local:4317/v1/logs"`
+    <br><br>These settings do **not** work when setting up DDOT with the OpenTelemetry source, because the OpenTelemetry Collector's `collector-config.yaml` must be configured with those settings using the exporter.


The real explanation here is that we aren't using the logs agent here at all, we're using the bundled OTel collector aka DDOT, so standard logs isn't carrying any payload so these environment variables won't affect how DDOT is working. I don't know if you want to try to wordsmith that. Maybe @krlv can phrase/explain it better than I can.

Thanks! I updated it a bit, is that correct? I said Datadog Agent, instead of the log agent, does that work?

ckelner · 2025-11-11T19:04:51Z

content/en/observability_pipelines/sources/opentelemetry.md

+            otlphttp:
+                endpoint: http://opw-observability-pipelines-worker.default.svc.cluster.local:4318
+        ...
+        service: pipelines: logs: exporters: [otlphttp]


@maycmlee I might have misled you but I would put each of these on a new line and indent them yaml style

Oh lol, I was wondering what that was. I fixed it.

ckelner

LGTM! Thanks for going a few rounds with me 💪 🎉

add info

d150620

maycmlee requested a review from a team as a code owner November 7, 2025 21:50

maycmlee added the WORK IN PROGRESS No review needed, it's a wip ;) label Nov 7, 2025

maycmlee changed the title ~~[DOCS-11765] Add OP Splunk HEC Dist of OTel and DDOT~~ [DOCS-11765] Add OP Splunk HEC Distribution of OTel and DDOT Nov 7, 2025

small edits

d132577

mikhacim reviewed Nov 10, 2025

View reviewed changes

content/en/observability_pipelines/sources/opentelemetry.md Show resolved Hide resolved

mikhacim reviewed Nov 10, 2025

View reviewed changes

maycmlee commented Nov 10, 2025

View reviewed changes

Apply suggestions from code review

071770e

ckelner reviewed Nov 10, 2025

View reviewed changes

apply suggestions

2ce1fc0

maycmlee requested review from ckelner and mikhacim November 11, 2025 16:07

maycmlee commented Nov 11, 2025

View reviewed changes

content/en/observability_pipelines/sources/opentelemetry.md Outdated Show resolved Hide resolved

Update content/en/observability_pipelines/sources/opentelemetry.md

f85fc26

ckelner reviewed Nov 11, 2025

View reviewed changes

apply suggestions

4fc82d9

ckelner reviewed Nov 11, 2025

View reviewed changes

apply suggestions

8c9b4f1

ckelner approved these changes Nov 11, 2025

View reviewed changes

Merge branch 'master' into may/op-otel

721f229


		## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines

		To send logs from DDOT:


		{{% observability_pipelines/log_source_configuration/splunk_hec %}}

		## Send logs from the Splunk Distributor of the OpenTelemetry collector to Observability Pipelines


		## Send logs from the Splunk Distributor of the OpenTelemetry collector to Observability Pipelines

		To send logs from the Splunk Distributor of the OpenTelemetry Collector:


		Use Observability Pipelines' Datadog Agent source to receive logs from the Datadog Agent. Select and set up this source when you [set up a pipeline][1].

		Note: If you are using the Datadog Distribution of OpenTelemetry Collector, you must [use the OpenTelemetry source to send logs to Observability Pipelines][4].

[DOCS-11765] Add OP Splunk HEC Distribution of OTel and DDOT #32704

Are you sure you want to change the base?

[DOCS-11765] Add OP Splunk HEC Distribution of OTel and DDOT #32704

Conversation

maycmlee commented Nov 7, 2025

What does this PR do? What is the motivation?

Merge instructions

Additional notes

Uh oh!

github-actions bot commented Nov 7, 2025

Preview links (active after the build_preview check completes)

Modified Files

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mikhacim Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ckelner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ckelner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ckelner left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Preview links (active after the `build_preview` check completes)

mikhacim Nov 10, 2025 •

edited

Loading