Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ disable_toc: false

Use Observability Pipelines' Datadog Agent source to receive logs from the Datadog Agent. Select and set up this source when you [set up a pipeline][1].

**Note**: If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), you must [use the OpenTelemetry source to send logs to Observability Pipelines][4].

## Prerequisites

{{% observability_pipelines/prerequisites/datadog_agent %}}
Expand Down Expand Up @@ -38,4 +40,5 @@ Use the Agent configuration file or the Agent Helm chart values file to connect

[1]: /observability_pipelines/configuration/set_up_pipelines/
[2]: /containers/docker/log/?tab=containerinstallation#linux
[3]: /containers/guide/container-discovery-management/?tab=helm#setting-environment-variables
[3]: /containers/guide/container-discovery-management/?tab=helm#setting-environment-variables
[4]: /observability_pipelines/sources/opentelemetry/#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines
23 changes: 22 additions & 1 deletion content/en/observability_pipelines/sources/opentelemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ disable_toc: false

Use Observability Pipelines' OpenTelemetry (OTel) source to collect logs from your OTel Collector through HTTP or gRPC. Select and set up this source when you set up a pipeline. The information below is configured in the pipelines UI.

**Notes**:
- If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), [use the OpenTelemetry source to send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines).
- If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, use the [Splunk HEC source][4] to send logs to Observability pipelines.

### When to use this source

Common scenarios when you might use this source:
Expand Down Expand Up @@ -73,6 +77,23 @@ Based on these example configurations, these are values you enter for the follow
- HTTP listener address: `worker:4317`
- gRPC listener address: `worker:4318`

## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines

To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT):
1. Deploy the DDOT Collector using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions.
1. [Set up a pipeline][6] on Observability Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui).
1. (Optional) Datadog recommends adding an [Edit Fields processor][7] to the pipeline that appends the field `op_otel_ddot:true`.
1. When you install the Worker, for the OpenTelemetry source environment variables:
1. Set your HTTP listener to `0.0.0.0:4318`.
1. Set your gRPC listener to `0.0.0.0:4317`.

**Note**: Logs sent from DDOT might have nested objects that prevent Datadog from parsing the log into the prettified structure. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object.

[1]: https://opentelemetry.io/docs/collector/
[2]: /observability_pipelines/sources/
[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/#bootstrap-options
[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/#bootstrap-options
[4]: /observability_pipelines/sources/splunk_hec/#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines
[5]: /opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=datadogoperator
[6]: /observability_pipelines/configuration/set_up_pipelines/
[7]: /observability_pipelines/processors/edit_fields#add-field
[8]: /observability_pipelines/processors/custom_processor
24 changes: 24 additions & 0 deletions content/en/observability_pipelines/sources/splunk_hec.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ disable_toc: false

Use Observability Pipelines' Splunk HTTP Event Collector (HEC) source to receive logs from your Splunk HEC. Select and set up this source when you [set up a pipeline][1].

**Note**: Use the Splunk HEC source if you want to [send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines](#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines).

## Prerequisites

{{% observability_pipelines/prerequisites/splunk_hec %}}
Expand All @@ -21,4 +23,26 @@ Select and set up this source when you [set up a pipeline][1]. The information b

{{% observability_pipelines/log_source_configuration/splunk_hec %}}

## Send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines

To send logs from the Splunk Distribution of the OpenTelemetry Collector:

1. Install the Splunk OpenTelemetry Collector based on the your environment:
- [Kubernetes][2]
- [Linux][3]
2. Configure the Splunk OpenTelemetry Collector:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should swap steps two and three as seen here: https://github.com/DataDog/logs-psa-private/tree/main/POCs-Opps-hacks/splunk-otel-op#install-and-setup-op -- because you will need the IP Address / Load Balancer URL of OP before you can configure your Splunk OTel Collector.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swapped the steps and added a note about firewalls.

```bash
cp /etc/otel/collector/splunk-otel-collector.conf.example etc/otel/collector/splunk-otel-collector.conf
```
```bash
# Splunk HEC endpoint URL, if forwarding to Splunk Observability Cloud
# SPLUNK_HEC_URL=https://ingest.us0.signalfx.com/v1/log
# If you're forwarding to a Splunk Enterprise instance running on example.com, with HEC at port 8088:
SPLUNK_HEC_URL=http://0.0.0.0:8088/services/collector
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should replace this with placeholder text. 0.0.0.0 will never work here. It should be the actual addressable IP address or Load balancer URL for OP(s). So something like where we've done this elsewhere in the docs like <IP_ADDRESS_OR_LOAD_BALANCER_URL_FOR_OP> -- though that's a bit of a mouthful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I updated it to the <OPW_HOST> placeholder and used the explanation we use in the other parts of the doc. Let me know if that correct.

```
3. [Set up a pipeline][4] using the [Splunk HEC source](#set-up-the-source-in-the-pipeline-ui).

[1]: /observability_pipelines/configuration/set_up_pipelines/
[2]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-kubernetes
[3]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-linux
[4]: /observability_pipelines/configuration/set_up_pipelines
Loading