Exporting Audit Events to Datadog with Fluentd in Kubernetes

[pnrao1983]

In this guide, we’ll walk through the setup of Fluentd to forward logs to Datadog using a Kubernetes deployment. This setup utilizes two pods: one for the Teleport Event Handler authenticates to the Teleport Auth Service to receive audit events over a gRPC stream, then second POD sends those events to Fluentd as JSON payloads over a secure channel established via mutual TLS.

Setup Overview

To follow this guide, refer to the Datadog Integration Documentation.

Important Notes

Make sure you adjust the URL:
• https://fluentd.fluentd.svc.cluster.local/events.log
This DNS name should reflect your specific namespace.

Datadog Integration Documentation Update

Host Update Based on Datadog Region:
When configuring the Datadog integration, ensure the host field for log intake is updated based on your Datadog region. Here’s the mapping:
US (Default): app.datadoghq.com
US3: us3.datadoghq.com
US5: us5.datadoghq.com
Europe (EU): app.datadoghq.eu
Federal: app.ddog-gov.com
Asia-Pacific (AP1): ap1.datadoghq.com
Asia-Pacific (AP2): ap2.datadoghq.com

Determine Your Datadog Region
To find out which Datadog region you’re using after logging in, you can follow these steps:

Method 1: Check the URL in the Browser

1. Log in to Datadog.
2. Check the URL in your browser’s address bar. The region is indicated by the subdomain.

Method 2: View the API and Application Keys Page

1. Go to the “Integrations” section in the Datadog dashboard.
2. Select “APIs” or go to “API and Application Keys” under “Organization Settings”.
3. The base URL for API requests displays the region information.

Deployment Instructions

Set Environment Variables:
TELEPORT_CLUSTER_ADDRESS=mytenant.teleport.sh:443 # update your cluster name
Run Docker Command:
docker run -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:16.4.0 configure . ${TELEPORT_CLUSTER_ADDRESS?}

Create Certificates:
Modify the certificates based on your Fluentd forwarder pod’s DNS name. Before you generate new server.crt & server.key rename the existing server.crt and server.key files. In this test case, I am using datadog namespace, so my DNS: fluentd.datadog.svc.cluster.local

mv server.crt server.crt_orig
mv server.key server.key_orig

Create updated Server Certificates:
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt -subj "/CN=localhost" -addext "subjectAltName=DNS:fluentd.datadog.svc.cluster.local" -CA "ca.crt" -CAkey "ca.key"

Here is the complete list of config we are going to use

[root@test-services test]# ll
total 76
-rw-r--r--. 1 root root 1765 Oct  8 10:21 ca.crt
-rw-r--r--. 1 root root 3243 Oct  8 10:21 ca.key
-rw-r--r--. 1 root root 1769 Oct  8 10:21 client.crt
-rw-r--r--. 1 root root 3243 Oct  8 10:21 client.key
-rw-------. 1 root root 1227 Oct  8 10:21 fluent.conf
-rw-r--r--. 1 root root 6797 Oct  8 13:38 fluentd-certs.yaml
-rw-r--r--. 1 root root 1344 Oct  8 10:23 fluentd-config.yaml
-rw-r--r--. 1 root root  180 Oct  8 10:20 fluentd-service.yaml
-rw-r--r--. 1 root root 1153 Oct  8 13:46 fluentd_deploy_pod.yaml
-rw-r--r--. 1 root root 6823 Oct  8 10:24 identity
-rw-r--r--. 1 root root 1513 Oct  8 13:26 server.crt
-rw-r--r--. 1 root root 1765 Oct  8 10:21 server.crt_orig
-rw-------. 1 root root 1704 Oct  8 13:26 server.key
-rw-r--r--. 1 root root 3326 Oct  8 10:21 server.key_orig
-rw-------. 1 root root  271 Oct  8 10:21 teleport-event-handler-role.yaml
-rw-------. 1 root root  360 Oct  8 10:21 teleport-event-handler.toml
-rw-r--r--. 1 root root  955 Oct  8 10:27 teleport-plugin-event-handler-values.yaml

Create Kubernetes Secrets and ConfigMaps:
Define your certificates and configurations in YAML files.

cat fluentd-certs.yaml
apiVersion: v1
kind: Secret
metadata:
  name: datadog-certs
  namespace: datadog
type: Opaque
data:
  ca.crt: "<base64_encoded_ca_crt>"
  server.crt: "<base64_encoded_server_crt>"
  server.key: "<base64_encoded_server_key>"

To decode the .crt and .key files using base64 command:

cat ca.crt | base64 -w 0
cat server.crt | base64 -w 0
cat server.key | base64 -w 0

**If you are running from mac os:**

cat ca.crt | base64 -b 0
cat server.crt | base64 -b 0
cat server.key | base64 -b 0

Here is the ConfigMap for fluentd-config:
cat fluentd-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: datadog
data:
  fluent.conf: |
    <source>
      @type http
      port 8888
      <transport tls>
        client_cert_auth true
        ca_path "/root/datadog/certs/ca.crt"
        cert_path "/root/datadog/certs/server.crt"
        private_key_path "/root/datadog/certs/server.key"
        private_key_passphrase "690ea31abd41a8068f8e0705b99e8e26a35843cb06cb4e8f8b8ba8a487ef8438"  # Replace with a secure method to handle secrets
      </transport>
      <parse>
        @type json
        json_parser oj
        time_type string
        time_format %Y-%m-%dT%H:%M:%S
      </parse>
      <buffer>
        @type file
        flush_thread_count 8
        flush_interval 1s
        chunk_limit_size 10M
        queue_limit_length 16
        retry_max_interval 30
        retry_forever true
      </buffer>
    </source>
    <match events.log>
      @type datadog
      @id awesome_agent
      api_key "7c4fXXXXcd50d9c7685277396e5de2"  # Replace with your Datadog API key
      host "http-intake.logs.datadoghq.com"
      dd_source "teleport"
      <buffer>
        @type memory
        chunk_limit_size 1M
        total_limit_size 100M
        flush_interval 10s
        retry_forever true
        retry_max_interval 60
      </buffer>
    </match>
    <match session.*.log>
      @type stdout
    </match>
    <system>
      log_level debug
    </system>

Fluentd POD config to run as a deployment
cat fluentd_deploy_pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd-deployment
  namespace: datadog
  labels:
    app: fluentd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      securityContext:
        runAsUser: 0
      containers:
        - name: fluentd
          image: fluent/fluentd:v1.16
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8888
          volumeMounts:
            - name: config-volume
              mountPath: /fluentd/etc/fluent.conf
              subPath: fluent.conf
            - name: certs-volume
              mountPath: /root/datadog/test
          env:
            - name: FLUENTD_CONF
              value: "fluent.conf"
          command: ["/bin/sh", "-c", "gem install fluent-plugin-datadog && fluentd -c /fluentd/etc/fluent.conf"]
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-config
        - name: certs-volume
          secret:
            secretName: datadog-certs

fluentd-service
cat fluentd-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: datadog
spec:
  selector:
    app: fluentd
  ports:
    - protocol: TCP
      port: 8888
      targetPort: 8888

teleport-plugin-event-handler-values
cat teleport-plugin-event-handler-values.yaml

eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  namespace: "datadog"
  #The window size configures the duration of the time window for the event handler
  #to request events from Teleport. By default, this is set to 24 hours.
  #Reduce the window size if the events backend cannot manage the event volume
  #for the default window size.
  #The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
  windowSize: "24h"

teleport:
  address: "nara-cluster.pnrao.rocks:443"
  identitySecretName: teleport-event-handler-identity
  identitySecretPath: "identity"

fluentd:
  url: "https://fluentd.teleport.svc.cluster.local:8888/events.log"
  sessionUrl: "https://fluentd.teleport.svc.cluster.local:8888/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

Apply Kubernetes Resources:

kubectl -n datadog create secret generic --from-file=identity teleport-event-handler-identity
kubectl -n datadog create secret generic teleport-event-handler-client-tls --from-file=ca.crt=ca.crt,client.crt=client.crt,client.key=client.key
kubectl apply -f fluentd-certs.yaml
kubectl apply -f fluentd-config.yaml
kubectl apply -f fluentd_deploy_pod.yaml
kubectl apply -f fluentd-service.yaml

Install the Teleport Plugin Event Handler:
helm install -n datadog teleport-plugin-event-handler teleport/teleport-plugin-event-handler --values teleport-plugin-event-handler-values.yaml --version 16.3.0

Check Deployment Status:
kubectl get all -n datadog

Check both the pod logs:

[root@nara-services certs]# k get all
NAME                                                 READY   STATUS    RESTARTS        AGE
pod/fluentd-deployment-5bc5f4c5f4-bzrbp              1/1     Running   0               38m
pod/teleport-plugin-event-handler-67669c99c5-zkhcj   1/1     Running   0               76m

NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/fluentd   ClusterIP   10.108.102.120   <none>        8888/TCP   23h

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fluentd-deployment              1/1     1            1           77m
deployment.apps/teleport-plugin-event-handler   1/1     1            1           76m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/fluentd-deployment-5bc5f4c5f4              1         1         1       77m
replicaset.apps/teleport-plugin-event-handler-67669c99c5   1         1         1       76m
[root@nara-services certs]#

========================================================================

[root@nara-services fluentd_pod_config]# k logs -f deployment.apps/fluentd-deployment
2024-09-20 16:04:57 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-20 16:04:57 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.4'
2024-09-20 16:04:57 +0000 [info]: gem 'fluentd' version '1.12.4'
2024-09-20 16:04:57 +0000 [debug]: No fluent logger for internal event
2024-09-20 16:04:57 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type http
    port 8888
    <transport tls>
      client_cert_auth true
      ca_path "/root/datadog/certs/ca.crt"
      cert_path "/root/datadog/certs/server.crt"
      private_key_path "/root/datadog/certs/server.key"
      private_key_passphrase xxxxxx
    </transport>
    <parse>
      @type "json"
      json_parser oj
      time_type string
      time_format "%Y-%m-%dT%H:%M:%S"
    </parse>
    <buffer>
      @type file
      flush_thread_count 8
      flush_interval 1s
      chunk_limit_size 10M
      queue_limit_length 16
      retry_max_interval 30
      retry_forever true
    </buffer>
  </source>
  <match events.log>
    @type datadog
    @id awesome_agent
    api_key xxxxxx
    host "http-intake.logs.datadoghq.com"
    dd_source "teleport"
    <buffer>
      @type "memory"
      chunk_limit_size 1M
      total_limit_size 100M
      flush_interval 10s
      retry_forever true
      retry_max_interval 60
    </buffer>
  </match>
  <match session.*.log>
    @type stdout
  </match>
  <system>
    log_level debug
  </system>
</ROOT>
2024-09-20 16:04:57 +0000 [info]: starting fluentd-1.12.4 pid=8 ruby="2.6.7"
2024-09-20 16:04:57 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "--under-supervisor"]
2024-09-20 16:04:58 +0000 [info]: adding match pattern="events.log" type="datadog"
2024-09-20 16:04:58 +0000 [info]: adding match pattern="session.*.log" type="stdout"
2024-09-20 16:04:58 +0000 [info]: adding source type="http"
2024-09-20 16:04:58 +0000 [debug]: #0 No fluent logger for internal event
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [warn]: section <buffer> is not used in <source> of http plugin
2024-09-20 16:04:58 +0000 [info]: #0 starting fluentd worker pid=15 ppid=8 worker=0
2024-09-20 16:04:58 +0000 [debug]: #0 [awesome_agent] buffer started instance=69938226575600 stage_size=0 queue_size=0
2024-09-20 16:04:58 +0000 [info]: #0 [awesome_agent] Starting HTTP connection to https://http-intake.logs.datadoghq.com:443 with compression enabled using v2 routes
2024-09-20 16:04:58 +0000 [debug]: #0 listening http bind="0.0.0.0" port=8888
2024-09-20 16:04:58 +0000 [debug]: #0 [awesome_agent] flush_thread actually running
2024-09-20 16:04:58 +0000 [debug]: #0 [awesome_agent] enqueue_thread actually running
2024-09-20 16:04:58 +0000 [info]: #0 fluentd worker is now running worker=0
========================================================================
[root@nara-services certs]# k logs -f replicaset.apps/teleport-plugin-event-handler-67669c99c5
INFO   Using batch size batch:20 event-handler/cli.go:249
INFO   Using type filter types:[] event-handler/cli.go:250
INFO   Using type exclude filter skip-event-types:map[] event-handler/cli.go:251
INFO   Skipping session events of type types:map[print:{}] event-handler/cli.go:252
INFO   Using start time value:<nil> event-handler/cli.go:253
INFO   Using timeout timeout:10s event-handler/cli.go:254
INFO   Using Fluentd url url:https://fluentd.teleport.svc.cluster.local:8888/events.log event-handler/cli.go:255
INFO   Using Fluentd session url url:https://fluentd.teleport.svc.cluster.local:8888/session.log event-handler/cli.go:256
INFO   Using Fluentd ca ca:/var/lib/teleport/plugins/event-handler/ca.crt event-handler/cli.go:257
INFO   Using Fluentd cert cert:/var/lib/teleport/plugins/event-handler/client.crt event-handler/cli.go:258
INFO   Using Fluentd key key:/var/lib/teleport/plugins/event-handler/client.key event-handler/cli.go:259
INFO   Using window size window-size:24h0m0s event-handler/cli.go:260
INFO   Using Teleport identity file file:/var/lib/teleport/plugins/event-handler/teleport-identity/identity event-handler/cli.go:263
INFO   Using Teleport identity file refresh interval:1m0s event-handler/cli.go:266
INFO   Created storage directory dir:storage/nara-cluster.pnrao.rocks_443 event-handler/state.go:125
INFO   Using initial cursor value cursor: event-handler/app.go:208
INFO   Using initial ID value id: event-handler/app.go:209
INFO   Using start time from state value:2024-09-20 15:26:47 +0000 UTC event-handler/app.go:210
INFO   event processing rate events_per_minute:4 event-handler/events_job.go:118
INFO   event processing rate events_per_minute:2 event-handler/events_job.go:118
INFO   event processing rate events_per_minute:2 event-handler/events_job.go:118
INFO   event processing rate events_per_minute:0 event-handler/events_job.go:118
INFO   event processing rate events_per_minute:0 event-handler/events_job.go:118
INFO   event processing rate events_per_minute:0 event-handler/events_job.go:118
========================================================================

My events are getting forwarded now:

Troubleshooting
During setup, you may encounter some errors. Here are common issues and their resolutions:

1. No Patterns Matched:
   [warn]: #0 no patterns matched tag="events.log"
2. TLS Verification Errors:
   tls: failed to verify certificate: x509: certificate is valid for localhost, not fluentd.teleport.svc.cluster.local
3. Context deadline exceeded

handler/app.go:125 ERRO Error sending event to fluentd: Post "https://fluentd.teleport.svc.cluster.local/events.log": context deadline exceeded (Client.Timeout exceeded while awaiting headers) event-handler/app.go:125

Conclusion

After following these steps, your events should be successfully uploaded to Datadog. If you encounter issues, refer to the troubleshooting section above or consult the logs for more details.