Add runbook for Sectigo root CA rotation #3944

louis-cqrl · 2025-10-15T13:55:11Z

Description

This pull request adds a new runbook under runbooks/sectigo-root-ca-rotation-2025/ to assist users of Datadog Agent v5 in resolving connectivity issues caused by upcoming SSL certificate authority changes (Sectigo root CA rotation).

The runbook provides step-by-step automated scripts for both Linux and Windows environments that:

Download and install the updated Datadog certificate bundle
Update the agent configuration to enable use_curl_http_client: true
Restart the Datadog Agent and verify connectivity
Detect and report SSL verification errors since restart

This ensures continued connectivity for Agent v5 instances affected by the root CA update.

Motivation

Older versions of Datadog Agent v5 use a bundled certificate store that does not include the new Sectigo root CA.
Without intervention, these agents may lose the ability to send data to Datadog intake endpoints after the rotation.
This runbook provides an automated remediation path for affected users.

Testing Guidelines

TODO

Additional Notes

This runbook is intended as a temporary mitigation for legacy Agent v5 users.
Datadog strongly recommends upgrading to Agent v6 or v7 for long-term compatibility and security updates.
The runbook creates backups of configuration files and prints detailed error messages if issues occur.

jeremy-hanna

Great work on this! I left some comments mostly on the linux script, tbh the Powershell is totally unfamiliar to me and we might want someone else to look at it

jeremy-hanna · 2025-10-15T19:44:54Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+
+The scripts require outbound HTTPS connectivity to:
+
+- `raw.githubusercontent.com` (to download the certificate)


is this something we would want to have available on our actual domain like our linux install script in case of firewall restrictions? I'm not sure how we would go about doing that but the build team probably knows

jeremy-hanna · 2025-10-15T20:02:15Z

runbooks/sectigo-root-ca-rotation-2025/linux.sh

+  elif command -v wget &>/dev/null; then
+    DOWNLOADER="wget"
+  else
+    install_curl


a couple thoughts on this:

if curl or wget don't exist, how did they download the install script to the host?

I don't feel good about installing curl or wget if the customer has explicitly removed it (not sure why they would), imo this should be part of the prerequisites to running this so the customer can proceed how they want to

runbooks/sectigo-root-ca-rotation-2025/linux.sh

jeremy-hanna · 2025-10-15T20:38:30Z

runbooks/sectigo-root-ca-rotation-2025/linux.sh

+}
+
+ensure_target_directory() {
+  if ! sudo test -d "$TARGET_DIR"; then


the command is being run with sudo, I'm not sure we need to prefix these commands as such

jeremy-hanna · 2025-10-15T20:41:15Z

runbooks/sectigo-root-ca-rotation-2025/linux.sh

+ensure_target_directory() {
+  if ! sudo test -d "$TARGET_DIR"; then
+    echo "Directory $TARGET_DIR does not exist. Creating it..."
+    sudo mkdir -p "$TARGET_DIR" || error_exit "Error: Failed to create $TARGET_DIR."


do we want /opt/datadog-agent/agent to be owned by root or the dd-agent user? I'm not familiar with agent 5 installations but agent 7 installations make this owned by dd-agent

possibly this should be run as sudo -u dd-agent mkdir -p "$TARGET_DIR"?

jhanna@fedora:~$ ls -al /opt total 0 drwxr-xr-x. 1 root root 58 Jul 16 12:09 . dr-xr-xr-x. 1 root root 180 Sep 29 12:01 .. drwxr-xr-x. 1 dd-agent dd-agent 472 Sep 29 12:01 datadog-agent

I'm not sure about this behavior for the Agent 5 I try an other way let me know what you think of it

jeremy-hanna · 2025-10-15T21:19:50Z

runbooks/sectigo-root-ca-rotation-2025/linux.sh

+main() {
+  check_downloader
+  ensure_target_directory
+  download_certificate


should we test that the downloaded certificate works here via wget --ca-certificate or curl --cert before restarting the agent? it could indicate a failure earlier in the update process here

good point, added something

jeremy-hanna · 2025-10-15T21:52:54Z

runbooks/sectigo-root-ca-rotation-2025/linux.sh

+  while IFS= read -r line; do
+    log_file="${line%%:*}"
+    pre_size="${line##*:}"
+    if sudo test -f "$log_file"; then
+      if [ "$pre_size" -ge 0 ]; then
+        sudo tail -c "+$((pre_size+1))" "$log_file" 2>/dev/null | grep -qiE "$pattern" \
+          && error_exit "Detected SSL/cert verification failure in $(basename "$log_file") since restart."
+      else
+        sudo tail -n 500 "$log_file" 2>/dev/null | grep -qiE "$pattern" \
+          && error_exit "Detected SSL/cert verification failure in new $(basename "$log_file") since restart."
+      fi
+    fi
+  done < "$OFFSETS_FILE"


why are we saving and reading log offsets to a file?

it was mainly to only have the log since the restart to avoid false negatives

jeremy-hanna · 2025-10-15T22:03:34Z

runbooks/sectigo-root-ca-rotation-2025/linux.sh

+      if [ "$pre_size" -ge 0 ]; then
+        sudo tail -c "+$((pre_size+1))" "$log_file" 2>/dev/null | grep -qiE "$pattern" \
+          && error_exit "Detected SSL/cert verification failure in $(basename "$log_file") since restart."
+      else
+        sudo tail -n 500 "$log_file" 2>/dev/null | grep -qiE "$pattern" \
+          && error_exit "Detected SSL/cert verification failure in new $(basename "$log_file") since restart."
+      fi


imo we should just rotate these logs files instead of trying to manage an offset, if something bad happens it'd be nice to just have a "here is everything since the attempted restart" log for customers to give us / troubleshoot with

if we can't rotate the logs another option might just to be opening these in a tail -f stream for 30s or something on the restart instead of trying to find our point in time but that is more difficult to troubleshoot

really good idea, I applied it, let me know if this fits your idea

Personally, I do not mind the offset mechanism since it is fairly well scoped and not too complicated in this use cae. The problem with log rotation is cleanup and bloat over time. Customers really do not like dirty hosts on uninstall.

jeremy-hanna · 2025-10-15T22:10:07Z

runbooks/sectigo-root-ca-rotation-2025/windows.ps1

+    }
+    $filtered += $line
+  }
+  $filtered += 'use_curl_http_client: true'


does the .conf file care about indentation levels (is it yaml)?

yep this is yaml

alexn-dd · 2025-10-16T17:21:16Z

runbooks/sectigo-root-ca-rotation-2025/windows.ps1

+}
+
+function Enable-Tls12 {
+  try { [Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12 } catch { }


It is probably better to at least write a warning on failure.

alexn-dd · 2025-10-16T17:26:35Z

runbooks/sectigo-root-ca-rotation-2025/windows.ps1

+  $filtered += 'use_curl_http_client: true'
+  $updated = (($filtered -join "`r`n") + "`r`n")
+
+  try { $updated | Set-Content -LiteralPath $ConfFile -Encoding ASCII }


Probably UTF8 is better.

alexn-dd · 2025-10-16T17:31:52Z

runbooks/sectigo-root-ca-rotation-2025/windows.ps1

+      Write-Warning "Failed to restart service '$name': $($_.Exception.Message)"
+    }
+  }
+  if (-not $restarted) { Error-Exit "Error: Failed to restart the Datadog Agent (service not found or restart failed)." }


If the intention is to Error-Exit if any service fails to restart, this condition will not work.
In the foreach loop, the first service that succeeds restart will permanently set $restarted = $true and ignore other subsequent failures.

iadjivon

Hi there, thanks for looping the docs team into this PR. I have left some comments/suggestions. Let me know if you have any questions!

iadjivon · 2025-10-17T22:43:41Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+
+## Why This Matters
+
+Agent v5 uses an embedded certificate bundle for SSL/TLS verification. When Datadog's SSL certificates are updated to use newer certificate authorities, older Agent v5 installations may not recognize these certificates, causing the agent to lose connectivity with Datadog.


Suggested change

Agent v5 uses an embedded certificate bundle for SSL/TLS verification. When Datadog's SSL certificates are updated to use newer certificate authorities, older Agent v5 installations may not recognize these certificates, causing the agent to lose connectivity with Datadog.

Agent v5 uses an embedded certificate bundle for SSL/TLS verification. When Datadog's SSL certificates are updated to use newer certificate authorities, older Agent v5 installations may not recognize these certificates, causing the Agent to lose connectivity with Datadog.

iadjivon · 2025-10-17T22:44:02Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+This runbook provides automated scripts for both Linux and Windows that will:
+
+1. Download and install an updated certificate bundle
+2. Configure the agent to use your operating system's certificate store as a fallback


Suggested change

2. Configure the agent to use your operating system's certificate store as a fallback

2. Configure the Agent to use your operating system's certificate store as a fallback

iadjivon · 2025-10-17T22:48:10Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+If certificate errors persist after running the script:
+
+1. Verify your operating system is receiving security updates
+2. Check the agent logs for detailed error messages:


Suggested change

2. Check the agent logs for detailed error messages:

2. Check the Agent logs for detailed error messages:

iadjivon · 2025-10-17T22:50:28Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+
+## Verification
+
+After running the script, verify your agent is reporting metrics:


Suggested change

After running the script, verify your agent is reporting metrics:

After running the script, verify your Agent is reporting metrics:

iadjivon · 2025-10-17T22:53:40Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+2. Check your host in the Datadog Infrastructure List
+3. Verify the "Last Seen" timestamp is recent
+
+You can also manually check agent status:


Suggested change

You can also manually check agent status:

You can also manually check the Agent status:

iadjivon · 2025-10-17T22:54:05Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+
+## Long-Term Recommendation
+
+While this runbook provides a working solution, **we strongly recommend upgrading to Datadog Agent v6 or v7**:


Suggested change

While this runbook provides a working solution, **we strongly recommend upgrading to Datadog Agent v6 or v7**:

While this runbook provides a working solution, **Datadog strongly recommends upgrading to Datadog Agent v6 or v7** to benefit from:

iadjivon · 2025-10-17T22:59:52Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+- Improved performance and new features
+- Long-term support
+
+Agent v5 reached end-of-life and no longer receives updates. For migration guidance, visit the [Datadog documentation](https://docs.datadoghq.com/agent/).


Suggested change

Agent v5 reached end-of-life and no longer receives updates. For migration guidance, visit the [Datadog documentation](https://docs.datadoghq.com/agent/).

Agent v5 reached end-of-life and no longer receives updates. For migration guidance, visit the [Datadog documentation](https://docs.datadoghq.com/agent/guide/upgrade_agent_fleet_automation).

This new link directs users to steps for upgrading their Agent. See if there is a guide here that you deem more appropriate for this section.

iadjivon · 2025-10-17T23:00:32Z

runbooks/sectigo-root-ca-rotation-2025/README.md

+1. Your Agent version
+2. Operating system and version
+3. Complete output from the script
+4. Recent agent log excerpts showing any errors


Suggested change

4. Recent agent log excerpts showing any errors

4. Recent Agent log excerpts showing any errors

renantelo · 2025-10-23T15:36:59Z

lgtm

Add runbook for sectigo root CA rotation

a8ae2b1

louis-cqrl requested a review from a team as a code owner October 15, 2025 13:55

louis-cqrl requested a review from jeremy-hanna October 15, 2025 13:55

louis-cqrl added 2 commits October 15, 2025 15:56

fix documentation

39f844f

fix documentation

006c53d

jeremy-hanna requested changes Oct 15, 2025

View reviewed changes

alexn-dd reviewed Oct 16, 2025

View reviewed changes

Apply comments

9b8d916

iadjivon reviewed Oct 17, 2025

View reviewed changes


		The scripts require outbound HTTPS connectivity to:

		- `raw.githubusercontent.com` (to download the certificate)


		## Why This Matters

		Agent v5 uses an embedded certificate bundle for SSL/TLS verification. When Datadog's SSL certificates are updated to use newer certificate authorities, older Agent v5 installations may not recognize these certificates, causing the agent to lose connectivity with Datadog.

	2. Configure the agent to use your operating system's certificate store as a fallback
	2. Configure the Agent to use your operating system's certificate store as a fallback

	2. Check the agent logs for detailed error messages:
	2. Check the Agent logs for detailed error messages:


		## Verification

		After running the script, verify your agent is reporting metrics:

	You can also manually check agent status:
	You can also manually check the Agent status:


		## Long-Term Recommendation

		While this runbook provides a working solution, we strongly recommend upgrading to Datadog Agent v6 or v7:

	Agent v5 reached end-of-life and no longer receives updates. For migration guidance, visit the [Datadog documentation](https://docs.datadoghq.com/agent/).
	Agent v5 reached end-of-life and no longer receives updates. For migration guidance, visit the [Datadog documentation](https://docs.datadoghq.com/agent/guide/upgrade_agent_fleet_automation).

	4. Recent agent log excerpts showing any errors
	4. Recent Agent log excerpts showing any errors

Add runbook for Sectigo root CA rotation #3944

Are you sure you want to change the base?

Add runbook for Sectigo root CA rotation #3944

Uh oh!

Conversation

louis-cqrl commented Oct 15, 2025

Description

Motivation

Testing Guidelines

Additional Notes

Uh oh!

jeremy-hanna left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iadjivon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

renantelo commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants