Fix macOS pcap traceroute ~100 ms latency floor by AlexandreYang · Pull Request #134 · DataDog/datadog-traceroute

AlexandreYang · 2026-07-07T20:48:09Z

Summary

Fixes #133.

On macOS, libpcap/BPF capture was configured with a 100 ms read timeout but not immediate mode. In practice, captured packets could be delivered to userspace on that timeout cadence, so traceroute RTT accounting included capture delivery delay. This produced an artificial ~100 ms RTT floor, including for directly reachable LAN targets.

This PR enables pcap immediate mode for the macOS BPF source so packets are delivered as soon as they arrive, while keeping the existing timeout as a fallback/read timeout setting.

Changes

Enables InactiveHandle.SetImmediateMode(true) in the Darwin pcap source before activation.
Names the existing pcap read timeout constant for clarity.
Adds a Darwin/root tagged regression test that captures a localhost TCP SYN/ACK and asserts delivery happens before the old 100 ms timeout floor.
Adjusts root-tagged packet test helpers so macOS capture-only tests use NewBpfDevice directly instead of requiring an unused raw packet sink.

Validation

Reproduced before the fix:

datadog-traceroute reported ~101-105 ms RTT to a local gateway over TCP/80.
Independent curl TCP connect timing to the same gateway was ~5 ms.
tcpdump -ttt showed SYN to SYN/ACK at ~3 ms.

Checks run after the fix:

go test -tags 'test root' ./packets -run TestPcapSourceDeliversPacketsWithoutReadTimeoutDelay -count=10 -v
go test -tags 'test root' ./packets -count=1 -v
go test ./...
make build

All passed.

Risk

Scoped to macOS packet capture setup. Linux uses AF_PACKET and Windows uses raw socket/driver completion paths, so this does not change their capture behavior.

Follow-up

This PR intentionally keeps the fix scoped to immediate mode plus regression coverage. Timestamp-based RTT accounting with packet capture metadata is broader hardening and is tracked separately in #135.

AlexandreYang · 2026-07-07T21:06:11Z

Version impact audit for datadog-agent consumers:

The traceroute regression starts in github.com/DataDog/datadog-traceroute v1.0.16 and is also present in v1.0.17. The triggering change is the Darwin libpcap migration in traceroute v1.0.16; the macOS pcap path used a 100 ms read timeout without immediate mode, which created the observed ~100 ms RTT floor.
datadog-agent appears to have skipped importing traceroute v1.0.16 directly. The Agent dependency bump I found is v1.0.15 -> v1.0.17 in DataDog/datadog-agent@a22956514edc4fe4fcff5c868013f514ad223d56 (Update module github.com/DataDog/datadog-traceroute to v1.0.17 (#52411), committed 2026-06-18 14:13:15 UTC).
Affected Agent builds: macOS Network Path builds from datadog-agent main / the 7.82.0 development stream after that dependency bump, until Agent consumes a fixed traceroute release.
Not affected based on the refs/tags I audited: 7.82.0-devel tag (v1.0.15), 7.81.x, 7.81.0, 7.81.0-rc.* (v1.0.15), 7.80.x, 7.80.4 (v1.0.15), and older sampled 7.x release tags such as 7.79.x (v1.0.13 or older).
I also scanned local 7.* tags for datadog-traceroute v1.0.16 / v1.0.17; no official Agent 7 release tag in that scan imported either affected traceroute version.

So the current release impact looks limited to macOS development/custom Agent builds from main after 2026-06-18, not stable Agent 7 release tags available in the audited refs. A future 7.82.0 RC/release would be affected if cut before Agent bumps to the fixed traceroute release.

AlexandreYang · 2026-07-07T21:22:32Z

The affected Agent builds are macOS Network Path builds that import github.com/DataDog/datadog-traceroute v1.0.16 or v1.0.17.

For datadog-agent, my audit shows:

Agent ref/version	traceroute version	Affected?
`main` / `7.82.0` development stream after `a22956514ed`	`v1.0.17`	Yes, on macOS
current `alex/netpath-rc` HEAD	`v1.0.17`	Yes, on macOS
`7.82.0-devel` tag	`v1.0.15`	No
`7.81.x`, `7.81.0`, `7.81.0-rc.*`	`v1.0.15`	No
`7.80.x`, `7.80.4`	`v1.0.15`	No
`7.79.x` / older sampled release tags	`v1.0.13` or older	No

So the practical answer is: no official released Agent 7 tag I scanned is affected; affected Agent builds are development/custom builds from main after the June 18, 2026 bump to datadog-traceroute v1.0.17. The Agent dependency bump is explicitly v1.0.15 -> v1.0.17, and the included traceroute v1.0.16 release notes contain the Darwin libpcap change that introduced this class of behavior.

Reference: DataDog/datadog-agent@a229565

The current Agent release metadata at the checked main SHA says milestone 7.82.0 and last stable Agent 7 is 7.80.4; 7.80.4 was released on 2026-07-01.

Reference: https://raw.githubusercontent.com/DataDog/datadog-agent/e77237492bfb80c0116ef82dedd623b0026ae043/release.json

Also: Agent appears to have skipped importing traceroute v1.0.16 directly. The bad traceroute code starts in v1.0.16, but Agent becomes affected when it jumps from v1.0.15 to v1.0.17.

Copilot

Pull request overview

This PR addresses a macOS-specific RTT inflation issue by configuring the libpcap/BPF capture path to deliver packets immediately instead of batching delivery on the read timeout cadence, which previously introduced an artificial ~100ms latency floor in traceroute RTT accounting.

Changes:

Enable pcap.InactiveHandle.SetImmediateMode(true) for the Darwin pcap source prior to activation.
Introduce a named pcapReadTimeout constant for clarity and reuse.
Add a Darwin/root regression test asserting captured loopback SYN/ACK delivery occurs well before the prior 100ms floor; refactor packet-source test helpers to avoid requiring an unused sink on macOS.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
packets/test_source_notdarwin_test.go	Adds a non-Darwin helper to construct/close a loopback test `Source` via `NewSourceSink`.
packets/pcap_filter_test.go	Switches to the new loopback-source helper and centralizes close logic.
packets/bpfdev_darwin.go	Names the read timeout and enables pcap immediate mode before activation.
packets/bpfdev_darwin_test.go	Adds a macOS/root regression test validating packet delivery latency is not dominated by the pcap read timeout.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

AlexandreYang · 2026-07-08T07:44:27Z

@codex review

chatgpt-codex-connector · 2026-07-08T07:49:31Z

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: c980c5d637

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a542ac5e3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-08T08:02:00Z

 		inactive.CleanUp()
 		return nil, fmt.Errorf("NewBpfDevice failed to set timeout: %w", err)
 	}
+	if err := inactive.SetImmediateMode(true); err != nil {


Preserve macOS read deadlines

When a probe is dropped or the BPF filter matches no packets, this can make macOS traceroutes hang indefinitely. PcapSource.Read() only checks p.deadline before/after ReadPacketData() returns, and gopacket documents SetImmediateMode as overriding SetTimeout (https://pkg.go.dev/github.com/google/gopacket/pcap#InactiveHandle.SetImmediateMode), so the 100ms timeout no longer wakes the loop to return ReceiveProbeNoPktError.

Useful? React with 👍 / 👎.

[AI Generated]
I do not think this is a valid issue in this PR. SetImmediateMode(true) does override libpcap's packet-buffer timeout behavior, but gopacket still preserves its Go-side timeout after activation: InactiveHandle.Activate copies p.timeout to the active handle and calls setNonBlocking() when the timeout is positive. ReadPacketData then polls via waitForPacket() using that Go-side timeout and returns NextErrorTimeoutExpired; PcapSource.Read already converts repeated timeout wakeups into ReceiveProbeNoPktError once its deadline has elapsed.

So immediate mode removes the 100 ms batching delay for packets, but it should not make dropped/no-match reads block indefinitely. No code change needed.

AlexandreYang added 2 commits July 7, 2026 22:47

empty

88f2bf4

Fix macOS pcap traceroute latency floor

51dfdcd

AlexandreYang changed the title ~~alex/fix-macos-100ms~~ Fix macOS pcap traceroute latency floor Jul 7, 2026

AlexandreYang changed the title ~~Fix macOS pcap traceroute latency floor~~ Fix macOS pcap traceroute ~100 ms latency floor Jul 7, 2026

AlexandreYang marked this pull request as ready for review July 8, 2026 07:16

AlexandreYang requested review from a team as code owners July 8, 2026 07:16

Copilot AI review requested due to automatic review settings July 8, 2026 07:16

Copilot started reviewing on behalf of AlexandreYang July 8, 2026 07:16 View session

qsl73 approved these changes Jul 8, 2026

View reviewed changes

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread packets/bpfdev_darwin_test.go

Comment thread packets/bpfdev_darwin_test.go Outdated

AlexandreYang mentioned this pull request Jul 8, 2026

Use packet capture timestamps for RTT accounting #135

Closed

Refine macOS pcap latency test

c980c5d

AlexandreYang requested a review from Copilot July 8, 2026 07:46

Copilot started reviewing on behalf of AlexandreYang July 8, 2026 07:46 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread packets/bpfdev_darwin_test.go

Match macOS pcap test SYNACK port

a542ac5

chatgpt-codex-connector Bot reviewed Jul 8, 2026

View reviewed changes

AlexandreYang merged commit 89cd02d into main Jul 8, 2026
19 checks passed

AlexandreYang deleted the alex/fix-macos-100ms branch July 8, 2026 08:24

qsl73 mentioned this pull request Jul 10, 2026

Use packet capture timestamps for RTT accounting #136

Merged

5 tasks

Navigation Menu

Uh oh!

Fix macOS pcap traceroute ~100 ms latency floor - #134

Fix macOS pcap traceroute ~100 ms latency floor#134
AlexandreYang merged 4 commits into
mainfrom
alex/fix-macos-100ms

AlexandreYang commented Jul 7, 2026 •

edited

Loading

Uh oh!

AlexandreYang commented Jul 7, 2026

Uh oh!

AlexandreYang commented Jul 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

AlexandreYang commented Jul 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Uh oh!

AlexandreYang Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AlexandreYang commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Risk

Follow-up

Uh oh!

AlexandreYang commented Jul 7, 2026

Uh oh!

AlexandreYang commented Jul 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

AlexandreYang commented Jul 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

AlexandreYang Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexandreYang commented Jul 7, 2026 •

edited

Loading