libbpf-tools/functrace: Add uprobe-based function tracer - #5537
Draft
ekyooo wants to merge 3 commits into
Draft
Conversation
Add functrace, a BPF CO-RE tool that traces userspace function calls via uprobes/uretprobes and reconstructs call trees with per-function latency measurement. Features: - uprobe/uretprobe-based function entry/exit tracing - per-thread shadow stack for call tree reconstruction - --root-func for trace boundary detection (transaction grouping) - configurable max call depth (--max-depth) - configurable ring buffer size (--ringbuf-size) - top-down text diagram output to stdout or file - PID filtering support - duration-based tracing Signed-off-by: Eunseon Lee <es.lee@lge.com>
Add shared helpers for emitting OTLP-compatible span records: - struct span / enum span_kind / enum span_status: Userspace representation of an OTLP-compatible span record with 128-bit traceId, 64-bit spanId/parentSpanId, timing, and status fields. - print_span(): Emit a span record as OTLP-compatible JSONL (one JSON object per line), suitable for piping to an OTLP collector. - struct time_sync / sync_time() / convert_to_realtime_ns(): Convert BPF ktime (CLOCK_MONOTONIC) to Unix epoch nanoseconds for OTLP timestamp fields. These utilities enable BPF tools to produce trace output that can be ingested by OTLP backends such as Grafana Tempo, Jaeger, or any OpenTelemetry-compatible collector. Signed-off-by: Eunseon Lee <es.lee@lge.com>
Integrate trace_helpers OTLP utilities into functrace, enabling OTLP-compatible span output for distributed tracing backends. New option: -F otel Output OTLP-compatible JSONL (one span per line) The OTLP output includes: - 128-bit traceId (pre-injected from userspace for consistency) - 64-bit spanId and parentSpanId for call tree reconstruction - Unix epoch nanosecond timestamps (converted from BPF ktime) - Function name, pid/tid, and status fields Example pipeline: functrace -b ./app -f func1 -f func2 -F otel | otlp-forwarder Signed-off-by: Eunseon Lee <es.lee@lge.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add functrace, a BPF CO-RE tool that traces userspace function calls
via uprobes/uretprobes and reconstructs call trees with per-function
latency measurement.
Features:
Why this approach
Why this approach
Many production issues — especially on embedded/edge devices or with immutable binaries — require
reconstructing a single process execution after the fact. Rebuilding the binary, adding SDK
instrumentation, or shipping a compiler toolchain to the target is frequently not possible.
functracetherefore attaches to running binaries from the outside with zero source access andzero recompilation, reconstructs the call tree in-kernel to minimize userspace event overhead, and
keeps export format decisions (e.g., OpenTelemetry) out of the BPF tool itself so it stays usable
standalone.
This design prioritizes deployability and post-hoc observability over the richer but invasive
guarantees of compile-time instrumentation.
Checklist
tools/toolname:,libbpf-tools/toolname:,src/cc:,docs:,build:,tests/python:)For new tools only
man/man8/) with an OVERHEAD section*_example.txt)tests/python/test_tools_smoke.py