[Feat]: Design proposal for tracing daemon-backed build workflows

[stupendoussuperpowers]

First-pass implementation can be seen here for reference: https://github.com/stupendoussuperpowers/go-witness/tree/docker-filetrace/attestation/workloadrun

Describe the solution you'd like:

Witness currently traces command execution primarily through the process tree started by command-run, with file activity represented through openedfiles and network activity represented through network-trace. That model works when the command being attested performs all relevant work in its own descendant processes. It breaks down for common artifact generation workflows that delegate work to daemons, service managers, containers, or sandbox runtimes. Examples for this include docker build that creates cgroups and mounts filesystems through dockerd and containerd, or snap install which mounts a filesystem through snapd.

This creates gaps for SBOMit and similar consumers that want Witness attestations for common artifact generation flows. For these use cases, the attestation must cover the effective workload, not only the CLI process that initiated it.

The proposed solution is the workload-run (final name TBD) attestor. It observes Linux workload boundaries such as cgroups and mounts while the command-run command is executing, captures low-level workload activity from those boundaries, then lets provider plug-ins interpret and enrich the raw facts into useful runtime-specific attestation data.

Proposed solution:

This is an initial proposal for the workload-run attestor. A reference implementation is linked above, but the final attestation shape, process-coordination model, and even the name for the attestor still warrant some discussions. Any help/insights are appreciated!

workload-run has two cooperating pieces:

1. A "collector" that observes common Linux workload primitives.
2. Workflow-specific metadata Providers that decide which observed primitives matter and how to interpret them.

Briefly, the lifecycle of tracing is:

1. eBPF-backed collectors begin watching for newly created cgroups, mount activity, and file/process events once the command-run execution window starts.
2. Newly observed cgroups are passed through all enabled providers. A cgroup is traced if at least one provider accepts it.
3. Activity from accepted cgroups is recorded as raw workload data. After tracing stops, each provider's Enrich function can group, add, or drop data based on the collected system traces.

We make the following assumptions about the system and the build processes in general:

• Newly created workload boundaries during the traced command are treated as potentially relevant to that command. This might lead to the attestation containing broad host-level events or even parallel builds. These are expected to be pruned out by individual Providers (or the lack thereof).
• Only newly observed cgroups and mounts inside the tracing window are considered by default. Existing long-lived daemon state is not attributed unless a provider later adds explicit correlation for it.

The reason for a collector/provider split is that most runtimes use the same small set of Linux primitives, while the meaning of those primitives is runtime-specific. The collector can therefore stay focused on cgroups, mounts, file opens, process metadata, and eventually network activity. Providers can stay focused on questions like "is this Docker?", "is this snap?", and "do these raw cgroups belong to the same higher-level workload?".

This allows for a few key benefits such as allowing for reuse of existing eBPF codebases avoiding implementation drift, and allowing for a more convenient way to both add new Providers or update existing ones without needing huge diffs. For instance, a cleverer Docker Provider may be able to differentiate between two parallel docker build commands, and under the existing architecture it would not be required to touch anything other than the DockerProvider struct itself.

Providers

Providers define runtime-specific policy and enrichment:

type Provider interface {
	Name() string
	CgroupFilter(cgroup string) bool
	FileFilter(file string) bool
	MountFilter(mount Mount) bool
	Enrich(ctx context.Context, workloads []Workload) ([]Workload, error)
}

The filter methods run during collection. They are used to accept or reject newly observed cgroups, opened files, and mounts. Enrich runs after collection and can label workloads, merge related cgroups, attach runtime metadata, or drop records that are no longer useful.

The reference implementation provides a minimal Docker provider. It accepts Docker and containerd-looking cgroup paths, keeps file-open activity from those cgroups, and labels matching workloads as docker or containerd during enrichment. It does not yet attempt deeper BuildKit or image-layer correlation.

A new provider such as one for snap would follow the same model: recognize snapd-created cgroups, snap mount patterns, squashfs paths, or snapd state, then enrich the raw workload into a snap-specific record.

Attestation Schema

The proposed attestation type is:

https://witness.dev/attestations/workload-run/v0.1

The current schema is:

type WorkloadRun struct {
	Providers  []string   `json:"providers,omitempty"`
	Collectors []string   `json:"collectors,omitempty"`
	Scope      Scope      `json:"scope,omitempty"`
	Workloads  []Workload `json:"workloads,omitempty"`
}

type Scope struct {
	StartedAt time.Time `json:"started_at,omitempty"`
	EndedAt   time.Time `json:"ended_at,omitempty"`
	Trigger   Trigger   `json:"trigger,omitempty"`
}

type Trigger struct {
	Attestor  string `json:"attestor,omitempty"`
	ProcessID int    `json:"processid,omitempty"`
}

type Workload struct {
	ID          string                          `json:"id,omitempty"`
	Kind        string                          `json:"kind,omitempty"`
	Runtime     string                          `json:"runtime,omitempty"`
	Name        string                          `json:"name,omitempty"`
	Image       string                          `json:"image,omitempty"`
	Cgroups     []Cgroup                        `json:"cgroups,omitempty"`
	Mounts      []Mount                         `json:"mounts,omitempty"`
	Processes   []ProcessInfo                   `json:"processes,omitempty"`
	OpenedFiles map[string]cryptoutil.DigestSet `json:"openedfiles,omitempty"`
	Metadata    map[string]string               `json:"metadata,omitempty"`
}

providers and collectors describe how the attestation was produced. scope describes when tracing was active and which command-run process triggered it. workloads contains the raw and enriched workload records.

This is a rough shape for the attestation, as the first obvious improvement we need to decide where to record network traces. They could either be a top level entry similar to OpenedFiles or both of these can be turned into a nested entry of Cgroups

The workload fields are intentionally a mix of generic collection data and provider-enriched data. cgroups, mounts, processes, and openedfiles come from the collector. runtime, name, image, and metadata are mainly provider-owned fields. There is room to add fields as providers become more precise, for example runtime IDs, build step IDs, network activity, layer IDs, package names, or stronger provenance links to the initiating command.

Example Use

witness run -a command-run -a workload-run --attestor-workload-run-provider docker -- docker build .

Testing, Documentation and Follow-up work required:

• Add Linux integration coverage for Docker/containerd cgroup detection and opened file attribution.
• Provide documentation around Linux/eBPF requirements, experimental status, provider authoring, and known over-attribution cases.

---

CC: @Vyom-Yadav @SantiagoTorres @jkjell

[Vyom-Yadav]

Hmm, just from a quick read I see you are proposing adding filter for mounts, files, cgroups which can contain any business logic.

The filter methods run during collection. They are used to accept or reject newly observed cgroups, opened files, and mounts. Enrich runs after collection and can label workloads, merge related cgroups, attach runtime metadata, or drop records that are no longer useful.

How would that work? Filter method work at userspace level. Which means record all and filter some. How about exposing filtering operations (include/exclude/matches, etc.) at bpf code level? I feel we can capture all this information in a new command run version instead of a new attestor. I am also of the opinion that witness should not do a lot of post processing of data. That is what things like SBOMit are meant for.

type Workload struct { ID string `json:"id,omitempty"` Kind string `json:"kind,omitempty"` Runtime string `json:"runtime,omitempty"` Name string `json:"name,omitempty"` Image string `json:"image,omitempty"` Cgroups []Cgroup `json:"cgroups,omitempty"` Mounts []Mount `json:"mounts,omitempty"` Processes []ProcessInfo `json:"processes,omitempty"` OpenedFiles map[string]cryptoutil.DigestSet `json:"openedfiles,omitempty"` Metadata map[string]string `json:"metadata,omitempty"` }

For this, shouldn’t it be opened files per process per cgroup? Basically mapping process to files to which cgroup was the process running under? Also, I don’t see namespaces being mentioned here. That would be crucial as well imo. Another thing is tying this with network trace as well. What I am structuring on that side is network traffic per process in a namespace running in a particular cgroup. I am not sure how that data would fit in. Just some initial thoughts.

…

On Thu, 25 Jun 2026 at 10:20 PM, Sanchit Sahay ***@***.***> wrote: *stupendoussuperpowers* created an issue (in-toto/go-witness#767) <#767> First-pass implementation can be seen here for reference: https://github.com/stupendoussuperpowers/go-witness/tree/docker-filetrace/attestation/workloadrun *Describe the solution you'd like:* Witness currently traces command execution primarily through the process tree started by command-run, with file activity represented through openedfiles and network activity represented through network-trace. That model works when the command being attested performs all relevant work in its own descendant processes. It breaks down for common artifact generation workflows that delegate work to daemons, service managers, containers, or sandbox runtimes. Examples for this include docker build that creates cgroups and mounts filesystems through dockerd and containerd, or snap install which mounts a filesystem through snapd. This creates gaps for SBOMit and similar consumers that want Witness attestations for common artifact generation flows. For these use cases, the attestation must cover the effective workload, not only the CLI process that initiated it. The proposed solution is the workload-run (final name TBD) attestor. It observes Linux workload boundaries such as cgroups and mounts while the command-run command is executing, captures low-level workload activity from those boundaries, then lets provider plug-ins interpret and enrich the raw facts into useful runtime-specific attestation data. *Proposed solution:* This is an initial proposal for the workload-run attestor. A reference implementation is linked above, but the final attestation shape, process-coordination model, and even the name for the attestor still warrant some discussions. Any help/insights are appreciated! workload-run has two cooperating pieces: 1. A "collector" that observes common Linux workload primitives. 2. Workflow-specific metadata Providers that decide which observed primitives matter and how to interpret them. Briefly, the lifecycle of tracing is: 1. eBPF-backed collectors begin watching for newly created cgroups, mount activity, and file/process events once the command-run execution window starts. 2. Newly observed cgroups are passed through all enabled providers. A cgroup is traced if at least one provider accepts it. 3. Activity from accepted cgroups is recorded as raw workload data. After tracing stops, each provider's Enrich function can group, add, or drop data based on the collected system traces. We make the following assumptions about the system and the build processes in general: - Newly created workload boundaries during the traced command are treated as potentially relevant to that command. This might lead to the attestation containing broad host-level events or even parallel builds. These are expected to be pruned out by individual Providers (or the lack thereof). - Only newly observed cgroups and mounts inside the tracing window are considered by default. Existing long-lived daemon state is not attributed unless a provider later adds explicit correlation for it. The reason for a collector/provider split is that most runtimes use the same small set of Linux primitives, while the meaning of those primitives is runtime-specific. The collector can therefore stay focused on cgroups, mounts, file opens, process metadata, and eventually network activity. Providers can stay focused on questions like "is this Docker?", "is this snap?", and "do these raw cgroups belong to the same higher-level workload?". This allows for a few key benefits such as allowing for reuse of existing eBPF codebases avoiding implementation drift, and allowing for a more convenient way to both add new Providers or update existing ones without needing huge diffs. For instance, a cleverer Docker Provider may be able to differentiate between two parallel docker build commands, and under the existing architecture it would not be required to touch anything other than the DockerProvider struct itself. *Providers* Providers define runtime-specific policy and enrichment: type Provider interface { Name() string CgroupFilter(cgroup string) bool FileFilter(file string) bool MountFilter(mount Mount) bool Enrich(ctx context.Context, workloads []Workload) ([]Workload, error) } The filter methods run during collection. They are used to accept or reject newly observed cgroups, opened files, and mounts. Enrich runs after collection and can label workloads, merge related cgroups, attach runtime metadata, or drop records that are no longer useful. The reference implementation provides a minimal Docker provider. It accepts Docker and containerd-looking cgroup paths, keeps file-open activity from those cgroups, and labels matching workloads as docker or containerd during enrichment. It does not yet attempt deeper BuildKit or image-layer correlation. A new provider such as one for snap would follow the same model: recognize snapd-created cgroups, snap mount patterns, squashfs paths, or snapd state, then enrich the raw workload into a snap-specific record. *Attestation Schema* The proposed attestation type is: https://witness.dev/attestations/workload-run/v0.1 The current schema is: type WorkloadRun struct { Providers []string `json:"providers,omitempty"` Collectors []string `json:"collectors,omitempty"` Scope Scope `json:"scope,omitempty"` Workloads []Workload `json:"workloads,omitempty"` } type Scope struct { StartedAt time.Time `json:"started_at,omitempty"` EndedAt time.Time `json:"ended_at,omitempty"` Trigger Trigger `json:"trigger,omitempty"` } type Trigger struct { Attestor string `json:"attestor,omitempty"` ProcessID int `json:"processid,omitempty"` } type Workload struct { ID string `json:"id,omitempty"` Kind string `json:"kind,omitempty"` Runtime string `json:"runtime,omitempty"` Name string `json:"name,omitempty"` Image string `json:"image,omitempty"` Cgroups []Cgroup `json:"cgroups,omitempty"` Mounts []Mount `json:"mounts,omitempty"` Processes []ProcessInfo `json:"processes,omitempty"` OpenedFiles map[string]cryptoutil.DigestSet `json:"openedfiles,omitempty"` Metadata map[string]string `json:"metadata,omitempty"` } providers and collectors describe how the attestation was produced. scope describes when tracing was active and which command-run process triggered it. workloads contains the raw and enriched workload records. This is a rough shape for the attestation, as the first obvious improvement we need to decide where to record network traces. They could either be a top level entry similar to OpenedFiles or both of these can be turned into a nested entry of Cgroups The workload fields are intentionally a mix of generic collection data and provider-enriched data. cgroups, mounts, processes, and openedfiles come from the collector. runtime, name, image, and metadata are mainly provider-owned fields. There is room to add fields as providers become more precise, for example runtime IDs, build step IDs, network activity, layer IDs, package names, or stronger provenance links to the initiating command. *Example Use* witness run -a command-run -a workload-run --attestor-workload-run-provider docker -- docker build . *Testing, Documentation and Follow-up work required:* - Add Linux integration coverage for Docker/containerd cgroup detection and opened file attribution. - Provide documentation around Linux/eBPF requirements, experimental status, provider authoring, and known over-attribution cases. ------------------------------ CC: @Vyom-Yadav <https://github.com/Vyom-Yadav> @SantiagoTorres <https://github.com/SantiagoTorres> @jkjell <https://github.com/jkjell> — Reply to this email directly, view it on GitHub <#767?email_source=notifications&email_token=ARTVXPIRRFIGXJV7WWH5ETT5BVJ33A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBXGQ2TOOBVGE4DFJTSMVQXG33OU5WWK3TUNFXW5JLFOZSW45FMMZXW65DFOJPWG3DJMNVQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARTVXPLA5W5NY2M2M5ZHULT5BVJ33AVCNFSNUABFKJSXA33TNF2G64TZHM2DSNJZGU2TOMJSHNEXG43VMU5TINZUGU3TQNJRHAZKC5QC> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[stupendoussuperpowers]

How would that work? Filter method work at userspace level. Which means record all and filter some. How about exposing filtering operations (include/exclude/matches, etc.) at bpf code level?

It maintains a bpf-map of "white listed" cgroups. Pretty similar to what the current eBPF command-run tracer does. Once a new cgroup is created, if a provider wants to trace it we add it to the white list.

I feel we can capture all this information in a new command run version instead of a new attestor.

For the most part we can, in fact it uses the same BPF code to track a lot of these things. One of the more compelling reasons to have an attestor like this is simply to capture cgroup/mount/etc-lifecycles similar to how we capture processes themselves (while not having to compete with the existing command-run schema that can be used for cases where daemon tracking is not needed). Most daemon based builds are going to rely on a common set of building blocks like these so capturing information about them should be sufficient for later processing.

For this, shouldn’t it be opened files per process per cgroup? Basically mapping process to files to which cgroup was the process running under?

Whoops yeah, oversight on my part. The test code I had tagged the opened file string with info about the PID. It should go something like Cgroups -> Processes -> {Network Traces, Opened Files,...}. Will fix and update both here and on the code.

I am also of the opinion that witness should not do a lot of post processing of data. That is what things like SBOMit are meant for.

Tend to agree/don't have overtly strong feelings on this. The main reason a "Provider" exists is to provide a way to filter for relevant cgroups and allow for optional pruning later. The pruning may as well be just differentiating between two parallel builds. The filtering is just to ensure that we don't end up capturing mounts or cgroups that have nothing to do with the docker build. This filtering can be as simple as regex matching on the name.

Might also be worth considering how much granularity we would expect from this (for instance, does it matter for us to tag cgroups spawned for two different RUN commands in the same image?).

---

I am pretty open to trying out suggestions for alternate versions/improvements to this general model, it's very experimental right now and I'm therefore open to experimenting.

Besides that, do we think there are more primitives that need to be tracked for popular build workloads?