devices: drop cilium/ebpf{,link} deps by kolyshkin · Pull Request #64 · opencontainers/cgroups

kolyshkin · 2026-06-23T06:30:52Z

Replace the use of the cilium/ebpf and cilium/ebpf/link with direct
bpf(2) syscalls. Keep cilium/ebpf/asm for instruction assembly.

Notes:

The eBPF device-filter programs are now tracked by raw file
descriptors instead of *ebpf.Program handles;
asm.Instructions.Marshal requires a concrete binary.LittleEndian or
binary.BigEndian, thus endian_{le,be}.go are introduced as a
workaround.

This reduces the runc binary size by about ~1M.

NOTE that this is probably limited to runc, because for other users (k8s, cri-o) this was already solved by opencontainers/runc#4248.

Being tested in opencontainers/runc#5340.

For initial discussion about this, see opencontainers/runc#5218.

…ranch Temporarily point the opencontainers/cgroups dependency at the drop-cilium-ebpf branch (opencontainers/cgroups#64) via a go.mod replace, and re-vendor, so CI can exercise the cilium/ebpf main+link package removal end-to-end in runc. This must not be merged: the replace directive points at a personal fork branch. Once opencontainers/cgroups#64 lands and is tagged, this should be replaced by a normal dependency bump. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Copilot

Pull request overview

This PR refactors the devices eBPF cgroup-device filter implementation to stop importing github.com/cilium/ebpf’s main and link packages, and instead perform the required operations via direct bpf(2) syscalls while continuing to use github.com/cilium/ebpf/asm for instruction assembly. This aligns with the stated goal of reducing consumer binary size by avoiding heavy transitive dependencies.

Changes:

Replaced cilium/ebpf program/link usage with thin wrappers around BPF_PROG_* commands and raw program fds.
Updated cgroup device filter attach/query logic to operate on fds (with explicit closes) instead of *ebpf.Program.
Added nativeEndian selection via build-tagged endian-specific files to satisfy asm.Instructions.Marshal requirements.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
go.sum	Removes now-unused transitive dependencies after dropping `cilium/ebpf` main/link usage.
devices/endian_le.go	Provides `nativeEndian = binary.LittleEndian` under little-endian arch build tags.
devices/endian_be.go	Provides `nativeEndian = binary.BigEndian` under big-endian arch build tags.
devices/ebpf_linux.go	Replaces `ebpf`/`link` usage with direct `bpf(2)` syscall wrappers and fd-based program management.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+	err = bpfProgAttach(dirFd, progFd, attachFlags, replaceFd)
 	if err != nil {
 		return fmt.Errorf("failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): %w", err)
 	}


Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin · 2026-06-23T16:34:25Z

@cyphar @rata PTAL 🙏🏻

rata

@kolyshkin nice, thanks! Did you c&p the definitions? In that case, can you link them so it's simpler to review?

Also, don't we have overlapping functionality in the cgroups package?

Also, I guess no go.mod changes because it is used for the asm part? Nice that the size is decreased anyways :)

kolyshkin · 2026-07-10T01:36:53Z

Also, don't we have overlapping functionality in the cgroups package?

Again I'm afraid I fail to understand what you mean here @rata, can you please elaborate or point to whatever you have in mind?

kolyshkin · 2026-07-10T02:06:58Z

Did you c&p the definitions? In that case, can you link them so it's simpler to review?

I did not, I just recreated a bare minimum (poor boy) version of functionality that we used from cilium/ebpf and cilium/ebpf/link, basically wiring BPF_PROG_LOAD, BPF_PROG_GET_FD_BY_ID, BPF_PROG_ATTACH, BPF_PROG_DETACH and adding related data structures (also as bare minimum).

I have added a separate second commit, linking to the original cilium/ebpf functions. Let me know if you want it or not so I will squash or remove it.

kolyshkin · 2026-07-13T19:59:02Z

@AkihiroSuda @thaJeztah PTAL (I think I've addressed all of your comments).

The second commit is optional and can either be squashed or removed.

kolyshkin

Looking into kernel's tools/lib/bpf/bpf.c, I see that BPF_PROG_LOAD is retried up to 5 times when EAGAIN is received. Also, cilium/ebpf does that (indefinitely).

Implemented the same.

kolyshkin · 2026-07-27T20:25:00Z

@thaJeztah @rata @cyphar PTAL

kolyshkin · 2026-07-28T00:28:58Z

Did you c&p the definitions? In that case, can you link them so it's simpler to review?

I did not, I just recreated a bare minimum (poor boy) version of functionality that we used from cilium/ebpf and cilium/ebpf/link, basically wiring BPF_PROG_LOAD, BPF_PROG_GET_FD_BY_ID, BPF_PROG_ATTACH, BPF_PROG_DETACH and adding related data structures (also as bare minimum).

I have added a separate second commit, linking to the original cilium/ebpf functions. Let me know if you want it or not so I will squash or remove it.

@rata please let me know what you want to do here; I'm fine either way

rata

@kolyshkin Sorry I missed the notifications. I spent quite some time to review this now, to compensate :)

It seems there are quite some subtleties to replace cilium :-/

rata · 2026-07-28T10:19:56Z

+			continue
+		}
+
+		return r1, os.NewSyscallError("bpf", err)


os.NewSyscallError() returns syscall.Errno, not an error type. Don't callers of this function always see error is not nil?

No, it returns an error (for the last 15 years or so). It also returns nil if err == nil.

https://github.com/golang/go/blob/591ed42daf93672b56c786e89272425ffca7dca4/src/os/error.go#L64-L72

Ah! You probably meant that unix.Syscall returns syscall.Errno not error, and thus it needs to be handled correctly (can't compare to nil, need to compare to 0).

Just found it myself (when testing in opencontainers/runc#5340) and fixed.

Yes, exactly that! Sorry I wasn't clear :)

rata · 2026-07-28T10:31:18Z


-func findAttachedCgroupDeviceFilters(dirFd int) (_ []*ebpf.Program, retErr error) {
+func bpf(cmd uintptr, attr unsafe.Pointer, size uintptr) (uintptr, error) {
+	for {


I think we should bound this loop. The verifier returns EAGAIN if there are pending signals: torvalds/linux@c349480

It seems cilium has a workaround for this: https://github.com/cilium/ebpf/blob/v0.17.3/internal/sys/syscall.go#L18-L24

I think we should at least bound the loop, but probably handle it properly as cilium does. What do you think? Am I missing something?

Yes I saw that code in cilium/ebpf and decided to omit it as no one uses profiler in production.

Maybe I was wrong.

I guess maybe some software like kubernetes might use oc/cgroups/devices (currently it does not -- the only user is runc) and have pprof enabled.

Implemented the same "disable SIGPROF when using BPF_PROG_LOAD" approach by copying code from cilium/ebpf.

Because it is any signal, can't the go runtime trigger this also? It sends SIGUSR to preempt and schedule goroutines.

I think I'd still bound the loop, if it doesn't work in a few retries, we can fail and let the higher level container creation be started again. We don't want an infinite loop on any circumstances.

rata · 2026-07-28T10:51:44Z

+
+	// The load failed. Retry with the verifier log enabled so we can include
+	// it in the error (the first attempt skips it, as it is the fast path).
+	log := make([]byte, 64*1024)


If this is still small, it seems in some kernels it can still return a non useful error. Cilium seems to retry until ENOSPC is not returned: https://github.com/cilium/ebpf/blob/v0.17.3/prog.go#L439-L443

Shall we add some bounded retry mechanism here?

Implemented.

rata · 2026-07-28T10:57:46Z

+			logrus.WithFields(logrus.Fields{
+				"fd": oldFd,
+			}).Logf(logLevel, "removing old filter %d from cgroup", idx)


Not sure logging the fd is useful here. Before we had quite some data to log. Can't we recover some?

Agree that listing fd is useless. TBH everything else we printed is kind of useless, too; for example:

=== RUN TestFreeze === RUN TestFreeze/SystemdViaSet time="2026-07-26T05:13:46Z" level=info msg="found more than one filter (2) attached to a cgroup -- removing extra filters!" time="2026-07-26T05:13:46Z" level=info msg="removing old filter 0 from cgroup" id=101 name= run_count=0 runtime=0s tag=531db05b114e9af3 type=CGroupDevice time="2026-07-26T05:13:46Z" level=info msg="removing old filter 1 from cgroup" id=102 name=sd_devices run_count=0 runtime=0s tag=3b0b81b071f088cd type=CGroupDevice time="2026-07-26T05:13:46Z" level=info msg="found more than one filter (2) attached to a cgroup -- removing extra filters!" time="2026-07-26T05:13:46Z" level=info msg="removing old filter 0 from cgroup" id=103 name= run_count=0 runtime=0s tag=531db05b114e9af3 type=CGroupDevice time="2026-07-26T05:13:46Z" level=info msg="removing old filter 1 from cgroup" id=104 name=sd_devices run_count=0 runtime=0s tag=3b0b81b071f088cd type=CGroupDevice === RUN TestFreeze/SystemdViaPauseResume === RUN TestFreeze/FSViaSet time="2026-07-26T05:13:46Z" level=info msg="found more than one filter (2) attached to a cgroup -- removing extra filters!" time="2026-07-26T05:13:46Z" level=info msg="removing old filter 0 from cgroup" id=108 name= run_count=0 runtime=0s tag=531db05b114e9af3 type=CGroupDevice time="2026-07-26T05:13:46Z" level=info msg="removing old filter 1 from cgroup" id=109 name= run_count=0 runtime=0s tag=531db05b114e9af3 type=CGroupDevice === RUN TestFreeze/FSViaPauseResume

So maybe we should just remove this logging entirely? WDYT @cyphar ?

Removed the "removing old filter ..." log message for now.

kolyshkin · 2026-07-28T22:19:30Z

@rata thanks a lot for your review, I think I addressed all your comments; PTAL

Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Replace the use of the cilium/ebpf and cilium/ebpf/link with direct bpf(2) syscalls. Keep cilium/ebpf/asm for instruction assembly. Notes: - the eBPF device-filter programs are now tracked by raw file descriptors instead of *ebpf.Program handles; - asm.Instructions.Marshal requires a concrete binary.LittleEndian or binary.BigEndian, so endian.go detects the native byte order at runtime as a workaround. This could be done during compile time but requires maintaining a list of all GOARCHes; - the "removing old filter %d from cgroup" log messages are removed: this always happens when using systemd and the messages are not useful, plus obtaining the details would add more code; This reduces the runc binary size by about ~1M. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Link to cilium/ebpf analogs. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

rata

@kolyshkin thanks! Left another round of comments. It's definitely tricky, I haven't looked into all of it yet

rata · 2026-07-29T09:23:31Z

+	// bpf(2) treats fd 0 as unset, so we need to dup it.
+	newFD, err := unix.FcntlInt(fd, unix.F_DUPFD_CLOEXEC, 1)


Sorry, I don't follow this. Why do we need to dup it unconditionally?

rata · 2026-07-29T09:28:52Z

+		if err == nil {
+			return fd, nil
+		}


Isn't this a rather unexpected thing to happen? This failed before without logs, we are trying to run it again with log, to give a proper error. I'm unsure what we should do if this happens, shall we say all is fine? Or shall we just fail saying we couldn't get a log of the failure?

Sometimes I think the latter is better, so we can't get tricked into something completely unexpected (and have a security bug, maybe?). And if it's a problem, we can always change it and backport it.

What do you think?

rata · 2026-07-29T09:32:55Z

+	const minLogSize = 64 * 1024
+	// If the log does not fit into the buffer, the kernel returns ENOSPC, in
+	// which case retry with a bigger buffer, up to maxLogSize (the kernel
+	// rejects a log_size larger than UINT_MAX >> 8 anyway).


It seems newer kernels support a bigger buffer (https://github.com/torvalds/linux/blob/v5.2/kernel/bpf/verifier.c#L8215). I think it's compeltely fine to keep it to 16MB, but I'd amend the comment :D

rata · 2026-07-29T09:35:39Z

+
+		fd, err = bpfFD(unix.BPF_PROG_LOAD, unsafe.Pointer(&attr), unsafe.Sizeof(attr))
+		runtime.KeepAlive(insnsBytes)
+		runtime.KeepAlive(licensePtr)


Don't we need to keep alive log too?

rata · 2026-07-29T09:38:58Z

+// The code below is copied from
+// github.com/cilium/ebpf/internal/sys/signals.go@v0.22.0.


MIT requires some attribution, I think, but not sure if this code is a "significant portion". Shall we add them?

rata · 2026-07-29T09:51:09Z


-func findAttachedCgroupDeviceFilters(dirFd int) (_ []*ebpf.Program, retErr error) {
+func bpf(cmd uintptr, attr unsafe.Pointer, size uintptr) (uintptr, error) {
+	for {


Because it is any signal, can't the go runtime trigger this also? It sends SIGUSR to preempt and schedule goroutines.

I think I'd still bound the loop, if it doesn't work in a few retries, we can fail and let the higher level container creation be started again. We don't want an infinite loop on any circumstances.

rata · 2026-07-29T10:14:06Z

 			ProgIds:    uint64(uintptr(unsafe.Pointer(&progIds[0]))),
 			ProgCnt:    uint32(len(progIds)),
 		}

 		// Fetch the list of program ids.
-		_, _, errno := unix.Syscall(unix.SYS_BPF,
-			uintptr(unix.BPF_PROG_QUERY),
-			uintptr(unsafe.Pointer(&query)),
-			unsafe.Sizeof(query))
+		_, err := bpf(unix.BPF_PROG_QUERY, unsafe.Pointer(&query), unsafe.Sizeof(query))
+		runtime.KeepAlive(progIds)


Okay, my AI friends caught a weird bug that existed before but we are making slightly more possible to hit now.

The thing is: ProgIds is on the stack and is not tracked as a pointer, as it is uint64 and the pointer it has is gotten via unsafe. If the go runtime decides it needs more memory for this goroutine, it will allocate a bigger stack and move things there. Pointers are re-written, but as ProgIds is not a pointer for the runtime, it is not. And if calling bpf() makes something that needs a bigger stack, we hit this issue.

The way out of this is using https://pkg.go.dev/runtime#Pinner.Pin, it seems. But I'm not very familiar with it.

kolyshkin requested a review from a team as a code owner June 23, 2026 06:30

kolyshkin force-pushed the drop-cilium-ebpf branch 2 times, most recently from d5f40e9 to 733c596 Compare June 23, 2026 06:56

kolyshkin force-pushed the drop-cilium-ebpf branch from 733c596 to cf4738d Compare June 23, 2026 07:12

kolyshkin requested review from AkihiroSuda, Copilot and cyphar June 23, 2026 07:14

Copilot started reviewing on behalf of kolyshkin June 23, 2026 07:15 View session

This was referenced Jun 23, 2026

[DO NOT MERGE] vendor: test opencontainers/cgroups#64 opencontainers/runc#5340

Draft

[dnm/rfc] remove cilium/ebpf dependency to save 1.1M opencontainers/runc#5218

Closed

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread devices/ebpf_linux.go

Comment thread devices/ebpf_linux.go

Comment on lines +304 to 307

err = bpfProgAttach(dirFd, progFd, attachFlags, replaceFd)

if err != nil {

return fmt.Errorf("failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): %w", err)

}

kolyshkin changed the title ~~devices: drop cilium/ebpf main and link package dependency~~ devices: drop cilium/ebpf{,link} deps Jun 23, 2026

kolyshkin force-pushed the drop-cilium-ebpf branch from cf4738d to 0fd3600 Compare June 23, 2026 07:24

thaJeztah reviewed Jun 23, 2026

View reviewed changes

Comment thread devices/ebpf_linux.go

thaJeztah reviewed Jun 23, 2026

View reviewed changes

Comment thread devices/endian_be.go Outdated

kolyshkin force-pushed the drop-cilium-ebpf branch from 0fd3600 to 8decd05 Compare June 23, 2026 16:28

kolyshkin requested review from dims and haircommander June 24, 2026 20:48

kolyshkin mentioned this pull request Jun 25, 2026

v2: set memory.oom.group => OOMPolicy in systemd #41

Merged

rata reviewed Jun 25, 2026

View reviewed changes

Comment thread devices/ebpf_linux.go

kolyshkin force-pushed the drop-cilium-ebpf branch from 8decd05 to ae8789d Compare July 10, 2026 02:06

kolyshkin commented Jul 13, 2026

View reviewed changes

kolyshkin force-pushed the drop-cilium-ebpf branch 2 times, most recently from 3779064 to e52ab80 Compare July 13, 2026 22:27

kolyshkin requested a review from thaJeztah July 14, 2026 02:46

thaJeztah reviewed Jul 14, 2026

View reviewed changes

Comment thread devices/ebpf_linux.go Outdated

kolyshkin force-pushed the drop-cilium-ebpf branch from e52ab80 to eba30e4 Compare July 14, 2026 17:16

kolyshkin requested a review from thaJeztah July 27, 2026 20:25

rata reviewed Jul 28, 2026

View reviewed changes

kolyshkin force-pushed the drop-cilium-ebpf branch from eba30e4 to 17c1710 Compare July 28, 2026 22:18

kolyshkin force-pushed the drop-cilium-ebpf branch from 17c1710 to 9b49352 Compare July 28, 2026 22:21

kolyshkin added 2 commits July 28, 2026 17:18

[wip] devices: annotate new bpf functions

9da2685

Link to cilium/ebpf analogs. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin force-pushed the drop-cilium-ebpf branch from 9b49352 to 9da2685 Compare July 29, 2026 00:18

rata reviewed Jul 29, 2026

View reviewed changes

		// bpf(2) treats fd 0 as unset, so we need to dup it.
		newFD, err := unix.FcntlInt(fd, unix.F_DUPFD_CLOEXEC, 1)

		// The code below is copied from
		// github.com/cilium/ebpf/internal/sys/signals.go@v0.22.0.

Uh oh!

Conversation

kolyshkin commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kolyshkin commented Jun 23, 2026

Uh oh!

rata left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kolyshkin commented Jul 10, 2026

Uh oh!

kolyshkin commented Jul 10, 2026

Uh oh!

kolyshkin commented Jul 13, 2026

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kolyshkin commented Jul 27, 2026

Uh oh!

kolyshkin commented Jul 28, 2026

Uh oh!

rata left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented Jul 28, 2026

Uh oh!

rata left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kolyshkin commented Jun 23, 2026 •

edited

Loading

rata left a comment •

edited

Loading

rata left a comment •

edited

Loading

kolyshkin Jul 29, 2026 •

edited

Loading