scap kernel module fails to build on Linux >= 6.13

[dkogan]

Describe the bug

scap fails to build. Here's a bug report showing the failure with falcosecurity-libs 0.18.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1093098

I just tried with 0.20.0, and I see the same failure

How to reproduce it

Build against a recent-enough kernel

Environment

Debian amd64.

Thanks.

[FedeDP]

Hi!
I think the build issues you are facing come from:

[configure-kmod] Including /var/lib/dkms/scap/0.18.1/build//configure/ACCESS_OK_2/Makefile.inc /var/lib/dkms/scap/0.18.1/build//configure/CLASS_CREATE_1/Makefile.inc /var/lib/dkms/scap/0.18.1/build//configure/DEVNODE_ARG1_CONST/Makefile.inc
[configure-kmod] Build output for HAS_ACCESS_OK_2:
[configure-kmod] make: Entering directory '/var/lib/dkms/scap/0.18.1/build/configure/ACCESS_OK_2' make -C /var/lib/dkms/scap/0.18.1/build M=/var/lib/dkms/scap/0.18.1/build/configure/ACCESS_OK_2 modules make[1]: Entering directory '/var/lib/dkms/scap/0.18.1/b
uild' make[1]: *** No rule to make target 'modules'.  Stop. make[1]: Leaving directory '/var/lib/dkms/scap/0.18.1/build' make: *** [Makefile:15: all] Error 2 make: Leaving directory '/var/lib/dkms/scap/0.18.1/build/configure/ACCESS_OK_2'
[configure-kmod] Build output for HAS_CLASS_CREATE_1:
[configure-kmod] make: Entering directory '/var/lib/dkms/scap/0.18.1/build/configure/CLASS_CREATE_1' make -C /var/lib/dkms/scap/0.18.1/build M=/var/lib/dkms/scap/0.18.1/build/configure/CLASS_CREATE_1 modules make[1]: Entering directory '/var/lib/dkms/scap/0.
18.1/build' make[1]: *** No rule to make target 'modules'.  Stop. make[1]: Leaving directory '/var/lib/dkms/scap/0.18.1/build' make: *** [Makefile:15: all] Error 2 make: Leaving directory '/var/lib/dkms/scap/0.18.1/build/configure/CLASS_CREATE_1'
[configure-kmod] Build output for HAS_DEVNODE_ARG1_CONST:
[configure-kmod] make: Entering directory '/var/lib/dkms/scap/0.18.1/build/configure/DEVNODE_ARG1_CONST' make -C /var/lib/dkms/scap/0.18.1/build M=/var/lib/dkms/scap/0.18.1/build/configure/DEVNODE_ARG1_CONST modules make[1]: Entering directory '/var/lib/dkms
/scap/0.18.1/build' make[1]: *** No rule to make target 'modules'.  Stop. make[1]: Leaving directory '/var/lib/dkms/scap/0.18.1/build' make: *** [Makefile:15: all] Error 2 make: Leaving directory '/var/lib/dkms/scap/0.18.1/build/configure/DEVNODE_ARG1_CONST'

No rule to make target 'modules'

Basically, the build/configure/* modules are small kernel modules that gets built as part of the kmod build process to check whether various changes are present or not (mostly in case of backports where we cannot rely upon kernel version macros); here they are: https://github.com/falcosecurity/libs/tree/master/driver/configure

In your case, all these modules fail to build thus the driver build proceeds as if all of these checks failed, not enabling the usage of correct APIs.

Am not sure about what's causing the No rule to make target 'modules' failure though.

[dkogan]

Hi. This is building with dkms. Worked ok for a long time, and I see it building OK with 6.12, but it fails with 6.13.

You are correct in that the configuration is the part that is failing: with 6.12 I see these compile flags being used: -DHAS_ACCESS_OK_2 -DHAS_CLASS_CREATE_1 -DHAS_DEVNODE_ARG1_CONST. But with 6.13 these flags do not end up being used.

[dkogan]

Hi. The bug is here:

libs/driver/Makefile.in

Line 32 in 51410de

KERNELDIR ?= $(CURDIR)

Prior to 6.13, this is evaluated in the kernel source directory, and the code works as intended. However, in 6.13 the kbuild logic changed, and this is evaluated in the scap/ driver directory instead. This incorrect value of KERNELDIR then cascades, and eventually causes this build problem. Can you set the KERNELDIR in some other way?

Thanks

[dkogan]

One "fix" is to replace $(CURDIR) above with $(OLDPWD). Works with both 6.12 and 6.13 in my tests. Might fail in other places. Found by reporting all the non-CONFIG variables in that spot:

$(foreach v,$(.VARIABLES),$(if $(filter CONFIG%,$v),,$(info $v='$($v)')))

[FedeDP]

Hi! Thanks, this is the commit that introduced the change, i think: torvalds/linux@13b2548
Will look for a solution asap, thanks for reporting!

[FedeDP]

/milestone next-driver

[FedeDP]

@dkogan would you mind testing using:

KERNELDIR=$(srctree)

?
Given the torvalds/linux@13b2548 commit message, that should be it; quoting relevant part:

The command for building external modules maintains backward
compatibility, but Makefiles that rely on working in the kernel
directory may break. In such cases, $(objtree) and $(srctree) should
be used to refer to the output and source directories of the kernel.

[FedeDP]

It won't work; can you try invoking dkms with:

--kernelsourcedir ${KERNELDIR}

where KERNELDIR is of course the target kernel build folder, ie: /lib/modules/$VERSION/build.

[dkogan]

Hi. Notes:

• My hacky "fix" of using $(OLDPWD) doesn't actually work
• Using $(srctree) doesn't work either. I see srctree=/usr/src/linux-headers-6.12.8-common while we want /usr/src/linux-headers-6.12.8-amd64. These are not the same thing, and I don't see links from anywhere in -common to the main directory
• invoking dkms with an extra option wouldn't be useful: 99% of the time you don't run dkms yourself: apt does this for you in a hook. So you can't pass it extra options.

[dkogan]

Also, Linus's commit says to use either $(srctree) or $(objtree). In my builds there is no $(objtree)

[dkogan]

ALSO, I'd like to clearly point out that I'm using Debian and its packaged kernel sources (in the linux-headers-... packages. This is a very normal thing to do, but the kernel devs probably don't use these for the most part; probably the mostly use the kernel sources from git directly. So their proposed ways to work around this may not work for the Debian packaging (they probably don't have a separate -common, and thus for them maybe $(srctree) IS a usable workaround.

[dkogan]

I just tried something else that apparently does not work:

diff --git a/driver/Makefile.in b/driver/Makefile.in
index 7b1fdc2..1a261ea 100644
--- a/driver/Makefile.in
+++ b/driver/Makefile.in
@@ -26,13 +26,22 @@ clean:
 
 install: all
 	$(MAKE) -C $(KERNELDIR) M=$(TOP) modules_install
 
 else
 
-KERNELDIR 	?= $(CURDIR)
+ifeq ($(KERNELDIR),)
+  TAIL := /include/config/auto.conf
+
+  # sort to uniqueify the list
+  KERNELDIR := $(sort $(patsubst %$(TAIL),%,$(filter %$(TAIL),$(MAKEFILE_LIST))))
+
+  $(if $(KERNELDIR),,$(error "Couln't find the KERNELDIR by looking for %$(TAIL) in MAKEFILE_LIST: no matches; MAKEFILE_LIST is '$(MAKEFILE_LIST)'"))
+  $(if $(word 2,$(KERNELDIR)),$(error "Couln't find the KERNELDIR by looking for %$(TAIL) in MAKEFILE_LIST: too many matches; MAKEFILE_LIST is '$(MAKEFILE_LIST)'"))
+endif
+
 #
 # Get the path of the module sources
 #
 FIRST_MAKEFILE := $(firstword $(MAKEFILE_LIST))
 FIRST_MAKEFILE_FILENAME := $(notdir $(FIRST_MAKEFILE))
 FIRST_MAKEFILE_DIRNAME := $(shell basename $(dir $(FIRST_MAKEFILE)))

It looks at MAKEFILE_LIST for any Makefiles whose name ends in /include/config/auto.conf. In 6.13 it uniquely pinned it down to the correct (not -common) directory. But in 6.12 the MAKEFILE_LIST doesn't have the correct directory in it at all

[FedeDP]

Did you try #2277 (comment) by chance? I 🙏 that can fix the issue, but am not 💯 sure!

EDIT: we use this very same method to make dkms build in local builder for driverkit: https://github.com/falcosecurity/driverkit/blob/6e5086a9b05baad23148654f78a0898c368f8021/pkg/driverbuilder/builder/templates/local.sh#L42
, that's why i hope it is the correct one (since it sets the correct kernelsourcedir)

[dkogan]

Hi. I tried that; it didn't help. But even if it did, it isn't a viable solution because dkms isn't something humans run: most of the time apt runs it in a hook, and you don't get to give it extra options.

In the commit in the linux tree, they say to use $(srctree). It doesn't work on Debian because of the split arch/common directories. Does that work in a non-Debian situation? What are you using, and does it work for you? If it's Debian only that can't make it work with $(srctree), then we can figure something out

[FedeDP]

I am not on 6.13 unfortunately, so i cannot test :/
But normal make is working fine against 6.13/6.14; it is just the dkms build that is failing, that's why i though that perhaps our dkms.conf might need some adjustment.

[dkogan]

OK. It sounds like we need a bit of time to find a generic solution. In the meantime I added a patch to the Debian build that works on Debian: https://salsa.debian.org/debian/falcosecurity-libs/-/blob/5c10309ed6c56b92954a6d24ff957486f725f393/debian/patches/kernel-module-builds-in-6.13.patch

Here I use $(MODLIB)/build, which is a symlink to the sources we want. Presumably this is a Debian-only thing, but I actually do not know for sure

[FedeDP]

Thanks for spotting the bug and taking the time to report and debug it. Let's see if we get more feedback on this one to understand the best way to proceed.

[unknowndevQwQ]

I also encountered an error when compiling the ebpf probe:

sysdig --bpf "(proc.name contains filter or proc.name in (bash))"
* Running scap-driver-loader for: driver version=0.18.1, arch=x86_64, kernel release=6.13.3-zen1-1-zen, kernel version=1
* Running scap-driver-loader with: driver=bpf, compile=yes, download=yes
* Filename 'scap_arch_6.13.3-zen1-1-zen_1.o' is composed of:
 - driver name: scap
 - target identifier: arch
 - kernel release: 6.13.3-zen1-1-zen
 - kernel version: 1
* Trying to download a prebuilt eBPF probe from https://download.sysdig.com/scap-drivers/0.18.1/x86_64/scap_arch_6.13.3-zen1-1-zen_1.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt scap eBPF probe
* Trying to compile the eBPF probe (scap_arch_6.13.3-zen1-1-zen_1.o)
expr: syntax error: unexpected argument '1'
In file included from probe.c:17:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/sched.h:12:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/current.h:10:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/cache.h:6:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/cache.h:5:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/linkage.h:8:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/linkage.h:6:
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/ibt.h:77:8: warning: 'nocf_check' attribute ignored; use -fcf-protection to enable the attribute [-Wignored-attributes]
   77 | extern __noendbr u64 ibt_save(bool disable);
      |        ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/ibt.h:32:34: note: expanded from macro '__noendbr'
   32 | #define __noendbr       __attribute__((nocf_check))
      |                                        ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/ibt.h:78:8: warning: 'nocf_check' attribute ignored; use -fcf-protection to enable the attribute [-Wignored-attributes]
   78 | extern __noendbr void ibt_restore(u64 save);
      |        ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/ibt.h:32:34: note: expanded from macro '__noendbr'
   32 | #define __noendbr       __attribute__((nocf_check))
      |                                        ^
In file included from probe.c:17:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/sched.h:12:
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
   47 |                 return this_cpu_read_const(const_pcpu_hot.current_task);
      |                        ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:578:36: note: expanded from macro 'this_cpu_read_const'
  578 | #define this_cpu_read_const(pcp)                        __raw_cpu_read_const(pcp)
      |                                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:163:35: note: expanded from macro '__raw_cpu_read_const'
  163 | #define __raw_cpu_read_const(pcp)       __raw_cpu_read(, , pcp)
      |                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:155:30: note: expanded from macro '__raw_cpu_read'
  155 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:94:40: note: expanded from macro '__my_cpu_type'
   94 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:45:32: note: expanded from macro '__percpu_seg_override'
   45 | # define __percpu_seg_override  __seg_gs
      |                                 ^
<built-in>:366:33: note: expanded from macro '__seg_gs'
  366 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from probe.c:17:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/sched.h:12:
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:578:36: note: expanded from macro 'this_cpu_read_const'
  578 | #define this_cpu_read_const(pcp)                        __raw_cpu_read_const(pcp)
      |                                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:163:35: note: expanded from macro '__raw_cpu_read_const'
  163 | #define __raw_cpu_read_const(pcp)       __raw_cpu_read(, , pcp)
      |                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:155:9: note: expanded from macro '__raw_cpu_read'
  155 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:94:40: note: expanded from macro '__my_cpu_type'
   94 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:45:32: note: expanded from macro '__percpu_seg_override'
   45 | # define __percpu_seg_override  __seg_gs
      |                                 ^
<built-in>:366:33: note: expanded from macro '__seg_gs'
  366 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from probe.c:17:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/sched.h:13:
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/processor.h:563:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
  563 |                 return this_cpu_read_const(const_pcpu_hot.top_of_stack);
      |                        ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:578:36: note: expanded from macro 'this_cpu_read_const'
  578 | #define this_cpu_read_const(pcp)                        __raw_cpu_read_const(pcp)
      |                                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:163:35: note: expanded from macro '__raw_cpu_read_const'
  163 | #define __raw_cpu_read_const(pcp)       __raw_cpu_read(, , pcp)
      |                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:155:30: note: expanded from macro '__raw_cpu_read'
  155 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:94:40: note: expanded from macro '__my_cpu_type'
   94 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:45:32: note: expanded from macro '__percpu_seg_override'
   45 | # define __percpu_seg_override  __seg_gs
      |                                 ^
<built-in>:366:33: note: expanded from macro '__seg_gs'
  366 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from probe.c:17:
In file included from /usr/lib/modules/6.13.3-zen1-1-zen/build/include/linux/sched.h:13:
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/processor.h:563:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:578:36: note: expanded from macro 'this_cpu_read_const'
  578 | #define this_cpu_read_const(pcp)                        __raw_cpu_read_const(pcp)
      |                                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:163:35: note: expanded from macro '__raw_cpu_read_const'
  163 | #define __raw_cpu_read_const(pcp)       __raw_cpu_read(, , pcp)
      |                                         ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:155:9: note: expanded from macro '__raw_cpu_read'
  155 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:94:40: note: expanded from macro '__my_cpu_type'
   94 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
/usr/lib/modules/6.13.3-zen1-1-zen/build/arch/x86/include/asm/percpu.h:45:32: note: expanded from macro '__percpu_seg_override'
   45 | # define __percpu_seg_override  __seg_gs
      |                                 ^
<built-in>:366:33: note: expanded from macro '__seg_gs'
  366 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from probe.c:27:
./fillers.h:873:56: error: member reference base type 'struct percpu_counter[4]' is not a structure or union
  873 |         bpf_probe_read_kernel(&val, sizeof(val), &mm->rss_stat.count[member]);
      |                                                   ~~~~~~~~~~~~^~~~~~
./fillers.h:2285:48: warning: passing 'volatile long *' to parameter of type 'long *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
 2285 |                 res = bpf_accumulate_argv_or_env(data, argv, &args_len);
      |                                                              ^~~~~~~~~
./fillers.h:1895:61: note: passing argument to parameter 'args_len' here
 1895 |                                                       long *args_len) {
      |                                                             ^
7 warnings and 1 error generated.
make[4]: *** [Makefile:74: probe.o] Error 1
make[3]: *** [/usr/lib/modules/6.13.3-zen1-1-zen/build/Makefile:1987: .] Error 2
make[2]: *** [/usr/lib/modules/6.13.3-zen1-1-zen/build/Makefile:251: __sub-make] Error 2
make[1]: *** [Makefile:251: __sub-make] Error 2
make: *** [Makefile:23: all] Error 2
mv: cannot stat '/usr/src/scap-0.18.1/bpf/probe.o': No such file or directory
Unable to load the scap eBPF probe
Unable to load the BPF probe
BPF probe is compiled for 6.11.9-zen1-1-zen, but running version is 6.13.3-zen1-1-zen