modern BPF driver build fails: BPF stack limit of 512 bytes is exceeded in sendmmsg.bpf.c:113:14

[apteryks]

Describe the bug

When building from source, I get the build error:

[ 36%] [MODERN BPF] Building BPF object: /home/maxim/src/falcosecurity-libs/newbuild/driver/modern_bpf/sendmmsg.bpf.o
cd /home/maxim/src/falcosecurity-libs/newbuild/driver/modern_bpf && /gnu/store/md62rkk5f1v8kg5qrzg80rzlj9n896ys-profile/bin/clang -g -O2 -target bpf -D____ -D__TARGET_ARCH_x86 -D__USE_VMLINUX__ -I/home/maxim/src/falcosecurity-libs/newbuild/libbpf-prefix/src/libbpf-build/root/usr/include -I/home/maxim/src/falcosecurity-libs/driver/modern_bpf -I/home/maxim/src/falcosecurity-libs -isystem "-idirafter /gnu/store/md62rkk5f1v8kg5qrzg80rzlj9n896ys-profile/include -idirafter /gnu/store/c7y194jwjsqv63lnyp35rrqc1a87qnbr-clang-13.0.1/lib/clang/13.0.1/include -idirafter /gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/include" -c /home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c -o /home/maxim/src/falcosecurity-libs/newbuild/driver/modern_bpf/sendmmsg.bpf.o
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:113:14: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.
int BPF_PROG(sendmmsg_x, struct pt_regs *regs, long ret) {
             ^
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:113:14: note: could not determine the original source location for driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:0:0
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:113:14: note: could not determine the original source location for driver/modern_bpf/helpers/store/auxmap_store_params.h:0:0
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:113:14: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.
int BPF_PROG(sendmmsg_x, struct pt_regs *regs, long ret) {
             ^
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:113:14: note: could not determine the original source location for driver/modern_bpf/helpers/store/auxmap_store_params.h:0:0
In file included from /home/maxim/src/falcosecurity-libs/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/sendmmsg.bpf.c:10:
In file included from /home/maxim/src/falcosecurity-libs/driver/modern_bpf/helpers/interfaces/variable_size_event.h:11:
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/helpers/store/auxmap_store_params.h:1045:44: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.
        auxmap->payload_pos = initial_payload_pos + total_size_to_read;
                                                  ^
/home/maxim/src/falcosecurity-libs/driver/modern_bpf/helpers/store/auxmap_store_params.h:149:37: error: Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.
                compute_event_types_stats(auxmap->event_type, counter);
                                                  ^
4 errors generated.
make[2]: *** [driver/modern_bpf/CMakeFiles/ProbeSkeleton.dir/build.make:3781: driver/modern_bpf/sendmmsg.bpf.o] Error 1
make[2]: Leaving directory '/home/maxim/src/falcosecurity-libs/newbuild'
make[1]: *** [CMakeFiles/Makefile2:1578: driver/modern_bpf/CMakeFiles/ProbeSkeleton.dir/all] Error 2
make[1]: Leaving directory '/home/maxim/src/falcosecurity-libs/newbuild'
make: *** [Makefile:136: all] Error 2

How to reproduce it

1. git clone this repo
2. mkdir build && cd build
3. cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=$PWD/install "-DBUILD_DRIVER=OFF" "-DENABLE_DKMS=OFF" "-DBUILD_LIBSCAP_MODERN_BPF=ON" ..
4. make -j$(nproc)

Expected behaviour

Build should succeed.

Environment

• Falco version: From git commit 6c46ed3.
• Cloud provider or hardware configuration: x86_64
• OS: Guix System
• Kernel: Linux terra 6.12.10 update: delete notices about chisels #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux
• Installation method: From source.

[apteryks]

This seems to work around this problem:

modified   driver/modern_bpf/helpers/base/shared_size.h
@@ -27,7 +27,7 @@
 #define MAX_IOVCNT 32
 
 /* Maximum number of supported sendmmsg/recvmmsg messages */
-#define MAX_SENDMMSG_RECVMMSG_SIZE 8
+#define MAX_SENDMMSG_RECVMMSG_SIZE 1
 
 /* Maximum number of `pollfd` structures that we can analyze. */
 #define MAX_POLLFD 16

Any higher value such as 2 triggers the original build failure.

[apteryks]

Reverting commit 8362ae9 resolves the issue.

[FedeDP]

Hi! Thanks for opening this issue!
Unfortunately we did not catch this problem with our kernel-testing matrix: https://falcosecurity.github.io/libs/matrix/.
Can you share your bpftool version?

[FedeDP]

/milestone 0.21.0

[Andreagit97]

The issue is probably with the clang version, which clang are you using to compile the probe?

[apteryks]

clang --version in my build container says I'm using 13.0.1. For bpftool:

$ bpftool --version
bpftool v7.5.0
using libbpf v1.5
features:

[apteryks]

It appears to build fine with Clang 14; confirmed!

[FedeDP]

Yep, the issue is that we should really come up with an architecture that provides us a way to runtime "probe" the system for bpf features; in your case, using bpf_loop would do the trick (and would be the best choice); but since we cannot guarantee that the system supports it (see #2027 (comment)) we are not able to make use of that.
We will try to address this for 0.21.0 🙏

[FedeDP]

Hi! Can you test HEAD of #2255 branch?
It contains some changes to properly use bpf_loop where available and fallback at sending just the first message where bpf_loop cannot be used. It should hopefully solve your issue!

[FedeDP]

Any news on this one @apteryks ?

[FedeDP]

Closing, as this should be fixed in master.
Feel free to reopen if needed!
/close