Document compilation options

[qmonnet]

Context

I've been looking at why -g -O2 is necessary when compiling from C to eBPF with clang. It's not documented as far as I know, and I haven't been able to find the specific reasons again. I'm sure I saw them on mailing lists in the past, but I can't find it now.

Disclaimer

I realise this may be outside of the scope of this documentation, I note that you don't really mention how to compile with clang, or much of the ecosystem in general. I thought I'd open the issue as a suggestion, anyway - feel free to close if it's not relevant.

Description

So it would be nice to have some documentation about why one needs to pass -g and -O2 to clang when compiling.

For -g it's easy, it's necessary to get 1) debug information about the program, in particular the C source code that we dump with bpftool or llvm-objdump, for inspection, and 2) more importantly for many use case, BTF info.

For -O2 it's more tricky, everybody agrees it's necessary, but it's hard to find out why, precisely. From what I remember - and I may be totally wrong here - it may be related to:

• more dead code elimination, although the verifier can now suppress dead code for programs loaded by root, but removing it beforehand still reduces the size of the program to validate
• simplification of the flow graph, meaning again a smaller program to validate
• loop unrolling, although this is less relevant nowadays
• more inlining, meaning less stack usage
• improved register usage, fewer spills to stack
• something to do with variables relocation, but I can't remember the details
• ...?

[Tropicao]

Hi,
I totally agree with the observation about the lack of doc on those requirements, especially the one related to optimization level.
FWIW, I've also done this kind of search on mailing lists and docs for some clear statements about the matter, and failed to find anything. The only statement I have found is the the following one: https://docs.kernel.org/bpf/clang-notes.html#jump-instructions.

[dylandreimerink]

I think there is a place for this in the docs. I have been wanting to add more verbose sections such as guides or tutorials, and to tie that in with #8. I see this being a section about project and toolchain setup.

For -O2 it's more tricky, everybody agrees it's necessary, but it's hard to find out why, precisely

From what I recall there are BPF specific compiler passes which are only enabled under O2 that perform work to make the output pass the verifier. I also cannot recall the exact details. When adding/removing -O2 in compiler explorer it shows the following passed being added under O2:

• https://github.com/llvm/llvm-project/blob/53b26971f5192713bd3cc01ed31e48b4ea0b4d2c/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp
• https://github.com/llvm/llvm-project/blob/53b26971f5192713bd3cc01ed31e48b4ea0b4d2c/llvm/lib/Target/BPF/BPFMISimplifyPatchable.cpp
• https://github.com/llvm/llvm-project/blob/53b26971f5192713bd3cc01ed31e48b4ea0b4d2c/llvm/lib/Target/BPF/BPFMIPeephole.cpp

Not sure if these do a critical role on their own or if its the combination of other passes and these that make programs better verifiable.