bpf: Support aggregate return values up to 16 bytes - #8695
Open
kernel-patches-daemon-bpf-rc[bot] wants to merge 13 commits into
Open
bpf: Support aggregate return values up to 16 bytes#8695kernel-patches-daemon-bpf-rc[bot] wants to merge 13 commits into
kernel-patches-daemon-bpf-rc[bot] wants to merge 13 commits into
Conversation
Author
|
Upstream branch: d114bb9 |
Author
|
Upstream branch: d114bb9 |
kernel-patches-daemon-bpf-rc
Bot
force-pushed
the
series/1143715=>bpf-next
branch
from
August 11, 2026 01:28
1ff1905 to
a506b60
Compare
kernel-patches-daemon-bpf-rc
Bot
force-pushed
the
bpf-next_base
branch
from
August 11, 2026 22:08
818432c to
d922dc0
Compare
added 13 commits
August 11, 2026 15:11
check_global_subprog_return_code() verifies that a global subprogram returns void, an arena pointer, or register R0 holding a scalar value. Later patches in this series add 16-byte aggregate return support, whose second half is returned in R2 and needs the same validation. Factor the per-register check into check_global_ret_scalar_reg(env, regno) so that it can be reused for R2. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes: such a value comes back in the R0:R2 register pair, with R2
holding the upper half. See LLVM patches [1] and [2].
Later patches teach the JIT, precision backtracking, live register analysis
and the verifier itself about that convention. All of them need to answer
the same question: does this subprogram return its value in a register
pair? Add the shared helpers up front so that those patches can be ordered
independently of each other:
- subprog_ret_type() resolves a subprogram's BTF return type. It is
factored out of subprog_returns_void(). The verifier_bug_if(!func) and
!func_proto checks it replaces are redundant, since
check_btf_func_early() already rejects a func_info record whose type_id
is not a BTF_KIND_FUNC pointing at a BTF_KIND_FUNC_PROTO. A check on
prog->aux->{btf,func_info} is added instead: unlike
subprog_returns_void(), which is only used for global subprograms, later
callers ask about static subprograms too, and those may belong to a
program loaded without BTF.
- ret_regs_cnt() maps the size of a return value to the number of
registers holding it.
- bpf_ret_reg_pair() answers the question above. Its users query it at
every subprogram call and at every subprogram exit, that is once per
verifier state rather than once per subprogram, so the answer is
precomputed into bpf_subprog_info->ret_reg_pair by
bpf_compute_subprog_ret_regs() and the helper itself is a flag test.
It lives in bpf_verifier.h because kernel/bpf/backtrack.c and
kernel/bpf/liveness.c need it as well.
bpf_compute_subprog_ret_regs() runs in bpf_check() right before
bpf_compute_live_registers(), which is the first of those users: by then
BTF func_info has been validated and the subprogram list is final.
No functional change, bpf_ret_reg_pair() has no callers yet.
[1] llvm/llvm-project#190894
[2] llvm/llvm-project#206876
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
LLVM 23 returns an __int128, or a struct/union larger than 8 bytes and no larger than 16 bytes, in the BPF R0:R2 register pair. The previous patch taught the verifier about that convention; wire up the JIT side so that the second half of the return value actually lands in R2. A kfunc returning more than 8 bytes hands the second half of the result back in RDX, the native x86-64 ABI's second return register. BPF R0 maps to RAX so it needs no move, but BPF R2 maps to RSI, so emit a RDX->RSI move after a BPF_PSEUDO_KFUNC_CALL whose function model reports ret_size > 8. Placing the second return half into R2 is possible on any JIT, but it needs architecture-specific JIT work. Rather than requiring every JIT to implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair() capability, defaulting to false in the generic core; an architecture opts in once its JIT handles the R0:R2 pair, and the remaining ones are left for future work. The verifier enforces it in bpf_add_kfunc_call(), rejecting a kfunc whose return is larger than 8 bytes with -EOPNOTSUPP when the JIT lacks the capability. Only x86, arm64 and riscv are supported so far. On arm64 and riscv the native second return register is already BPF R2 (x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2 register pair on return with no extra move, unlike x86 (RDX->RSI). This has been tested on x86 and arm64. The riscv path is expected to work by the same register-mapping reasoning as arm64 but has not been tested. bpf_add_kfunc_call() also rejects a kfunc that is marked KF_FASTCALL and returns more than 8 bytes. The bpf_fastcall contract implemented by mark_fastcall_pattern_for_call() assumes a call clobbers R0 plus the registers holding its arguments, so a return in the R0:R2 pair would clobber an R2 the caller expects the fastcall pattern to preserve. Such a kfunc is rejected with -EOPNOTSUPP as well. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
A function returning a value larger than 8 bytes (a struct/union, or an __int128) uses R2 as a second return register alongside R0. Precision backtracking treats only R0 as a return register at a call/return boundary, so once the verifier starts modeling R2 that way, marking the second half of such a return precise would trip the "unexpected regs" checks in backtrack_insn() and reject a valid program with -EFAULT. Handle it here, ahead of the patch that introduces the modeling. Marking the upper half precise, for example by branching on it after a call to a static subprogram, walks backtracking into the callee and reaches its BPF_EXIT with R2 still set in the mask. R2 is part of BPF_REGMASK_ARGS, so this hits "backtracking exit unexpected regs". Returning the pair from a global subprogram or from a kfunc instead hits the equivalent check at the call site. Handle R2 like R0 in the three boundaries where a call defines the return registers: - static subprog exit (BPF_EXIT): when the callee returns a pair, R2 is a return register rather than a clobbered argument, so its precision has to cross the frame boundary just like R0's: clear it from the caller's mask before the R1-R5 check, then set it again in the callee's mask after bt_subprog_enter(). The clear has to be conditional, which is why the subprogram containing the exit insn is looked up and queried. For a callee that does not return a pair, check_func_call() has already invalidated the caller's R1-R5 and prepare_func_exit() copies back only R0, so nothing after the call can depend on R2 and backtracking should never still be asking for it here. Clearing it unconditionally would turn that into a silent no-op instead of reporting it through the existing "backtracking exit unexpected regs" check. - global subprog call: a global subprog returning >8 bytes also sets R2; clear it before the args check. - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8) also sets R2; clear it like R0. All three are gated on R2 actually being in the mask, so the extra BTF and kfunc descriptor lookups stay off the common backtracking path. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a struct/union or an __int128) reads R2 as well as R0, since the second half of the return value is passed back in R2. compute_insn_live_regs() only marked R0 used at exit, so a callee's R2 could be considered dead and cleaned from checkpointed states, which would allow unsound state pruning. Mark R2 as read at the BPF_EXIT of a subprogram that does return a register pair. bpf_compute_live_registers() walks the instructions in order and env->subprog_info[] is sorted by subprogram start, so the containing subprogram is tracked with a running index and its return convention is queried once per subprogram through bpf_ret_reg_pair(). Marking R2 at every exit instead would be simpler, but R2 would then stay live backwards across any call that is not followed by a write to R2, which is nearly every program, and would needlessly hurt state pruning. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
A callback handed to a helper or a kfunc (bpf_loop(), bpf_timer_set_callback(), bpf_for_each_map_elem(), ...) is invoked through bpf_callback_t, and an exception callback is invoked by bpf_throw() through u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64); Both prototypes yield a single u64 in R0, and neither caller has any notion of a second return register, so a callback returning a value in the R0:R2 pair would have the upper half of its return value silently dropped. Reject both at load time: - check_ld_imm(): a callback is materialized as PTR_TO_FUNC by an ld_imm64 pointing at its subprogram, so the subprogram's return convention can be checked where the callback pointer is created, before it ever reaches a helper or kfunc argument. - do_check_common(): an exception callback is not referenced by a PTR_TO_FUNC, it is named by a BTF decl_tag and verified on its own, so check it as its frame is set up, next to the existing "cannot return void" and single-argument checks. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
LLVM 23 added support for returning a value in two registers for an __int128, or a struct/union whose size is greater than 8 but not more than 16 bytes. See LLVM patches [1] and [2]. Before LLVM 23 the BPF backend could not return these values at all. A by-value struct or union return (of any size) was rejected at compile time with: error: aggregate returns are not supported and an __int128 return failed later in the backend with: fatal error: error in backend: unable to allocate function return #1 Both are resolved in LLVM 23, which lowers such returns into the R0:R2 register pair. This patch adds handling for returns greater than 8 bytes in several places: BPF subprogram returns (the main program, and both global and static subprograms) and kfunc returns. The R0:R2 convention is only implemented in the JIT. The BPF interpreter has no notion of a second return register: a BPF-to-BPF call goes through JMP_CALL_ARGS and a BPF_EXIT hands back BPF_R0 alone, so a caller reading R2 would see a stale value. Force the JIT wherever a caller can observe the pair, that is at the call to a global subprogram in check_func_call() and at the return from a static subprogram in prepare_func_exit(). Kfunc calls need no separate handling since bpf_add_kfunc_call() already sets jit_required for every kfunc call. A by-value struct or union returned by a kfunc must be composed only of scalars, since the verifier models the returned register bits as an unknown scalar and a pointer field would otherwise be laundered into one, escaping provenance and reference tracking. A global subprogram must return a scalar in every return register. The existing exemption for arena pointers now applies only when the return value fits in R0 alone: both halves of a register pair carry a piece of a >8 byte scalar, so an arena pointer in either of them is a leak rather than a legitimate return value. A subprogram whose whole return value is an arena pointer is unaffected. A static subprogram is handled differently. The verifier walks into its frame, so prepare_func_exit() propagates the return register(s) to the caller. R0 holding a stack pointer has long been rejected outright there, but R2 is deliberately not treated the same way. LLVM owns both sides of a static call and is not bound by the ABI, so even with a 9..16 byte declared return type it may leave R2 untouched when the caller only consumes the low half; R2 can then hold an incidental stack pointer that is not a return value at all, and rejecting the program would be a false positive. Propagating the register as is would be worse: the callee frame is freed immediately afterwards, leaving the caller with a PTR_TO_STACK that refers to a frame which no longer exists. So the caller's R2 is marked uninitialized instead, and only a caller that actually reads the returned upper half fails. As with R0, a pointer into the caller's own frame is scrubbed too, which is conservative but keeps the two registers consistent. Once callers read R0:R2, an extension program can no longer replace a function with a >8 byte return value: an extension's own return is capped at 8 bytes by the program-exit check above, so it would leave R2 stale for the target's callers. btf_check_type_match() cannot catch this, as it compares return types by btf_type->info only and an int carries no vlen, so a 16-byte __int128 and an 8-byte long compare equal. Reject such an attach in bpf_check_attach_target() instead. [1] llvm/llvm-project#190894 [2] llvm/llvm-project#206876 Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
The R0:R2 return convention is derived from the BTF function prototype: bpf_compute_subprog_ret_regs() inspects the return type of every subprogram and records whether its value comes back in a register pair. btf_check_subprog_call() can decide, at a call site, that this BTF is not to be trusted and mark the subprogram unreliable, which happens when compiler optimizations remove arguments from a static function or when a mismatched type is passed to a global one. From that point on the verifier falls back to conservative, R0-only, semantics for the subprogram, while the compiled code keeps returning a pair and leaves the upper half in R2 behind the verifier's back. Rather than silently mistracking R2, reject a return value larger than 8 bytes as soon as the prototype it was derived from becomes unreliable. Add subprog_ret_pair_unreliable() and test it at the two places that can observe the flag: check_func_call(), for the call itself, and prepare_func_exit(), for the return from an inlined static subprogram. Note that the main program needs no such check: a >8 byte return from subprog 0 is rejected at BPF_EXIT regardless of whether its BTF is reliable. Callbacks need none either: a callback address only becomes a PTR_TO_FUNC through check_ld_imm(), which already rejects any callback returning more than 8 bytes. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Relax btf_distill_func_proto() to accept a by-value struct or union that the R0:R2 convention added in earlier patches can carry: - a struct or union larger than 8 and up to 16 bytes, returned in the R0:R2 register pair, matching what LLVM emits for the BPF target; - a struct or union up to 8 bytes, returned in R0 alone. A >8 byte scalar (__int128) was already accepted and is unchanged. Everything else stays rejected: a return type larger than 16 bytes, and any type that __get_type_size() cannot return in registers at all (e.g. an array), which it already reports as ret < 0. btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret) and struct_ops function models, so relaxing it widens what those can attach to. A >8 byte return stays rejected on every path that reads the target's return value: commit c48796a ("bpf: Reject >8 byte return values on return-reading trampoline paths") covers fexit, fmod_ret and fsession plus their _multi variants, and struct_ops, and an fentry-only trampoline never sets BPF_TRAMP_F_CALL_ORIG so it does not touch the return value at all. A struct or union of 8 bytes or less is newly accepted for those paths; its single eightbyte is returned in R0 like any other scalar. btf_validate_return_type() is relaxed as well, so that it accepts a by-value struct or union up to 16 bytes in addition to void and scalars. With btf_distill_func_proto() and btf_validate_return_type() relaxed, the verifier, JIT, precision-backtracking and live-register support from the earlier patches becomes reachable: this final patch enables <=16 byte aggregate return values end to end. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Add selftests that exercise a 16-byte return value passed in the R0:R2 register pair, written in C so that they depend on the compiler lowering the register-pair return. Covered are an __int128 return, a 16-byte struct return (from a static and from a global subprogram) and a 16-byte union return, plus __int128 and 16-byte struct returns from a kfunc, for which bpf_kfunc_call_test_i128() and bpf_kfunc_call_test_ret_pair() are added to bpf_testmod. The R0:R2 convention is only emitted by LLVM 23 and newer. Each object records in a read-only has_reg_pair_ret flag which compiler built it; where that is false the programs are stubs and subtests report a skip rather than a pass. The two kfunc subtests further depend on the JIT: bpf_add_kfunc_call() rejects a kfunc returning more than 8 bytes with -EOPNOTSUPP where bpf_jit_supports_kfunc_ret_reg_pair() is false. Those calls therefore live in an object of their own, and that load failing with -EOPNOTSUPP is what turns the two subtests into skips, so no list of the JITs implementing the pair needs to be kept here. A register-pair return from a BPF subprogram needs no JIT support, so the remaining subtests run everywhere. Both kfuncs are restricted to x86_64 and arm64, and the two subtests report a skip elsewhere. pahole only BTF-encodes a function whose declared arguments sit in the ABI's argument registers, and an architecture that returns a value larger than 8 bytes through a hidden pointer (sret) shifts every one of them by a register. s390x is such an architecture: there pahole drops the function, resolve_btfids leaves the kfunc ID at 0, and register_btf_kfunc_id_set() then fails at module init, so bpf_testmod does not load at all and every test that needs it fails. This is not a property of the compiler -- __SIZEOF_INT128__ is defined by gcc on s390x, and a by-value struct return has nothing to do with __int128 in the first place -- so the guard is on the architecture. prog_tests/tracing_failure.c already restricts a __int128 return the same way. Signed-off-by: Yonghong Song <yonghong.song@linux.dev> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Add inline-asm tests, which do not depend on the compiler lowering a register-pair return and so run regardless of the LLVM version, covering what the C tests cannot reach. aggregate_ret_func.c exercises BPF-to-BPF returns: a global subprogram whose R2 the caller may read, ones that leave R2 uninitialised or holding a pointer, a static subprogram whose R2 stays precise under backtracking, R2 liveness across a call, and a >8 byte return at program exit. Six kfuncs returning aggregates by value are added to bpf_testmod, and aggregate_ret_run.c calls them from inline asm to check what comes back in R0:R2. A negative arena test is added as well: a global subprogram with a register-pair return that leaves an arena pointer in R2 is rejected, since an arena pointer is only a valid return value when it is returned in R0 alone. Three cases cover the boundaries of the new convention: - A return value larger than 16 bytes does not fit in R0:R2 and is rejected by btf_distill_func_proto(), ahead of the KF_FASTCALL and JIT-capability checks; one of the new kfuncs returns a 24-byte struct for this. The equivalent for a BPF subprogram cannot be written in C: from LLVM 23 on, a by-value return larger than 16 bytes is lowered to an sret pointer argument and the BTF the verifier reads says the function returns void, so the size bound in btf_validate_return_type() only guards hand-crafted BTF. - A static subprogram returning a struct that contains a pointer is accepted, and the caller can use the returned pointer. Unlike a global subprogram, whose caller models the return as an opaque scalar pair, a static one is verified inline, so prepare_func_exit() hands the caller real register state and the pointer stays tracked. - An extension cannot replace a function returning more than 8 bytes. btf_check_type_match() does not catch this, since it compares return types by btf_type->info alone and both an __int128 and a __u64 are BTF_KIND_INT with no vlen, so the rejection has to come from bpf_check_attach_target(). The test reuses the freplace failure harness in fexit_bpf2bpf.c, with aggregate_ret_target.c providing a target whose global subprogram returns in R0:R2. The kfunc tests need the JIT to place the second half of a return value into R2, which bpf_add_kfunc_call() only allows where bpf_jit_supports_kfunc_ret_reg_pair() is true. In aggregate_ret_kfunc.c the two tests that depend on getting past that check are tagged __arch_x86_64/__arch_arm64/__arch_riscv64; the others are rejected earlier (on KF_FASTCALL, on a >16 byte return, and on reading R2 after an 8-byte struct return). In aggregate_ret_run.c the kfunc-calling programs are dropped from the object when the load reports -EOPNOTSUPP and their subtests are skipped, and the __int128 inline-asm test is split into a BPF-to-BPF half, which needs no JIT capability and runs everywhere, and a kfunc half. Five of the six new kfuncs return a struct or union by value and take arguments, so they join the x86_64/arm64 guard added in the previous patch; see the comment there. That includes the one returning only 8 bytes: s390x hands back a by-value aggregate through an sret pointer whatever its size, so its declared arguments are displaced just the same and pahole skips the function. The sixth returns 24 bytes but takes no argument, leaving nothing for the sret pointer to displace, and pahole does encode it there. Both objects calling the guarded kfuncs reference every one of them from __kfunc_btf_root(), so neither can load at all where those kfuncs are absent; the aggregate_ret_kfunc tests and the four aggregate_ret_run subtests are skipped as a group elsewhere. aggregate_ret_func.c calls no kfunc and keeps running everywhere. R0 holds bytes 0..7 of a return value and R2 bytes 8..15, so where a member sits inside a register depends on the endianness of the target. The checks in aggregate_ret_run.c that read a member out of half a register are built for a little-endian target only; arm64 is the one JIT implementing the pair that can be built big endian, and that configuration is left for later. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Add two __failure tests covering the callback return-size checks: - timer_ret_pair_fail: a bpf_timer callback declared to return more than 8 bytes, rejected by check_ld_imm() where the callback's PTR_TO_FUNC is created, with "callback function with >8-byte return value is not supported". - exceptions_ret_pair_fail: an exception callback declared to return more than 8 bytes, rejected by do_check_common() when the callback subprogram is verified, with "exception cb cannot return value larger than 8 bytes". Both callback bodies are written in inline asm so that the tests do not depend on LLVM 23 R0:R2 codegen and run on any compiler. The verifier reads the return type from BTF rather than from the instructions, so the >8 byte return prototype is supplied through __btf_func_path(), pointing at a companion btf__*.c program that exists only to carry that BTF. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a scalar-only struct or union, or an __int128), passed back in the R0:R2 register pair. Add a kfunc return-value section documenting this, including that a struct or union up to 8 bytes is returned in R0 alone, which struct and union members are accepted, that the R0:R2 register pair requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a return value larger than 16 bytes is unsupported. Also note that the same convention applies to BPF subprogram returns, and document the consequence for a global subprogram: it must assign both halves of a register-pair return, since an unassigned R2 may be left holding a pointer argument and is then rejected as a leak. Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Author
|
Upstream branch: 07cb86a |
kernel-patches-daemon-bpf-rc
Bot
force-pushed
the
series/1143715=>bpf-next
branch
from
August 11, 2026 22:11
a506b60 to
c730b74
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: bpf: Support aggregate return values up to 16 bytes
version: 4
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=1143715