bpf, arm64: __arena kfunc and struct_ops arguments - #8688
Open
kernel-patches-daemon-bpf-rc[bot] wants to merge 7 commits into
Open
bpf, arm64: __arena kfunc and struct_ops arguments#8688kernel-patches-daemon-bpf-rc[bot] wants to merge 7 commits into
kernel-patches-daemon-bpf-rc[bot] wants to merge 7 commits into
Conversation
Author
|
Upstream branch: d114bb9 |
kernel-patches-daemon-bpf-rc
Bot
force-pushed
the
bpf-next_base
branch
from
August 11, 2026 22:08
818432c to
d922dc0
Compare
save_args() reads stack-passed arguments relative to FP assuming the trampoline is entered through the fentry call from a traced function, in which case both the parent frame (FP/x9) and the traced function frame (FP/LR) are saved before FP is set, so the arguments start at FP + 32. An indirect trampoline for a struct_ops callback is entered through a function pointer (blr), so only the FP/LR frame is pushed and the arguments start at FP + 16, not FP + 32. Every stack-passed argument of a struct_ops callback with more than eight argument slots is read two slots off. This went unnoticed because no struct_ops member passed arguments on the stack until bpf_testmod_ops3::test_arena_stack, added by commit 2d4de9a ("selftests/bpf: Test stack-passed struct_ops arena arguments"). That member covers this on arm64 once the JIT gains arena argument support later in this series. Pass is_struct_ops into save_args() and pick the offset accordingly, mirroring the x86 fix. Fixes: 9014cf5 ("bpf, arm64: Support up to 12 function arguments") Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
The insn library encodes the immediate and shifted-register forms of ADD/SUB but not the extended-register form. The BPF JIT wants it to rebase a 32-bit arena offset onto the arena kernel base in a single instruction, add xN, xBASE, wN, uxtw, instead of a separate zero-extend followed by a plain add. Add aarch64_insn_gen_add_sub_extended_reg(), modeled on the shifted-register generator. The option and imm3 fields occupy the same bits as the shifted form's shift amount, so they are encoded through the existing IMM_6 field type. Note that register 31 does not mean the same thing in the two forms: in the extended-register encoding it is SP for Rn, and for Rd unless the instruction sets the flags, while it stays XZR for Rm. Callers porting a shifted-register site that passes A64_ZR need to be aware of that, so say so above the function. Signed-off-by: Tejun Heo <tj@kernel.org> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Implement arena argument rebasing for kfunc calls on arm64. x28 already holds kern_vm_start whenever the prog has an arena, and the newly added extended-register add zero-extends the 32-bit arena offset in place, so an unconditional argument costs a single instruction emitted right before the call: add xN, x28, wN, uxtw A nullable argument first truncates into wN so that a zero offset leaves xN holding a real NULL, then tests it and jumps over the add: mov wN, wN cbz wN, 1f add xN, x28, wN, uxtw 1: The rebase is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees it and needs no special handling. The emitted count depends only on the kfunc model, so it is identical across JIT passes. bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well. Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Implement the struct_ops arena argument conversion on arm64. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through x10 with the low half of the base materialized once into x11: sub w10, wsrc, w11 /* truncate and clear the upper 32 bits */ str x10, [sp, #slot] A nullable argument tests the full 64-bit kernel pointer first: mov x10, xsrc cbz x10, 1f sub w10, w10, w11 1: str x10, [sp, #slot] The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through x10, so only the subtraction (and the NULL test) is inserted there. The register loop now walks arguments rather than registers so that the per-argument flags line up with the slots a multi-slot argument occupies; the sequence of stores is otherwise unchanged. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline, so a tracing trampoline emits exactly what it did before and never touches x11. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction. Conversion must never reach the original function, which takes kernel addresses. That holds because BPF_TRAMP_F_INDIRECT is incompatible with BPF_TRAMP_F_CALL_ORIG, so pass 0 rather than the base to the call-origin save_args() and assert the flag combination the same way x86 does, rather than leaving the invariant to a comment. With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for arm64 and drop the x86-64-only qualifier from the kfunc documentation. Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Pin the arm64 counterparts of the x86-64 rebase sequences: the single extended-register add for an unconditional argument, the nullable truncate-test-and-skip variant, and all five argument registers in one call. The nullable cases use a local label so the branch is pinned to the instruction right after the add, and the label line does not spell out the call because arm64 emits either a direct bl or a materialize- and-blr pair depending on the distance to the kfunc. Note that on arm64 an unconditional argument is one instruction with nothing to anchor it against, so arena_arg_jit_rebase alone cannot tell the two forms apart; it only requires that nothing is emitted between the rebase and the call. The args5 test is what pins the distinction, since its four consecutive adds leave no room for a nullable truncate-and-branch pair between them. Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
The arena kfunc and struct_ops argument tests were restricted to x86-64 because it was the only JIT that implemented the conversions. arm64 does now, so let them run there too: tag every program in arena_kfunc.c with __arch_arm64 in addition to __arch_x86_64, and widen the __x86_64__ guards in the struct_ops arena test. Without this the tests report SKIP on arm64 rather than exercising the newly added JIT support. Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
…rgument The trampoline reads the __arena flag from the btf_func_model per argument but stores the ctx one register slot at a time, so the two only line up if every preceding argument occupies exactly one slot. Every arena-bearing member of bpf_testmod_ops3 takes single-slot arguments, so nothing exercises the mapping and a mis-indexed arg_flags lookup would go unnoticed on any architecture. Add test_arena_multislot(), whose first argument is a 16-byte struct passed by value. It fills ctx[0] and ctx[1], putting the arena pointer at argument index one but slot two. The callback checks both halves of the struct before dereferencing ctx[2], so a JIT that walks registers instead of arguments converts the wrong slot and fails the test. Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Author
|
Upstream branch: 07cb86a |
kernel-patches-daemon-bpf-rc
Bot
force-pushed
the
series/1143632=>bpf-next
branch
from
August 11, 2026 22:13
2642a43 to
576a385
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: bpf, arm64: __arena kfunc and struct_ops arguments
version: 1
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=1143632