[rocjitsu] B0-to-A0 translation rejects a valid lane-saved `s_set_pc_i64` return

[RattataKing]

[rocjitsu][gfx1250] B0-to-A0 translation rejects a valid lane-saved s_set_pc_i64 return

Summary

RocJitsu's gfx1250 B0-to-A0 translator rejects device-linked CTS code objects
when a non-leaf function saves its incoming return PC in VGPR lanes, makes a
nested call, restores the return PC, and returns through s_set_pc_i64.

The first failure is:

error: legalization .text+0x2264 s_set_pc_i64:
indirect branch or call target recovery is not implemented for relocated kernel text

The instruction is decoded correctly, and the surrounding machine code is a
valid compiler-generated call/return sequence. RocJitsu fails to classify the
final s_set_pc_i64 s[30:31] as a recovered return and rejects the entire code
object.

This affects 22 of the 29 gfx1250 CTS cases in the attached failure set. The
issue is reproducible in rj_dbt_translate without pytest, CTest, ROCr, or the
hotswap plugin, which isolates it to RocJitsu translation.

Environment

rocm-systems commit: e80ab425ea9087a2b4a6e27c6fa97bb476464a48
ROCm SDK:            10.1.0a20260812
host:                Linux 7.0.0-28-generic x86_64
translation:         gfx1250 B0 -> gfx1250 A0

Reproduction through the CTS corpus

Use the checkout-built hotswap plugin explicitly so ROCr cannot select an SDK-adjacent
copy. Run this from a workspace containing the rocm-systems,
rocjitsu-build, and rocjitsu-test-corpus directories:

WORKSPACE="$(pwd)"
ROCM_SYSTEMS_SOURCE_DIR="${WORKSPACE}/rocm-systems"
ROCJITSU_BUILD_DIR="${WORKSPACE}/rocjitsu-build"
ROCM_PATH="$(rocm-sdk path --root)"
export ROCM_PATH ROCM_SYSTEMS_SOURCE_DIR ROCJITSU_BUILD_DIR
export LD_LIBRARY_PATH="${ROCM_PATH}/lib:${LD_LIBRARY_PATH:-}"
mkdir -p /tmp/rocjitsu-cts-dump

env \
  -u LD_PRELOAD \
  -u HSA_HOTSWAP_ENABLE \
  HSA_HOTSWAP_DISABLE=1 \
  HSA_TOOLS_DISABLE_REGISTER=1 \
  HSA_TOOLS_LIB="${ROCJITSU_BUILD_DIR}/lib/rocjitsu/src/rocjitsu/hooks/libhsa_hotswap_rocjitsu.so" \
  HSA_HOTSWAP_VERBOSE=1 \
  HSA_HOTSWAP_DUMP_SOURCE=1 \
  HSA_HOTSWAP_DUMP_DIR=/tmp/rocjitsu-cts-dump \
  "${ROCJITSU_BUILD_DIR}/tools/rocjitsu/rocjitsu" \
    --config "${ROCM_SYSTEMS_SOURCE_DIR}/emulation/rocjitsu/configs/gfx1250.json" \
    -- \
    pytest rocjitsu-test-corpus/tests/test_corpus.py \
      --target gfx1250 \
      --suite cts \
      --case cts.gfx1250.fpsan.fpsan_amdgcn_ldexp_test \
      --case cts.gfx1250.fpsan.fpsan_amdgcn_math_extra_test \
      --case cts.gfx1250.fpsan.fpsan_amdgcn_math_test \
      --case cts.gfx1250.fpsan.fpsan_atomic_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_scalef32_pk_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_hip_device_test \
      --case cts.gfx1250.fpsan.fpsan_swmmac_16x16x128_fp8_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_swmmac_16x16x64_16_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_amdgcn_bf16_trans_test \
      --case cts.gfx1250.fpsan.fpsan_classify_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_f16_fp8_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_fp8_e5m3_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_fp8_pack_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_fp8_pk_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_fp8_sr_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_fp8_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_scale_unpack_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_sr_pack_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_sr_pk_f16_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_cvt_test \
      --case cts.gfx1250.fpsan.fpsan_wave_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_16x16x4_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_16x16x128_f8f6f4_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_16x16x128_fp8_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_16x16x32_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_16x16x64_fp8_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_32x16x128_f4_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_wmma_scale_f8f6f4_gfx1250_test \
      --case cts.gfx1250.fpsan.fpsan_xlane_test \
      --timeout 15 \
      -q --tb=short -n 8

This exact selector form was verified locally. Pytest selected all 29 requested
cases without a usage or collection error and completed with:

29 failed in 31.26s

Observed hotswap output:

[hsa-hotswap-rj] installed eager gfx1250 B0-to-A0 hook
[hsa-hotswap-rj] error: translation diagnostic
source_id=fnv1a64:07dd843f24f66a72
severity=error
kind=translator-legalization
guest_offset=.text+0x2264
mnemonic=s_set_pc_i64
message=indirect branch or call target recovery is not implemented for relocated kernel text
[hsa-hotswap-rj] error: eager translation
input_revision=b0 output_revision=a0
outcome=translation_failed changed=0
translation_status=4 status=4112

CTest subsequently labels the process as SEGFAULT, but translation has already
failed before the CTS correctness checks run.

Standalone translator reproduction

Attach the code object captured by HSA_HOTSWAP_DUMP_SOURCE=1 to this issue and
run:

rj_dbt_translate \
  rocjitsu-gfx1250-b0-to-a0-07dd843f24f66a72.elf \
  --input-target gfx1250 \
  --output-target gfx1250 \
  --input-revision b0 \
  --output-revision a0 \
  --output-mode diff

Observed result:

error: legalization .text+0x2264 s_set_pc_i64: indirect branch or call target recovery is not implemented for relocated kernel text
rj_dbt_translate: failed
source_code_object_id: fnv1a64:07dd843f24f66a72
input_revision: b0
output_revision: a0
output_elf_bytes: 525648

Captured artifact metadata:

size:   525648 bytes
sha256: b1fb43c4dc7541dac3569c6d5473bd77f75512d55f31848d02fbba87cbad82fe

This standalone reproduction does not load ROCr or the hotswap plugin.

Relevant machine code

The code object's .text section has VMA 0x38700, making
.text+0x2264 address 0x3a964. That address is the final instruction of
__ockl_fprintf_stderr_begin.

The caller constructs the callee's concrete address and calls it through
s_swap_pc_i64:

000000000003ec60: s_get_pc_i64  s[0:1]
000000000003ec64: s_add_nc_u64  s[0:1], s[0:1], 0xffffffffffffbc1c
000000000003ec70: s_swap_pc_i64 s[30:31], s[0:1]

The computed target is 0x3a880, the symbol
__ockl_fprintf_stderr_begin.

The callee saves the incoming return pair, makes a nested call, restores it,
and returns:

000000000003a8ac: v_writelane_b32 v24, s30, 0
000000000003a8b4: v_writelane_b32 v24, s31, 1
000000000003a8bc: s_get_pc_i64    s[0:1]
000000000003a8c0: s_add_nc_u64    s[0:1], s[0:1], 0xfffffffffffffe40
000000000003a910: s_swap_pc_i64   s[30:31], s[0:1]
...
000000000003a930: v_readlane_b32  s30, v24, 0
000000000003a938: v_readlane_b32  s31, v24, 1
...
000000000003a964: s_set_pc_i64    s[30:31]

LLVM decodes the final word 0xBE80481E as
s_set_pc_i64 s[30:31], so this is not an instruction-decoder failure.

Expected behavior

RocJitsu should prove that the final s_set_pc_i64 s[30:31] is the return
corresponding to the recovered call, preserve or relocate the call/return
semantics, and continue translating the code object.

If the current translation strategy cannot safely support this pattern, the
diagnostic should identify the unsupported return-address preservation pattern
rather than treating it as an arbitrary unresolved branch. The preferred fix is
to support this compiler-generated sequence so the CTS reaches its correctness
checks.

Suspected RocJitsu path

The intended flow appears to be:

1. discover_indirect_branch_edges() proves bounded s_getpc-derived targets.
2. BasicBlock::build() creates recovered indirect call metadata and call edges.
3. scoped_call_return_offsets() identifies matching s_set_pc_i64 returns.
4. BinaryTranslator::translate_impl() admits offsets present in
   valid_call_return_offsets.

Relevant locations:

emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/analysis/indirect_branch_discovery.cpp:35
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/analysis/indirect_branch_discovery.h:139
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/basic_block.cpp:196
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/basic_block.cpp:350
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/dbt/scoped_cfg_edges.cpp:48
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/dbt/scoped_cfg_edges.cpp:137
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/dbt/binary_translator.cpp:1210
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/dbt/binary_translator.cpp:3423
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/code/dbt/binary_translator.cpp:3883

At the final guard in binary_translator.cpp, this instruction has no recovered
call, relocation-table call, recovered return, direct branch delta, or
whole-object relocated-address proof. The translator emits the observed
diagnostic because .text+0x2264 was not added to
valid_call_return_offsets.

One likely false-negative is the conservative lane-stash rule in
adopted_root_return_offsets(). It rejects a return address saved across a
nested call unless the carrier VGPR is ABI callee-saved:

if (is_call && !is_callee_saved_vgpr(vgpr))
  return false;

The CTS sequence uses v24, while the current predicate accepts its
callee-saved pattern beginning at v40:

bool is_callee_saved_vgpr(uint16_t phys_vgpr) {
  return phys_vgpr >= 40 && phys_vgpr <= 255 && ((phys_vgpr - 40) % 16) < 8;
}

The nested callee chain in the captured object does not reference v24, so the
compiled sequence preserves the value in this object even though the ABI-only
proof cannot establish that. Instrumenting the scope/call-edge construction
would confirm whether this is the first point that loses the return proof or
whether the call and return are separated earlier during scope construction.

Impact

• At least 22 selected gfx1250 CTS executables fail before correctness testing.
• The ordinary hotswap run reports downstream process segfaults after
  translation failure.
• Setting LD_LIBRARY_PATH or selecting the branch-built plugin explicitly does
  not address this failure.
• The same code object consistently fails through the standalone translator.