Problems SMP an new SoC(ARM64: Cortex-A53) based board.

[dylan-hong-nc]

Environment

• HOST : ubnutu-20.04(x86_64)
• Zephyr:
  • sdk-version: v0.17.0
  • version: v4.1.0
  • note: zephyr rtos operates in non-secure area and activates secondary core with psci via TF-A (BL31).
    For reference, the same TF-A is shared and used by Linux and commercial rtos.
• SoC:
  • vendor: Nextchip
  • model: nvs2900(apache5)
    • core: cortex-a53

Describe Problem

Hello, this is my first time contacting you.
We are porting zephyr v4.1.0 to our SoC and are having trouble with SMP bring-up.

For testing, I built and ran the test code in the following path:
tests/kernel/smp

However, the test stops after the log below.
The location where the test stops is different each time you test.

NOTICE:  DRAM_FREQ: 1866 [MHz]
NOTICE:  EXT OSC     : 24 MHz
NOTICE:  DDR DENSITY : 2048 MB
NOTICE:  BL31: v2.10.4(release):v2.7.0-3025-g018cbe6ae-dirty
NOTICE:  BL31: Built : 15:18:34, Jun 20 2025
NOTICE:  APACHE5 SIP SVC Version 0.8
NOTICE:  verify header done - rsd: 2, sclk: 67000000, pol(H)-pha(H), quad
NOTICE:  leave nc_loader_read_and_verify_header...


U-Boot 2025.04-00011-g9289c08cd026 (Jun 24 2025 - 14:41:08 +0900)

DRAM:  2 GiB
Core:  39 devices, 16 uclasses, devicetree: fit
MMC:   mmc@34000000: 0, mmc@35000000: 1
Loading Environment from SPIFlash... default rx sample delay: 2
SF: Detected mt25ql512a with page size 256 Bytes, erase size 4 KiB, total 64 MiB
OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: gmac0@43020000
Hit any key to stop autoboot:  0
PHY Reset
MEGACHIPS MAAE1003S ID OK : 0x1a d414
Using gmac0@43020000 device
TFTP from server 192.168.13.31; our IP address is 192.168.13.35
Filename 'dylan7h/zephyr/zephyr.bin'.
Load address: 0x80000000
Loading: #####################
         4.5 MiB/s
done
Bytes transferred = 103016 (19268 hex)
## Starting application at 0x80000000 ...
*** Booting Zephyr OS build v4.1.0-27-g334c975fffdd ***
Secondary CPU core 1 (MPID:0x1) is up
Secondary CPU core 2 (MPID:0x2) is up
Secondary CPU core 3 (MPID:0x3) is up
Running TESTSUITE smp
===================================================================
START - test_coop_resched_threads
 PASS - test_coop_resched_threads in 0.051 seconds
===================================================================
START - test_coop_switch_in_abort
 PASS - test_coop_switch_in_abort in 0.201 seconds
===================================================================
START - test_cpu_id_threads
 PASS - test_cpu_id_threads in 1.001 seconds
===================================================================
START - test_fatal_on_smp
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x010000008002a0d0
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x0000000000000040
E: x4:  0x0000000000000001  x5:  0x0000000000000000
E: x6:  0x0000000000000000  x7:  0x0000000000000000
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012dbe0  x11: 0x0000000080002f98
E: x12: 0x000000008012dc38  x13: 0x0000000000000000
E: x14: 0x0000000000000000  x15: 0xfffffff7ffdfffff
E: x16: 0x0000000000000000  x17: 0x0000000000000000
E: x18: 0x0000000000000000  lr:  0x0000000080002f90
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
E: Current thread: 0x80027e00 (unknown)
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x010000008002a110
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x000000000000032b
E: x4:  0x0000000080015838  x5:  0x0000000080014912
E: x6:  0x0000000000000001  x7:  0x0000000000000002
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012e7e0  x11: 0x0000000080002f98
E: x12: 0x000000008012e838  x13: 0x000000008002a000
E: x14: 0x000000008012e730  x15: 0x0000000000000000
E: x16: 0x0000000000000001  x17: 0x00000000000000b1
E: x18: 0x0000000080028180  lr:  0x0000000080001d64
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 1
E: Current thread: 0x80028180 (test_fatal_on_smp)
 PASS - test_fatal_on_smp in 0.321 seconds
===================================================================
START - test_get_cpu
 PASS - test_get_cpu in 0.051 seconds
===================================================================
START - test_inc_concurrency
type 0: cnt 60000, spend 15 ms
type 1: cnt 60000, spend 570 ms
type 2: cnt 60000, spend 565 ms

After quite some time of debugging, I found out via jtag debugger that the flush_owned_fpu function in arch/arm64/core/fpu.c was stuck in an infinite loop due to while (atomic_ptr_get(&_kernel.cpus[i].arch.fpu_owner) == thread).
I inserted the following memory dump code to analyze the cause.

I checked the buffer printed to the memory and confirmed that it was recorded as follows.
(I set it to record all cores from 0 to 3, but the cores actually recorded were 0 and 1.)

[CPU-0]

[CPU-1]

I ran the test with FPU_SHARING disabled with CONFIG_FPU_SHARING=n because fpu_ipi didn't seem to work properly. The test went through to the end as follows:
However, the test_smp_ipi test fails.
(If I repeat the test, it often succeeds.)

NOTICE:  DRAM_FREQ: 1866 [MHz]
NOTICE:  EXT OSC     : 24 MHz
NOTICE:  DDR DENSITY : 2048 MB
NOTICE:  BL31: v2.10.4(release):v2.7.0-3025-g018cbe6ae-dirty
NOTICE:  BL31: Built : 15:18:34, Jun 20 2025
NOTICE:  APACHE5 SIP SVC Version 0.8
NOTICE:  verify header done - rsd: 2, sclk: 67000000, pol(H)-pha(H), quad
NOTICE:  leave nc_loader_read_and_verify_header...


U-Boot 2025.04-00011-g9289c08cd026 (Jun 24 2025 - 14:41:08 +0900)

DRAM:  2 GiB
Core:  39 devices, 16 uclasses, devicetree: fit
MMC:   mmc@34000000: 0, mmc@35000000: 1
Loading Environment from SPIFlash... default rx sample delay: 2
SF: Detected mt25ql512a with page size 256 Bytes, erase size 4 KiB, total 64 MiB
OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: gmac0@43020000
Hit any key to stop autoboot:  0
PHY Reset
MEGACHIPS MAAE1003S ID OK : 0x1a d414
Using gmac0@43020000 device
TFTP from server 192.168.13.31; our IP address is 192.168.13.35
Filename 'dylan7h/zephyr/zephyr.bin'.
Load address: 0x80000000
Loading: ####################
         4.3 MiB/s
done
Bytes transferred = 98888 (18248 hex)
## Starting application at 0x80000000 ...
*** Booting Zephyr OS build v4.1.0-27-g334c975fffdd ***
Secondary CPU core 1 (MPID:0x1) is up
Secondary CPU core 2 (MPID:0x2) is up
Secondary CPU core 3 (MPID:0x3) is up
Running TESTSUITE smp
===================================================================
START - test_coop_resched_threads
 PASS - test_coop_resched_threads in 0.051 seconds
===================================================================
START - test_coop_switch_in_abort
 PASS - test_coop_switch_in_abort in 0.201 seconds
===================================================================
START - test_cpu_id_threads
 PASS - test_cpu_id_threads in 1.001 seconds
===================================================================
START - test_fatal_on_smp
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x0100000080027230
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x0000000000000040
E: x4:  0x0000000000000001  x5:  0x0000000000000000
E: x6:  0x0000000000000000  x7:  0x0000000000000000
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012afd0  x11: 0x0000000080002f98
E: x12: 0x000000008012b028  x13: 0x0000000000000000
E: x14: 0x0000000000000000  x15: 0xffffffffffffffff
E: x16: 0x0000000000000000  x17: 0x0000000000000000
E: x18: 0x0000000000000000  lr:  0x0000000080002f90
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
E: Current thread: 0x80026778 (unknown)
E: ELR_ELn: 0x00000000800014e0
E: ESR_ELn: 0x0000000056000002
E:   EC:  0x15 (Unknown)
E:   IL:  0x1
E:   ISS: 0x2
E: TPIDRRO: 0x0100000080027258
E: x0:  0x0000000000000000  x1:  0x0000000000000000
E: x2:  0x0000000000000000  x3:  0x000000000000032b
E: x4:  0x0000000080014838  x5:  0x0000000080013912
E: x6:  0x0000000000000001  x7:  0x0000000000000002
E: x8:  0x0000000000000003  x9:  0x0000000000000000
E: x10: 0x000000008012bbd0  x11: 0x0000000080002f98
E: x12: 0x000000008012bc28  x13: 0x0000000080027000
E: x14: 0x000000008012bb20  x15: 0x0000000000000000
E: x16: 0x0000000000000001  x17: 0x00000000000000b1
E: x18: 0x0000000080026948  lr:  0x0000000080001d64
E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 1
E: Current thread: 0x80026948 (test_fatal_on_smp)
 PASS - test_fatal_on_smp in 0.321 seconds
===================================================================
START - test_get_cpu
 PASS - test_get_cpu in 0.051 seconds
===================================================================
START - test_inc_concurrency
type 0: cnt 60000, spend 15 ms
type 1: cnt 60000, spend 547 ms
type 2: cnt 60000, spend 547 ms
 PASS - test_inc_concurrency in 1.116 seconds
===================================================================
START - test_preempt_resched_threads
 PASS - test_preempt_resched_threads in 0.512 seconds
===================================================================
START - test_sleep_threads
 PASS - test_sleep_threads in 1.001 seconds
===================================================================
START - test_smp_affinity
 PASS - test_smp_affinity in 0.001 seconds
===================================================================
START - test_smp_coop_threads
 PASS - test_smp_coop_threads in 0.613 seconds
===================================================================
START - test_smp_ipi
cpu num=4
    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/smp/src/main.c:753: smp_test_smp_ipi: (sched_ipi_has_called != 0 is false)
did not receive IPI.(0)
 FAIL - test_smp_ipi in 0.115 seconds
===================================================================
START - test_smp_release_global_lock
 PASS - test_smp_release_global_lock in 0.021 seconds
===================================================================
START - test_smp_switch_torture
 PASS - test_smp_switch_torture in 15.001 seconds
===================================================================
START - test_wakeup_threads
 PASS - test_wakeup_threads in 0.201 seconds
===================================================================
START - test_workq_on_smp
 PASS - test_workq_on_smp in 0.051 seconds
===================================================================
START - test_yield_threads
 PASS - test_yield_threads in 0.301 seconds
===================================================================
TESTSUITE smp failed.

------ TESTSUITE SUMMARY START ------

SUITE FAIL -  93.75% [smp]: pass = 15, fail = 1, skip = 0, total = 16 duration = 20.558 seconds
 - PASS - [smp.test_coop_resched_threads] duration = 0.051 seconds
 - PASS - [smp.test_coop_switch_in_abort] duration = 0.201 seconds
 - PASS - [smp.test_cpu_id_threads] duration = 1.001 seconds
 - PASS - [smp.test_fatal_on_smp] duration = 0.321 seconds
 - PASS - [smp.test_get_cpu] duration = 0.051 seconds
 - PASS - [smp.test_inc_concurrency] duration = 1.116 seconds
 - PASS - [smp.test_preempt_resched_threads] duration = 0.512 seconds
 - PASS - [smp.test_sleep_threads] duration = 1.001 seconds
 - PASS - [smp.test_smp_affinity] duration = 0.001 seconds
 - PASS - [smp.test_smp_coop_threads] duration = 0.613 seconds
 - FAIL - [smp.test_smp_ipi] duration = 0.115 seconds
 - PASS - [smp.test_smp_release_global_lock] duration = 0.021 seconds
 - PASS - [smp.test_smp_switch_torture] duration = 15.001 seconds
 - PASS - [smp.test_wakeup_threads] duration = 0.201 seconds
 - PASS - [smp.test_workq_on_smp] duration = 0.051 seconds
 - PASS - [smp.test_yield_threads] duration = 0.301 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION FAILED

I tried running the shell (not smp test) with FPU_SHARING disabled and the options below.

CONFIG_SHELL=y
CONFIG_SHELL_BACKEND_SERIAL=y

As an experiment, I tried entering kernel sleep 100 as shell input, but it did not wake up from sleep.

However, I confirmed that if I apply the scheduling options below, it wakes up normally regardless of the sleep value.

CONFIG_TIMESLICE_PER_THREAD=y
CONFIG_TIMESLICE_SIZE=1
CONFIG_TICKET_SPINLOCKS=y

Unfortunately, I tried re-enabling FPU_SHARING while keeping this option enabled, but the result was the same.

What else can I test?
Does SMP work properly on other ARM64 based boards?
I've attached some of my config files.

Please help,
Thank you.

zephyr_smp_debug.zip

++++++++++++++++++++++++++
I tried to reflect the changes ARM64_DCACHE_ALL_OPS, ARM64_BOOT_DISABLE_DCACHE and arm/arm64/core/switch.S from the main branch to my workspace, but it didn't give me any meaningful results.

[arm/arm64/core/switch.S]
#ifdef CONFIG_SMP
	/* Write barrier: ensure all preceding writes are executed
	* before writing the switch handle
	*/
	dmb sy
#endif

When jumping from u-boot to zephyr I used the following commands:

dcache flush; icache flush; dcache off; icache off; go ${image_addr_r}