Problems booting an i.MX RT1170 based board

[tyalie]

I've an issue getting a board that uses the i.MX RT1170 chipset to boot with Zephyr. I've no problem doing so with an mcux based example project, but when trying it with Zephyr I get very random and faulty behaviour from the chip. The board is a TQ TQMa117xL, which is on the MBa117xL dev board.

I've already implemented the adaptations that need to be made for the board to the best of my knowledge (mostly what I learned from getting the chip to run with the mcux SDK in mcuXpresso). But when trying it out (even with the mimxrt1170_evk_cm7 board) Zephyr seems to crash at very weird places.

1. When attaching a (JLink) debugger, Zephyr will crash the chip when trying to set the VTOR in reset.S (first few instructions) - except when I use 'step instruction' in gdb.
2. depending on what i've in the my main.c I'll reach the start of the method or the chip will crash beforehand. I've had success landing in a forever loop in the main.c when halting for each instruction like described in (1) and continuing the program after the VTOR has been set.
3. when setting $pc to the __start address (reset handler) and continuing the program, the chip will land in the ROM bootloader memory region quite frequently (if it hasn't crashed before)

I'm assuming here that the chipset has crashed when JLink will only give me 0xdeadbeee as values for the registers.

My work so far is documented here: https://github.com/tyalie/tq-zephyr-board.
I've already written @danieldegrasse privately and he linked me here.

[DerekSnell]

Hi @tyalie ,
This sounds like a challenging problem. What memory are you executing from when you have all these issues? Based on your description, and that you are bringing up a new board, my initial suspicion is that you are executing from external memory, and maybe there is an issue with that memory interface. When you say this same board runs fine using MCUXpresso SDK-based applications, are those apps executing from the same memory you are trying to use with Zephyr? Have you confirmed the address ranges of the executable code are similar between them?

Have you tried executing from the internal ITCM? This may help confirm the issue you have is memory related. This discussion #36800 has tips for building the app to execute from SRAM, and move data to DTCM. You can use the debugger to load the executable to ITCM, and run it from there.

And can you clarify what you mean by:

But when trying it out (even with the mimxrt1170_evk_cm7 board) Zephyr seems to crash at very weird places.

Does this mean you are building a Zephyr app using board mimxrt1170_evk_cm7 and running it on your board? Or are you seeing the same issue when running a Zephyr app on the MIMXRT1170-EVK board?

Since single-stepping helps the app execute farther, and with the location of the crashes being random, these also point to memory issues. A single-step will use a breakpoint to halt after that line, so the core may only fetch and execute a few instructions between halts. Running the app can stress that external memory interface more, and increase the chances of a problem causing an issue.

You say you had to make some changes for the SDK-based app to run on this board. What changes were needed? And what did you do in Zephyr's board support to make the same changes?

And yes, when the debugger reports reading 0xdeadbeef, it has lost connection with the MCU. This means it cannot really read that memory location, and the debugger uses that default deadbeef to show there is an issue. Best regards

[tyalie]

Okay so having a bit of distance to the project I got some new ideas that I tried out and I have been able to get it more reliable and more of a working state.

What memory are you executing from when you have all these issues?

The chip itself doesn't have any internal memory and the NXP nvk and the TQ board both chip with a single nor flash connected to flexspi 1. I've also looked into the binaries to verify that the FCFB and IVT is correct and can confirm that zephyr output is a valid bootable binary.

Does this mean you are building a Zephyr app using board mimxrt1170_evk_cm7 and running it on your board? Or are you seeing the same issue when running a Zephyr app on the MIMXRT1170-EVK board?

Yeah I tried that but it seems to have caused more issues than I anticipated. The flexspi_nor_config.c seems to be more relevant than I thought. Using the EVK target will result in the deadbeee behaviour or similar. I've now exchanged my custom frankenstein config and the one from the EVK with the version I got from TQ.

I have also stopped using the MIMXRT1176xxxA_M7 JLink device and instead went to using the TQ provided one. Together with the --no-reset-after-load the chip finally played nice (the TQ Jlink device doesn't recover the PC and SP after a reset).

And yes, when the debugger reports reading 0xdeadbeef, it has lost connection with the MCU.

Thank you for confirming this, I only assumed it so far as I wasn't able to find a lot of documentation here.

[tyalie]

After all those changes I now land reliably in the Zephyr fault handler. Reason of entry is 0 and it seems to be a IBUSERR fault. The issue seems to be the return from z_bss_init

zephyr/kernel/init.c

Lines 109 to 145 in 24984de

	void z_bss_zero(void)
	{
	if (IS_ENABLED(CONFIG_ARCH_POSIX)) {
	/* native_posix gets its memory cleared on entry by
	* the host OS, and in any case the host clang/lld
	* doesn't emit the __bss_end symbol this code expects
	* to see
	*/
	return;
	}
	z_early_memset(__bss_start, 0, __bss_end - __bss_start);
	#if DT_NODE_HAS_STATUS(DT_CHOSEN(zephyr_ccm), okay)
	z_early_memset(&__ccm_bss_start, 0,
	(uintptr_t) &__ccm_bss_end
	- (uintptr_t) &__ccm_bss_start);
	#endif
	#if DT_NODE_HAS_STATUS(DT_CHOSEN(zephyr_dtcm), okay)
	z_early_memset(&__dtcm_bss_start, 0,
	(uintptr_t) &__dtcm_bss_end
	- (uintptr_t) &__dtcm_bss_start);
	#endif
	#if DT_NODE_HAS_STATUS(DT_CHOSEN(zephyr_ocm), okay)
	z_early_memset(&__ocm_bss_start, 0,
	(uintptr_t) &__ocm_bss_end
	- (uintptr_t) &__ocm_bss_start);
	#endif
	#ifdef CONFIG_CODE_DATA_RELOCATION
	extern void bss_zeroing_relocation(void);
	bss_zeroing_relocation();
	#endif /* CONFIG_CODE_DATA_RELOCATION */
	#ifdef CONFIG_COVERAGE_GCOV
	z_early_memset(&__gcov_bss_start, 0,
	((uintptr_t) &__gcov_bss_end - (uintptr_t) &__gcov_bss_start));
	#endif
	}

The stack gets corrupted and the assembly instruction in address 0x300045b2 seems to set the PC to a non code location (< 0x10000000), resulting in the fault handler to be called.

Dump of assembler code for function z_bss_zero:
   0x30004598 <+0>:	ldr	r0, [pc, #24]	; (0x300045b4 <z_bss_zero+28>)
   0x3000459a <+2>:	movs	r1, #0
   0x3000459c <+4>:	ldr	r2, [pc, #24]	; (0x300045b8 <z_bss_zero+32>)
   0x3000459e <+6>:	push	{r3, lr}
   0x300045a0 <+8>:	subs	r2, r2, r0
   0x300045a2 <+10>:	bl	0x3000569e <z_early_memset>
   0x300045a6 <+14>:	ldr	r0, [pc, #20]	; (0x300045bc <z_bss_zero+36>)
   0x300045a8 <+16>:	ldr	r2, [pc, #20]	; (0x300045c0 <z_bss_zero+40>)
   0x300045aa <+18>:	movs	r1, #0
   0x300045ac <+20>:	subs	r2, r2, r0
   0x300045ae <+22>:	bl	0x3000569e <z_early_memset>
=> 0x300045b2 <+26>:	pop	{r3, pc}
   0x300045b4 <+28>:	lsls	r0, r1, #2
   0x300045b6 <+30>:	strh	r0, [r0, #0]
   0x300045b8 <+32>:	lsls	r4, r0, #10
   0x300045ba <+34>:	strh	r0, [r0, #0]
   0x300045bc <+36>:	movs	r0, r0
   0x300045be <+38>:	movs	r0, #0
   0x300045c0 <+40>:	movs	r0, r0
   0x300045c2 <+42>:	movs	r0, #0

I'll continue looking into it maybe this is something simple.

[DerekSnell]

Hi @tyalie ,
I'm glad you are making progress. Where is the data memory and stack located? Is that also in external memory, or the internal SRAMs? It seems the stack is getting corrupted here, so when it returns from a sub-function, the PC is loaded with a bad address.

[tyalie]

Hi @DerekSnell
I'll need to figure the exact details out tomorrow. What baffels me here is that PC will be set to mostly small but implausible numbers. If my zero bss function would overwrite the memory area I would expect zeros to appear instead. But that's a problem I'll tackle tomorrow.

[tyalie]

The issue was that the SEMC initialization for the SDRAM was incorrect in the TQ provided device configuration data. A 16bit I/O device was talked to like it was a 32bit one so all stuff on the stack was heavily corrupted.

[tyalie]

Okay. That has taken me a few more hours but Zephyr runs now. The only issue I have remaining is that I don't know how to change the CONFIG_ADJUST_DCDC from a board config and I don't wanna fork the IMXRT1176 soc file as the SOC name seems to be important in the nxp hal for further configuration. The TQ board will get stuck when adjusting the DCDC even on the CM7 core.

This configuration seems to be hard pinned for the soc

zephyr/soc/arm/nxp_imx/rt/Kconfig.soc

Line 323 in 9a3d762

select ADJUST_DCDC

Is there any way of changing soc configs in a board config?

[DerekSnell]

Hi @tyalie ,
Good point, this Kconfig symbol should be selected at the board level, not SoC. NXP will work on submitting a PR to address this soon. Thanks for pointing this out

[tyalie]

Thank you. I can also submit the PR to this myself.

[DerekSnell]

Hi @tyalie ,
FYI, after discussing, we feel the best option is to move the DCDC enable and voltage level to the devicetree. Similar to how this PR #49782 is moving some clock settings from Kconfig to devicetree.

You can still use your fork for now to remove these from the SOC files. But I don't think you need to submit a PR, since we will be improving the configuration. Best regards

[tyalie]

Hi. Are there any updates on this?

[decsny]

#92203