raspberrypi: Proper A/B kernel Updates and Atomic Boot Partition Updates

[om26er]

Currently we have been using meta-rauc-community and it works fine, the A/B partitioning scheme and OTA updates work as well (we use RAUC).

Since there is only a single partition for boot, I wonder how to implement OTA updates for the kernel ?

[leon-anavi]

Hi @om26er,

You should be able to achieve this using:

IMAGE_INSTALL_append = " kernel-image kernel-devicetree"

Btw meta-mender-raspberrypi for Mender A/B updates could be used as a reference example for such feature.

Best regards, Leon

[timemaster5]

Hi,
I am also interested in this topic, and I am not sure whether that suggested solution helps.

On raspberry pi, the kernel and devicetree is stored on boot partition, which is not included in update process. The update process using rauc only updates rootfs. So even if you include kernel image and devicetree files in the rootfs, rasperry won't use it after update.

Maybe it is a good first step which could be later on extended about copying these files to the boot partition, maybe using a post update hook. But it is not a recommended way because the update is not atomic. So if there is a problem during update, the raspberry pi will not boot again at all. Also there is no fallback to previously known good kernel image.

What I think is the best way is to implement boot-mbr-switch method described in the rauc documentation here, which consists of two boot partitions stored on the card and switching between them by manipulating partition table. Which is atomic and therefore more safe. This way, we can also implement a fallback in case of boot fail which is probably the most important reason. Because otherwise the only change of fixing the pi is to reflash the sd card or at least the boot partition.

The setup should not be hard, the only thing I have a problem with is where to store the u-boot environment file. This file is currently on the /boot partition, so we need to move it out, somewhere to the bare unallocated space of the sd card. Because we will have two different boot partitions. U-boot can also save two copies of this file to be sure that we won't use a broken one on boot.

[om26er]

I found some useful information on this page as well https://github.com/mendersoftware/meta-mender/tree/master/meta-mender-raspberrypi#meta-mender-raspberrypi

It seems mender only supports kernel updates (by putting the kernel on the rootfs) and DTB updates are not supported. For our project, being able to update the kernel and DTB is very important so we'd explore more and see what could be done.

[om26er]

maybe if @timemaster5 can find a solution to this stuff, it'll make life easier for us ;-)

[timemaster5]

Problem with me is that I don't have any time right now. I am working on a project where I must deliver something under the deadline. So for now I end up with extensive research on this topic, and I hope someone else will be able to take over and do the implementation. If not, I will definitely get back at some point, because I would like this feature implemented too, but currently can't say when.

From what I know, I am pretty sure the required changes are:

1. Add the second /boot into wic, but exclude it from the partition table

part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label bootA --active --align 4096 --size 20
part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label bootB --align 4096 --size 20 --no-table
part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --label systemA --align 4096 --size 1536
part / --ondisk mmcblk0 --fstype=ext4 --label systemB --align 4096 --size 1536

2. Patch u-boot to store env file, and redundant env file on the SD card outside of our /boot partitions. It could even be exactly what mender requires in this patch

3. Update system.conf to contain newly created boot partitions, something like:

[slot.bootloader.0]
device=/dev/mmcblk0
type=boot-mbr-switch
region-start=164352 <- this value needs to be fixed
region-size=20M

[slot.bootloader.1]
device=/dev/mmcblk0
type=boot-mbr-switch
region-start=164352 <- this value needs to be fixed
region-size=20M

4. Add bootimg-partition FS into an update bundle. It should be only a matter of adding it into RAUC_BUNDLE_SLOTS in the update bundle recipe

5. Update u-boot startup script a little bit. Here I have no idea what would be the best to do..
   Maybe it could be taken from another project where updating boot-loader partition works. We probably want to switch to a second boot-loader slot only if the boot process fails after the update of boot-loader partition, otherwise only switch rootfs one

Region start values, corresponding partition sizes and alignments changes in wic file, and in the patch could be based on the suggested layout from the rauc documentation.

And that's probably it.

Sorry I can't help more at this point

[leon-anavi]

@timemaster5 thank you for the valuable feedback and sharing all these details. I believe this will be very useful in future.

Thanks,
Leon

[timemaster5]

You are welcome, I hope I will have time for the actual integration soon :)

Anyway, just to let you know. I just successfully updated the device in a remote location in a similar way to what was originally suggested @leon-anavi. I uploaded a rootfs image, did the update of rootfs image and then manually copied over a new content of /boot partition.

Even though it is not my favourite way, it went ok and I rebooted to the new kernel and device tree. But there are no guarantees. So I would consider this way as best-effort only.

[om26er]

@timemaster5 thanks for your pointers. This is on my TODO list for this week as well. I got some help from the RAUC upstream as well and they say they have successfully update RPi boot partition as well.

[timemaster5]

Oh great, if you could share some other information on this topic from other channels it would be appreciated

[om26er]

I have this stuff mostly implemented. One thing I need to mention is that we only need a single entry for bootloader slot in the system.conf

[slot.bootloader.0]
device=/dev/mmcblk0
type=boot-mbr-switch
region-start=4194304
region-size=218103808

The main thing to figure out was to create a vfat image based on the files listed in ${IMAGE_BOOT_FILES} variable.

[om26er]

So the only missing thing for me currently is that the uboot.env is being written to /boot, which means on updating the boot partition, that files gets removed and gets recreated on next boot. This causes rootfs.0 as default. I have applied this patch https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-raspberrypi/recipes-bsp/u-boot/patches/0001-configs-rpi-enable-mender-requirements.patch however it seems uboot.env is still being written to the /boot partition

[om26er]

Here is the fdisk output.

Before

root@raspberrypi4-64:~# fdisk /dev/mmcblk0 -l
Disk /dev/mmcblk0: 14.43 GiB, 15485370368 bytes, 30244864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2b883497

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *        8192   221183   212992  104M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        434176  6578175  6144000    3G 83 Linux
/dev/mmcblk0p3       6578176 12722175  6144000    3G 83 Linux
/dev/mmcblk0p4      12722176 30244863 17522688  8.4G 83 Linux

Flash the image

root@raspberrypi4-64:~# rauc write-slot bootloader.0 /opt/rpi-ota-packages/boot_bundle.img 
rauc-Message: 20:08:46.772: Config option 'statusfile=<path>/per-slot' unset, falling back to per-slot status
rauc-Message: 20:08:46.776: Using per-slot statusfile
rauc-Message: 20:08:46.776: Checking image type for slot type: boot-mbr-switch
rauc-Message: 20:08:46.776: Image detected as type: *.img
rauc-Message: 20:08:46.776: Found inactive boot partition 1 (pos. 113246208B, size 109051904B)
rauc-Message: 20:08:46.776: Clearing inactive boot partition 1 on /dev/mmcblk0
rauc-Message: 20:08:58.704: Opening inactive boot partition 1 on /dev/mmcblk0
rauc-Message: 20:08:58.705: Copying image to inactive boot partition 1 on /dev/mmcblk0
rauc-Message: 20:09:06.306: Setting MBR to switch boot partition
rauc-Message: 20:09:06.306: Slot written successfully

After flashing

root@raspberrypi4-64:~# fdisk /dev/mmcblk0 -l
Disk /dev/mmcblk0: 14.43 GiB, 15485370368 bytes, 30244864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2b883497

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      221184   434175   212992  104M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        434176  6578175  6144000    3G 83 Linux
/dev/mmcblk0p3       6578176 12722175  6144000    3G 83 Linux
/dev/mmcblk0p4      12722176 30244863 17522688  8.4G 83 Linux

[timemaster5]

I have this stuff mostly implemented. One thing I need to mention is that we only need a single entry for bootloader slot in the system.conf

[slot.bootloader.0]
device=/dev/mmcblk0
type=boot-mbr-switch
region-start=4194304
region-size=218103808

The main thing to figure out was to create a vfat image based on the files listed in ${IMAGE_BOOT_FILES} variable.

Thank you, could you share your .wks file and maybe some other info on how to build that vfat image?

[timemaster5]

So the only missing thing for me currently is that the uboot.env is being written to /boot, which means on updating the boot partition, that files gets removed and gets recreated on the next boot. This causes rootfs.0 as default. I have applied this patch https://github.com/mendersoftware/meta-mender/blob/master/meta-mender-raspberrypi/recipes-bsp/u-boot/patches/0001-configs-rpi-enable-mender-requirements.patch however it seems uboot.env is still being written to the /boot partition

This seems to me like the patch has not been applied. Maybe bitbake -c cleansstate u-boot could help? There could be some other reason, but that patch specifically defines sectors on the flash drive to store the env file in and disables storing on VFAT partition. So the only reasonable explanation to me is that u-boot still uses the original settings.

Out of the curiosity, what RaspberryPi do you use? For example, I use Compute module 3 with custom defconfig for the u-boot. So this patch wouldn't work me out of the box.

[timemaster5]

Here is the fdisk output.

Before

root@raspberrypi4-64:~# fdisk /dev/mmcblk0 -l
Disk /dev/mmcblk0: 14.43 GiB, 15485370368 bytes, 30244864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2b883497

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *        8192   221183   212992  104M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        434176  6578175  6144000    3G 83 Linux
/dev/mmcblk0p3       6578176 12722175  6144000    3G 83 Linux
/dev/mmcblk0p4      12722176 30244863 17522688  8.4G 83 Linux

Flash the image

root@raspberrypi4-64:~# rauc write-slot bootloader.0 /opt/rpi-ota-packages/boot_bundle.img 
rauc-Message: 20:08:46.772: Config option 'statusfile=<path>/per-slot' unset, falling back to per-slot status
rauc-Message: 20:08:46.776: Using per-slot statusfile
rauc-Message: 20:08:46.776: Checking image type for slot type: boot-mbr-switch
rauc-Message: 20:08:46.776: Image detected as type: *.img
rauc-Message: 20:08:46.776: Found inactive boot partition 1 (pos. 113246208B, size 109051904B)
rauc-Message: 20:08:46.776: Clearing inactive boot partition 1 on /dev/mmcblk0
rauc-Message: 20:08:58.704: Opening inactive boot partition 1 on /dev/mmcblk0
rauc-Message: 20:08:58.705: Copying image to inactive boot partition 1 on /dev/mmcblk0
rauc-Message: 20:09:06.306: Setting MBR to switch boot partition
rauc-Message: 20:09:06.306: Slot written successfully

After flashing

root@raspberrypi4-64:~# fdisk /dev/mmcblk0 -l
Disk /dev/mmcblk0: 14.43 GiB, 15485370368 bytes, 30244864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2b883497

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      221184   434175   212992  104M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        434176  6578175  6144000    3G 83 Linux
/dev/mmcblk0p3       6578176 12722175  6144000    3G 83 Linux
/dev/mmcblk0p4      12722176 30244863 17522688  8.4G 83 Linux

And this is really cool :)

Also, your region starts and offset values looks good to me. They don't seem to collide with the u-boot env file defined in the patch.