Converting a standard Raspberry Pi image to PINN format

[Ircama]

The following procedure allows converting a specific Raspberry Pi OS image (in the form ".img.xz"), so that it can be installed via PINN. The description also covers the O.S. installation process using a standard PINN setup via a local web server.

The procedure can be run with Linux or with Windows on WSL (Ubuntu O.S.; for instance Ubuntu 22.04.5 LTS).

It can be used if a specific Raspberry Pi O.S. version has to be installed instead of the ones already covered by the set of preconfigured repositories in PINN (e.g., an old legacy kernel version).

It is tested with images in https://downloads.raspberrypi.com/ and specifically with https://downloads.raspberrypi.com/raspios_armhf/images/raspios_armhf-2022-04-07/2022-04-04-raspios-bullseye-armhf.img.xz.

It is recommended to run it manually, step by step.

### CONFIGURATION #####################################################
IMAGE_URL=https://downloads.raspberrypi.com/raspios_armhf/images/raspios_armhf-2022-04-07/2022-04-04-raspios-bullseye-armhf.img.xz
OS_NAME="custom_raspbian"  # Change this to your preferred OS name
OS_DESCRIPTION="A port of Debian Bullseye with the Raspberry Pi Desktop"  # Change this to your preferred description
DELTA_SIZE=5  # Additional size in MB for each partition vs. the minimum calculated dimension
IMAGE_NAME=$(basename "$IMAGE_URL" .xz)
INFO_URL="$(dirname "$IMAGE_URL")/$(basename "$IMAGE_URL" .img.xz).info"
KERNEL="unknown"  # if not discovered from $INFO_URL
WEB_SERVER="http://192.168.178.46:8081/os/${OS_NAME}"  # Your local web server; see below
#######################################################################

# Prerequisites
sudo apt-get install -y p7zip-full  # 7z
sudo apt-get install xz-utils  # xz
sudo apt install libarchive-tools  # bsdtar
sudo apt install -y acl  # getfacl

# Create working directories
mkdir -p pinn/os/${OS_NAME}
mkdir -p pinn/mount
cd pinn

# Download the .xz image
wget $IMAGE_URL

# Extract the .xz image
7z x ${IMAGE_NAME}.xz  # long-running

# List content (two partitions found)
fdisk -l ${IMAGE_NAME}

# Extract 0.fat and 1.img
7z x ${IMAGE_NAME}  # long-running

# Convert 0.fat to boot.tar.xz (sudo commands might ask the password)
sudo mount 0.fat mount
cd mount
sudo bsdtar --numeric-owner --format gnutar --one-file-system -cpvf ../os/${OS_NAME}/boot.tar .
cd ..
sudo umount mount
BOOT_UNCOMPRESSED_SIZE=$(stat -c %s os/${OS_NAME}/boot.tar)
echo $BOOT_UNCOMPRESSED_SIZE
xz -9 -e os/${OS_NAME}/boot.tar
BOOT_COMPRESSED_SIZE=$(stat -c %s os/${OS_NAME}/boot.tar.xz)

# Convert 1.img to root.tar.xz (sudo commands might ask the password)
sudo mount 1.img mount
cd mount
sudo find . -type s -exec rm {} \;
sudo getfacl -s -R . |sudo tee acl_permissions.pinn
sudo bsdtar --numeric-owner --format gnutar --one-file-system -cpvf ../os/${OS_NAME}/root.tar .  # long-running
cd ..
sudo umount mount
ROOT_UNCOMPRESSED_SIZE=$(stat -c %s os/${OS_NAME}/root.tar)
echo $ROOT_UNCOMPRESSED_SIZE
xz -9 -e os/${OS_NAME}/root.tar  # long-running  # considerably long-running
ROOT_COMPRESSED_SIZE=$(stat -c %s os/${OS_NAME}/root.tar.xz)

# Get partition information and sizes (in sectors)
SECTOR_SIZE=$(fdisk -l ${IMAGE_NAME} | grep "Sector size" | awk '{print $4}')
BOOT_SIZE=$(fdisk -l ${IMAGE_NAME} | grep "img1" | awk '{print $4}')  # (mmcblk0p6)
ROOT_SIZE=$(fdisk -l ${IMAGE_NAME} | grep "img2" | awk '{print $4}')

# Compute SHA Checksums
BOOT_SHASUM="$(sha256sum "os/${OS_NAME}/boot.tar.xz" | cut -f1 -d' ')"
ROOT_SHASUM="$(sha256sum "os/${OS_NAME}/root.tar.xz" | cut -f1 -d' ')"

# Extract other information
DATE=$(echo $IMAGE_NAME | sed 's/^\([0-9-]*\)-.*/\1/')
test "$DATE" || DATE="UNRELEASED"
VERSION=$(echo $(echo $IMAGE_NAME | sed 's/^[0-9-]*-\(.*\).img/\1/'))
test "$VERSION" || VERSION="RELEASE"
if [ "$INFO_URL" ];then
    wget -O os.info "$INFO_URL"
    KERNEL=$(grep -oP 'Linux version \K[^ ]+' os.info || echo "$DATE")
fi

# Add "partition_setup.sh"
wget -O os/${OS_NAME}/partition_setup.sh http://downloads.raspberrypi.org/raspbian/partition_setup.sh

# Add "Raspbian.png"
wget -O os/${OS_NAME}/Raspbian.png http://downloads.raspberrypi.org/raspbian/Raspbian.png

# Add "marketing.tar"
wget -O os/${OS_NAME}/marketing.tar http://downloads.raspberrypi.org/raspbian/marketing.tar

# Add "release_notes.txt" (optional)
wget -O os/${OS_NAME}/release_notes.txt https://downloads.raspberrypi.org/raspios_armhf/release_notes.txt

# Add "terms" (optional splash page with Terms and Conditions to be accepted)
echo "This is a custom OS Image.
Name: "$OS_NAME".
Description: "$OS_DESCRIPTION".
Release date: "$DATE".
Version: "$VERSION".
Kernel: "$KERNEL".
Standard terms and conditions apply." | tee os/${OS_NAME}/terms

# Create partitions.json
cat > os/${OS_NAME}/partitions.json << EOF
{
    "partitions": [
        {
            "label": "boot",
            "filesystem_type": "FAT",
            "mkfs_options": "-F 32",
            "partition_size_nominal": $((BOOT_SIZE*SECTOR_SIZE/1024/1024+DELTA_SIZE)),
            "uncompressed_tarball_size": $((BOOT_UNCOMPRESSED_SIZE/1024/1024+1)),
            "want_maximised": false,
            "sha256sum": "$BOOT_SHASUM"
        },
        {
            "label": "root",
            "filesystem_type": "ext4",
            "mkfs_options": "-O ^huge_file",
            "partition_size_nominal": $((ROOT_SIZE*SECTOR_SIZE/1024/1024+DELTA_SIZE)),
            "uncompressed_tarball_size": $((ROOT_UNCOMPRESSED_SIZE/1024/1024+1)),
            "want_maximised": true,
            "sha256sum": "$ROOT_SHASUM"
        }
    ]
}
EOF

# Create os.json
cat > os/${OS_NAME}/os.json << EOF
{
    "name": "$OS_NAME",
    "description": "$OS_DESCRIPTION",
    "release_date": "$DATE",
    "supported_models": [
        "Pi Model",
        "Pi 2",
        "Pi 3",
        "Pi Zero",
        "Pi Zero 2",
        "Pi 4"
    ],
    "version": "$VERSION",
    "kernel": "$KERNEL",
    "supports_backup": true,
    "url": "http://www.raspbian.org/",
    "group": "Custom",
    "username": "pi",
    "password": "raspberry"
}
EOF

TOTAL_DOWNLOAD_SIZE=$(du -sb os/${OS_NAME} | awk '{print $1}')

# Create os_list_v3.json
cat > os/os_list_v3.json << EOF
{
    "os_list": [
        {
            "os_name":	    "$OS_NAME",
            "description":  "$OS_DESCRIPTION",
            "release_date": "$DATE",
            "version": "$VERSION",
            "supported_models": [
                "Pi Model",
                "Pi 2",
                "Pi 3",
                "Pi Zero",
                "Pi Zero 2",
                "Pi 4"
            ],
            "url":              "http://www.raspbian.org/",
            "group":            "Custom",
            "download_size":    $TOTAL_DOWNLOAD_SIZE,
            "os_info":          "$WEB_SERVER/os.json",
            "partitions_info":  "$WEB_SERVER/partitions.json",
            "icon":             "$WEB_SERVER/Raspbian.png",
            "marketing_info":   "$WEB_SERVER/marketing.tar",
            "partition_setup":  "$WEB_SERVER/partition_setup.sh",
            "tarballs": [
                                "$WEB_SERVER/boot.tar.xz",
                                "$WEB_SERVER/root.tar.xz"
            ],
            "nominal_size":  $((BOOT_SIZE*SECTOR_SIZE/1024/1024+DELTA_SIZE+ROOT_SIZE*SECTOR_SIZE/1024/1024+DELTA_SIZE))
        }
    ]
}
EOF

echo "Optional flavors that can be added in os/${OS_NAME} directory:
${OS_NAME}_boot.tar
${OS_NAME}_boot.tar.xz
${OS_NAME}_boot.txt
${OS_NAME}_root.tar
${OS_NAME}_root.tar.xz
${OS_NAME}_root.txt

See https://github.com/procount/pinn/blob/master/noobsconfig.md
"

wslpath -w .  # If you are in WSL, get the corresponding Windows pathname of the current directory.

At this step, the installation package is completed.

Now we need to run a web server to serve the content of the previously created os directory. There are various methods to do this, and the following is one of the simplest.

Avoid running a web server on WSL, unless you configure bridging in order to allow outbound access from other devices; use Windows instead: install Python from the Microsoft Store. Then run a Windows CMD.

cd <Windows Path as produced by wslpath -w .>

python -m http.server 8081

Check with a browser that the web server can serve the URLs included in os_list_v3.json. Use a browser in LAN (e.g., a smartphone) to verify that the web server can be accessed via LAN.

The subsequent process is standard PINN administration.

• Format an SD card with Raspberry Pi Imager and use it to install PINN as Operating System ("Misc Utility Images", "PINN"; use e.g., the latest version).
• Copy os_list_v3.json from the os directory to the RECOVERY fat partition
• Useful to also add a valid wpa_supplicant.conf
• Useful to also add forcetrigger vncinstall ssh provision=... at the end of cmdline.txt, so that you can use VNC and you can also login to the rescue shell via ssh.

Set PINN to boot to VNC and to the resque shell (set forcetrigger vncinstall ssh at the end of cmdline.txt).

• Eject the SD card and boot PINN with a Raspberry.
• Connect to PINN via VNC client.
• Select "Custom".
• Select your OS.
• Press Install.
• Proceed with download and installation via GUI. The installation shall create the partitions, then download and extract data from the web server.
• Wait for the process to complete (and do not block the web server meanwhile).

At the end of the installation, from the rescue shell change the password of the pi user (generally required to allow login via ssh) and optionally set the wifi network (for recent OS versions which do not use wpa_supplicant.conf). To perform this, at the "OS(es) Installed Successfully" message, do the following:

• Login to the rescue shell via ssh, root user, password raspberry
• Run the following commands:

umount /mnt
mount /dev/mmcblk0p7 /mnt
chroot /mnt bash
mount -t proc proc /proc
su - pi  # You should see the pi@recovery:~ $ prompt.
sudo raspi-config  # change password and other essential settings; exit from raspi-config without rebooting
exit
exit
umount /mnt/proc
umount /mnt
umount -a

rebootp.sh 6  # This will boot the O.S.

After the OS boot, you can then ssh with pi user and with the previously added password.

In order to perform an autoboot, if you added forcetrigger, you also need to create the autoboot.txt file in the RECOVERY FAT filesystem; from the rescue shell via SSH:

# Enable root boot
mount -o remount,rw /mnt
# echo "boot_partition=6" > /mnt/autoboot.txt
umount /mnt

You can also:

mv /mnt/autoboot.txt /mnt/autoboot-bkp.txt  # disable autoboot
mv /mnt/autoboot-bkp.txt /mnt/autoboot.txt  # enable autoboot

References:

• https://github.com/procount/pinn/wiki/How-to-Create-a-Multi-Boot-SD-card-out-of-2-existing-OSes-using-PINN
• https://github.com/procount/pinn/blob/master/README_PINN.md#how-to-create-a-custom-os-version
• https://github.com/procount/pinn/blob/master/noobsconfig.md

Default repo_list.json:

• https://raw.githubusercontent.com/procount/pinn-os/master/os/repo_list.json

Default ancillary downloads for raspbian:

• http://downloads.raspberrypi.org/raspbian/os.json
• http://downloads.raspberrypi.org/raspbian/Raspbian.png
• http://downloads.raspberrypi.org/raspbian/partitions.json
• http://downloads.raspberrypi.org/raspbian/marketing.tar
• http://downloads.raspberrypi.org/raspbian/partition_setup.sh

[lurch]

Default ancillary downloads for raspbian:
http://downloads.raspberrypi.org/raspbian/Raspbian.png

It's probably "better" to grab those ancillary files from https://downloads.raspberrypi.com/raspios_armhf/ rather than http://downloads.raspberrypi.org/raspbian/ 🙂 (although please be aware that the .json and .sh files in there were only used by NOOBS, which has been deprecated for a long time (PINN is an enhanced fork of NOOBS), and so those files are no longer being updated.)

[thetechstoner]

I think it would be cool to
train a Local ai model
to convert raspberry pi image to PINN format