README-internals: add more multipath documentation

This is a complex topic, so let's add more docs.

Author: jlebon

internals/README-internals.md

@@ -66,32 +66,60 @@ For other operating systems which do "per cloud" disk images, it would have been
 natural to just change `/etc/chrony.conf` per platform.  But that would mean we have a different
 ostree commit checksum per platform, breaking our "image based" update model.
 
-# multipath
+# Multipath
 
-A lot of history here.  A TL;DR is that nontrivial multipath setups conceptually conflict
-a bit with the "CoreOS model" of booting into the desired configuration from the start.
-There's also a long related issue in that we want to use a "pristine" initramfs in
-general, and nontrivial multipath configuration needs to be in the initramfs.
+Multipath differs from other storage configurations by a major aspect: it is usually not
+configured by Ignition. If we mount an individual path for e.g. `/sysroot`, multipathd will
+not be able to take ownership afterwards. Furthermore, directly accessing individual paths
+before `multipathd` takes over is unsafe (e.g. it could be a non-optimized path). And since
+we need to mount `/boot` very early on, this naturally pushes multipath configuration into
+kernel arguments (and ideally soon, initramfs overlays).
 
-What we ended up with is adding an `rd.multipath=default` kernel argument which 
+What we ended up with is adding an `rd.multipath=default` kernel argument which
 triggers dracut to do "basic" automatic multipath setup in the stock initramfs:
 https://github.com/dracutdevs/dracut/pull/780
 
-So we still have a model then where the host boots up in a non-multipath
-configuration, Ignition runs and the kernel arguments are applied, then we reboot into the
-final configuration.
+By the nature of multipath, a tricky aspect is that e.g. the `by-label/root` symlink is
+valid both *before* and *after* multipathd takes ownership. In order to safely wait for the
+multipathed rootfs to show up, we have these udev rules which create, for example,
+`by-label/dm-mpath-root`:
+
+https://github.com/coreos/fedora-coreos-config/blob/94e0daa567a658f023d48ac5929c72ed910792bd/overlay.d/05core/usr/lib/udev/rules.d/90-coreos-device-mapper.rules#L1
+
+This is why we require the `root=/dev/disk/by-label/dm-mpath-root` kernel argument; so that
+the mount generated by `systemd-fstab-generator` waits for the the multipath version to show
+up and doesn't just mount an individual path.
+
+Firstboot (day-1) support is usually done at coreos-installer time by doing:
+
+```
+coreos-installer install \
+  --append-karg rd.multipath=default \
+  --append-karg root=/dev/disk/by-label/dm-mpath-root \
+  --append-karg rw
+  ...
+```
+
+The `rw` bit is necessary because `systemd-fstab-generator` will create a read-only mount by
+default (usually, `rw` is injected by `rdcore rootmap` for subsequent boots, but this does
+not happen if there is already a `root` karg).
+
+That said, turning on multipath on a subsequent (day-2) boot is still supported if the
+multipath setup itself is compatible with this. This is done by appending the same kargs as
+above using e.g.  `rpm-ostree kargs`. (Appending the kargs can also be done via
+`ignition-kargs`, though this still counts as "day-2" since on first boot we'd still access
+the boot partition directly.)
 
 We don't yet document multipath for FCOS, but we do document this setup for
 OpenShift that has a kola test:
 
-- https://github.com/coreos/coreos-assembler/blob/60f675ec5037b84c01f17192d773a14166dc6a14/mantle/kola/tests/misc/multipath.go#L57
+- https://github.com/coreos/coreos-assembler/blob/f5d003d2ebb81283c3e071ce2ac268884aa7232b/mantle/kola/tests/misc/multipath.go
+
+We also support multipath on an individual non-root partition. See the test above for how
+this works.
 
 More links:
 
 - https://github.com/coreos/ignition-dracut/issues/154
 - https://bugzilla.redhat.com/show_bug.cgi?id=1944660
-
-
-An example issue seems to be rooted in our use of labels to find `boot`
-and `root`. The labels seem to be racy in our current code because
-`multipathd.service` may take over the block devices.
+- https://github.com/coreos/fedora-coreos-config/pull/1011