Repository navigation
Conversation
7b61907 to
e41bfc2
Compare
e41bfc2 to
bb305fc
Compare
bb305fc to
86a65e0
Compare
|
@jmarrero-bot can you review this and check the behavior on the dev space? |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved migration defects can leave preserved state inaccessible under SELinux or abort after boot configuration changes, while an unsupported test-image layout blocks validation.
7 open issues
Relabel migrated files on separately mounted /var · New Allow full-copy fallback for reflink failures · New Place the systemd unit under /etc/systemd/system · New Export the configured image to the guest · New Split mountinfo fields on literal spaces · New Pull the pinned Podman LBI reference · New Fix shell quoting in the tmpfiles rule command · New
What changed in this PR
Adds opt-in /var preservation to OSTree-based bootc install to-existing-root, supporting package-mode migrations independently of the broader work in #2384.
Changes:
- Adds preservation flags and configurable exclusions, with checks for incompatible installation options.
- Implements reflink-aware copying, nested-mount filtering, metadata preservation, and staged publication.
- Adds unit and reboot integration coverage, local and Packit test setup, and user documentation.
| File | Description |
|---|---|
tmt/tests/test-50-to-existing-root-preserve-var.fmf |
Registers the package-mode preservation test. |
tmt/tests/booted/package-mode/test-to-existing-root-preserve-var.sh |
Tests migration, mount exclusions, and persistence across reboot. |
tmt/plans/package-mode.fmf |
Prepares local or Packit images for migration testing. |
Justfile |
Adds a local package-mode test recipe. |
hack/provision-packit.sh |
Allows image preparation without immediate conversion. |
docs/src/man/bootc-install-to-existing-root.8.md |
Documents preservation options and limitations. |
docs/src/man/bootc-install-config.5.md |
Documents configurable exclusion paths. |
docs/src/bootc-installation.7.md |
Explains preservation behavior and data-consistency limits. |
crates/lib/src/store/reflink.rs |
Adds pairwise reflink probing and tests. |
crates/lib/src/store/mod.rs |
Exposes the reflink helper module. |
crates/lib/src/install/migrate/mountinfo.rs |
Parses mount information to identify nested mounts. |
crates/lib/src/install/migrate.rs |
Implements filtered, staged /var copying and unit tests. |
crates/lib/src/install/config.rs |
Adds exclusion configuration and merge coverage. |
crates/lib/src/install.rs |
Integrates preservation, validation, and deferred cleanup. |
crates/lib/src/cli.rs |
Updates command help and installer dispatch. |
.packit.yaml |
Adds package-mode migration CI jobs. |
Review effort: Balanced
| migrate::preserve_var(&root_path.join("var"), &new_var, &preserve_var_skip) | ||
| .context("Post-install /var preservation")?; |
| ) -> Result<()> { | ||
| with_staging_dir(destination, |_, staging| { | ||
| let options = if reflink { | ||
| &["--reflink=always", "-a"][..] |
| mkdir -p /usr/etc/systemd/system/local-fs.target.wants && \ | ||
| printf '%s\n' \ | ||
| '[Unit]' \ | ||
| 'Description=bootc TMT preserve-var bind mount' \ | ||
| '' \ | ||
| '[Mount]' \ | ||
| 'What=/var/lib/bootc-tmt-preserve-var-bind/source' \ | ||
| 'Where=/var/lib/bootc-tmt-preserve-var-bind/target' \ | ||
| 'Type=none' \ | ||
| 'Options=bind' \ | ||
| '' \ | ||
| '[Install]' \ | ||
| 'WantedBy=local-fs.target' \ | ||
| > "/usr/etc/systemd/system/$unit" && \ | ||
| ln -s "../$unit" "/usr/etc/systemd/system/local-fs.target.wants/$unit" |
| echo "Refusing to overwrite existing archive: $archive_path" >&2 | ||
| exit 1 | ||
| fi | ||
| podman save --format oci-archive --output "$archive_path" localhost/bootc |
| index + 1 | ||
| ); | ||
| let fields = line | ||
| .split(|byte| byte.is_ascii_whitespace()) |
| podman --remote=false image exists localhost/bootc | ||
| podman --remote=false pull quay.io/curl/curl:latest | ||
| podman --remote=false pull quay.io/curl/curl-base:latest | ||
| podman --remote=false pull registry.access.redhat.com/ubi9/podman:latest |
| printf '%s\n' "${tmpfiles_rule[@]}" > "$rule" | ||
| printf '%s\n' "${tmpfiles_rule[@]}" | cmp - "$rule" |
jmarrero-bot
left a comment
There was a problem hiding this comment.
Not ready yet for hosts with mounts under /var. In Fedora 44 and CentOS Stream 10 VMs, a host with nothing (or just rpc_pipefs) mounted under /var migrates fine: PostgreSQL data and app state in /var/lib survive the reboot. But a separate /var/log disk, or a tmpfs below /var, makes the copy fail after Installation complete!. By then the host's boot entries are already gone, so it boots the image without the host's /var or with only part of it. Details are inline. jmarrero-forge#11 has fixes on top of this head, and with them both of those scenarios pass.
Also:
- On these cloud-init hosts the migrated system had no SSH host keys, so sshd fails (see the inline comment on the docs). The package-mode Testing Farm jobs here stop the same way:
Failed to start sshd.serviceafter the reboot, thenSSH master process failed to start(e.g. https://artifacts.dev.testing-farm.io/0791d950-98ff-4a8e-9de6-b24531a9e433/). - I reproduced two of Copilot's tmt comments with
just test-tmt-package-mode. package-mode.fmf:133: the install fails withresolving bound image registry.access.redhat.com/ubi9/podman:9.8-1793032728: ... does not resolve to an image ID. Test L313: the trace showsprintf %sn, so the tmpfiles rule is written as one line. - Composefs:
--preserve-var --composefs-backendis refused before anything is touched, so ostree is the only backend this supports.
VM matrix, exact errors, unit/tmt results: https://gist.github.com/jmarrero-bot/16d461d0ed7a43ab4086d24e17152e17
Generated-by: https://github.com/jmarrero-bot#llms
| ) | ||
| .await?; | ||
|
|
||
| if preserve_var { |
There was a problem hiding this comment.
Every failure I hit came from this block, after clean_boot_directories and bootupd had already replaced the host's boot entries (only ostree-1.conf left). The host then boots the image without its /var (mountinfo error) or with part of it (placeholder errors). The data is still on disk in the old /var, but the new system doesn't look there. Opening the source, reading mountinfo and checking each nested mount could all happen before install_to_filesystem; 637e8d1 in jmarrero-forge#11 does that. It catches the mountinfo case, but the other two depend on the destination. The copy itself can still fail late, e.g. a full copy running out of space, so a free-space check would fit there too.
| format!("Malformed mountinfo line {}: missing separator", index + 1) | ||
| })?; | ||
| anyhow::ensure!( | ||
| separator >= 6 && fields.len() == separator + 4, |
There was a problem hiding this comment.
This rejects valid records with an empty source: mount -t tmpfs "" /var/lib/x shows up as ... - tmpfs rw,..., which has only three fields after -. Any such record anywhere in the installer's mountinfo fails the migration, not just ones below /var. F44 and CS10 VMs: Post-install /var preservation: Malformed mountinfo line 42: invalid field count (line 36 on CS10). Only the mountpoint (field 5) is used, so parsing just that avoids it.
| let mut entries = destination.read_dir(".")?; | ||
| anyhow::ensure!( | ||
| entries.next().transpose()?.is_none(), | ||
| "Mountpoint placeholder {} is not empty", |
There was a problem hiding this comment.
ostree seeds the new stateroot's /var from the image, so a nested mountpoint where the image has content lands here. A Fedora 44 host with /var/log on its own disk and a fedora-bootc:44 image fails with Reflink-copying /var into new deployment (Strategy C): Mountpoint placeholder log is not empty. Keeping the image's content (the mount covers it at boot) and rejecting only a type mismatch works on the same host image.
| .collect::<BTreeSet<_>>(); | ||
| for name in destination_names { | ||
| if !source_names.contains(name.as_os_str()) { | ||
| fremovexattr(&destination, &name).with_context(|| { |
There was a problem hiding this comment.
On CentOS Stream 10 the root of a tmpfs mounted below /var has no security.selinux, so this tries to strip the label from the new placeholder and SELinux refuses: restoring directory metadata failed: xattrs: Removing destination xattr security.selinux: Permission denied (os error 13). Keeping the destination's label when the source has none fixes it.
|
|
||
| /// Relative paths below `/var` to leave out while preserving `/var`. | ||
| #[clap(long = "preserve-var-skip", value_name = "PATH")] | ||
| pub(crate) preserve_var_skip: Vec<String>, |
There was a problem hiding this comment.
Nothing ties this to --preserve-var, so without it the paths are silently ignored. And install_to_existing_root now loads and validates preserve-var-skip from the install config on every run (L3185-3190), so a bad entry there breaks to-existing-root even without --preserve-var. Maybe requires = "preserve_var", and validate only when it's set?
| useful for preserving network settings, user accounts, or application data from the | ||
| previous installation. | ||
|
|
||
| With the OSTree backend, `--preserve-var` copies the old `/var` into the new |
There was a problem hiding this comment.
This could use a sentence on SSH, or a default exclusion. fedora-bootc:44 and centos-bootc:stream10 both ship cloud-init, and where cloud-init is active sshd-keygen@ stands down for it. The host's /var/lib/cloud is carried over, so cloud-init sees the same instance and doesn't create host keys, and the old keys stay in the old /etc. After each successful migration I could check (F44 and CS10, NoCloud), there were no host keys, so sshd fails and the host is unreachable over SSH. In the one run where nothing was copied, cloud-init generated new keys.
The /var copy needs careful filtering around exclusions and nested mounts, and must avoid publishing partial entries or overwriting existing data. Isolating the copy engine and its tests makes these filesystem guarantees reviewable before adding installer orchestration. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
A signal or failed fsfreeze could leave /var/log frozen because the cleanup handlers were installed afterward. Install traps before the freeze attempt and use one EXIT cleanup path that preserves the original status. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode runner currently asks TMT to discover directly from a large checkout, which makes TMT 1.69 traverse ignored build output. Discover from a temporary sanitized snapshot instead, and retain the local OCI archive for guest sync. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The local package-mode test image must use OSTree regardless of the caller's variant. Forward the other configured build settings and override the variant only on the nested build. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Keep root-level Podman build secrets out of the guest snapshot and clean them up if the build or test setup fails. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode test currently removes every root build-secret pathname, including files that were present before the test began. Snapshot matching pathnames before installing cleanup and remove only new matches while preserving command status. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Podman build secret paths are shared temporary paths, so cleanup from this runner could delete files owned by concurrent builds or users. Leave lifecycle management to Podman and exclude stale artifacts from the test snapshot. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Fedora package-mode guests can keep /home outside /var, so preserve the complete lexical TMT run tree under /var/home before migration. Boot the source with SELinux disabled and verify the target relabels fresh application and PostgreSQL state when it starts enforcing. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode VM can expose /home as a real directory, so the lexical controller workdir may not be preserved by migration. Derive the destination from the resolved TMT root and run ID, and assert test SSH key labels against target policy in both SELinux phases. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The Fedora package-mode test mounts /var/lib/pgsql from the source guest, but that path is not guaranteed to exist. Create it before running the target-image PostgreSQL initializer. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The guest creates the PostgreSQL mountpoint as root, but initdb needs it to be writable by postgres. Set its numeric ownership from the target image and restrict its mode before database setup. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The PostgreSQL data directory must be a normal directory beneath the bind mount, not the mountpoint itself. Seed it through the target image as its postgres user so the preserved data uses the target UID/GID and remains compatible with the post-migration service. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The one-shot target-image container does not start systemd, so PostgreSQL's default socket directory may be unavailable. Configure a /tmp socket and connect the seed queries over loopback TCP. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The target image already provides /var/lib/pgsql, while /mnt may not exist. Seed in the mounted data directory so the state survives preserve-var and is available to the target service. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Check that the target image already contains /var/lib/pgsql before attaching a bind mount there. This ensures the seed run cannot pass by relying on Podman's bind-mount behavior to create a missing destination. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The full /boot tar snapshot is sensitive to incidental metadata changes, which can make a rejected preflight appear to mutate boot configuration. Compare only the paths and contents of boot-critical files instead. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The rejected preflight check should detect changes to all active EFI loader entries, including the architecture-independent fallback path. Record symlink targets alongside referent content digests so retargeting links is visible. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode guest has a normal BLS entry before migration, so the preflight snapshot must not require an OSTree-prefixed name. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
A leftover OSTree BLS entry must not satisfy the package-mode baseline check. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The source kernel may disable SELinux, preventing restorecon from writing security.selinux on the mounted target deployment. Use the target policy to label the generated test-only keys directly and verify them before reboot. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
bootc status reports the local test image with an explicit :latest tag, while the test variable omits the implicit tag. Accept that exact normalized reference without loosening the assertion to other tags or images, and print status details if it still fails. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Mountpoint placeholders represent empty nested-mount boundaries and may have no target-policy label. Preserve the staging-created label for those placeholders only; copied data still requires a policy match. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Only empty mountpoint placeholders can lack a target policy label. Verify the staged directory through its no-follow handle before accepting an unmatched label, so malformed or unexpectedly populated placeholders fail before publication. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Treat preserve-var as experimental until the interface is stabilized. Require an explicit --experimental acknowledgement, document the opt-in, and exercise it in the package-mode TMT test. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Keep the full behavior and safety caveats for /var migration in the experimental documentation, and point to it from the stable installation and configuration references. Assisted-by: AI Signed-off-by: ckyrouac <ckyrouac@redhat.com>
79113f2 to
d1b21bc
Compare


Splitting
--preserve-varoff of #2384