Skip to content

install: Preserve /var when installing to existing root - #2586

Open
ckyrouac wants to merge 44 commits into
bootc-dev:mainfrom
ckyrouac:preserve-var
Open

ckyrouac wants to merge 44 commits into
bootc-dev:mainfrom
ckyrouac:preserve-var

Conversation

@ckyrouac

Copy link
Copy Markdown
Collaborator

Splitting --preserve-var off of #2384

@github-actions github-actions Bot added area/install Issues related to `bootc install` area/documentation Updates to the documentation labels Oct 22, 2026
@bootc-bot
bootc-bot Bot requested a review from henrywang October 22, 2026 16:21
@ckyrouac ckyrouac added ci/tier-1 Run CI for tier-1 OS (centos-10) only and removed area/install Issues related to `bootc install` area/documentation Updates to the documentation labels Oct 22, 2026
@ckyrouac
ckyrouac marked this pull request as ready for review November 2, 2026 18:34
@github-actions github-actions Bot added area/install Issues related to `bootc install` area/documentation Updates to the documentation labels Nov 2, 2026
@ckyrouac
ckyrouac requested a review from jmarrero November 2, 2026 18:35
@jmarrero
jmarrero requested a balanced review from Copilot November 10, 2026 19:31
@jmarrero

Copy link
Copy Markdown
Contributor

@jmarrero-bot can you review this and check the behavior on the dev space?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved migration defects can leave preserved state inaccessible under SELinux or abort after boot configuration changes, while an unsupported test-image layout blocks validation.

7 open issues
What changed in this PR

Adds opt-in /var preservation to OSTree-based bootc install to-existing-root, supporting package-mode migrations independently of the broader work in #2384.

Changes:

  • Adds preservation flags and configurable exclusions, with checks for incompatible installation options.
  • Implements reflink-aware copying, nested-mount filtering, metadata preservation, and staged publication.
  • Adds unit and reboot integration coverage, local and Packit test setup, and user documentation.
File Description
tmt/​tests/​test-50-to-existing-root-preserve-var.fmf Registers the package-mode preservation test.
tmt/​tests/​booted/​package-mode/​test-to-existing-root-preserve-var.sh Tests migration, mount exclusions, and persistence across reboot.
tmt/​plans/​package-mode.fmf Prepares local or Packit images for migration testing.
Justfile Adds a local package-mode test recipe.
hack/​provision-packit.sh Allows image preparation without immediate conversion.
docs/​src/​man/​bootc-install-to-existing-root.8.md Documents preservation options and limitations.
docs/​src/​man/​bootc-install-config.5.md Documents configurable exclusion paths.
docs/​src/​bootc-installation.7.md Explains preservation behavior and data-consistency limits.
crates/​lib/​src/​store/​reflink.rs Adds pairwise reflink probing and tests.
crates/​lib/​src/​store/​mod.rs Exposes the reflink helper module.
crates/​lib/​src/​install/​migrate/​mountinfo.rs Parses mount information to identify nested mounts.
crates/​lib/​src/​install/​migrate.rs Implements filtered, staged /var copying and unit tests.
crates/​lib/​src/​install/​config.rs Adds exclusion configuration and merge coverage.
crates/​lib/​src/​install.rs Integrates preservation, validation, and deferred cleanup.
crates/​lib/​src/​cli.rs Updates command help and installer dispatch.
.packit.yaml Adds package-mode migration CI jobs.

Review effort: Balanced

Comment thread crates/lib/src/install.rs Outdated
Comment on lines +3233 to +3234
migrate::preserve_var(&root_path.join("var"), &new_var, &preserve_var_skip)
.context("Post-install /var preservation")?;
Comment thread crates/lib/src/install/migrate.rs Outdated
) -> Result<()> {
with_staging_dir(destination, |_, staging| {
let options = if reflink {
&["--reflink=always", "-a"][..]
Comment thread tmt/plans/package-mode.fmf Outdated
Comment on lines +139 to +153
mkdir -p /usr/etc/systemd/system/local-fs.target.wants && \
printf '%s\n' \
'[Unit]' \
'Description=bootc TMT preserve-var bind mount' \
'' \
'[Mount]' \
'What=/var/lib/bootc-tmt-preserve-var-bind/source' \
'Where=/var/lib/bootc-tmt-preserve-var-bind/target' \
'Type=none' \
'Options=bind' \
'' \
'[Install]' \
'WantedBy=local-fs.target' \
> "/usr/etc/systemd/system/$unit" && \
ln -s "../$unit" "/usr/etc/systemd/system/local-fs.target.wants/$unit"
Comment thread Justfile Outdated
echo "Refusing to overwrite existing archive: $archive_path" >&2
exit 1
fi
podman save --format oci-archive --output "$archive_path" localhost/bootc
index + 1
);
let fields = line
.split(|byte| byte.is_ascii_whitespace())
Comment thread tmt/plans/package-mode.fmf Outdated
podman --remote=false image exists localhost/bootc
podman --remote=false pull quay.io/curl/curl:latest
podman --remote=false pull quay.io/curl/curl-base:latest
podman --remote=false pull registry.access.redhat.com/ubi9/podman:latest
Comment on lines +312 to +313
printf '%s\n' "${tmpfiles_rule[@]}" > "$rule"
printf '%s\n' "${tmpfiles_rule[@]}" | cmp - "$rule"

@jmarrero-bot jmarrero-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ready yet for hosts with mounts under /var. In Fedora 44 and CentOS Stream 10 VMs, a host with nothing (or just rpc_pipefs) mounted under /var migrates fine: PostgreSQL data and app state in /var/lib survive the reboot. But a separate /var/log disk, or a tmpfs below /var, makes the copy fail after Installation complete!. By then the host's boot entries are already gone, so it boots the image without the host's /var or with only part of it. Details are inline. jmarrero-forge#11 has fixes on top of this head, and with them both of those scenarios pass.

Also:

  • On these cloud-init hosts the migrated system had no SSH host keys, so sshd fails (see the inline comment on the docs). The package-mode Testing Farm jobs here stop the same way: Failed to start sshd.service after the reboot, then SSH master process failed to start (e.g. https://artifacts.dev.testing-farm.io/0791d950-98ff-4a8e-9de6-b24531a9e433/).
  • I reproduced two of Copilot's tmt comments with just test-tmt-package-mode. package-mode.fmf:133: the install fails with resolving bound image registry.access.redhat.com/ubi9/podman:9.8-1793032728: ... does not resolve to an image ID. Test L313: the trace shows printf %sn, so the tmpfiles rule is written as one line.
  • Composefs: --preserve-var --composefs-backend is refused before anything is touched, so ostree is the only backend this supports.

VM matrix, exact errors, unit/tmt results: https://gist.github.com/jmarrero-bot/16d461d0ed7a43ab4086d24e17152e17

Generated-by: https://github.com/jmarrero-bot#llms

Comment thread crates/lib/src/install.rs Outdated
)
.await?;

if preserve_var {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every failure I hit came from this block, after clean_boot_directories and bootupd had already replaced the host's boot entries (only ostree-1.conf left). The host then boots the image without its /var (mountinfo error) or with part of it (placeholder errors). The data is still on disk in the old /var, but the new system doesn't look there. Opening the source, reading mountinfo and checking each nested mount could all happen before install_to_filesystem; 637e8d1 in jmarrero-forge#11 does that. It catches the mountinfo case, but the other two depend on the destination. The copy itself can still fail late, e.g. a full copy running out of space, so a free-space check would fit there too.

format!("Malformed mountinfo line {}: missing separator", index + 1)
})?;
anyhow::ensure!(
separator >= 6 && fields.len() == separator + 4,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rejects valid records with an empty source: mount -t tmpfs "" /var/lib/x shows up as ... - tmpfs rw,..., which has only three fields after -. Any such record anywhere in the installer's mountinfo fails the migration, not just ones below /var. F44 and CS10 VMs: Post-install /var preservation: Malformed mountinfo line 42: invalid field count (line 36 on CS10). Only the mountpoint (field 5) is used, so parsing just that avoids it.

Comment thread crates/lib/src/install/migrate.rs Outdated
let mut entries = destination.read_dir(".")?;
anyhow::ensure!(
entries.next().transpose()?.is_none(),
"Mountpoint placeholder {} is not empty",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ostree seeds the new stateroot's /var from the image, so a nested mountpoint where the image has content lands here. A Fedora 44 host with /var/log on its own disk and a fedora-bootc:44 image fails with Reflink-copying /var into new deployment (Strategy C): Mountpoint placeholder log is not empty. Keeping the image's content (the mount covers it at boot) and rejecting only a type mismatch works on the same host image.

.collect::<BTreeSet<_>>();
for name in destination_names {
if !source_names.contains(name.as_os_str()) {
fremovexattr(&destination, &name).with_context(|| {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On CentOS Stream 10 the root of a tmpfs mounted below /var has no security.selinux, so this tries to strip the label from the new placeholder and SELinux refuses: restoring directory metadata failed: xattrs: Removing destination xattr security.selinux: Permission denied (os error 13). Keeping the destination's label when the source has none fixes it.

Comment thread crates/lib/src/install.rs

/// Relative paths below `/var` to leave out while preserving `/var`.
#[clap(long = "preserve-var-skip", value_name = "PATH")]
pub(crate) preserve_var_skip: Vec<String>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing ties this to --preserve-var, so without it the paths are silently ignored. And install_to_existing_root now loads and validates preserve-var-skip from the install config on every run (L3185-3190), so a bad entry there breaks to-existing-root even without --preserve-var. Maybe requires = "preserve_var", and validate only when it's set?

Comment thread docs/src/bootc-installation.7.md Outdated
useful for preserving network settings, user accounts, or application data from the
previous installation.

With the OSTree backend, `--preserve-var` copies the old `/var` into the new

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could use a sentence on SSH, or a default exclusion. fedora-bootc:44 and centos-bootc:stream10 both ship cloud-init, and where cloud-init is active sshd-keygen@ stands down for it. The host's /var/lib/cloud is carried over, so cloud-init sees the same instance and doesn't create host keys, and the old keys stay in the old /etc. After each successful migration I could check (F44 and CS10, NoCloud), there were no host keys, so sshd fails and the host is unreachable over SSH. In the one run where nothing was copied, cloud-init generated new keys.

The /var copy needs careful filtering around exclusions and nested mounts,
and must avoid publishing partial entries or overwriting existing data.
Isolating the copy engine and its tests makes these filesystem guarantees
reviewable before adding installer orchestration.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
A signal or failed fsfreeze could leave /var/log frozen because the cleanup handlers were installed afterward. Install traps before the freeze attempt and use one EXIT cleanup path that preserves the original status.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode runner currently asks TMT to discover directly from a large checkout, which makes TMT 1.69 traverse ignored build output. Discover from a temporary sanitized snapshot instead, and retain the local OCI archive for guest sync.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The local package-mode test image must use OSTree regardless of the caller's variant. Forward the other configured build settings and override the variant only on the nested build.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Keep root-level Podman build secrets out of the guest snapshot and clean them up if the build or test setup fails.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode test currently removes every root build-secret pathname, including files that were present before the test began. Snapshot matching pathnames before installing cleanup and remove only new matches while preserving command status.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Podman build secret paths are shared temporary paths, so cleanup from this runner could delete files owned by concurrent builds or users. Leave lifecycle management to Podman and exclude stale artifacts from the test snapshot.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Fedora package-mode guests can keep /home outside /var, so preserve the complete lexical TMT run tree under /var/home before migration. Boot the source with SELinux disabled and verify the target relabels fresh application and PostgreSQL state when it starts enforcing.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode VM can expose /home as a real directory, so the lexical controller workdir may not be preserved by migration. Derive the destination from the resolved TMT root and run ID, and assert test SSH key labels against target policy in both SELinux phases.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The Fedora package-mode test mounts /var/lib/pgsql from the source guest, but that path is not guaranteed to exist. Create it before running the target-image PostgreSQL initializer.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The guest creates the PostgreSQL mountpoint as root, but initdb needs it to be writable by postgres. Set its numeric ownership from the target image and restrict its mode before database setup.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The PostgreSQL data directory must be a normal directory beneath the bind mount, not the mountpoint itself. Seed it through the target image as its postgres user so the preserved data uses the target UID/GID and remains compatible with the post-migration service.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The one-shot target-image container does not start systemd, so PostgreSQL's default socket directory may be unavailable. Configure a /tmp socket and connect the seed queries over loopback TCP.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The target image already provides /var/lib/pgsql, while /mnt may not exist. Seed in the mounted data directory so the state survives preserve-var and is available to the target service.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Check that the target image already contains /var/lib/pgsql before attaching a bind mount there. This ensures the seed run cannot pass by relying on Podman's bind-mount behavior to create a missing destination.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The full /boot tar snapshot is sensitive to incidental metadata changes, which can make a rejected preflight appear to mutate boot configuration. Compare only the paths and contents of boot-critical files instead.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The rejected preflight check should detect changes to all active EFI loader entries, including the architecture-independent fallback path. Record symlink targets alongside referent content digests so retargeting links is visible.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The package-mode guest has a normal BLS entry before migration, so the preflight snapshot must not require an OSTree-prefixed name.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
A leftover OSTree BLS entry must not satisfy the package-mode baseline check.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
The source kernel may disable SELinux, preventing restorecon from writing security.selinux on the mounted target deployment. Use the target policy to label the generated test-only keys directly and verify them before reboot.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
bootc status reports the local test image with an explicit :latest tag, while the test variable omits the implicit tag. Accept that exact normalized reference without loosening the assertion to other tags or images, and print status details if it still fails.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Mountpoint placeholders represent empty nested-mount boundaries and may have no target-policy label. Preserve the staging-created label for those placeholders only; copied data still requires a policy match.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Only empty mountpoint placeholders can lack a target policy label. Verify the staged directory through its no-follow handle before accepting an unmatched label, so malformed or unexpectedly populated placeholders fail before publication.

Assisted-by: AI
Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Treat preserve-var as experimental until the interface is stabilized. Require an explicit --experimental acknowledgement, document the opt-in, and exercise it in the package-mode TMT test.

Assisted-by: AI

Signed-off-by: ckyrouac <ckyrouac@redhat.com>
Keep the full behavior and safety caveats for /var migration in the experimental documentation, and point to it from the stable installation and configuration references.

Assisted-by: AI

Signed-off-by: ckyrouac <ckyrouac@redhat.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/documentation Updates to the documentation area/install Issues related to `bootc install` ci/tier-1 Run CI for tier-1 OS (centos-10) only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants