Hardened Docker base images fail due to usrmerge incompatibility on Debian 12

[suii2210]

While experimenting with switching Airflow CI images to Docker Hardened Images (dhi.io/python:*), CI builds consistently fail during OS dependency installation on Debian 12.
The failure is caused by usrmerge being implicitly installed as a dependency (via lsb-release) and breaking during its post-installation step due to filesystem restrictions in hardened images.
This issue is deterministic and cannot be resolved through script-level workarounds.

Environment

• Base image: dhi.io/python:3.12-debian12-dev
• OS: Debian 12 (bookworm)
• Context: Airflow CI Docker build
• Affected script: install_os_dependencies.sh

Observed Failure
During apt-get install, usrmerge is pulled in automatically:

Unpacking usrmerge (37~deb12u1) ...
Setting up usrmerge (37~deb12u1) ...
removed '/lib64'
/var/lib/dpkg/info/usrmerge.postinst: line 70: /usr/bin/rmdir: cannot execute: required file not found
dpkg: error processing package usrmerge (--configure)

This leaves dpkg in a half-configured state and causes the Docker build to fail permanently.

Root Cause Analysis

• usrmerge is installed indirectly, primarily via lsb-release
• Debian executes usrmerge.postinst immediately during unpack
• Hardened Docker images intentionally restrict filesystem operations
• usrmerge attempts to:
• remove /lib64
• execute /usr/bin/rmdir
• These operations are blocked by the hardened image

Once usrmerge reaches the unpack phase, no cleanup, purge, or hold can recover the build.

What Was Tried (and Why It Does Not Work)

The following approaches were tested and confirmed ineffective:

• apt-mark hold usrmerge
• apt-get purge usrmerge
• Post-install cleanup or symlink fixes
• Conditional logic inside install_os_dependencies.sh
• Reordering or splitting apt installs
  All of these fail because usrmerge executes its post-install script before any script-level logic can intervene.

Validation

Switching back to a non-hardened base image confirms the diagnosis:

FROM python:3.12-slim-bookworm

• CI build completes successfully
• No usrmerge installation
• No dpkg corruption
  This proves the failure is caused by base-image incompatibility, not Airflow scripts.

Conclusion

This is not a bug in Airflow’s Docker scripts.

It is an incompatibility between:
Debian 12’s usrmerge lifecycle
and Docker Hardened Images’ filesystem restrictions

As long as usrmerge is allowed to unpack, CI builds on hardened images will fail.

[potiuk]

I guess you could replace lsb-release - > from what I remember, it's only used to easier retrieve the release version and can be likely replaced with direct reading of the release version

[suii2210]

I guess you could replace lsb-release - > from what I remember, it's only used to easier retrieve the release version and can be likely replaced with direct reading of the release version

I did try replacing lsb-release entirely.
Unfortunately, on dhi.io/python:3.12-debian12-dev, usrmerge is still pulled in during apt-get install and executes its postinst during unpack, leaving dpkg in a broken state.
This suggests the failure isn’t tied to lsb-release specifically, but to usrmerge being incompatible with the hardened image filesystem constraints.

[potiuk]

I looked it up - and I would be very surprised to see usrmerge not working- https://wiki.debian.org/UsrMerge seems to be required by Debian starting from bookworm and there is a long discussion about adopting it, so I guess hardened image based on debian has to support it this way or the other.

[suii2210]

That matches what I observed. Since usrmerge is required starting with Debian 12, a Debian-based hardened image that blocks its postinst lifecycle is effectively incompatible. From Airflow’s side, this doesn’t seem solvable without either avoiding hardened Debian images or relying on upstream fixes in the base image.

[shahar1]

@suii2210 hey, did you manage to overcome the latest issue?

[suii2210]

hello @shahar1
Since Debian 12 requires usrmerge, and the hardened image blocks its post-install lifecycle, any Debian-based hardened image is effectively incompatible. Once usrmerge starts unpacking, dpkg is left in a broken state and there’s no script-level way to recover.
At this point the only viable paths seem to be:
avoiding Debian-based hardened images, or
relying on an upstream fix in the hardened base image to properly support usrmerge.
Happy to test again if the base image behavior changes, but I don’t see a downstream workaround.