Ubuntu and emergency mode because of device timeout/systemd dependency cycle (OpenSSH-Configuration problem mentioned)

[Marcel-Scherzinger]

Hello and thanks for making it possible to manage servers on other distros with Nix and sorry if this is not the right place to ask such questions!

I recently got two VMs from my institution and after some config copying I used Nix with system-manager to manage them. Since then, sometimes when I reboot the VMs, they go into emergency mode and I have to go to the VM program to press Ctrl+D making them boot.
The logs then show that there is a timeout waiting for one device and also some cyclic dependencies in systemd services.

As I already had a problem with one of system-manager's systemd services (after managing openssh with it, I suddenly got a post-quantum warning as it looks like the default config sets insecure algorithms. Setting the nix option to null removes the option from the generated config and enables openssh to freely choose, resolving the error.) I wanted to ask if this could be related to this project in some way (because of some systemd service definitions that collide with Ubuntu's).
If not, then I know that I have to search elsewhere, as I'm by now quite clueless how to resolve those sporadic errors.

May 12 17:36:56 vmName systemd[1]: var.mount: Found ordering cycle on systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZz>
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on local-fs-pre.target/start
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on systemd-tmpfiles-setup-dev.service/start
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on userborn.service/start
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on var.mount/start
May 12 17:36:56 vmName systemd[1]: var.mount: Job systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZzs9qeCDhdEfZr3WBGB5iy>
May 12 17:36:56 vmName systemd[1]: var.mount: Found ordering cycle on systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZz>
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on local-fs-pre.target/start
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on systemd-tmpfiles-setup-dev.service/start
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on userborn.service/start
May 12 17:36:56 vmName systemd[1]: var.mount: Found dependency on var.mount/start
May 12 17:36:56 vmName systemd[1]: var.mount: Job local-fs-pre.target/start deleted to break ordering cycle starting with var.mount/start
May 12 17:36:56 vmName systemd[1]: srv.mount: Job srv.mount/start failed with result 'dependency'.
May 12 17:36:56 vmName systemd[1]: Dependency failed for systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZzs69EENWS3fEmU>
░░ Subject: A start job for unit systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZzs69EENWS3fEmUvfrdB7KPb8p4onnXFha2.ser>
░░ Defined-By: systemd

[picnoir]

To me, it seems like it could be related to userborn depending on var.mount/start. I'm not entirely sure why we did not hit that anywhere else.

The logs you sent are truncated, could you paste the full lines for:

May 12 17:36:56 vmName systemd[1]: var.mount: Job systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZzs9qeCDhdEfZr3WBGB5iy>
May 12 17:36:56 vmName systemd[1]: var.mount: Found ordering cycle on systemd-fsck@dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dXCXZTuMnP0jNLoZCojkmzEgoRgS2pZz>

Could you also paste the content of the failing mount unit (or your fstab file if you did not use the systemd mount units directly). Feel free to DM me that on Matrix or elsewhere if there's some sensitive data in there.

[Marcel-Scherzinger]

Thanks for your help. I sent some files to you (@ninjatrappeur:alternativebit.fr) I didn't touch the storage units by myself but I have some details to add:

• After such an emergency reboot I have to manually reenable some systemd services, most importantly systemctl restart systemd-resolved.service to reenable dns
• After further reboots the issue doesn't occur again, only if I run system-manager again for deployment when it really needs to change something. I tested it e. g. by reordering one line in my Caddyfile that wouldn't have impact on caddy but would present the file as changed to nix (caddy with systemd service from nixpkgs.)

[picnoir]

Woops, my website is not up to date, I don't have access to this account anymore 😓 picnoir at the same domain is the right one.

Hmm, maybe the loops is related to system-manager.target somehow? Can you send the output of systemd-analyze plot?

[picnoir]

Overall, the best way to move forward here would be for you to create a minimal vm test reproducing this issue.

[Marcel-Scherzinger]

I sent you everything to the other account and the output of systemd-analyze plot when the system is running normally, tell me if you also need it after an emergency mode start.

The minimal working example is clearly a good idea but I'm not able to do it as my institution only provides me with these two machines and both have the issue in the same situations. They are by now quite minimal, I only installed docker, openssh, nix and the rest using a very small nix flake. So I don't have many changes but I sadly don't have a recovery point to the time before installing Nix for the first time.

[picnoir]

Just had a look at the boot logs and disk layout. I'm not sure what the issue here, but I have two remarks:

1. the unit dependency cycle posted up there is triggered by the unit of a LVM mount.
2. some LVM mounts are timing out during the boot. Not sure why, not sure it's related to the dependency cycle.

Overall, I'm not sure if it's an issue between system-manager and LVM mounts (not sure we tested that so far) or if it's an issue related to the hypervisor/disk mounts.

I think the best next move would be trying to replicate this disk layout in the container test base ubuntu image and see if we're facing the same issue or not.

[Marcel-Scherzinger]

Thanks for your help. I uploaded the files for others interested in the topic. If I find something I will post it here but I'm unsure because of the LVM thing https://github.com/Marcel-Scherzinger/vm-problem-with-system-manager-logs

[Marcel-Scherzinger]

I now changed the fstab entry for /var to defaults 0 0 and now it booted normally after deploying with system-manager. In my run, there was this time also no timeout and it rebooted very quickly. I will watch the situation and will write another time if it still occurs:

New line:

# /var was on /dev/templateVG/var during curtin installation
/dev/disk/by-id/dm-uuid-LVM-XCXZTuMnP0jNLoZCojkmzEgoRgS2pZzs9qeCDhdEfZr3WBGB5iyDAyFgTa8D6DSO /var ext4 defaults 0 0

Then I ran: sudo systemctl daemon-reload and rebooted