AppImage not unmounting when running as systemd service

[nickjmeyer]

I have an AppImage that when run on the command line, works just fine. Executes as expected and cleans up the mount point on exit. However, when run as a systemd service, regardless if it exists cleanly or with a failure, the mount point is left behind in /tmp. This ends up accumulating a bunch of stale fuse mounts and after long enough the mount limit is reached and no more appimages can be run.

Any ideas why running as a systemd service might prevent the mount point from getting cleaned up?

[probonopd]

While the service is running, the AppImage should be mounted. Once the application inside the AppImage exits, the AppImage should get unmounted. This only works when the application inside the AppImage is quit properly. How does the application inside your AppImage get quit?

[nickjmeyer]

I was able to reproduce this behavior with the following minimal example program:

int main() { return 0; }

[probonopd]

Then I don't know the answer to your question as I have never tried to run applications with systemd.

[nickjmeyer]

I've noticed fuse mounts are fussy with users. For example, I've tried to shell out to a sub-command prefixed with sudo and from some research it seems the allow_user config flag is needed to allow other users besides the one who mounted it to execute within the mount.

Any chance there is something similar regarding the unmounting?

[probonopd]

Just to be sure, in your minimal example program your AppImage properly unmounts when NOT launched by systemd?

As which user is systemd running your AppImage?

[nickjmeyer]

Just to be sure, in your minimal example program your AppImage properly unmounts when NOT launched by systemd?

That's correct. The AppImage works just as expected when run outside systemd (e.g., on the command line) and on exit the fuse mount is properly unmounted.

As which user is systemd running your AppImage?

I might be misunderstanding the question, but I'm running the service with this line in the service file to specify the user:

[Service]
User=myuser

[probonopd]

Unfortunately I don't know the answer, or how to debug this. But we can leave this issue open in case someone more knowledgeable about systemd sees this.

[nickjmeyer]

Sounds good. Thanks for chiming in so quickly!

[probonopd]

Please retry with an AppImage made using the latest https://github.com/AppImage/appimagetool/releases/tag/continuous, it might have been fixed in the lastest AppImage runtime by

• auto_unmount #93

[nickjmeyer]

I extracted the appimage I used before and created a new one from that AppDir using the latest appimagetool. However, I'm still seeing the same issue. It appears to be the same issue @lalten saw in #92.

I tested this service file (appimage.service):

[Unit]
Description=Testing appimages

[Service]
Type=oneshot
ExecStart=/home/nickjmeyer/appimage_test_new.AppImage

[Install]
WantedBy=multi-user.target

Using the following:

sudo systemctl enable /path/to/appimage.service
sudo systemctl start appimage.service

It works sometimes, but then other times the appimage hangs on exit. I'll see stdout in journalctl -u appimage.service so I can see the tool ran and tried to exit, but the call to systemctl start hangs. So I'm assuming it's hanging while trying to unmount. When this occurs, I'll see an orphaned mount in /tmp.

[mehmetakbulut]

@nickjmeyer Could you add [Service] RemainAfterExit=true in appimage.service and share outputs of following?

export SYSTEMD_PAGER=cat
sudo -E systemctl status appimage.service
sudo -E systemctl start appimage.service 
sudo -E systemctl status appimage.service
sudo -E systemctl stop appimage.service
sudo -E systemctl status appimage.service

[nickjmeyer]

By adding RemainAfterExit=true, it seemed to be more reliable. I didn't see the same hanging when the service was started like I did prior. Did you intend for that to be a possible fix? Or were you using that to get better output in status?

Here is the service file with the change:

nickjmeyer@penguin:~$ cat appimage_test.service 
[Unit]
Description=Testing appimages

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/home/nickjmeyer/appimage_test_new.AppImage

[Install]
WantedBy=multi-user.target

And here's the output:

nickjmeyer@penguin:~$ export SYSTEMD_PAGER=cat
nickjmeyer@penguin:~$ sudo -E systemctl status appimage_test.service
○ appimage_test.service - Testing appimages
     Loaded: loaded (/etc/systemd/system/appimage_test.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead) since Fri 2024-12-13 13:00:30 EST; 24s ago
   Duration: 6.922s
   Main PID: 2847 (code=exited, status=0/SUCCESS)

Dec 13 13:00:23 penguin systemd[1]: Starting appimage_test.service - Testing appimages...
Dec 13 13:00:23 penguin appimage_test_new.AppImage[2849]: fusermount3 version: 3.14.0
Dec 13 13:00:23 penguin appimage_test_new.AppImage[2847]: Success!
Dec 13 13:00:23 penguin systemd[1]: Finished appimage_test.service - Testing appimages.
Dec 13 13:00:30 penguin systemd[1]: appimage_test.service: Deactivated successfully.
Dec 13 13:00:30 penguin systemd[1]: Stopped appimage_test.service - Testing appimages.
nickjmeyer@penguin:~$ sudo -E systemctl start appimage_test.service
nickjmeyer@penguin:~$ sudo -E systemctl status appimage_test.service
● appimage_test.service - Testing appimages
     Loaded: loaded (/etc/systemd/system/appimage_test.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (exited) since Fri 2024-12-13 13:01:05 EST; 15s ago
    Process: 2914 ExecStart=/home/nickjmeyer/appimage_test_new.AppImage (code=exited, status=0/SUCCESS)
   Main PID: 2914 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4915)
     Memory: 4.0K
     CGroup: /system.slice/appimage_test.service

Dec 13 13:01:05 penguin systemd[1]: Starting appimage_test.service - Testing appimages...
Dec 13 13:01:05 penguin appimage_test_new.AppImage[2916]: fusermount3 version: 3.14.0
Dec 13 13:01:05 penguin appimage_test_new.AppImage[2914]: Success!
Dec 13 13:01:05 penguin systemd[1]: Finished appimage_test.service - Testing appimages.
nickjmeyer@penguin:~$ sudo -E systemctl stop appimage_test.service
nickjmeyer@penguin:~$ sudo -E systemctl status appimage_test.service
○ appimage_test.service - Testing appimages
     Loaded: loaded (/etc/systemd/system/appimage_test.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead) since Fri 2024-12-13 13:01:23 EST; 3s ago
   Duration: 18.753s
    Process: 2914 ExecStart=/home/nickjmeyer/appimage_test_new.AppImage (code=exited, status=0/SUCCESS)
   Main PID: 2914 (code=exited, status=0/SUCCESS)

Dec 13 13:01:05 penguin systemd[1]: Starting appimage_test.service - Testing appimages...
Dec 13 13:01:05 penguin appimage_test_new.AppImage[2916]: fusermount3 version: 3.14.0
Dec 13 13:01:05 penguin appimage_test_new.AppImage[2914]: Success!
Dec 13 13:01:05 penguin systemd[1]: Finished appimage_test.service - Testing appimages.
Dec 13 13:01:23 penguin systemd[1]: appimage_test.service: Deactivated successfully.
Dec 13 13:01:23 penguin systemd[1]: Stopped appimage_test.service - Testing appimages.
nickjmeyer@penguin:~$ sudo -E systemctl start appimage_test.service
nickjmeyer@penguin:~$ sudo -E systemctl status appimage_test.service
● appimage_test.service - Testing appimages
     Loaded: loaded (/etc/systemd/system/appimage_test.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (exited) since Fri 2024-12-13 13:01:33 EST; 1s ago
    Process: 2935 ExecStart=/home/nickjmeyer/appimage_test_new.AppImage (code=exited, status=0/SUCCESS)
   Main PID: 2935 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4915)
     Memory: 4.0K
     CGroup: /system.slice/appimage_test.service

Dec 13 13:01:33 penguin systemd[1]: Starting appimage_test.service - Testing appimages...
Dec 13 13:01:33 penguin appimage_test_new.AppImage[2937]: fusermount3 version: 3.14.0
Dec 13 13:01:33 penguin appimage_test_new.AppImage[2935]: Success!
Dec 13 13:01:33 penguin systemd[1]: Finished appimage_test.service - Testing appimages.
nickjmeyer@penguin:~$ sudo -E systemctl stop appimage_test.service
nickjmeyer@penguin:~$ sudo -E systemctl status appimage_test.service
○ appimage_test.service - Testing appimages
     Loaded: loaded (/etc/systemd/system/appimage_test.service; enabled; preset: enabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead) since Fri 2024-12-13 13:01:39 EST; 1s ago
   Duration: 6.270s
    Process: 2935 ExecStart=/home/nickjmeyer/appimage_test_new.AppImage (code=exited, status=0/SUCCESS)
   Main PID: 2935 (code=exited, status=0/SUCCESS)

Dec 13 13:01:33 penguin systemd[1]: Starting appimage_test.service - Testing appimages...
Dec 13 13:01:33 penguin appimage_test_new.AppImage[2937]: fusermount3 version: 3.14.0
Dec 13 13:01:33 penguin appimage_test_new.AppImage[2935]: Success!
Dec 13 13:01:33 penguin systemd[1]: Finished appimage_test.service - Testing appimages.
Dec 13 13:01:39 penguin systemd[1]: appimage_test.service: Deactivated successfully.
Dec 13 13:01:39 penguin systemd[1]: Stopped appimage_test.service - Testing appimages.

[mehmetakbulut]

Did you intend for that to be a possible fix?

I suspected that the MR by @probonopd might have fixed the underlying issue and a bit systemd configuration magic might make service state more aligned with what you expect.

By default, a service goes to inactive state when main PID exits. In the case of a hello world example, (i.e. print & exit) you expect the process to exit and that to be considered a success, i.e. RemainAfterExit=true. Otherwise it is as if you execute systemctl stop on process exit which complicates things. The behavior you want depends entirely on your use case. 🙂

Or were you using that to get better output in status?

export SYSTEMD_PAGER=cat was to disable pager behavior so we can get the output in its entirety.

And here's the output:

It looks like no processes were left behind at any point. So I think this is good?
Is /tmp clear of mountpoints as well?

If /tmp still has mountpoints, that might be an additional bug, potentially with appimage runtime or further upstream, because we know with absolute certainty that appimage runtime has exited. By default, systemd runs each service in a dedicated cgroup which allows it to cull all processes when service goes inactive.

[nickjmeyer]

By default, a service goes to inactive state when main PID exits.

This is probably my lack of understanding of fuse, but I don't quite understand how this would affect the service on exit. Is the fuse subsystem waiting for the main PID to exit in order to unmount, but it's not exiting for some reason? Curious if you have any insight into why this behavior change allows the unmounting to happen cleanly.

Is /tmp clear of mountpoints as well?

Yup! I ran this

for i in {1..100}; do echo "> attempt $i"; echo "start"; sudo -E systemctl start appimage_test.service; echo "stop"; sudo -E systemctl stop appimage_test.service; echo "orphaned: $(ls -a /tmp/ | grep -i .mount_appim | wc -l)"; sleep 5; done

to stress test it. Each of the 100 iterations started / stopped as expected and there were no orphaned mounts. So this definitely seems like a fix. I can easily accommodate needing to explicitly stop before the next start.

Thanks a ton for the help!