Problems running `apt` on deployed Jumpbox

[dohq]

Hi
Thanks for great job.
I updated v4.4.6 to v4.5.0
but use uaac comannd return Error message.

$ uaac
/var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/yaml.rb:5:in `<top (required)>':
It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
/var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require': libyaml-0.so.2: cannot open shared object file: No such file or directory - /var/vcap/store/
jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/x86_64-linux/psych.so (LoadError)
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/psych.rb:8:in `<top (required)>'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/yaml.rb:6:in `<top (required)>'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/gems/ruby-2.4.4/gems/cf-uaac-4.1.0/lib/uaa/cli/config.rb:14:in `<top (required)>'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/gems/ruby-2.4.4/gems/cf-uaac-4.1.0/lib/uaa/cli/common.rb:15:in `<top (required)>'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/gems/ruby-2.4.4/gems/cf-uaac-4.1.0/lib/uaa/cli/runner.rb:14:in `<top (required)>'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/gems/ruby-2.4.4/gems/cf-uaac-4.1.0/lib/uaac_cli.rb:15:in `<top (required)>'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /var/vcap/store/jumpbox/home/pfmanager/.rvm/rubies/ruby-2.4.4/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /u/pfmanager/.rvm/gems/ruby-2.4.4/gems/cf-uaac-4.1.0/bin/uaac:17:in `<top (required)>'
        from /u/pfmanager/.rvm/gems/ruby-2.4.4/bin/uaac:23:in `load'
        from /u/pfmanager/.rvm/gems/ruby-2.4.4/bin/uaac:23:in `<main>'
        from /u/pfmanager/.rvm/gems/ruby-2.4.4/bin/ruby_executable_hooks:24:in `eval'
        from /u/pfmanager/.rvm/gems/ruby-2.4.4/bin/ruby_executable_hooks:24:in `<main>'

I think was the need libyaml-2-0 package.
sudo apt install libyaml-2-0

but return new error

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  libyaml-0-2
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 0 B/47.6 kB of archives.
After this operation, 166 kB of additional disk space will be used.
dpkg: unrecoverable fatal error, aborting:
 unknown group 'messagebus' in statoverride file
W: No sandbox user '_apt' on the system, can not drop privileges
E: Sub-process /usr/bin/dpkg returned an error code (2)

Have you noticed the cause?

sorry my poor engilish.

thanks

[jhunt]

Hi!

I have not personally seen this, but I don't know that I've used the UAA ruby CLI since we switched to Xenial. What happens if you redeploy on an Ubuntu Trusty stemcell?

[dohq]

oops!
I totally thought default installed uaac cli.
I manually installed cf-uaac ago...
sorry.
but I think that it is also a problem that the apt command can not be executed.
Should I create another issue?
thanks.

[jhunt]

I have no idea what's wrong with apt.

Can you provide more information about stemcell version / APT repository configuration? I'll see if I can reproduce this issue.

(we can keep using this GH issue; i'll retitle it)

[dohq]

Thankyou fo reply.
ok, I use version

stemcell

bosh-aws-xen-hvm-ubuntu-xenial-go_agent 170.19

and
apt repositoty source

$ cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu xenial main universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main universe multiverse
deb http://security.ubuntu.com/ubuntu xenial-security main universe multiverse

My first use create-env jumpbox-boshrelease version is v4.4.5.
after that updating 4.4.6 -> 4.5.0

Let me know if we can give you any other information.

[jhunt]

Deploying 4.5.0 directly, I get the following:

# apt-get update
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial InRelease [17.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [600 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [249 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [415 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [163 kB]
Get:9 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial/main amd64 Packages [6,648 B]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [902 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [365 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [719 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [295 kB]
Fetched 3,948 kB in 2s (1,570 kB/s)
Reading package lists... Done

# apt install libyaml-2-0
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libyaml-2-0

[dohq]

I trying install is libyaml-0-2
sorry I missed package name.

but apt-get update output diffarent my jumpbox...

jumpbox# apt update
Get:1 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial InRelease [17.5 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial/main amd64 Packages [6,648 B]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [902 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [600 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [720 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [295 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.6 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [415 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [163 kB]
Get:13 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [5,600 B]
Get:14 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [2,676 B]
Fetched 3,363 kB in 3s (841 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
13 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: No sandbox user '_apt' on the system, can not drop privileges

W: No sandbox user '_apt' on the system, can not drop privileges

Hmm...

[jhunt]

I don't believe that the _apt sandbox user warning is the issue here; it's just a warning, and there's another error about statoverrides in the original post. Also see https://askubuntu.com/questions/882039/no-sandbox-user-apt-on-the-system-can-not-drop-privileges

Some digging around on the 'net makes it sound like something was installed, in the past, that references a user that has since been removed from /etc/passwd.

From my fresh 4.5.0 jumpbox, I get this:

[13:44:09] bosh_696c4995e6ef487@jumpbox ~
$ id messagebus
uid=106(messagebus) gid=110(messagebus) groups=110(messagebus)

Looking for any files owned by UID 106 nets me this:

[13:44:24] bosh_696c4995e6ef487@jumpbox ~
$ sudo find / -uid 106 2>/dev/null
/proc/507
/proc/507/task
/proc/507/task/507
/proc/507/task/507/net
/proc/507/task/507/attr
/proc/507/task/507/attr/selinux
/proc/507/task/507/attr/smack
/proc/507/task/507/attr/apparmor
/proc/507/net
/proc/507/attr
/proc/507/attr/selinux
/proc/507/attr/smack
/proc/507/attr/apparmor

And finally, checking the process table for PID 507 (using the '[p]id trick'):

[13:45:20] bosh_696c4995e6ef487@jumpbox ~
$ ps -ef | grep ' [5]07 '
message+   507     1  0 Jan22 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation

Which makes sense; dbus is the message bus, so it's user should be messagebus.

A few questions:

1. Do you have a dbus-daemon process on your system, and if so, what is its effective UID?
2. Do you have a messagebus user in /etc/passwd? Feel free to post id messagebus and getent passwd messagebus output, it's safe.
3. Do you have this issue if you install a fresh 4.4.x jumpbox deployment, and then upgrade immediately to 4.5.x?

[dohq]

Thnkyou for reply.
I will try that this weekend.

[dohq]

Sorry very very late for reply...

From then on 4.4 → 4.5, I tried as much as I can think of, but did not reproduce. . .
Perhaps there was a problem with my operation.
You can close this issue once.
I am sorry for taking the trouble.

[jhunt]

No worries. Glad the update worked out for you!

[krutten]

More details on the issue. On boxes upgraded from Trusty to Xenial, packages installed via trusty may not correctly work on Xenial. Trying to install new packages (or replace missing packages) fails. This is often noticed when trying to Ruby after libyaml goes missing and RVM can't apt install it again but happens for any package, for example nmap.

# apt install nmap
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libblas-common libblas3 liblinear3 liblua5.2-0 lua-lpeg ndiff python-bs4 python-chardet python-html5lib python-lxml python-six
Suggested packages:
  liblinear-tools liblinear-dev python-genshi python-lxml-dbg python-lxml-doc
The following NEW packages will be installed:
  libblas-common libblas3 liblinear3 liblua5.2-0 lua-lpeg ndiff nmap python-bs4 python-chardet python-html5lib python-lxml python-six
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 6,056 kB of archives.
After this operation, 27.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libblas-common amd64 3.6.0-2ubuntu2 [5,342 B]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libblas3 amd64 3.6.0-2ubuntu2 [147 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 liblinear3 amd64 2.1.0+dfsg-1 [39.3 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 liblua5.2-0 amd64 5.2.4-1ubuntu1 [106 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 lua-lpeg amd64 0.12.2-1 [28.3 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 python-bs4 all 4.4.1-1 [64.2 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial/main amd64 python-chardet all 2.3.0-2 [96.3 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 python-six all 1.10.0-3 [10.9 kB]
Get:9 http://archive.ubuntu.com/ubuntu xenial/main amd64 python-html5lib all 0.999-4 [83.1 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python-lxml amd64 3.5.0-1ubuntu0.1 [818 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial/main amd64 ndiff all 7.01-2ubuntu2 [20.1 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/main amd64 nmap amd64 7.01-2ubuntu2 [4,638 kB]
Fetched 6,056 kB in 2min 3s (49.1 kB/s)
dpkg: unrecoverable fatal error, aborting:
 unknown group 'messagebus' in statoverride file
W: No sandbox user '_apt' on the system, can not drop privileges
E: Sub-process /usr/bin/dpkg returned an error code (2)
==[]=[ 14:37:09 ]=[ jumpbox/1 ]=[ ~ ]=[]==
#

This issue is something removed messagebus and _apt users and groups from the /etc/passwd and /etc/group files. apt install can be fixed by properly determining the UID and GID of messagebus and adding it back

# ps auwwx |grep [d]bus
106         522  0.0  0.0  42888  3784 ?        Ss   Sep13   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
==[]=[ 14:38:41 ]=[jumpbox/1 ]=[ ~ ]=[]==
#

# cat /var/lib/dpkg/statoverride
root crontab 2755 /usr/bin/crontab
root messagebus 4754 /usr/lib/dbus-1.0/dbus-daemon-launch-helper
==[]=[ 14:41:52 ]=[jumpbox/1 ]=[ ~ ]=[]==

# ls -la /usr/lib/dbus-1.0/dbus-daemon-launch-helper
-rwxr-xr-- 1 root 110 42992 Jun 10 19:46 /usr/lib/dbus-1.0/dbus-daemon-launch-helper
==[]=[ 14:41:58 ]=[jumpbox/1 ]=[ ~ ]=[]==
#

Then run

addgroup --system -gid 110 messagebus
Adding group `messagebus' (GID 110) ...
Done

adduser --system --uid 106 --gid 110 --home /var/run/dbus messagebus
Adding system user `messagebus' (UID 106) ...
Adding new user `messagebus' (UID 106) with group `messagebus' ...

adduser --force-badname --system --home /nonexistent --no-create-home --quiet _apt || true

Apt installs will work for a while, but some process will remove the messagebus and _apt again.

If you try and create the messagebus user with a different UID/GID then originally used, the jumpbox will have permission issues on reboot and can no longer boot.

Does not seem to happen to Jumpboxes created on Xenial, just upgraded from Trusty.