Upgrade of nightly might fail if package was upgraded manually

[tailhook]

The output of upgrade command:

admin@ip-172-26-3-249:~$ ./edgedb server upgrade --nightly
Job for [email protected] failed because the control process exited with error code.
See "systemctl status [email protected]" and "journalctl -xe" for details.
Error: failed to dump "default": process "systemctl" "--user" "start" "edgedb-server@default" failed: exit code: 1

Log is:

Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > sys.exit(main())
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > return self.main(*args, **kwargs)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > rv = self.invoke(ctx)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > return ctx.invoke(self.callback, **ctx.params)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > return callback(*args, **kwargs)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > server_main(**kwargs)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > run_server(ServerConfig(**kwargs))
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > need_cluster_restart = _init_cluster(cluster, args)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > need_restart = asyncio.run(bootstrap.bootstrap(cluster, bootstrap_arg
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/asyncio/runn
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > return loop.run_until_complete(main)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     uvloop/loop.pyx, line 1456, in uvloop.loop.Loop.run_until_complete
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > await _check_data_dir_compatibility(conn)
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:     /usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-package
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]:         > raise errors.ConfigurationError(
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]: ---- Details ----
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]: Details: The database instance was initialized with EdgeDB format version 20200
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]: Hint: You need to recreate the instance and upgrade using dump/restore.
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]: edb.errors.ConfigurationError: database instance incompatible with this version
Jul 15 10:15:57 ip-172-26-3-249 edgedb-server-1-alpha4[25945]: DETAILS: The database instance was initialized with EdgeDB format version 20200
Jul 15 10:15:58 ip-172-26-3-249 systemd[25715]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Jul 15 10:15:58 ip-172-26-3-249 systemd[25715]: Failed to start EdgeDB Database Service, instance "default".

To show nicer error this we might consider running edgedb in foreground and analyze output. Or alternatively, we can implement edgedb server logs and parse output of that command internally.

[elprans]

The most reliable solution here would be to slot each nightly separately so they're treated as major releases.

[elprans]

Moving to edgedb-pkg

[elprans]

@ambv Let's switch nightlies to encode nightly-{date and hour} in the package slot, e.g. instead of edgedb-server-1-alpha5 we would have edgedb-server-1-alpha5-nightly2020061720

[tailhook]

1. I'll need to upgrade logic of upgrade --nightly after that
2. I'm still not sure if this is a best thing to do, as we then need to garbage-collect those packages installed on users machines. And this may interfere with any instances that aren't tracked by server install (and I expect nightly users to have a much larger percentage of advanced users), so garbage collecting nightlies automatically can be a problem for some users.

[elprans]

We can make the garbage-collection into an explicit command and advise users to run it if we detect no instances using that version. This should also be useful for non-nighly releases too.

On the other hand, breakage caused by apt-get upgrade is significantly worse, and I see no other reliable way to avoid it.

[tailhook]

Maybe apt-mark hold package right after installing? With yum it looks like the plugin is needed.

But yes, given that we need garbage collection anyway it might be a good idea to have separate nightly packages.