Start specific Julia version if it is specified in Project.toml/Manifest.toml

[davidanthoff]

This is probably a bit speculative at this point, but it would be fairly simply to look at a Project.toml/Manifest.toml if one is specified via the --project command line arguments and see whether these specify a specific Julia version. If they do, run things in that specific version.

Of course, right now there is no official way to specify a Julia version in a Manifest.toml. For Project.toml, the situation is a bit different: at least for packages one can specify a version in the compat section. The mybinder integration for example looks for a julia entry in the compat section to decide what Julia version to use. We could do something similar, for example if there was an entry julia = "=1.3.1" in the Project.toml, we could start Julia 1.3.1.

But maybe this should also just wait until there is an official way to record Julia versions in Manifest.toml files.

[davidanthoff]

The entire handling of this from rustup is interesting: https://rust-lang.github.io/rustup/overrides.html. We might want to add the same options here as well.

[ericphanson]

Of course, right now there is no official way to specify a Julia version in a Manifest.toml

Just to say, Pkg in Julia 1.7+ already does this! See JuliaLang/Pkg.jl#2561 and followups.

[davidanthoff]

Ah, nice!

I think the big question is how we would want this to actually behave. One option would be that we always try to use the exact Julia version that is recorded in the manifest, but maybe that is a bit too drastic, as one probably would want to normally at least use the closest minor matching version, or something like that... Not really sure what the right design here is :)

[johnnychen94]

When this is supported, I would imagine adding some alias

alias julia='juliaup --project'
alias julia-1='juliaup launch 1'

and everything just works smarter.

[wolthom]

@davidanthoff Is someone already working on this?

We briefly discussed this topic in the juliaup-dev Slack channel. There you mentioned that it would be great to offer the options (or something similar) in the rustup overrides section.

I think the implementation of a layered configuration approach itself shouldn't be too bad. Two crates I've used for such purposes before are figment and config. If you prefer fewer dependencies in this instance, it should be possible to emulate it via HashMaps as well.

Could you outline some of the already existing parts?

What I'm aware of:

• The existing manual override via + exists (this should be the highest priority, right?)
• Handling of Manifest.toml-files with a julia_version is not yet implemented
• A default channel can be set globally in the juliaup.json file
• Local channel overrides are not yet possible (right?)

Where I'm not sure:

• Is it possible to use environment variables already?
• How should a local override and a Manifest.toml entry interact? My guess would be to give the local override a higher priority but warn in case of conflicts.
• Similarly regarding environment variables
• Where / which configuration files should be considered? I remember some discussions about .julia vs .juliaup vs something else for config locations.

[davidanthoff]

No one is working on this, as far as I know. I thought about it, but that is it :)

In my mind, the override hierarchy should probably look like this, highest priority first:

1. A channel selector used on the command-line, such as julia +beta (implemented).
2. The JULIAUP_CHANNEL environment variable (not implemented).
3. A directory override that is stored in the central juliaup.json config file in the Julia depot (not implemented).
4. The .juliaup.toml file (not implemented, could also come later).
5. The Julia version stored in the Manifest.toml (not implemented).
6. The default Juliaup channel, stored in the Juliaup.json config file in the Julia depot (implemented).

I think generally everything except 5 would store a channel name, and therefore be valid only on a specific user machine, i.e. not info that could be checked into source control.

For 4 things are very unclear to me right now. Questions are: is this a file that would be checked into git? Or is this a local config file that we would add to .gitignore? What exactly would we actually store in there? My gut feeling is that we would just not add this in a first implementation and think more about it.

I think 3 could also come later, although that is fairly simple to figure out how it should work, so maybe we should just add it now. We would need to provide a command line interface to use it, as we never want users to manually modify the juliaup.json file.

5 is going to be tricky :) The primary problem is that Manifest.toml doesn't store a channel name, but a Julia version. In general a user can have many channels installed that all provide the same Julia version. We probably also want to consider what should happen if a Manifest.toml stores say Julia version 1.7.2, but the user has a channel with Julia 1.7.3. Should that automatically be used? Should the Manifest.toml be updated? Or offered to be updated? What should happen if a Manifest.toml has a Julia version stored and there is no channel installed locally that matches even the minor version? Prompt that offers to install? In non-interactive mode, should we error and not start? There are also questions how the default project activation interacts with this. If a user does not specific --project=. on the command line, should the Manifest.toml even be considered at all?

I think maybe a pragmatic approach would be to start with an implementation of 2 and 3, and think a bit more about the other options.

I think the implementation of a layered configuration approach itself shouldn't be too bad. Two crates I've used for such purposes before are figment and config. If you prefer fewer dependencies in this instance, it should be possible to emulate it via HashMaps as well.

So as this implementation will sit on the "hot" path of everything, and also be included in the julialauncher executable which we are trying to keep small, I would probably go for a complete "by hand" implementation here... Where we for example don't even read config files that we don't need to read etc. I don't think that would be too cumbersome?

Also CCing @StefanKarpinski and @staticfloat, in particular on the order of overrides.

[staticfloat]

1. A channel selector used on the command-line, such as julia +beta (implemented).

👍

2. The JULIAUP_CHANNEL environment variable (not implemented).

Hmmm, I can see this being useful, but only in rare situations.

3. A directory override that is stored in the central juliaup.json config file in the Julia depot (not implemented).

I'm not sure why this would be used. Can you explain more about this?

4. The .juliaup.toml file (not implemented, could also come later).

I'm not sure why you would use this over a Project.toml or Manifest.toml approach.

5. The Julia version stored in the Manifest.toml (not implemented).

I like this, but I think it probably needs to be enabled somehow, as it's a breaking behavioral change. Perhaps julia +manifest, or perhaps an option set at install time.

6. The default Juliaup channel, stored in the Juliaup.json config file in the Julia depot (implemented).

The thing I'm most concerned about is that a user runs julia --project foo test.jl and sees something that works (because it uses v1.7, matching the Manifest), then they run julia, then run Pkg.activate("foo"); include("test.jl") and it breaks because their default channel is v1.8, or something like that. Also note the possibility for confusion if someone has defined JULIA_PROJECT.

I think for this manifest-matching mode to not confuse users we need one of the following:

• juliaup matches automatically, but it prints out that it's matching to a particular version when it does so.
• juliaup does not match automatically, it must be manually-enabled.

[StefanKarpinski]

Given that we already potentially autoupdate versions when starting julia via juliaup, I've increasingly been wondering what the point of having specific channels installed is. What I mean is: suppose I run julia +1.3 and I haven't added the 1.3 channel. Currently I get an error. And I can go ahead and add that channel and run it. But why make me do that? Why not just do whatever is necessary to install and run the latest version of 1.3 for me? At that point the idea of having a channel installed becomes a non-concept. You either have a version or you don't and when you use a channel it either has to get a version for that or not.

Ok, how is that related? Well it somewhat simplifies the logic of deciding what version to use since you don't have to worry about what channels are installed. You're either in "offline mode" and have to use a version that is already installed, in which case you pick the best installed version based on some policy, or you're allowed to download any version you want, in which case you pick the best of all published versions also according to some policy.

Btw, I got here because I started trying to write down an preference ordering on channels and that was hard. The ordering on versions already exists, why do we have to worry about channels at all when we're trying to pick a version not a channel?

Regarding policies, it seems like you want to choose how much of the recorded version to match exactly and how much to ignore and vary freely. The default would probably be to respect the major and minor versions and ignore the patch. That would mean that in offline mode you'd look at all the installed versions with the same major/minor version and then pick the latest one. In online mode you'd do the same but pick the latest of all patch releases with the same major/minor version. You could also pick a policy that only matches the major number or matches the whole version number.

[davidanthoff]

2. The JULIAUP_CHANNEL environment variable (not implemented).

Hmmm, I can see this being useful, but only in rare situations.

I think it would make it simple to integrate with something like https://asdf-vm.com/ for those folks that use that. It also should be super easy to implement ;)

3. A directory override that is stored in the central juliaup.json config file in the Julia depot (not implemented).

I'm not sure why this would be used. Can you explain more about this?

Initially I just copied it from the rust list ;) But here is one scenario where it might be useful: say I get a project from someone that has Julia 1.3.2 stored in the Manifest.toml. I have a 64 and 32 bit version of Julia 1.3.2 installed on my system, so at that point the launcher would presumably default to the 64 bit version, but maybe I want to temporarily (for a few days) test things with a 32 bit version of Julia. But I don't want to record that choice into the Manifest.toml because I don't want to commit this to the repo, I just want to switch this one folder for a few days over to some specific Juliaup channel. I think it would essentially just be a way to locally reconfigure a given directory without putting anything into that directory that might end up in source control. How useful that would really be in the end I don't know.

4. The .juliaup.toml file (not implemented, could also come later).

I'm not sure why you would use this over a Project.toml or Manifest.toml approach.

It would essentially be another way to locally override things. Or for example make a specific choice that can't be made in the Manifest.toml, like the 32 vs 64 bit binaries. Or maybe one could also configure auto-update behavior for this specific folder or something like that. But, this is all pretty vague in my mind right now, so I would just not do anything about this right now, we can always later add additional entries in this list.

5. The Julia version stored in the Manifest.toml (not implemented).

I like this, but I think it probably needs to be enabled somehow, as it's a breaking behavioral change. Perhaps julia +manifest, or perhaps an option set at install time.

6. The default Juliaup channel, stored in the Juliaup.json config file in the Julia depot (implemented).

The thing I'm most concerned about is that a user runs julia --project foo test.jl and sees something that works (because it uses v1.7, matching the Manifest), then they run julia, then run Pkg.activate("foo"); include("test.jl") and it breaks because their default channel is v1.8, or something like that. Also note the possibility for confusion if someone has defined JULIA_PROJECT.

I think for this manifest-matching mode to not confuse users we need one of the following:

• juliaup matches automatically, but it prints out that it's matching to a particular version when it does so.
• juliaup does not match automatically, it must be manually-enabled.

Yes, that all makes sense to me. Couple random other thoughts:

• Maybe 4 (.juliaup.toml) could actually help with this, i.e. it could be a way to locally store a Julia version that will always be used, irrespective of whether one activates a project or not. But it also feels like weird duplication...
• I do know what my preferred solution would be: make JULIA_PROJECT=@. the default :) I know that there were some security concerns around that, but I think one could probably solve them in other ways. In my mind, though, that would at the end of the day be the solution that is closest to what users would expect to happen.
• I hadn't really appreciated that with the ability to change project interactively during a running Julia session, we would then always have an aspect of a project that doesn't change when the user does that interactively... I don't like that but also don't really know how one could possibly get around that.

[davidanthoff]

@StefanKarpinski ah, interesting. A couple quick reactions:

1. I think we can most definitely do way, way more auto-things. julia +1.3 could just install the 1.3 channel etc., so I think much of the behavior you describe we could get while we still maintain the notion of "installed channels", installation of channels would just happen more often automatically. In my mind we should do that no matter what.
2. So then the question is do we need the notion of installed channels at all? I think it might still be useful for things like juliaup status, or simply the command juliaup update, but I'll have to think a bit more about that...

Btw, I got here because I started trying to write down an preference ordering on channels and that was hard. The ordering on versions already exists, why do we have to worry about channels at all when we're trying to pick a version not a channel?

I know, we already have one attempt at that in the Julia extension for the notebooks. We have the same problem there: we get a Jupyter notebook that has a Julia version recorded and then need to pick the right channel for that. This is the implementation. It is really pretty ad-hoc at the moment...

One very easy out of this would be to just be way more aggressive about auto-installing the x.y.z channels.