Code loading: add support for "portable scripts".

[KristofferC]

These are jl files that have project and manifest embedded inside them, similar to how Pluto notebooks have done for quite a while.

The delimiters are #!manifest begin and #!manifest end (analogous for the project data), and the content can either have single-line comments or be inside a multi-line comment.

When the active project is set to a portable script, the project and manifest data will be read from the inlined toml data.

Starting julia with a file (julia file.jl) will set the file as the active project if julia detects that it is a portable script.

The tests and the parser for the inline toml data were written by Claude 🤖 .

---

I still have to finish the Pkg part (edit JuliaLang/Pkg.jl#4479) that writes these files, so this cannot be tested "end-to-end" right now. However, if one creates a portable script manually like this (rename to .jl)
portable_script.txt, we can check that it works via:

❯ JULIA_LOAD_PATH="@" ./julia portable_script.jl
✓ Successfully ran portable script with Plots, CSV, and DataFrames!

❯ JULIA_LOAD_PATH="@" julia +nightly portable_script.jl
ERROR: LoadError: ArgumentError: Package Plots not found in current path.
- Run `import Pkg; Pkg.add("Plots")` to install the Plots package.
Stacktrace:
  [1] macro expansion   

where we can see that the manifest information inside the portable script is used.

Invitation to bike shedding:

• How should the start and end delimiters look
• Should Pkg default to putting them above or below the content.
• Should inline manifest information be needed? If not, what do we do on a Pkg.resolve()? Put it inline or in some temp environment?
• Is it ever ok to use the fle ending (.jl) to have semantic meaning for the code loading?
• Other things?

In addition, I think we need a --instantiate flag or something in Julia that checks if all the packages are available for the scripts and if not, side-loads Pkg to download them. Maybe that should even be automatic for portable scripts. I think this feature has been requested on the Pkg repo. Being able to get a file and run it in "one shot" is quite tempting to support.

---

Previous work:
https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies
https://internals.rust-lang.org/t/pre-rfc-cargo-script-for-everyone/18639

[MasonProtter]

What will happen here if there is only a #!project but no #!manifest? Is that okay? Sometimes when passing around a script you don't want to bundle the full manifest info.

[KristofferC]

Yeah, that's what the

Should inline manifest information be needed? If not, what do we do on a Pkg.resolve()? Put it inline or in some temp environment?

question is about. In order to run the script you at least need to do a resolve it to have manifest data available so you have some concrete versions to load. Maybe julia should do that automatically if it tries to run a portable script without manifest information. But then where should the result be stored? inline or in some hashed directory somewhere?

Also, when running a portable script julia should maybe disable the global env from the load path by default?

[MasonProtter]

Oops sorry, I missed you listing that.

Yeah, what I was picturing was that we could would have a folder for script environments in .julia/environments that stores the "real" project and manifest files. Then we'd just need a hook during Pkg.gc or whatever that when it checks and finds these Project files, it checks if the originating script still exists. If the script is gone, we delete the project/manifest.

Maybe there's a better way to handle that.

[KristofferC]

Yeah, what I was picturing was that we could would have a folder for script environments in .julia/environments that stores the "real" project and manifest files.

Only manifest files, right? If you don't even have an inline project section, then you are just a normal julia file, or?

[MasonProtter]

I was imagining that project info from the script would get copied there, and if there is a manifest it also gets copied. But maybe that's not necessary and you don't need copies, just writing out a manifest if it's not embedded.

[KristofferC]

I was imagining that project info from the script would get copied there

We just read it inline now, so no need for that really.

What should the .julia/environments/scripts/... be keyed on? Some hash based on the absolute path of the script?

[MasonProtter]

In the silly little implementation I played with recently, I was just replacing the path separators in the absolute path with underscores. But there's probably a bunch of reasons why that's a bad idea.

[JanisErdmanis]

Could the key for ~.julia/environments/script/ be a hash of the extracted Project.toml itself?

Also it is worth to add that if manifest were to added in the script itself that would make editing such scripts manually by hand much less pleasant. It doesn’t seem like users would want such a level of reproducibility in practice where compat bounds would suffice.

[KristofferC]

Could the key for ~.julia/environments/script/ be a hash of the extracted Project.toml itself?

I thought about that but then you get collisions if you have different scripts that happen to get the same project content.

It doesn’t seem like users would want such a level of reproducibility in practice where compat bounds would suffice.

I disagree, I think you would want to send someone a file and have them be able to reproduce stuff?

[JanisErdmanis]

Wouldn't interpreting the compat bounds as exact versions with which to resolve produce the same Manifest.toml when executed on the same Julia version?

[KristofferC]

I don't really want to have a discussion here on how the resolver works. The feature can be made to support both inline and outline manifest info, so not sure what there is to discuss more about this.

JuliaLang/Pkg.jl#4479 is a slightly WIP follow-up to this from the pkg side. With that, you can do:

julia> Pkg.activate("portable_script.jl")
  Activating project at `~/julia/FatEnv/portable_script.jl`

(portable_script.jl) pkg> st
  Installing known registries into `~/.julia`
       Added `General` registry to ~/.julia/registries
Status `~/julia/FatEnv/portable_script.jl`
  [a93c6f00] DataFrames v1.8.1
  [7876af07] Example v0.5.5
  [91a5bcdd] Plots v1.41.1

(portable_script.jl) pkg> rm Example
    Updating `~/julia/FatEnv/portable_script.jl`
  [7876af07] - Example v0.5.5
    Updating `~/julia/FatEnv/portable_script.jl`
  [7876af07] - Example v0.5.5

(portable_script.jl) pkg> st
Status `~/julia/FatEnv/portable_script.jl`
  [a93c6f00] DataFrames v1.8.1
  [91a5bcdd] Plots v1.41.1

(portable_script.jl) pkg> precompile
Precompiling packages  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━ 122/157
  ◐ StatsBase

and the file updates during pkg operations.

[tecosaur]

One preference from me: I think it would be nice to put/read the manifest from the bottom, so if you open the script and start looking at it you don't see a few hundred lines of TOML up front.

[KristofferC]

Yeah, I also got to that conclusion. But the project data should still be on top?

[KristofferC]

Regarding storing the manifest inline or not, I think a key in the project section could determine that.

[tecosaur]

Yeah, I also got to that conclusion. But the project data should still be on top?

On one hand, splitting it does seem a bit odd, but having project info at the top does feel right to me somehow.

Regarding storing the manifest inline or not, I think a key in the project section could determine that.

I'm guessing there may be some scripts which are a bit "package-like" in that they have compat bounds on packages, and it's fine for those to be resolved when run. In this way, having the manifest section be optional seems nice.

[KristofferC]

Actually for the outline manifest we can use the manifest = key in the project data. That way you can move around the script without losing track of the manifest (say if it was hashed by path)

[KristofferC]

Ok, so I have made the following changes (mostly in Pkg).

• By default, it starts writing manifest info inline.
• If you put inline_manifest = false in the project section, it puts manifest = "~/.julia/environments/scripts/portable_script_0686334e-56a7-4d0f-b85f-a9bbebe086df/Manifest.toml" in there and copies the existing inline manifest into that file.
• If you put inline_manifest = true, it removes the manifest = entry and copies in the information from that manifest inline and deletes the folder with the external manifest.
• You can then easily "ping pong" between storing manifest data inline or outline without losing any information.

A UUID feels a bit excessive in the path...

[tecosaur]

A UUID feels a bit excessive in the path...

How about <script_name>_<hash>/Manifest.toml? To me that seems like a decent blend of avoiding hash collisions while actually being somewhat informative if you peek at environments/scripts (as a freebee).

julia> let f="path/to/myscript.jl"; first(splitext(basename(f))) * '_' * string(hash(f), base=62) end
"myscript_Hkm5zPmRBLp"

A 64-bit discriminator seems plenty to me, with ~10k scripts there's a ~1 in a trillion chance of a hash collision.

Edit: I remembered case-insensitive filesystems exist. Make that string(hash(f), base=36).

[tecosaur]

Question: What do we expect julia --project=@script myscript.jl to do?

It may not make a lot of sense, but I'm sure that we'll see people try it sooner or later. My instinctive feeling is that doing anything other than using the project/manifest embedded in myscript.jl would be counterintuitive.

[KristofferC]

Yes (and it should also be what it does now). Should add a test :)

[KristofferC]

I dont think hashing only by path is good because you get a collision if you then move the script and create a new one in the same place with the same name. Since the path to the manifest is stored in the script itself it doesn't really need to have anything to do with the path (which is why I just grabbed for a uuid).

[fonsp]

Hey! This looks awesome, very cool to see this getting standardized! Later in the process, I would like to make sure that this can be made compatible with Pluto's format, so that you can activate and run Pluto notebooks in standalone Julia :)

@KristofferC asked me to share some experiences from Pluto (which already has a similar feature by default for notebook files), so here you go:

General reaction

Among Pluto's users (education, Julia end-users), people really appreciate this feature and it makes reproducibility much more accessible. It makes it much easier to share your work reliably with new Julia users, which is a huge win. Pluto enables this by default (when creating a notebook, and when opening a notebook). For this PR I would also suggest enabling it by default when opening a file with embedded project data, or at least showing a hint.

Surprises

Embedding the Project+Manifest also turned out less amazing than I thought. The main unexpected issues are:

Julia versions

In practice, the manifest is only useful with the same Julia version, and it is pretty common to change Julia versions. (E.g. when sharing work with someone else, or when opening old work.) We made https://github.com/JuliaPluto/GracefulPkg.jl to still get some value out of unresolvable environments in Pluto, but for base Julia you might want to address this.

I would be curious to hear your ideas for this issue, and maybe we can share a solution.

Setup times

Package setup times (install+precomp) are long in Julia, and they get longer with every release. Cache reuse is minimal across environments in practice, which means that each new script launch will probably trigger its own lengthy setup process. Recent Julia releases seem more optimised for the "big global env", rather than many specific environments.

This is very relevant in Pluto, where you often share your work, sometimes many notebooks. But if you use this feature less frequently, or only for personal work, then cache hit rates will be high.

[KristofferC]

In practice, the manifest is only useful with the same Julia version, and it is pretty common to change Julia versions. (E.g. when sharing work with someone else, or when opening old work.)

I agree with this. The "competition" has it easier in this case because they often don't use the language itself to launch a script (which immediately ties you to that version of the language), but instead have a launcher one abstraction level above, which can also decide what julia version to use. E.g., in uv run, uv has the choice to go and look at what Python version the script wants, and even go and download it. For julia script.jl, that isn't really possible.

We could have something like juliaup run perhaps but:

• I think it is expanding the responsibilities of juliaup a bit much.
• It is a little bit of a shame to have to download something other than julia to launch scripts like this.

Even for Pkg Apps I feel the pain of the non-flexibility of having the launcher of the app not being able to choose julia version.

(cc @davidanthoff)

[tecosaur]

but instead have a launcher one abstraction level above, which can also decide what julia version to use. E.g., in uv run, uv has the choice to go and look at what Python version the script wants, and even go and download it. For julia script.jl, that isn't really possible.

FWIW, I implemented this feature in Juliaup ~1y ago (in the julia launcher), but it wasn't merged because there weren't enough tests.

[vchuravy]

The actual julia binary (cli in the repository) is very small and is essentially a wrapper around loading libjulia. Years ago I was spit balling with Elliot the idea to put a toml parser in there and use that to choose if to start up with or without the compiler enabled or which system image, which libjulia to load.

Not something that we can do immediately, but it is feasible.

[KristofferC]

Years ago I was spit balling with Elliot the idea to put a toml parser in there and use that to choose if to start up with or without the compiler enabled or which system image, which libjulia to load.

Okay, but what we want to do is to choose what julia itself to use, which this doesn't seem to help with.

FWIW, I implemented this feature in Juliaup ~1y ago (in the julia launcher), but it wasn't merged because there weren't enough tests.

For reference: JuliaLang/juliaup#10, JuliaLang/juliaup#1095. I think it would be cool to take that further, especially for the case of portable scripts and Julia apps (installed via Pkg).

[tecosaur]

I think it would be cool to take that further

So do I. At some point though, it's worth being realistic about how many projects I'm trying to make progress with at once. I might take a look at this again late 2026/early 2027, but at moment I've got a small pile of things I'm similarly interested in that have much nicer git histories.

[KristofferC]

I'll try to get JuliaLang/juliaup#1095 off the ground again. I think we need that plus auto-instantiation and auto-precompilation (we have the latter) for julia script.jl to be a "one-shot" reproducible script run.

[tecosaur]

Other than tests, there's one feature that I wanted to add to JuliaLang/juliaup#1095 that bears mentioning: using Julia compat bounds in the Project.toml to determine whether it's ok to run a newer Julia version or not (e.g. behave differently with Julia = "1.11.2" vs Julia = "1.11").

[cjdoris]

I haven't had time to read through or try this PR in detail yet but just wanted to point out a package I wrote a short while ago which does pretty much the same thing: https://github.com/cjdoris/SelfContainedScripts.jl

It's cool that it may be baked in to Julia and not require an external package, although I was pleasantly surprised at how ergonomically it could be done as a package.

[KristofferC]

Cool! I think "first class" support, where you can use the package manager, etc, and drive it like a normal environment, and then run it with julia script.jl might make it ergonomic enough for more widespread use. Since you have experience with it I'dbe happy to take some comments once you have time to read through the stuff here.

[Keno]

Couple of thoughts:

1. Should the syntax have support for versioned manifests.
2. Should we insist that the project/manifest are at the start of the file (i.e. only scan lines that start with # from the start of the file).
3. Do we really need the multi-line comment version of this?

[ericphanson]

as an alternative to comments, rust's cargo-script feature seems to be going for frontmatter. They put the lockfile in their build outputs iiuc. Not sure that would work for us but wanted to share.

https://github.com/rust-lang/rfcs/blob/master/text/3502-cargo-script.md
https://github.com/rust-lang/rfcs/blob/master/text/3503-frontmatter.md

[MasonProtter]

Regarding 2., I think it'd be a readability disaster to demand that the project and manifest be at the start of the file. If scanning time of big files is a concern though, we could have a token at the start of the file like e.g.

#inline_project=true

or whatever to signify that there is a project / manifest somewhere that needs to be scanned for.

Regarding 3. I think a multiline comment version is nicer to read and write.

[Keno]

If scanning time of big files is a concern though

That, but it also seems very weird to me that a random comment somewhere in the middle of the file determines how it should be loaded.

I think a multiline comment version is nicer to read and write.

Then why not

#=!project begin

the proposed

#!project begin
#=

just looks extremely ugly. (Though I sill prefer the single-comment version, esp since these are mostly machine written and even if not, IDEs will definitely insert those characters for you).

[tecosaur]

it also seems very weird to me that a random comment somewhere in the middle of the file determines how it should be loaded.

I'd argue the manifest should be put specifically at the end of the file.

[Keno]

I'd argue the manifest should be put specifically at the end of the file.

I think it'd be reasonable to mandate that the project be at the beginning of the file (since that's usually what you care about seeing and editing and it's not that big) and that the manifest should be at the end of the file with no code following (because you probably won't scroll past it and you want to avoid some trickster - or an LLM putting some surprise extra code there).

[tecosaur]

Indeed, that's the project/manifest split I had in mind in my earlier comments.

[KristofferC]

One thought I had was if (by luck) all of the TOML spec can be parsed by the Julia parser and then we could use some @project begin idea to indicate portability (in the same way as @main is used) but stuff like 'abc' does not parse so that's not really workable. Maybe a bad idea anyway.

Requiring parsing support like in the Rust frontmatter suggestions feels a little bit too "intrusive" to me and would be annoying that you couldn't parse newer files with older julia etc.

Something like https://packaging.python.org/en/latest/specifications/inline-script-metadata/#inline-script-metadata feels more apt to me.
The issue with all of the above though is that they really want things to be at the top, even manifest data. I tried with Pkg.jl to emit #region ..., #endregion around these which make things easily foldable but I am not sure relying too much on editor support is a good idea.

I'm happy to restrict things initially (like not having support for multi-line comments)

[KristofferC]

the manifest should be at the end of the file with no code following

What happnes if you put a snippet of code after the manifest data? Does the script all of a sudden become non-portable and will use your default environment when running? Feels like a bit of a fot gun.

[Keno]

What happnes if you put a snippet of code after the manifest data?

The presence of #! in a contiguous block of a comments at the top of the file indicates that it's a portable script. On activation, it checks for the manifest. If there's any non-trivia after it, that's a project activation error.

[KristofferC]

that's a project activation error.

👍

[KristofferC]

I forgot, regarding

Should the syntax have support for versioned manifests.

I'd say initially no. I am not a fan of versioned manifest because the only reason normal manifests don't work in 99.99% of the cases is due to us looking up stdlib info from the manifest when it can arbitrarily change when changing julia versions. We could have chosen to look up this information from the stdlib project files in which case everything would be fine.

[KristofferC]

The presence of #! in a contiguous block of a comments at the top of the file indicates that it's a portable script.

Just want to say that even if the project block is missing you would still want to consider the script as "portable" if you start with julia --project=script.jl. That is to allow one to easily "initialize" a portable script with the package manager (start adding deps to it for example).

[KristofferC]

Also, I wonder if a portable script should run with a restrictive load path by default (not have the global environment in there). It isn't very much portable if you rely on the global env...

[Keno]

Just want to say that even if the project block is missing you would still want to consider the script as "portable" if you start with julia --project=script.jl. That is to allow one to easily "initialize" a portable script with the package manager (start adding deps to it for example).

Sure

[Keno]

Also, I wonder if a portable script should run with a restrictive load path by default (not have the global environment in there). It isn't very much portable if you rely on the global env...

May have an opt-in in for that in the Project.toml instead? It seems weird to special case this - the same applies to a Project.toml next to the script.

[MasonProtter]

One thought I had was if (by luck) all of the TOML spec can be parsed by the Julia parser and then we could use some @project begin idea to indicate portability (in the same way as @main is used) but stuff like 'abc' does not parse so that's not really workable. Maybe a bad idea anyway.

How about a string macro?

project"""
...
"""

manifest"""
...
"""

?

Could also have a @project begin that doesn't support / needs special handling of 'abc'

[KristofferC]

I'm now thinking that most of the decisions about portable scripts you want to take before executing the file and also if the manifest section is towards the end I'm not sure you will have macro expanded that before the code runs. And you also want the detection to be easy for other tools (like juliaup) so I don't think you get away from raw string processing to parse the inline content.

And at that point, maybe magic comments are OK...