Recommended best practice for including jupyter notebooks **without** their outputs, but enabling use of the _freeze cache

[jamesowers-cohere]

Description

I would like contributors of my quarto backed site to be able to include jupyter notebooks as pages. However, I have some requirements:

• The outputs of jupyter notebooks are very large so I do not want to include the output of the notebooks in the version controlled files pushed to github
• I want to ensure execute: enabled: true, such that every render will rerun the notebook from scratch if there has been a change in the source code
• I want my contributors to be able to develop using a jupyter notebook rather than a qmd file
• (this seems like a novel approach - it's not one that is in your documentation) the code which creates the virtual environment and jupyter kernel to use for the notebook is executed prior to the running of the notebook itself (this is generally done in a qmd)

However, because the "file source" which matches the _freeze cache hash check is that of the notebook including its outputs, it doesn't seem like I can satisfy my requirements (i.e. I am forced to version control notebook outputs if I want to be able to use _freeze cache).

Is there any other current approach that I'm missing that would enable me to use the cache without having to version control jupyter notebook outputs.

Some solutions I'm currently considering involve making use of a pre render script which could do one of the following:

• use quarto convert to convert all ipynb files to qmd pre-render, and prevent the rendering of ipynb altogether (I worry about weird edge cases where something would work in a notebook that doesn't in qmd)
• accept that freeze cache cannot be used for jupyter notebooks and either:
  • force execution of the recipes which create the jupyter kernels upon every render
  • roll my own cache solution and copy the notebook with outputs from some shared location if the notebook code has not changed

All advice well received, and I'm happy to provide more information about my setup as needed.

[cscheid]

The freezer contents, by definition, include the contents of the notebook you executed. So if I understand your situation correctly, the essential mismatch between your requirements and Quarto features need to be solved by disabling the freezer for the pages which you don't want to check in to version control.

Of course, in that case, everyone will need to re-execute those .ipynb files at every quarto render. But that's somewhat inevitable: either quarto has access to the computation results (in which case they need to be somewhere like version control), or quarto doesn't have access to the computation results (in which case Quarto needs to execute the code).

accept that freeze cache cannot be used for jupyter notebooks

It can be used for notebooks. It's just that freeze is, by design, meant to be used in conjunction with source control.

You might be able to split the difference by using the contents shortcode (referring to a file that you know to be in .gitignore or whatever the equivalent for your version control system) or a custom Lua filter that pulls contents in a similar manner. In this case, your .ipynb notebooks being rendered by Quarto would not contain the large outputs. They would simply contain placeholders that would be filled out by the Lua filter. As a result, the freezer contents would be relatively small (and you would become responsible for ensuring that the large notebook results are available somewhere).

[jamesowers-cohere]

Did my clarification here land at all? I think the misunderstanding is that my problem is not about what is stored in the _freeze directory, but the hash function that determines whether the file has changed since it was cached i.e. the process that determines whether the _freeze cached outputs should be used.

[cscheid]

Your misunderstanding is your assumption that there is a hash function, and your expectation that there should be a hash function. That's simply not how the freezer is designed.

[jamesowers-cohere]

Aha, thank you @cscheid !!!

Please may you explain how quarto decides whether or not to use the freezer cache? My experience is as documented, i.e.

You can use the freeze option to denote that computational documents should ... only be re-rendered when their source file changes:

https://quarto.org/docs/projects/code-execution.html#freeze

How does quarto decide whether the source file has changed?

[jamesowers-cohere]

To link this back to my question specifically - I'm wanting quarto to consider a jupyter notebook file unchanged if it is the same in all ways other than outputs.

[jamesowers-cohere]

@cscheid it is interesting that you say:

Your misunderstanding is your assumption that there is a hash function...

Because the entry under the "hash" key within the html.json files in _freeze exactly matches the md5sum hash of the .qmd or .ipynb file the cache is for.

[jamesowers-cohere]

I'd love to rekindle this discussion if possible. I have relented and am now version controlling _freeze (I am having issues with this around version controlling many files, but that is an aside!). My question about what determines whether the cache will get used when running, for example, quarto preview path/to/notebook.ipynb is still relevant and my point still stands:

• To "hit" the cache (i.e. to prevent the notebook being re-executed), I need the notebook to contain the output
• But the cache itself is a store of the output data

If I remove the output from the jupyter notebook, it no longer "hit"s the cache, and so is re-executed.

i.e. I'm having to version control the output data twice! Once within the jupyter notebook itself and once within the _freeze cache.

It's an important part of my project that the default setting is that jupyter notebooks get executed (unless there has been no change to them, and there exists a cache for them) - my project partly serves as a website of documentation of code, but also as validation that this code actually runs (the publication of the site is proof that the code runs). I get the behaviour I need out of the box if I use a .qmd file; I would love to get the same when using .ipynb files (because this is a much better format for my developers to use) without having to version control the notebook output data twice!

[cderv]

Before starting, I think wording is important here. You are mentioning using _freeze, which is a Quarto feature described in https://quarto.org/docs/projects/code-execution.html#freeze, and then mentioning cache - in quarto, cache is also something usable to manage execution - it is described in https://quarto.org/docs/projects/code-execution.html#cache

Cache uses an engine caching mechanism; for the jupyter engine, it will utilize Jupyter Cache. This means that when Quarto execute the jupyter, jupyter will reload some cached data.

This is different from freeze in Quarto where in that case, quarto won't run at all the execution engine. This is useful for example to render a website without having to setup some computation environment, or to completely freeze an old blog post. What quarto will do is reuse the resulting intermediate markdown it has created at first computation, and stored in _freeze. Quarto will use freeze always when opt-in with freeze: true, or only when source file hasn't change (when freeze: auto). Freezer can be update when rendering incrementally (https://quarto.org/docs/projects/code-execution.html#incremental-render)

At last, with .ipynb as input using jupyter engine, Quarto won't execute notebook by default. So if you have pre-computed output in the notebook, and then do quarto render my.ipynb, the cells should not be executed. It needs to be opt-in with quarto render my.ipynb --execute or execute.enabled: true config. See https://quarto.org/docs/computations/python.html#workflow

I get the behaviour I need out of the box if I use a .qmd file; I would love to get the same when using .ipynb files (because this is a much better format for my developers to use) without having to version control the notebook output data twice!

My hope is the explanation clarifies how it is supposed to work. Now, it seems you observe a different behavior.

From your original post in July, it seems you were trying to have store ipynb without output, but still wanted to have a way for the notebook to not recomputed by using _freeze or cache.

Which setting are you using for freeze ? Did you try with freeze: true and not freeze: auto. Or you could also set --use-freezer. In this case, the freezer should always be used (and the hash you saw not be used).

Does that help ?