binary images (general strategy; also in html exports)

[falkmielke]

The brms tutorial is provided by knitting html files locally and uploading them in a static folder.

However, knitting Rmd's will include binary sections for the images.

This causes severe overhead for git each time the tutorial receives changes and has to be re-run.

• We should find a better solution to host html files like this.
• Alternatively, store images in a static folder so that they don't get changed all the time (requires manual image linking).

[falkmielke]

Update: there is the more general issue of image file storage.
Although the repo is still manageable at this moment, it can be anticipated that the graphs created by Rmarkdown files will contribute a lot of binary waste to the repo. I am guilty myself, see here, for example.

Visualization is a fundamental aspect of tutorials. Whenever there is another "tutorials" stakeholder meeting at our institute, we should discuss solving this.

Some ways others found to solve this problem:

• git-filter-branch seems to be the official solution, but requires all copies of the repo to be manipulated (unrealistic, except if we fork)
• git filter-repo seems to be an alternative
• a "repository cleaner" could be locally tested.
• going further, we might test and implement git annex; there are similar tools

[florisvdh]

Also git LFS can be added to the list. https://git-lfs.com/

[hansvancalster]

See also https://github.com/github/git-sizer (I actually doubt this repo will ever become too big - the images are not really large files).

git maintenance start might be a general good practice to reduce git repo size.

[falkmielke]

Thank you both for the links!

See also https://github.com/github/git-sizer (I actually doubt this repo will ever become too big - the images are not really large files).

The problem is that they get re-calculated and change on every render of a tutorial.
The whole image binary changes although adjustments are minor; this quickly adds up (I experienced this on the variogram branch: 17 images, 3.0MB; times three for three review rounds).
Maybe some of the effect could be mitigated by consistently using random seeds so the images do not change.

Git is just not suited for binaries.

[hansvancalster]

The problem is that they get re-calculated and change on every render of a tutorial. The whole image binary changes although adjustments are minor; this quickly adds up (I experienced this on the variogram branch: 17 images, 3.0MB; times three for three review rounds). Maybe some of the effect could be mitigated by consistently using random seeds so the images do not change.

Oh, I see - missed that - good point 👍 But even in that case it should be evaluated if it really adds up too much - after taking mitigating measures like random seeds into account. Images are typically a couple of 100 kB and they are infrequently updated (only during a couple of review rounds, and when a tutorial is updated), which is something git can handle without hassle I would guess.

[falkmielke]

Two more notes:

on the original issue, Floris noticed that quarto offers the embed-resources option to choose whether binary images get included in html or not; very useful.
... but not in the context of a hugo project, where you might want to keep images in the static folder.
It is html only (does not work with markdown export, yet prevents image subfolder creation), and seems to mess the css style. More testing needed.

I attempted to clean up images from the variogram branch with the following command (reference):

git-filter-repo --path content/tutorials/spatial_variograms/spatial_variograms.markdown_strict_files --invert-paths --force

However, I do not dare to apply it to this repo; it diverges histories and removes origin for safety; consequences are hard to judge.

Using set.seed(123) in R already helps a lot to mitigate the issue; I will experiment with it and suggest additions to documentation of creating a tutorial.