Calculations in key-figures.ipynb take too long

[r-ford]

Similar to this issue in the CMIP6 Cookbook, key-figures.ipynb is too much for GitHub Actions to build, so the cell outputs are not included. In this case, though, I can't just cut out some material.

I'm not sure that Dask is actually implemented properly in this notebook, so a solution to this issue could just be using Dask to speed up the computation.

[brian-rose]

I doubt that we'll be able to use Dask (or at least use it effectively) on GitHub Actions.

It would be good to figure out why those cells are not running. According to the docs here:
https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits
each job should have a maximum of 6 hours execution time.

[brian-rose]

@r-ford do you want to take a look at updating the infrastructure in this repo (ProjectPythia/cookbook-gallery#96) and see if executing on the binder will fix this?

If so we can also close ProjectPythia/cookbook-gallery#12

[r-ford]

@brian-rose I just updated the infrastructure, but it looks like there is an error with the code in one of the notebooks:

CellExecutionError: An error occurred while executing the following cell:
------------------
uniques = col.unique(columns=["component", "frequency", "experiment", "variable"])
pprint.pprint(uniques, compact=True, indent=4)
------------------

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [6], line 1
----> 1 uniques = col.unique(columns=["component", "frequency", "experiment", "variable"])
      2 pprint.pprint(uniques, compact=True, indent=4)

TypeError: unique() got an unexpected keyword argument 'columns'

Maybe some of this code is outdated?

[brian-rose]

@r-ford ok, I guess we'll have to take a deeper dive into the code. Seems like some sort of conflict with the intake_esm catalog.

[brian-rose]

We'll try to get this running before the AGU meeting.

[ktyle]

@brian-rose @r-ford @clyne I am making some progress fixing the problematic code cells (the example-workflows/key-figures.ipynb notebook runs to completeion now) ... but the issue remains that the notebooks take extremely long to run (on our amped-up department server, there are several cells that each take ~25 minutes to complete, and that's with Dask).

[brian-rose]

Hmmm there appears to be a new Pangeo binder running on AWS: https://hub.aws-uswest2-binder.pangeo.io/

I wonder if that's a better platform for running this code, since the data are in S3 storage.

[brian-rose]

There's an authentication step that uses GitHub user account. I'm not sure what will happen if we just set

binderhub_url: https://hub.aws-uswest2-binder.pangeo.io

in the config file.

[ktyle]

@brian-rose Given that (at present) we don't know how to leverage binderbot to use authentication, and the fact that this notebook takes north of an hour to run, I think it would be better to present this notebook as HTML, in its fully-rendered format after it has successfully run locally, but not have it run via github actions or binderbot. This way we would at least be able to include it in the cookbook.

[r-ford]

@brian-rose Given that (at present) we don't know how to leverage binderbot to use authentication, and the fact that this notebook takes north of an hour to run, I think it would be better to present this notebook as HTML, in its fully-rendered format after it has successfully run locally, but not have it run via github actions or binderbot. This way we would at least be able to include it in the cookbook.

I think this makes sense for now. If we're expecting some more readers soon, it would be nice if they could at least see the output.

[brian-rose]

@ktyle @r-ford agreed that committing a pre-executed version of this notebook is an acceptable short-term solution, given that it has useful content and that we are actively pursuing solutions for reproducible health checks etc.

I left some notes in #12 about how to implement this and get the whole Cookbook successfully building and publishing itself.

[ktyle]

We will revisit when the new Binderhub is up and running