Resolve Sphinx/reST/MyST references to HTML hyperlinks to documentation pages

[agriyakhetarpal]

Problem

jupyterlite-sphinx is currently not able to process reST/MyST references to their URLs, like Sphinx/napoleon or Numpydoc can. This leads to class names, table of contents links, etc. getting added in their literal sense, and does not make for pretty URLs, as something like scipy.special.ndtri then shows up as {class}scipy.special.ndtri in the JupyterLite contents.

An interactive notebook's HTML form in SciPy's Sphinx documentation

Said notebook when viewed in a JupyterLite deployment

This has been surfaced before as well by @gabalafou in agriyakhetarpal/pywt#4 (comment) and I've thought a bit about it, but I've opened this issue now so that we can track it a bit better and further prioritise it for a 0.19 or 0.20 release.

Proposed Solution

• Suggestion from @rgommers today: use the right numpydoc or sphinx.ext.napoleon APIs to be able to create HTML anchor tags from references, pointing to a relative URL so that we always point to the correct version (stable/devdocs/particular version/local Sphinx deployment at any commit)
  • If those APIs are not public, make them public upstream
  • Otherwise, consider writing a separate minimal Sphinx extension to do only this part
• Run those APIs over the reST contents of the notebooks to resolve the references. This gives us reST-formatted links.
• Write a helper function to convert reST-formatted links (<link> _.) to Markdown-formatted links []() in _try_examples.py, or better, in a separate file to be reused across all directives. (I think we already have something related to this).
• Pipe the notebook contents through this helper function to get HTML links in them

Additional context

As a bonus, it will help us make use of the jupyterlite_sphinx_strip downstream in projects like SciPy more more sparingly, eliminating most of the changes in this previous PR: scipy/scipy#22042