Multi-root workspaces

[alcarney]

Up until now I've considered multi-root support to mean multiple sphinx projects - something that's probably non trivial due to a certain amount of global scope in Sphinx/docutils.

However, it's probably more likely that in a multi-root workspace, there would be one root that contains a sphinx project while the other roots are just the projects that are being documented? - in which case support would be more straightforward, just a case of finding the right root to work from.

I'd be interested to know at a high level the structure of people's multi-root setups.

[briantist]

My use case for multi-root is several Ansible-adjacent projects/repos available in a single workspace. None of these are what I'd call "sphinx projects", but they have some RST files to be edited. In most cases, the docsites generated from them are done in a central process that's outside the scope of these repositories, so I do not need to know or care about sphinx config; it's not even part of the repos.

My hope in installing the related(?)/dependent(?) project, vscode-restructuredtext, was to get better syntax highlighting and rendering of those RST files. I don't need it to be perfect or contain the complete theme or anything like that.

[chrisy]

Mixed bag; several git repos in the workspace and some of them have their own Sphinx documentation, though only one of them is the primary concern.

[lextm]

Just want to share some of my thoughts here.

Multi-root workspace is a good feature of VSCode but not mandatory. I have tons of sphinx sites so I used to have a single workspace to include all individual sphinx projects. The structure was like this,

A multi-root workspace
            |
            |-------> Sphinx project A
            |-------> Sphinx project B
            |-------> Sphinx project C
            |-------> Non sphinx project D (no conf.py)
            |-------> Sphinx project E

However, soon I found that I rarely need to open them all together. So I rarely opened this workspace, but each project when needed. For the past year I didn't even open the workspace once.

So I am ok to live without esbonio's multi-root workspace support, as long as its resources are well spent on other useful features (better error handling, preview improvements, and maybe better sync scrolling).

You probably know that snooty language server (and its VSCode) is a MongoDB internal project to drive all its documentation system, and it never supports multi-root workspaces. If huge commercial projects do not require multi-root workspace support (like my case), I wonder actually how many users are out there that really rely on this.

On the other hand, the architecture of esbonio/sphinx might be good for a single sphinx project, but to support multi-root workspaces, esbonio might have to spin out multiple sphinx instances at runtime. That does not only increase resource consumed (if you don't really need to work on all roots) but also bumps the complexity of esbonio (multi-process communication and error handling). It might take significant effort to implement (@alcarney can investigate and reveal more).

It is also hard to know how each user uses multi-root workspaces. The one I used was kind of the typical, but I can foresee some might have special setup. Without collecting such information upfront, it's not easy for Alex to determine the best approach to go and which are the use cases to cover.

[emailweixu]

Mixed bag; several git repos in the workspace and some of them have their own Sphinx documentation, though only one of them is the primary concern.

Same for me.

[resnickj]

I also have a multi-root workspace like so:

Multi-root workspace
            |
            |-------> Sphinx project A
            |-------> Sphinx project B
            |-------> Sphinx project C

All Sphinx projects in the workspace are part of larger documentation "suite", and it is not uncommon to be making changes across multiple projects at the same time, so it was very convenient to have them all open in one workspace, and be able to use the preview feature. Would be great if this could be made possible again, although I have no sense of how much effort that would require.

[davbeek]

I used to use multi-root workspaces with one or two sphinx documentation projects in combination with several non-sphinx projects. Currently, I only use single-folder workspaces. So, instead of one multi-root workspace I have several open single-folder projects.

[martinpakosch]

Hey, I started to use Esbonio LS and VSCode extension some days ago. And I am facing some strange issues which I cannot classify unambiguously as bug, coincident of missing multi-root support or a misconfiguration of esbonio. Sphinx itself runs perfectly at all builds. I think my issues might come from issues with the VSCode extension. Let me explain, why I post this here:

My environment:

• Windows 10
• Python 3.7.9 (venv)
  • Sphinx 5.2.3
  • esbonio lsp 0.14.1
• esbonio extension 0.10.1
• multi-root workspace

My multi-root setup:

root1 - general project/meta/unrelated files
root2 - secondary repository with specific content, no sphinx
root3 - primary repository, with sphinx docs
- .venv
- docs
  - source
    - conf.py
- src
  - mylib

Settings are in root1/myspace.code-workspace.

Although the explicit settings for src, build and conf dir with absolute paths seem to trigger the sphinx build (output panel) on each save, the vscode extension seems to have issues. The {} next to the language disappear from the statusbar after the first build and stay invisible for consecutive builds. Additionally the preview panel cannot be opened for the particular rst file. Nothing happens when clicking the corresponding icon.

This is the same even for single-root workspaces when I deactivate the root1 and root2 in the workspace config! When I remove the explicit esbonio paths, then everything works fine via auto path-detection for single-root workspaces.

As the extension has everything it needs in terms of explicit config paths for at least one sphinx project, I consider this to be more likely a bug, or at least a coincident due to my setup. I would expect, that the vscode extension/LSP is already able to handle one explicit sphinx project, regardless if it is part of a multi-root workspace or not.

Suggestion for Multi-Root enhancement:

So finally, I would suggest to fix the basic functionality of sphinx build including a working preview panel in the following situations:

1. Single-Root Workspace with explicit paths set (as auto path-detection without explicit paths works fine)
2. Multi-Root Workspace with explicit paths set (as this should be enough to detect the environment (maybe via VSCode itself) and the primary Sphinx project). As mentioned the build is already triggered correctly.

• Missing explicit paths might be handled the same as it is already implemented: disable all sphinx related features but keep IntelliSense etc. active.

By this you could provide basic multi-root support for at least one contained sphinx project. Anybody who needs true multi-roots with multiple Sphinx projects might have to wait for the full enhancement.

Kind regards, Martin

[alcarney]

Thanks for taking the time to share your experience!

It sounds like there could be a few issues going on...

The {} next to the language disappear from the statusbar after the first build and stay invisible for consecutive builds

I think I've actually seen this issue myself... just never got around to investigating it properly. It's likely due to the extension and/or server not handling Windows paths correctly. I've opened #458 to track this.

Additionally the preview panel cannot be opened for the particular rst file. Nothing happens when clicking the corresponding icon.

This is likely a symptom of your main issue - i.e. the extension/server not correctly handling your configuration.
The extension only opens a preview panel when it can find a corresponding HTML file in what it thinks is the build directory.
Have you tried enabling debug logging ("esbonio.server.logLevel": "debug")? Looking at the Esbonio Output channel in VSCode's panel should give a better indication of what's going on.

[client] Build start.
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 17 source files that are out of date
updating environment:
0 added, 1 changed, 0 removed
...
build succeeded.

The HTML pages are in docs/_build/html.    <--
[client] Build complete {
  "config": {
    "sphinx": {
      "buildDir": "/home/alex/Projects/esbonio/docs/_build/html",  <--
      "builderName": "html",
      ...
    }
  }
}

I assume you're seeing a mismatch in the reported build directories?

This is the same even for single-root workspaces when I deactivate the root1 and root2 in the workspace config! When I remove the explicit esbonio paths, then everything works fine via auto path-detection for single-root workspaces.

That's a surprise! Especially when I use some amount of explicit path configuration...

I would've said the most likely explanation for what you were seeing was the fact that the server assumes it is only ever given a single root, but since you say that you still have issues when using a single root, I'm not so sure...

Would it be possible for you to share the following section of the debug log which shows the root the server is using, the options given by the extension and what they get resolved to?

[esbonio.lsp] Workspace root 'file:///home/alex/Projects/esbonio'
[esbonio.lsp] User Config {
  "build_dir": "${confDir}/_build",
  "builder_name": "html",
  "conf_dir": null,
  "config_overrides": {},
  "doctree_dir": null,
  "force_full_build": false,
  "keep_going": false,
  "make_mode": true,
  "num_jobs": 1,
  "quiet": false,
  "silent": false,
  "src_dir": null,
  "tags": [],
  "verbosity": 0,
  "warning_is_error": false
}
[esbonio.lsp] Sphinx Args {
  "buildername": "html",
  "confdir": "/home/alex/Projects/esbonio/docs",
  "confoverrides": {},
  "doctreedir": "/home/alex/Projects/esbonio/docs/_build/doctrees",
  "freshenv": false,
  "keep_going": false,
  "outdir": "/home/alex/Projects/esbonio/docs/_build/html",
  "parallel": 1,
  "srcdir": "/home/alex/Projects/esbonio/docs",
  "status": null,
  "tags": [],
  "verbosity": 0,
  "warning": null,
  "warningiserror": false
}

Hopefully from that we'll be able to figure out what is going on 😄

[martinpakosch]

Hi @alcarney,

nice you picked it up so fast. :D So I ran some usecases with debug logs. Each time I set up a use case I reloaded the VSCode window (reload window command), so that I forced initialization of the esbonio extension. I've already noticed some path issues, but take a look yourself at the attached file. :) Good luck.

esbonio_debug.md

That's a surprise! Especially when I use some amount of explicit path configuration...

I did not try partial paths in the config, only with the full set of src, build and conf.

[martinpakosch]

Hi Alex, it seems that I've hit pandora's box. :D Today I noticed a really strange behaviour with built-process-scaling and I am not sure if that might be also a coincident due to the path issues.

While working (as a workaround) in a single root workspace with no explicit paths in esbonio config (refering to usecase #1 of previous debugging log) I see the extension suddenly scaling up the sphinx build processes. First it spawns one build process, then two, then three... I stopped at 11 simultaneously spawned sphinx builds!! Guess there is no limit.

The trigger for that insane behaviour is luckily quite obvious. It happens only when something changes in the conf.py, even just saving the file without changes triggers the built process, which scales up +1 each time! When I modify a .rst file in-between the scaling stops, but the current scale is preserved. So when it already scaled to 5 concurrent builds due to 5 saves of conf.py, it runs these 5 concurrent builds each time I edit any .rst source file. Scaling continues as soon as I continue changing/saving conf.py.

At some point before I started to debug this the LSP suddenly shut down itself. No idea why. Did not see it in my latest scaling debugging up to 11 or 12 concurrent builds. Maybe something different I encountered. Will monitor this and come back to it again.

However, to reset this scaling I have to restart the server via {} icon in the status bar. Then everything is fine and only one built is executed. This is reproducible at 100%.

Example with scale 2:

Running Sphinx v5.2.3
Running Sphinx v5.2.3
WARNING: node class 'toctree' is already registered, its visitors will be overridden
WARNING: node class 'toctree' is already registered, its visitors will be overridden
WARNING: node class 'desc' is already registered, its visitors will be overridden
WARNING: node class 'desc' is already registered, its visitors will be overridden
WARNING: node class 'desc_signature' is already registered, its visitors will be overridden
WARNING: node class 'desc_signature' is already registered, its visitors will be overridden
WARNING: node class 'desc_signature_line' is already registered, its visitors will be overridden
WARNING: node class 'desc_signature_line' is already registered, its visitors will be overridden
WARNING: node class 'desc_content' is already registered, its visitors will be overridden
WARNING: node class 'desc_content' is already registered, its visitors will be overridden
WARNING: node class 'desc_inline' is already registered, its visitors will be overridden
WARNING: node class 'desc_inline' is already registered, its visitors will be overridden
WARNING: node class 'desc_name' is already registered, its visitors will be overridden
WARNING: node class 'desc_name' is already registered, its visitors will be overridden
<...>
WARNING: directive 'autoattribute' is already registered, it will be overridden
WARNING: directive 'autoattribute' is already registered, it will be overridden
WARNING: directive 'autoproperty' is already registered, it will be overridden
WARNING: directive 'autoproperty' is already registered, it will be overridden
WARNING: directive 'autonewvarattribute' is already registered, it will be overridden
WARNING: directive 'autonewvarattribute' is already registered, it will be overridden
loading intersphinx inventory from https://docs.python.org/3.7/objects.inv...
loading intersphinx inventory from https://docs.python.org/3.7/objects.inv...
<...>
building [mo]: targets for 0 po files that are out of date
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 17 source files that are out of date
building [html]: targets for 17 source files that are out of date
updating environment:
updating environment:
[new config]
[new config]
17 added, 0 changed, 0 removed
17 added, 0 changed, 0 removed
reading sources... [  5%] api/mylib
<...>
reading sources... [100%] index
reading sources... [100%] index
<...>
build succeeded.
build succeeded.

The HTML pages are in ..\..\..\..\AppData\Roaming\Code\User\workspaceStorage\136dd8f8fc7ff9a64b42dc0bce04fa7c\swyddfa.esbonio\sphinx\html.
The HTML pages are in ..\..\..\..\AppData\Roaming\Code\User\workspaceStorage\136dd8f8fc7ff9a64b42dc0bce04fa7c\swyddfa.esbonio\sphinx\html.

Edit:

Oh and as a site note regarding multi-roots:

Performance was still good with 12 builts, also for the preview refresh :D (core i5, 8th gen), although my docs are quite small at the moment. Maybe you could indeed consider spawning multiple builds for multiple sphinx-roots in a later supported multi-root environment like mentioned by lextm. But it should be done smart: on each change spawn only the corresponding build of the corresponding sphinx project (needs multiple { confdir, builtdir, sourcedir } configs/auto-detects) and allow the user to configure and/or to trigger manually a full build of all roots if he needs to due to dependencies. Then performance-issues due to scale are up to the user and his project size. ;-)

[alcarney]

take a look yourself at the attached file

Thanks again for taking the time to pull all that together, it's going to be a big help!

I see the extension suddenly scaling up the sphinx build processes. First it spawns one build process, then two, then three...
It happens only when something changes in the conf.py.

Thankfully, I don't think the server is spawning multiple build processes.

Each time the project's conf.py is saved, the server recreates its Sphinx application instance so it can use the latest settings. After some investigatoin, it turns out during this process the server also registers a new logging handler to capture the build output but without removing the old one leading to duplicate log messages.

Which is probably why the performance remained pretty good :)

[martinpakosch]

Which is probably why the performance remained pretty good :)

That makes sense. :D Was already wondering how this "process-scaling" would have worked without conflicts in the same build-destination. :) Did not think about it while writing the latest post.

Thanks again for taking the time to pull all that together, it's going to be a big help!

If you need more, let's talk.

[alcarney]

Anyone still following this might be interested to hear that I just pushed a pre-release version of the Esbonio VSCode extension - v.90.1 - with an initial implementation of multi-root/multi Sphinx project support!

It's more of a proof of concept than anything, since the (now bundled) pre-release language server implementation doesn't implement any features - other than building your Sphinx project each time you save a file.

But hopefully it's just enough to gauge whether it can correctly support your multi-root setups! 🤞

See this issue for more details