Caching mechanism for large projects: how does it work across builds?

[mgaitan]

Hi,

I'm using sphinx-autodoc2 in a large monorepo, and while I appreciate the static analysis approach, I’ve noticed that parsing takes a long time, even when running consecutive builds with minimal changes (e.g., modifying a single .md file).

I investigated the implementation and found that the database is stored in-memory:

sphinx-autodoc2/src/autodoc2/sphinx/utils.py

Lines 67 to 73 in 1f55a88

	def get_database(env: BuildEnvironment) -> Database:
	"""Get the database from the environment."""
	# We store it persistently in the environment so that it can be
	# accessed by directives and other extensions.
	if not hasattr(env, "autodoc2_db"):
	env.autodoc2_db = InMemoryDb() # type: ignore
	return env.autodoc2_db # type: ignore

Given that the database is in memory, it seems like the analysis is lost between builds. However, the `README states that sphinx-autodoc2 is "optimized for rebuilds"

sphinx-autodoc2/README.md

Lines 13 to 15 in 1f55a88

	Optimized for rebuilds
	: Analysis of packages and file rendering are cached, so you can use `sphinx-autodoc2` in your development workflow.

Could you clarify how this caching mechanism is supposed to work across builds?
Is there a way to persist the parsed data between runs, or would this require a feature request to implement a persistent database?

[cmarqu]

I just came across https://pypi.org/project/diskcache/ (via https://www.youtube.com/watch?v=N7dpC60hDwo) which looks like it would be useful here.