config argument docs and developer readme

Author: yaugenst-flex

docs/configuration/index.rst

@@ -8,7 +8,7 @@ your API key, the active environment, and any local tweaks you make while
 experimenting. The ``tidy3d.config`` module keeps all of this in one place
 through a single object called ``config``. This page explains how it behaves,
 where values are stored, and how to keep your changes consistent across
-sessions.
+sessions. For a catalog of every option, see :doc:`reference`.
 
 Getting Started
 ---------------
@@ -174,5 +174,6 @@ updating scripts that depend on these names.
 Next Steps
 ----------
 
+- :doc:`reference`
 - :doc:`migration`
 - :doc:`../api/configuration`

docs/configuration/reference.rst

@@ -0,0 +1,185 @@
+Configuration Reference |:clipboard:|
+====================================
+
+All configuration for ``tidy3d`` is organized into named sections reachable
+through ``from tidy3d.config import config``. This page lists every built-in
+section and the options they expose so you can see everything that is
+configurable in one place.
+
+How to read this page
+---------------------
+
+- Environment variable overrides always win. Use the pattern
+  ``TIDY3D_<SECTION>__<FIELD>`` (additional ``__`` segments follow the same
+  nesting, for example ``TIDY3D_PLUGINS__SAMPLE__ENABLED``).
+- Fields tagged with **[persisted]** are written to the active ``config.toml``
+  (base or profile) when you call ``config.save()``. Fields without the tag are
+  runtime-only and revert to their defaults on the next import unless you use
+  ``config.save(include_defaults=True)`` or set them via environment variables
+  or profiles.
+- Types reference Python objects unless noted otherwise. Literal values are
+  quoted and tuples use the ``(x, y, z)`` notation seen in the API.
+
+
+Logging (``config.logging``)
+----------------------------
+
+Controls the verbosity and suppression behaviour of the global logger.
+
+``level`` — ``LogLevel``, default ``"WARNING"`` **[persisted]**
+    Lowest logging level that will be emitted. Accepts ``"DEBUG"``, ``"SUPPORT"``,
+    ``"USER"``, ``"INFO"``, ``"WARNING"``, ``"ERROR"``, and ``"CRITICAL"``.
+
+``suppression`` — ``bool``, default ``True``
+    Suppresses repeated log messages when ``True`` so only the first occurrence
+    of identical messages is shown.
+
+
+Simulation (``config.simulation``)
+----------------------------------
+
+Optional overrides that tweak solver behaviour at runtime.
+
+``use_local_subpixel`` — ``bool | None``, default ``None``
+    Forces or disables the local subpixel averaging routine. ``True`` enables
+    the algorithm, ``False`` disables it, and ``None`` keeps the solver default.
+
+
+Microwave (``config.microwave``)
+--------------------------------
+
+Applies to the microwave solver add-on.
+
+``suppress_rf_license_warning`` — ``bool``, default ``False``
+    When ``True`` the microwave solver skips the warning about RF license
+    availability.
+
+
+Adjoint (``config.adjoint``)
+----------------------------
+
+Parameters that shape autograd (adjoint) behaviour, including local execution
+settings and numerical tolerances.
+
+``min_wvl_fraction`` — ``float``, default ``0.05``
+    Minimum fraction of the smallest wavelength used when discretizing cylinders
+    for autograd derivatives. Must be ``>= 0``.
+
+``points_per_wavelength`` — ``int``, default ``10``
+    Default number of material sample points per wavelength for cylinder
+    discretization. Must be positive.
+
+``default_wavelength_fraction`` — ``float``, default ``0.1``
+    Fallback fraction of the minimum wavelength when adaptive spacing is needed.
+    Must be ``>= 0``.
+
+``minimum_spacing_fraction`` — ``float``, default ``0.01``
+    Smallest normalized spacing allowed when constructing adaptive finite
+    difference stencils. Must be ``>= 0``.
+
+``local_gradient`` — ``bool``, default ``False`` **[persisted]**
+    Enables local gradient evaluation. Remote gradients ignore the other adjoint
+    overrides unless this is ``True``.
+
+``local_adjoint_dir`` — ``pathlib.Path``, default ``"adjoint_data"`` **[persisted]**
+    Directory (relative to the working directory) where intermediate gradient
+    artifacts are stored when ``local_gradient`` is enabled.
+
+``gradient_precision`` — ``Literal["single", "double"]``, default ``"single"``
+    Floating-point precision used for gradient calculations.
+
+``monitor_interval_poly`` — ``tuple[int, int, int]``, default ``(1, 1, 1)``
+    Cell spacing between samples for polynomial autograd monitors.
+
+``monitor_interval_custom`` — ``tuple[int, int, int]``, default ``(1, 1, 1)``
+    Cell spacing between samples for custom autograd monitors.
+
+``quadrature_sample_fraction`` — ``float``, default ``0.4``
+    Fraction of uniform samples reused when building Gauss quadrature nodes.
+    Must be between ``0`` and ``1``.
+
+``gauss_quadrature_order`` — ``int``, default ``7``
+    Maximum Gauss–Legendre order used in composite quadrature rules. Must be
+    positive.
+
+``edge_clip_tolerance`` — ``float``, default ``1e-9``
+    Padding tolerance used when clipping polygon edges during surface
+    integrations. Must be ``>= 0``.
+
+``solver_freq_chunk_size`` — ``int | None``, default ``None``
+    Upper bound on the number of frequencies processed per chunk during gradient
+    evaluation. ``None`` disables chunking.
+
+``max_traced_structures`` — ``int``, default ``500``
+    Maximum number of structures whose fields may be traced in an adjoint run.
+
+``max_adjoint_per_fwd`` — ``int``, default ``10``
+    Maximum number of adjoint simulations dispatched per forward solve.
+
+
+Web (``config.web``)
+--------------------
+
+Settings for the cloud API client and related environment overrides.
+
+``apikey`` — ``SecretStr | None``, default ``None`` **[persisted]**
+    API key used for authentication. The value is masked when serialized. Also
+    accepts ``SIMCLOUD_APIKEY`` as a shortcut environment variable.
+
+``ssl_verify`` — ``bool``, default ``True``
+    Verifies SSL certificates for API requests.
+
+``enable_caching`` — ``bool``, default ``True`` **[persisted]**
+    Allows the web service to return cached simulation results when available.
+
+``api_endpoint`` — ``str``, default ``"https://tidy3d-api.simulation.cloud"`` **[persisted]**
+    Base URL for API calls. Must be an HTTP or HTTPS URL.
+
+``website_endpoint`` — ``str``, default ``"https://tidy3d.simulation.cloud"`` **[persisted]**
+    Base URL for the Tidy3D website. Must be an HTTP or HTTPS URL.
+
+``s3_region`` — ``str``, default ``"us-gov-west-1"``
+    AWS region used by the platform’s S3 storage.
+
+``timeout`` — ``int``, default ``120``
+    HTTP request timeout in seconds. Must be between ``0`` and ``300``.
+
+``ssl_version`` — ``ssl.TLSVersion | None``, default ``None``
+    Explicit TLS version to enforce. Accepts values such as
+    ``ssl.TLSVersion.TLSv1_2``. ``None`` lets ``requests`` negotiate the version.
+
+``env_vars`` — ``dict[str, str]``, default ``{}``
+    Additional environment variables exported before API calls. Handy for proxy
+    or credential helpers.
+
+
+Local Cache (``config.local_cache``)
+------------------------------------
+
+Controls the optional on-disk cache for simulation artifacts.
+
+``enabled`` — ``bool``, default ``False`` **[persisted]**
+    Turns the local cache on or off. When enabled, results are reused if the
+    inputs match.
+
+``directory`` — ``pathlib.Path``, default ``<base>/cache/simulations`` **[persisted]**
+    Directory where cached artifacts are stored. The path is expanded, resolved,
+    and created if missing. ``<base>`` comes from ``TIDY3D_BASE_DIR`` when set,
+    otherwise ``XDG_CACHE_HOME`` on Unix or ``~/.cache`` on other systems, so the
+    default resolves to ``~/.cache/tidy3d/simulations``.
+
+``max_size_gb`` — ``float``, default ``10.0`` **[persisted]**
+    Maximum cache size in gigabytes. ``0`` disables the size limit.
+
+``max_entries`` — ``int``, default ``0`` **[persisted]**
+    Maximum number of cached simulations retained. ``0`` means no limit and the
+    cache falls back to size-based eviction.
+
+
+Plugins (``config.plugins``)
+----------------------------
+
+Acts as a container for plugin-defined sections. After a plugin calls
+``@register_plugin("name")``, its configuration becomes available under
+``config.plugins.<name>`` and follows the same persistence and environment
+variable rules described above (for example ``TIDY3D_PLUGINS__NAME__FIELD``).

tidy3d/config/README.md

@@ -0,0 +1,183 @@
+# `tidy3d.config` Architecture
+
+The configuration subsystem wires together runtime defaults, environment
+overrides, profile files, and optional plugin hooks so that `from tidy3d.config
+import config` always exposes an up-to-date view of the active settings. This
+document focuses on the developer-facing design so you can confidently extend
+or debug the module.
+
+## Big Picture
+
+- **Pydantic schemas define sections.** Every built-in section lives in
+  `sections.py` and is registered via the `@register_section` decorator.
+- **A central manager composes state.** `ConfigManager` merges defaults,
+  environment variables, builtin/user profiles, and runtime overrides. It also
+  applies side-effect handlers whenever values change.
+- **Persistence is layered.** `ConfigLoader` handles filesystem reads/writes;
+  `serializer.py` keeps TOML files annotated with descriptions and stable key
+  ordering.
+- **Registries are dynamic.** `registry.py` tracks section schemas and handler
+  callables and notifies the manager whenever new items arrive (including from
+  plugins).
+- **Legacy wrappers exist for compatibility.** `legacy.py` exposes the previous
+  API surface while delegating to the modern manager.
+
+## Lifecycle Overview
+
+1. Importing `tidy3d.config` loads `sections.py`, which registers every built-in
+   section and handler.
+2. `ConfigManager` instantiates, attaches itself to the registry, and loads the
+   effective tree by merging builtin profiles, `config.toml`, profile overrides,
+   environment variables, and any runtime overrides.
+3. Handlers declared with `@register_handler` run to push settings into global
+   side effects (log level, environment variables, cache directories, etc.).
+4. The public `config` object is a `LegacyConfigWrapper` that proxies to the
+   active manager but still honours legacy attributes.
+5. Runtime mutations go through `ConfigManager.update_section`, which triggers a
+   reload+reapply cycle so memoized models always represent the latest state.
+
+## Component Map
+
+```mermaid
+flowchart LR
+    subgraph ImportTime["Import Time"]
+        sections_py["sections.py<br/>@register_section"] --> registry_py
+        sections_handlers["sections.py<br/>@register_handler"] --> registry_py
+    end
+
+    subgraph Registry
+        registry_py["registry.py<br/>section & handler registries"]
+    end
+
+    subgraph Manager
+        manager_py["manager.ConfigManager"] --> loader_py
+        manager_py --> handlers["Registered handlers"]
+        manager_py --> legacy_wrapper["legacy.LegacyConfigWrapper"]
+        manager_py --> plugins_accessor["plugins accessor"]
+    end
+
+    subgraph Persistence
+        loader_py["loader.ConfigLoader"] --> serializer_py
+        serializer_py["serializer.py<br/>annotated TOML builder"] --> filesystem["config.toml<br/>profiles/<name>.toml"]
+    end
+
+    env_vars["Environment variables"] --> loader_py
+    builtin_profiles["profiles.py<br/>BUILTIN_PROFILES"] --> manager_py
+    runtime_overrides["Runtime overrides"] --> manager_py
+    plugins["register_plugin(... )<br/>(plugin imports)"] --> registry_py
+    registry_py --> manager_py
+```
+
+## Module Reference
+
+### `sections.py`
+
+- Defines concrete `ConfigSection` subclasses (Pydantic models) for built-in
+  sections (`logging`, `simulation`, `microwave`, `adjoint`, `web`,
+  `local_cache`, `plugins` container).
+- Decorated with `@register_section("name")` so the schema is discoverable at
+  import time. For plugins, use `@register_plugin("plugin_name")`.
+- Optional `@register_handler("name")` functions apply side effects after the
+  manager reloads (e.g., update logger globals, export environment variables).
+- Fields can include `json_schema_extra={"persist": True}` to mark values that
+  should be written to disk by default.
+
+### `registry.py`
+
+- Holds the global section and handler dictionaries.
+- Exposes decorators (`register_section`, `register_plugin`, `register_handler`)
+  and accessors (`get_sections`, `get_handlers`).
+- Provides `attach_manager()` so the active `ConfigManager` is notified whenever
+  new sections or handlers register. This allows late imports (plugins, tests)
+  to automatically appear without manual refresh.
+
+### `manager.py`
+
+- `ConfigManager` orchestrates the entire system:
+  - Attaches to the registry and caches per-section Pydantic model instances.
+  - Loads data through `ConfigLoader` and merges layers using `deep_merge`.
+  - Tracks runtime overrides in memory per profile.
+  - Filters persisted values (`_filter_persisted`) so `config.save()` writes only
+    annotated fields unless `include_defaults=True`.
+  - Invokes handlers after every reload or targeted update.
+  - Exposes helper accessors (`plugins`, `profiles`, `get_section`, `format`,
+    etc.).
+- `SectionAccessor` proxies dot-attribute access back into `update_section`.
+- Normalizes profile names so builtin aliases like `"prod"` resolve
+  consistently.
+
+### `loader.py`
+
+- Resolves the configuration directory (`resolve_config_directory`) based on
+  platform, `$TIDY3D_BASE_DIR`, and legacy fallbacks.
+- Reads/writes `config.toml` and profile overrides atomically, including
+  temporary file + backup behaviour.
+- Parses environment variables into nested dictionaries (`load_environment_overrides`).
+- Supplies dictionary utilities (`deep_merge`, `deep_diff`) used throughout
+  the manager.
+- Coordinates with `serializer.build_document` to preserve inline comments and
+  descriptions when writing TOML.
+
+### `serializer.py`
+
+- Looks up field descriptions from registered Pydantic models so TOML files
+  stay annotated.
+- Converts nested dictionaries into `tomlkit` documents with stable formatting
+  and comment preservation—critical for user-friendly diffs.
+
+### `profiles.py`
+
+- Maintains the shipped profile presets (`default`, `prod`, `dev`, `uat`,
+  `pre`, `nexus`). The manager merges these into the baseline before applying
+  user overrides.
+
+### `legacy.py`
+
+- Provides backwards-compatible attribute access (`LegacyConfigWrapper`,
+  `LegacyEnvironment`, etc.) that forward to the modern manager.
+- Emits `DeprecationWarning`s to encourage migrations while keeping older code
+  functional.
+
+## Adding a New Section
+
+1. Define a Pydantic model in `sections.py` (or your plugin) that inherits from
+   `ConfigSection` and decorate it with `@register_section("name")`.
+2. Optionally decorate a function with `@register_handler("name")` to apply
+   side effects whenever the section changes.
+3. Add field descriptions and `json_schema_extra={"persist": True}` where
+   appropriate so they show up in the generated docs and persisted config.
+4. Import the module somewhere reachable so registration happens as part of
+   normal startup (built-ins live in `sections.py`; plugins should register at
+   import time).
+
+`ConfigManager` will automatically detect the new section, expose it under
+`config.<name>`, and include it in persistence and documentation without extra
+wiring.
+
+## Handler Side Effects
+
+- Handlers receive the fully validated Pydantic model for the section.
+- Only the sections explicitly updated trigger their handler; call
+  `config.reload_config()` or `ConfigManager._apply_handlers()` manually if you
+  need to reapply everything (usually done during initialization).
+- Handlers must be robust to repeated calls because they run on every reload.
+
+## Persistence and Serialization Notes
+
+- Only fields marked with `json_schema_extra={"persist": True}` are written by
+  default. Call `config.save(include_defaults=True)` to flush the entire model
+  tree to disk (useful for debugging).
+- TOML output preserves descriptions derived from field docstrings, so keep the
+  docstrings concise and informative.
+- The loader writes files atomically and stores a `.bak` during the swap to
+  prevent data loss. Beware of direct edits to `_loader._docs`; tests can use
+  `ConfigLoader` to manipulate the cached documents safely.
+
+## Debugging Tips
+
+- `config.format()` (or `print(config)`) renders a Rich tree of the current
+  effective configuration—useful when verifying merges.
+- To inspect persisted values without env overrides, call
+  `ConfigManager._compose_without_env()` in a debugger.
+- The registry exposes `get_sections()` / `get_handlers()` for quick sanity
+  checks that your plugin registered correctly.