[Feature]: Replace docval with better strict type and shape validation system with type hints

[rly]

What would you like to see added to HDMF?

I realized we have discussed this many times in the past year but we do not have an issue for it yet. This came up again today during the NWB Data Conversion Workshop.

PyNWB/HDMF uses docval which was developed before type hints were officially supported by Python and before Pydantic and these other strict type-checkers were popular. docval is now incompatible with systems that display type hints, like hovering over variables in Jupyter and auto-complete in IDEs. And it is cumbersome for new developers or anyone browsing the source code to learn. To improve usability and maintainability of the NWB and HDMF APIs, I suggest we replace docval with a more modern strict type-checking system and documentation system. This will be tedious but worth it in the end.

docval is used for documentation, type checking, and shape checking, and it is used in code that inspects other classes like the class generator and neuroconv's code that gets a json schema from a classes' constructor docval. The validator may also use docval args. We have hooks that allow you to create docval aliases like "array_data" that can be dynamically updated, e.g., in HDMF Zarr. We need to be careful when replacing docval to ensure we do not alter or lose significant functionality.

What solution would you like?

Replace docval with another system like pydantic in strict mode, beartype, or numpydantic. Need to research options. Pydantic is widely used and plays nicely with JSON schema, which will be useful in potential long-term integration with LinkML. beartype appears to be quite fast. I think neither plays nicely with numpy arrays, so we may need to use something like numpydantic

Do you have any interest in helping implement the feature?

Yes.

[h-mayorquin]

As a related background information. This has been an ongoing concern in numpy for a while:
numpy/numpy#16544

[mavaylon1]

@rly is this something we can start in August?

[rly]

Yes, as discussed in person, let's target August/September to start working on this together.

[sneakers-the-rat]

related to:
#994
NeurodataWithoutBorders/pynwb#1408 (comment)

i'd be down to help with this <3

[bendichter]

Implementation Plan: Replace docval with Modern Type Validation System

This plan proposes a phased approach to replace HDMF's custom docval decorator with modern Python type hints and validation libraries (Pydantic, beartype, and/or numpydantic). The migration will maintain backward compatibility during transition while improving IDE support, developer experience, and maintainability.

Current State Analysis

docval provides the following functionality:

1. Type checking - Runtime type validation for method arguments
2. Shape validation - Array dimension/shape checking
3. Documentation generation - Auto-generates docstrings (Google-style and Sphinx)
4. Default value handling - With deep-copying of mutable defaults
5. Macros - Extensible type aliases (e.g., 'array_data' → [np.ndarray, list, tuple, h5py.Dataset])
6. Enum validation - Restricting values to allowed sets
7. Helper functions - getargs(), popargs(), get_docval()

Scale of changes:

• ~300+ @docval usages across the codebase
• Used in core classes: Container, Data, Builder, Spec, ObjectMapper, TypeMap
• Class generator dynamically creates @docval decorated __init__ methods
• External tools (neuroconv, PyNWB) introspect docval for JSON schema generation

Proposed Technology Stack

Component	Recommendation	Rationale
Type hints	Python 3.10+ style	Native, IDE-friendly, uses X | Y union syntax
Runtime validation	beartype	Fast, minimal overhead, plays well with numpy
Array shape/dtype	numpydantic	Built for numpy/HDF5 array validation
Schema generation	Pydantic (for Spec classes)	JSON Schema support for LinkML integration

---

Phase 1: Foundation

• Create hdmf.validation module with new validation infrastructure
• Implement @validated decorator that wraps beartype + numpydantic
• Define type aliases to replace docval macros (ArrayData, ScalarData, etc.)
• Create docval-to-type-hints migration script/tool
• Add test helpers to verify behavioral equivalence between old and new validation

Phase 2: Spec Classes Migration

• Convert Spec base class to Pydantic model (continues PR [WIP] Replace Spec classes with Pydantic models #1331)
• Convert AttributeSpec to Pydantic model
• Convert DatasetSpec to Pydantic model
• Convert GroupSpec to Pydantic model
• Convert LinkSpec to Pydantic model
• Convert SpecNamespace and NamespaceCatalog to use type hints

Phase 3: Builder Classes Migration

• Add type hints to BaseBuilder class
• Add type hints to GroupBuilder class
• Add type hints to DatasetBuilder class
• Add type hints to LinkBuilder class
• Add type hints to ReferenceBuilder class

Phase 4: Container Classes Migration

• Add type hints to AbstractContainer base class
• Add type hints to Container class
• Add type hints to Data class
• Add type hints to MultiContainerInterface class
• Add type hints to Table and Row classes

Phase 5: Build System Migration

• Add type hints to ObjectMapper class
• Add type hints to TypeMap class
• Add type hints to BuildManager class
• Refactor ClassGeneratorManager to generate type-hinted __init__ methods
• Update _spec_dtype_map to use new type aliases

Phase 6: Common Module Migration

• Add type hints to DynamicTable and related classes
• Add type hints to VectorData, VectorIndex, ElementIdentifiers
• Add type hints to AlignedDynamicTable
• Add type hints to HERD / external resources classes

Phase 7: I/O Classes Migration

• Add type hints to HDMFIO base class
• Add type hints to HDF5IO class
• Add type hints to H5 utility classes

Phase 8: External API Compatibility

• Create adapter for get_docval() to return type hint info in compatible format
• Provide JSON Schema generation utility from type hints
• Add deprecation warnings with migration guidance
• Document new validation error message format

Phase 9: Cleanup & Documentation

• Remove deprecated docval code from utils.py
• Update API documentation
• Update tutorials and examples
• Write migration guide for downstream packages (PyNWB, neuroconv, hdmf-zarr)

---

Breaking Changes & Mitigation

Change	Impact	Mitigation
get_docval() return format	External tools introspecting docval	Provide adapter during deprecation period
Positional arg handling	Some user code may rely on current behavior	AllowPositional config option maintained
Shape validation syntax	Different error messages	Document new error format

Testing Strategy

1. Dual validation mode - Run both old and new validators during transition, compare results
2. Property-based testing - Use Hypothesis to generate test cases
3. Integration tests - Full read/write cycle tests with PyNWB

Dependencies to Add

[project.optional-dependencies]
validation = [
    "beartype>=0.15.0",
    "numpydantic>=1.0.0",
]