Skip to content

fix: DbtProcess/DbtColumnProcess parents + typing.get_type_hints shim (SHA-887)#922

Closed
Aryamanz29 wants to merge 1 commit into
mainfrom
mothership/SHA-887/dbt-process-superclass-and-get-type-hints-shim
Closed

fix: DbtProcess/DbtColumnProcess parents + typing.get_type_hints shim (SHA-887)#922
Aryamanz29 wants to merge 1 commit into
mainfrom
mothership/SHA-887/dbt-process-superclass-and-get-type-hints-shim

Conversation

@Aryamanz29
Copy link
Copy Markdown
Member

Three SHA-887 follow-ups that all independently break atlan-publish-app's ordering graph against pyatlan v9.x. Bundling them because the symptoms overlap (publish-app order regressions reproduce until the full set lands) and they share the same generator-level mechanism.

1. DbtProcess extends Process, not Dbt

  • Atlas typedef: superTypes=[\"Dbt\", \"Process\"]
  • Generator picks super_types[0]Dbt → publish-app's issubclass(cls, Process) check (used to identify lineage types that must publish after the entities they reference) returned False.
  • DbtProcess then landed in Batch 1 alongside its upstream entities → ATLAS-404 on inputs / outputs lookups in live dbt runs.
  • Fix: add \"DbtProcess\": \"Process\" to _SUPERCLASS_OVERRIDES in class_generator.py and patch the generated pyatlan/model/assets/dbt_process.py source.

2. DbtColumnProcess extends ColumnProcess, not Dbt

  • Same root cause: superTypes=[\"Dbt\", \"ColumnProcess\"].
  • Same fix applied to pyatlan/model/assets/dbt_column_process.py and the override map.

3. typing.get_type_hints() shim for core Attributes classes

  • Five core asset modules (s_q_l, schema, data_product, data_quality_rule, process) gate cross-module relationship-type imports under if TYPE_CHECKING: to avoid circular imports at module load. With from __future__ import annotations, those forward-ref strings can't be resolved by typing.get_type_hints() against the defining module's __dict__ and the call raises NameError — e.g. name 'DbtTest' is not defined for Table.Attributes.
  • Pydantic v1's update_forward_refs(**localns) in core/__init__.py already handles this for the SDK's own use, but reflection-based downstream tooling (notably atlan-publish-app's ordering graph) falls through to a degraded fallback that collapses all SQL types into one publish batch.
  • Fix: after every asset has loaded, core/__init__.py injects the resolved classes back into each affected module's __dict__ so typing.get_type_hints(cls.Attributes) succeeds without a caller-supplied localns.
  • A new _RUNTIME_INJECTION_OVERRIDES constant + _build_runtime_injection_block helper in class_generator.py plus a corresponding placeholder in templates/core/init.jinja2 ensure the shim survives regeneration.

Tests

  • tests/unit/model/dbt_process_test.py — 7 cases (issubclass guards, __bases__[0], type_name immutability, dbt_* + Process attribute round-trip).
  • tests/unit/model/dbt_column_process_test.py — same 7 cases for the ColumnProcess parent.
  • tests/unit/model/get_type_hints_compat_test.py — parametrised over all 5 affected core classes plus a concrete spot-check that SQL.Attributes.dbt_tests resolves to the real DbtTest class.

Verified

  • ./qa-checks — ruff format / ruff check / mypy all pass
  • pytest tests/unit -q --ignore=tests/unit/aio — 6033 passed, 2 skipped
  • Manual issubclass / get_type_hints checks against installed pyatlan
  • Downstream verification in atlan-publish-app once this ships: DbtProcess / DbtColumnProcess move to a strictly later batch than the entities they reference, and typing.get_type_hints() resolves on every core Attributes class without a workaround localns

Notes

Release notes will be added in the next version bump — HISTORY.md is left alone here so the already-shipped 9.7.0 section isn't retroactively edited.

🤖 Generated with Claude Code

…ype_hints shim

Three SHA-887 follow-ups, all of which independently break atlan-publish-app's
ordering graph against pyatlan v9.x.

1. DbtProcess extends Process, not Dbt
   - Atlas typedef: superTypes=["Dbt", "Process"]
   - Generator picks super_types[0] → Dbt → publish-app's
     issubclass(cls, Process) returned False → DbtProcess landed in
     Batch 1 alongside its upstream entities → ATLAS-404 on inputs/outputs
     in real dbt runs.
   - Fix: extend _SUPERCLASS_OVERRIDES with "DbtProcess": "Process";
     also patch the generated dbt_process.py source.

2. DbtColumnProcess extends ColumnProcess, not Dbt
   - Same root cause: superTypes=["Dbt", "ColumnProcess"]; same fix.

3. typing.get_type_hints() shim for core/__init__.py
   - Five core asset modules (s_q_l, schema, data_product,
     data_quality_rule, process) gate cross-module relationship-type
     imports under TYPE_CHECKING. With from __future__ import
     annotations, those forward refs can't be resolved by
     typing.get_type_hints() against the defining module's __dict__
     and the call raises NameError (e.g. "name 'DbtTest' is not
     defined" for Table.Attributes). Pydantic v1's
     update_forward_refs(**localns) already handles this for the SDK's
     own use, but reflection-based downstream tooling — notably
     atlan-publish-app's ordering graph — falls through to a degraded
     fallback that collapses all SQL types into one publish batch.
   - Fix: after every asset has loaded, core/__init__.py injects the
     resolved classes back into each affected module's __dict__ so
     typing.get_type_hints(cls.Attributes) succeeds without a
     caller-supplied localns. New _RUNTIME_INJECTION_OVERRIDES constant
     + _build_runtime_injection_block helper in class_generator.py and
     a corresponding placeholder in templates/core/init.jinja2 ensure
     the shim survives regeneration.

Tests
- tests/unit/model/dbt_process_test.py — 7 cases (issubclass guards,
  __bases__[0], type_name immutability, dbt_* + Process attribute
  round-trip).
- tests/unit/model/dbt_column_process_test.py — same 7 cases for the
  ColumnProcess parent.
- tests/unit/model/get_type_hints_compat_test.py — parametrised over
  all 5 affected core classes plus a concrete spot-check that
  SQL.Attributes.dbt_tests resolves to the real DbtTest class.

Verified
- ./qa-checks: ruff format / ruff check / mypy all pass.
- pytest tests/unit -q --ignore=tests/unit/aio: 6033 passed, 2 skipped.

Release notes will land in the next version bump (HISTORY.md is left
alone here so 9.7.0's already-shipped section isn't retroactively
edited).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines +430 to +454
# typing.get_type_hints() compatibility shim
#
# Some Attributes classes reference cross-module relationship types whose
# imports must stay under TYPE_CHECKING to avoid circular imports at module
# load time. Pydantic v1 resolves these forward refs via update_forward_refs()
# above, but typing.get_type_hints() does its own resolution against each
# class's defining module __dict__ — which doesn't see TYPE_CHECKING-only
# names. Inject the resolved classes back into the affected modules now that
# all assets have loaded so downstream tooling (e.g. atlan-publish-app's
# ordering graph) can call typing.get_type_hints() without a custom localns.
from . import data_product as _data_product_module
from . import data_quality_rule as _data_quality_rule_module
from . import process as _process_module
from . import s_q_l as _s_q_l_module
from . import schema as _schema_module

_data_product_module.StarburstDataset = StarburstDataset # type: ignore[misc]
_data_quality_rule_module.Column = Column # type: ignore[misc]
_process_module.Procedure = Procedure # type: ignore[misc]
_process_module.BigqueryRoutine = BigqueryRoutine # type: ignore[misc]
_process_module.FabricActivity = FabricActivity # type: ignore[misc]
_process_module.Function = Function # type: ignore[misc]
_s_q_l_module.DbtTest = DbtTest # type: ignore[misc]
_schema_module.SnowflakeDynamicTable = SnowflakeDynamicTable # type: ignore[misc]
_schema_module.Table = Table # type: ignore[misc]
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Duplicated shim block

Lines 404-428 and 430-454 contain identical runtime injection shim code. The main branch __init__.py ends at line 402 — this PR should add one shim block (26 lines), not two (52 lines).

The duplication suggests either:

  1. The file was generated twice without cleaning the intermediate state, or
  2. The generated output was manually edited after generation
Suggested change
# typing.get_type_hints() compatibility shim
#
# Some Attributes classes reference cross-module relationship types whose
# imports must stay under TYPE_CHECKING to avoid circular imports at module
# load time. Pydantic v1 resolves these forward refs via update_forward_refs()
# above, but typing.get_type_hints() does its own resolution against each
# class's defining module __dict__ — which doesn't see TYPE_CHECKING-only
# names. Inject the resolved classes back into the affected modules now that
# all assets have loaded so downstream tooling (e.g. atlan-publish-app's
# ordering graph) can call typing.get_type_hints() without a custom localns.
from . import data_product as _data_product_module
from . import data_quality_rule as _data_quality_rule_module
from . import process as _process_module
from . import s_q_l as _s_q_l_module
from . import schema as _schema_module
_data_product_module.StarburstDataset = StarburstDataset # type: ignore[misc]
_data_quality_rule_module.Column = Column # type: ignore[misc]
_process_module.Procedure = Procedure # type: ignore[misc]
_process_module.BigqueryRoutine = BigqueryRoutine # type: ignore[misc]
_process_module.FabricActivity = FabricActivity # type: ignore[misc]
_process_module.Function = Function # type: ignore[misc]
_s_q_l_module.DbtTest = DbtTest # type: ignore[misc]
_schema_module.SnowflakeDynamicTable = SnowflakeDynamicTable # type: ignore[misc]
_schema_module.Table = Table # type: ignore[misc]

Remove this entire block (lines 430-454). The first shim block at lines 404-428 is sufficient.

@Aryamanz29
Copy link
Copy Markdown
Member Author

Superseded — splitting into two cleaner PRs (atlan-python parent-class fixes + atlan-publish-app switch to pydantic ModelField). The get_type_hints shim added too much pyatlan-side maintenance burden; the better fix is publish-app moving off typing.get_type_hints entirely. New PRs incoming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant