fix: DbtProcess/DbtColumnProcess parents + typing.get_type_hints shim (SHA-887)#922
Closed
Aryamanz29 wants to merge 1 commit into
Closed
Conversation
…ype_hints shim
Three SHA-887 follow-ups, all of which independently break atlan-publish-app's
ordering graph against pyatlan v9.x.
1. DbtProcess extends Process, not Dbt
- Atlas typedef: superTypes=["Dbt", "Process"]
- Generator picks super_types[0] → Dbt → publish-app's
issubclass(cls, Process) returned False → DbtProcess landed in
Batch 1 alongside its upstream entities → ATLAS-404 on inputs/outputs
in real dbt runs.
- Fix: extend _SUPERCLASS_OVERRIDES with "DbtProcess": "Process";
also patch the generated dbt_process.py source.
2. DbtColumnProcess extends ColumnProcess, not Dbt
- Same root cause: superTypes=["Dbt", "ColumnProcess"]; same fix.
3. typing.get_type_hints() shim for core/__init__.py
- Five core asset modules (s_q_l, schema, data_product,
data_quality_rule, process) gate cross-module relationship-type
imports under TYPE_CHECKING. With from __future__ import
annotations, those forward refs can't be resolved by
typing.get_type_hints() against the defining module's __dict__
and the call raises NameError (e.g. "name 'DbtTest' is not
defined" for Table.Attributes). Pydantic v1's
update_forward_refs(**localns) already handles this for the SDK's
own use, but reflection-based downstream tooling — notably
atlan-publish-app's ordering graph — falls through to a degraded
fallback that collapses all SQL types into one publish batch.
- Fix: after every asset has loaded, core/__init__.py injects the
resolved classes back into each affected module's __dict__ so
typing.get_type_hints(cls.Attributes) succeeds without a
caller-supplied localns. New _RUNTIME_INJECTION_OVERRIDES constant
+ _build_runtime_injection_block helper in class_generator.py and
a corresponding placeholder in templates/core/init.jinja2 ensure
the shim survives regeneration.
Tests
- tests/unit/model/dbt_process_test.py — 7 cases (issubclass guards,
__bases__[0], type_name immutability, dbt_* + Process attribute
round-trip).
- tests/unit/model/dbt_column_process_test.py — same 7 cases for the
ColumnProcess parent.
- tests/unit/model/get_type_hints_compat_test.py — parametrised over
all 5 affected core classes plus a concrete spot-check that
SQL.Attributes.dbt_tests resolves to the real DbtTest class.
Verified
- ./qa-checks: ruff format / ruff check / mypy all pass.
- pytest tests/unit -q --ignore=tests/unit/aio: 6033 passed, 2 skipped.
Release notes will land in the next version bump (HISTORY.md is left
alone here so 9.7.0's already-shipped section isn't retroactively
edited).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment on lines
+430
to
+454
| # typing.get_type_hints() compatibility shim | ||
| # | ||
| # Some Attributes classes reference cross-module relationship types whose | ||
| # imports must stay under TYPE_CHECKING to avoid circular imports at module | ||
| # load time. Pydantic v1 resolves these forward refs via update_forward_refs() | ||
| # above, but typing.get_type_hints() does its own resolution against each | ||
| # class's defining module __dict__ — which doesn't see TYPE_CHECKING-only | ||
| # names. Inject the resolved classes back into the affected modules now that | ||
| # all assets have loaded so downstream tooling (e.g. atlan-publish-app's | ||
| # ordering graph) can call typing.get_type_hints() without a custom localns. | ||
| from . import data_product as _data_product_module | ||
| from . import data_quality_rule as _data_quality_rule_module | ||
| from . import process as _process_module | ||
| from . import s_q_l as _s_q_l_module | ||
| from . import schema as _schema_module | ||
|
|
||
| _data_product_module.StarburstDataset = StarburstDataset # type: ignore[misc] | ||
| _data_quality_rule_module.Column = Column # type: ignore[misc] | ||
| _process_module.Procedure = Procedure # type: ignore[misc] | ||
| _process_module.BigqueryRoutine = BigqueryRoutine # type: ignore[misc] | ||
| _process_module.FabricActivity = FabricActivity # type: ignore[misc] | ||
| _process_module.Function = Function # type: ignore[misc] | ||
| _s_q_l_module.DbtTest = DbtTest # type: ignore[misc] | ||
| _schema_module.SnowflakeDynamicTable = SnowflakeDynamicTable # type: ignore[misc] | ||
| _schema_module.Table = Table # type: ignore[misc] |
There was a problem hiding this comment.
Bug: Duplicated shim block
Lines 404-428 and 430-454 contain identical runtime injection shim code. The main branch __init__.py ends at line 402 — this PR should add one shim block (26 lines), not two (52 lines).
The duplication suggests either:
- The file was generated twice without cleaning the intermediate state, or
- The generated output was manually edited after generation
Suggested change
| # typing.get_type_hints() compatibility shim | |
| # | |
| # Some Attributes classes reference cross-module relationship types whose | |
| # imports must stay under TYPE_CHECKING to avoid circular imports at module | |
| # load time. Pydantic v1 resolves these forward refs via update_forward_refs() | |
| # above, but typing.get_type_hints() does its own resolution against each | |
| # class's defining module __dict__ — which doesn't see TYPE_CHECKING-only | |
| # names. Inject the resolved classes back into the affected modules now that | |
| # all assets have loaded so downstream tooling (e.g. atlan-publish-app's | |
| # ordering graph) can call typing.get_type_hints() without a custom localns. | |
| from . import data_product as _data_product_module | |
| from . import data_quality_rule as _data_quality_rule_module | |
| from . import process as _process_module | |
| from . import s_q_l as _s_q_l_module | |
| from . import schema as _schema_module | |
| _data_product_module.StarburstDataset = StarburstDataset # type: ignore[misc] | |
| _data_quality_rule_module.Column = Column # type: ignore[misc] | |
| _process_module.Procedure = Procedure # type: ignore[misc] | |
| _process_module.BigqueryRoutine = BigqueryRoutine # type: ignore[misc] | |
| _process_module.FabricActivity = FabricActivity # type: ignore[misc] | |
| _process_module.Function = Function # type: ignore[misc] | |
| _s_q_l_module.DbtTest = DbtTest # type: ignore[misc] | |
| _schema_module.SnowflakeDynamicTable = SnowflakeDynamicTable # type: ignore[misc] | |
| _schema_module.Table = Table # type: ignore[misc] |
Remove this entire block (lines 430-454). The first shim block at lines 404-428 is sufficient.
Member
Author
|
Superseded — splitting into two cleaner PRs (atlan-python parent-class fixes + atlan-publish-app switch to pydantic ModelField). The get_type_hints shim added too much pyatlan-side maintenance burden; the better fix is publish-app moving off typing.get_type_hints entirely. New PRs incoming. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three SHA-887 follow-ups that all independently break atlan-publish-app's ordering graph against pyatlan v9.x. Bundling them because the symptoms overlap (publish-app order regressions reproduce until the full set lands) and they share the same generator-level mechanism.
1.
DbtProcessextendsProcess, notDbtsuperTypes=[\"Dbt\", \"Process\"]super_types[0]→Dbt→ publish-app'sissubclass(cls, Process)check (used to identify lineage types that must publish after the entities they reference) returnedFalse.DbtProcessthen landed in Batch 1 alongside its upstream entities → ATLAS-404 oninputs/outputslookups in live dbt runs.\"DbtProcess\": \"Process\"to_SUPERCLASS_OVERRIDESinclass_generator.pyand patch the generatedpyatlan/model/assets/dbt_process.pysource.2.
DbtColumnProcessextendsColumnProcess, notDbtsuperTypes=[\"Dbt\", \"ColumnProcess\"].pyatlan/model/assets/dbt_column_process.pyand the override map.3.
typing.get_type_hints()shim for coreAttributesclassess_q_l,schema,data_product,data_quality_rule,process) gate cross-module relationship-type imports underif TYPE_CHECKING:to avoid circular imports at module load. Withfrom __future__ import annotations, those forward-ref strings can't be resolved bytyping.get_type_hints()against the defining module's__dict__and the call raisesNameError— e.g.name 'DbtTest' is not definedforTable.Attributes.update_forward_refs(**localns)incore/__init__.pyalready handles this for the SDK's own use, but reflection-based downstream tooling (notablyatlan-publish-app's ordering graph) falls through to a degraded fallback that collapses all SQL types into one publish batch.core/__init__.pyinjects the resolved classes back into each affected module's__dict__sotyping.get_type_hints(cls.Attributes)succeeds without a caller-suppliedlocalns._RUNTIME_INJECTION_OVERRIDESconstant +_build_runtime_injection_blockhelper inclass_generator.pyplus a corresponding placeholder intemplates/core/init.jinja2ensure the shim survives regeneration.Tests
tests/unit/model/dbt_process_test.py— 7 cases (issubclassguards,__bases__[0],type_nameimmutability,dbt_*+Processattribute round-trip).tests/unit/model/dbt_column_process_test.py— same 7 cases for theColumnProcessparent.tests/unit/model/get_type_hints_compat_test.py— parametrised over all 5 affected core classes plus a concrete spot-check thatSQL.Attributes.dbt_testsresolves to the realDbtTestclass.Verified
./qa-checks— ruff format / ruff check / mypy all passpytest tests/unit -q --ignore=tests/unit/aio— 6033 passed, 2 skippedissubclass/get_type_hintschecks against installed pyatlanDbtProcess/DbtColumnProcessmove to a strictly later batch than the entities they reference, andtyping.get_type_hints()resolves on every coreAttributesclass without a workaroundlocalnsNotes
Release notes will be added in the next version bump —
HISTORY.mdis left alone here so the already-shipped 9.7.0 section isn't retroactively edited.🤖 Generated with Claude Code