BLDX-1076: Add GENERIC_OPENLINEAGE connector type and make emit() accept connector_type#908
Merged
luke-hoffman1 merged 4 commits intoApr 29, 2026
Conversation
…ept connector_type
Add GENERIC_OPENLINEAGE = ("generic-openlineage", ELT) to AtlanConnectorType
so PyAtlan can target the marketplace generic-openlineage (GOLC) endpoint at
/events/openlineage/generic-openlineage/api/v1/lineage.
Make OpenLineageEvent.emit() / emit_async() accept an optional connector_type
kwarg (defaulting to SPARK for backwards compatibility), mirroring the
existing pattern on emit_raw()/emit_raw_async(). Applied to both the current
and pyatlan_v9 copies of event.py.
Unit tests cover:
- GENERIC_OPENLINEAGE enum value
- Routing to /events/openlineage/generic-openlineage/api/v1/lineage
- emit() honors the connector_type kwarg
- emit() default is still SPARK
- Fix ruff formatting in event.py and open_lineage_test.py - Add integration test for emit() with GENERIC_OPENLINEAGE connector type
luke-hoffman1
deleted the
mothership/BLDX-1076/generic-openlineage-connector
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Session owner: @mothershipbeta
Summary
GENERIC_OPENLINEAGE = ("generic-openlineage", AtlanConnectionCategory.ELT)toAtlanConnectorTypeso PyAtlan can target the marketplacegeneric-openlineage(GOLC) ingestion endpoint at/events/openlineage/generic-openlineage/api/v1/lineage.AtlanConnectorType.SPARKinOpenLineageEvent.emit()/emit_async()— both now accept an optionalconnector_typekwarg defaulting toSPARK(matching the pattern already used byemit_raw()/emit_raw_async()).pyatlan/model/open_lineage/event.pyandpyatlan_v9/model/open_lineage/event.py. The v9 enum module re-exports from the main module, so the new enum value is automatically available underpyatlan_v9as well.generic-openlineage/api/v1/lineage, the newconnector_typekwarg onemit(), and preservation of the SPARK default.Context
PyAtlan (v9.4.0) had no way to push OpenLineage events to the GOLC endpoint —
AtlanConnectorType.OPENLINEAGEresolves toevents/openlineage/openlineage/...which is not configured, andSPARKresolves toevents/openlineage/spark/...which routes to a different workflow. See BLDX-1076 for the full problem statement and endpoint routing reference (pyatlan/client/common/open_lineage.py::OpenLineageSend.prepare_request).Blast radius
Both changes are purely additive and backwards-compatible:
emit()/emit_async()gain a kwarg with the same effective default — all existing call-sites continue to work unchanged.Test plan
tests/unit/model/open_lineage/open_lineage_test.pypytest tests/unit/model/open_lineage/open_lineage_test.py— 27 passedpytest tests/unit/model/— 839 passed (no regressions in adjacent unit tests)🤖 Generated with Claude Code