Partitioned External Assets

[chximn]

Dagster lacks official support for partitioned external assets. However, this seems to be possible to implement since there is a workaround available in a previous discussion: #18211.

This would make it possible to include external assets that are not managed by Dagster while keeping the same partitioning externally.

Workaround:

class AssetSpecWithPartitionsDef(
    namedtuple(
        "AssetSpecWithPartitionsDef",
        AssetSpec._fields + ("partitions_def", ),
        defaults=(None,) * (1 + len(AssetSpec._fields)),
    )
):
    ...

def external_asset_from_spec_with_partition(*, partitions_def, **spec_kwargs):
    asset = external_asset_from_spec(
        AssetSpecWithPartitionsDef(
            **AssetSpec(**spec_kwargs)._asdict(),
            partitions_def=partitions_def,
        )
    )
    asset._partitions_def = partitions_def
    return asset


external_asset = external_asset_from_spec_with_partition(
    key="external_asset_key",
    partitions_def=custom_partition_def,
)

I have tested the above workaround and it seems to work fine but I can't say for sure if this the right way to do it.
It would be nice to have an official support for such feature.

[astronautas]

Up! It

Dagster lacks official support for partitioned external assets. However, this seems to be possible to implement since there is a workaround available in a previous discussion: #18211.

This would make it possible to include external assets that are not managed by Dagster while keeping the same partitioning externally.

Workaround:

class AssetSpecWithPartitionsDef(
    namedtuple(
        "AssetSpecWithPartitionsDef",
        AssetSpec._fields + ("partitions_def", ),
        defaults=(None,) * (1 + len(AssetSpec._fields)),
    )
):
    ...

def external_asset_from_spec_with_partition(*, partitions_def, **spec_kwargs):
    asset = external_asset_from_spec(
        AssetSpecWithPartitionsDef(
            **AssetSpec(**spec_kwargs)._asdict(),
            partitions_def=partitions_def,
        )
    )
    asset._partitions_def = partitions_def
    return asset


external_asset = external_asset_from_spec_with_partition(
    key="external_asset_key",
    partitions_def=custom_partition_def,
)

I have tested the above workaround and it seems to work fine but I can't say for sure if this the right way to do it. It would be nice to have an official support for such feature.

Hey, thanks for the example, but it doesn't seem to work in the sense that in the UI partitions are not visible :/. At least when I use HourlyPartitionsDefinition.

[chximn]

@astronautas
I have also encountered this issue of partitions in external assets not being visible in Dagster after updating to some version >=1.6.7. An issue related to this is still open : #20396

[astronautas]

Up! Is this still an issue? We're planning to use external assets.

[the4thamigo-uk]

We really need this. The workaround is broken in dagster 1.7.14 it seems, and in general such workarounds seem really flakey and temporary.

[ammerzon]

@the4thamigo-uk I agree that we need official support for this use case. Could you provide more details about the issue? Are you encountering a specific error message, or is the system simply not functioning as expected?

[chximn]

Partitioned external assets can now be declared directly with AssetSpec.
I can confirm that this works fine as of version 1.8.7, but it could probably also work in other earlier versions.

from dagster import AssetSpec
external_asset = AssetSpec(key=..., partitions_def=...)

[v1gnesh]

@chximn will they show up like regular assets (partition info, check info) in the UI?

[daviddwlee84]

Seems AssetSpec can accept partitions_def now. But not showing the "partition" tab somehow.
And it will be materialized entirely with AssetMaterialization(..., partition=...)

dagster                                  1.8.13
dagster-cloud                            1.8.13
dagster-cloud-cli                        1.8.13
dagster-graphql                          1.8.13
dagster-pipes                            1.8.13
dagster-webserver                        1.8.13