Export partition - allow non matching partition expressions in case we can prove the destination expression does not split the data by arthurpassos · Pull Request #2074 · Altinity/ClickHouse

arthurpassos · 2026-07-17T20:39:56Z

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Allow export partition through different partition expressions as long as the destination expression does not repartition the data. This is validated at schedule time through two mechanisms:

Structural match: in case the source and destination are identical, the destination expression is a subset of the source expression or the destination expression can be entirely computed using only constants and the exact values guaranteed (pinned) by the source.
Dynamic proof: the destination expression is monotonic over the source partition min/max range.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-07-17T20:41:02Z

Workflow [PR], commit [9a1cb91]

arthurpassos · 2026-07-18T17:35:56Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f3af7c4fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…n-monotonic-compat

ilejn · 2026-07-21T14:55:32Z

Consider mentioning 'export' in the title and/or description

arthurpassos · 2026-07-21T14:59:02Z

Consider mentioning 'export' in the title and/or description

Done, I'll soon add the description

…n-monotonic-compat

arthurpassos · 2026-07-24T13:04:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca43cf1f6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Selfeer · 2026-08-06T10:38:21Z

@Selfeer I would appreciate if you could "humanize" a bit more those AI reports. For example, the following issue is very hard to understand and doesn't explain what is happening: 1. 🔴 High — Argument order lost, monotonicity proof evaluates the wrong expression.

It says "anchor" pointing to some code location. Then it says the impact. Then the trigger. It is too much to read to understand the real problem. Brain energy required to process this is very high.

Instead, if you could understand the issue yourself first and then give me a few SQL instructions that repro the case and a human explanation of what's going on, that would be 100 times better.

I agree. I'd much rather do it the way you described: perform my own investigation on the findings and raise issues as needed-and we actually do that. But even in that case, I would still have to post this exact message first.

The purpose of the audit review has always been to perform a quick review of the PR without running any tests first, share the findings with the developer, and let you decide whether they are actual issues. If they are, we then raise separate issues afterward.

I can update the skills we use for the audit review to make the output easier to read, but overall, the audit review has always been a separate part of the verification process, separate from our actual testing.

…n-monotonic-compat

arthurpassos · 2026-08-20T13:07:46Z

@k-morozov hi, I have made some refactorings, fixed conflicts and updated the docs. Could you please re-review it?

arthurpassos · 2026-08-20T13:09:20Z

@Selfeer regarding the 5 AI findings you posted

Should be fixed
Seems like AI gibberish. At the very least, not related to this pr
Should be fixed
Outdated, that code path does not exist anymore
Not related to this pr

k-morozov · 2026-08-21T08:26:11Z

+            "To allow its usage, enable the setting `allow_insert_into_iceberg`.");
+    }
+
+    ExportPartitionUtils::verifyExportSchemaCastable(


Calls ExportPartitionUtils::verifyExportSchemaCastable twice here and below.

Thanks, bad rebase

k-morozov · 2026-08-21T08:39:24Z

    )


-def test_export_part_unnamed_tuple_partition_key_owner_matching_named_destination_is_allowed(cluster):


What's the reason?

k-morozov · 2026-08-21T08:42:36Z

+    /// so the same expression over the two types can produce different partitions.
+    bool isSameTypeForPartitioning(const DataTypePtr & lhs, const DataTypePtr & rhs)
+    {
+        return lhs->getName() == rhs->getName();


What about unnamed Tuple, Array, etc?
Does it related with test test_export_part_unnamed_tuple_partition_key_owner_matching_named_destination_is_allowed that was removed?
up: maybe haveSameTupleElementLayout will be usefull.

Answering #2074 (comment) here as it relates to this question.

The main reason I removed that test, and yes, it is affected by this name check, is to draw a line of "good enough" and move on. This PR needs to get merged so that we fix the broader and more common use cases, we can worry about named tuples later.

k-morozov · 2026-08-21T08:54:44Z

+
+        /// A NULL value forms its own destination partition, so a Nullable column may split the source
+        /// partition; min/max cannot rule that out. Require a structural match for such columns.
+        if (isNullableOrLowCardinalityNullable(source_type))


Seems that this behauvor should be in documentation (and motivation maybe?) Also I need to find tests for this condition..

k-morozov · 2026-08-21T09:02:23Z

+
+        /// Compute the global min/max index of the parts
+        IMergeTreeDataPart::MinMaxIndex minmax;
+        for (const auto & part : parts)


As I can see, we prepare mimax every time even if we have an equal partition key. But we really need it only for verifyOutputMapsToSinglePartition.

Right, but verifyOutputMapsToSinglePartition is called within a for loop. What is your suggestion?

For example, create 2 loops:
1 for happy path.
2. for verifyOutputMapsToSinglePartition.

So we could create minmax between 1 and 2 steps.

Another way - to fill minmax within the current loop only once before verifyOutputMapsToSinglePartition.

Imo, that's somewhat of an unecessary "optimization" that will not give us any speed up with the cost of more code. I vote for keeping it as is unless you strongly object.

k-morozov · 2026-08-21T09:22:20Z

        if (parquet_row_group_size_bytes)
            json.set("parquet_row_group_size_bytes", *parquet_row_group_size_bytes);
+        if (iceberg_partition_timezone)
+            json.set("iceberg_partition_timezone", *iceberg_partition_timezone);


We try to set this setting for export. But if we have several columns with different timezones, we would set the value from the setting for all of them, right?

I think that's how this setting was designed, @ianton-ru can confirm. AFAIK, this setting was created due to a misconception on the Apache Iceberg specs (the spec itself is poorly written) when it was believed we could tweak the timezone of an apache iceberg column. When in reality, according to 3rd party docs, the types are always UTC.

Selfeer · 2026-08-25T11:37:52Z

export partition regression tests are green, from audit review there is one low issue found:

Low: In StorageReplicatedMergeTree::exportPartitionToTable, export_merge_tree_partition_force_export = 1 removes an existing export znode via tryRemoveRecursive before the new plain-destination partition-compatibility validation runs, so an invalid forced request can cancel an in-progress export and then fail without scheduling a replacement.

arthurpassos · 2026-08-25T11:55:19Z

export partition regression tests are green, from audit review there is one low issue found:

Low: In StorageReplicatedMergeTree::exportPartitionToTable, export_merge_tree_partition_force_export = 1 removes an existing export znode via tryRemoveRecursive before the new plain-destination partition-compatibility validation runs, so an invalid forced request can cancel an in-progress export and then fail without scheduling a replacement.

This is not related to this PR specifically, but we should indeed fix it. Should be trivial

Selfeer · 2026-08-25T11:57:12Z

And here is a small report on the regression fails, seems safe to merge right now. These are not broken by the PR but will be addressed on the regression suite for the final version. This report is mostly for the purpose of revisiting it in the future if needed.

1. Regression test failures — all expected fallout of the behaviour change

1.1 `s3_export_part` — 1 scenario — stale error-message assertion

Failing test: /s3/minio/export tests/export part/error handling/different partition key

Setup: source MergeTree ... PARTITION BY i, destination S3(..., partition_strategy='hive') PARTITION BY p.

What CI observed:

Code: 36. DB::Exception: Cannot export partition: the destination partition expression
uses column 'p', which is not part of the source MergeTree partition key.. (BAD_ARGUMENTS)

with exitcode == 36 for every one of the 15 exported parts.

Why it failed: the scenario asserts the literal old wording.

    with Then("I should see an error related to the different partition key"):
        assert results[0].exitcode == 36, error()
        assert "Tables have different partition key" in results[0].output, error()

Verdict: not a bug — behaviour is correct, the assertion is stale.

The destination partitions by p, which is not in the source partition key at all, so it is rejected
before and after this PR, with the same BAD_ARGUMENTS / code 36, synchronously at schedule time.
Only the message text changed, from the removed AST string comparison to the new per-column message.
Nothing about the product regressed here.

Note — the export_partition twin was already fixed, export_part was missed. The mirror
scenario in s3/tests/export_partition/error_handling.py already accepts either wording:

    with Then("I should see an error related to the different partition key"):
        assert results[0].exitcode == 36, error()
        assert (
            "Tables have different partition key" in results[0].output
            or "not part of the source MergeTree partition key" in results[0].output
        ), error(results[0].output)

That is exactly why s3_export_partition was green in this same run while s3_export_part failed.
The fix is to apply the same both-wordings assertion (or the new wording alone, once the base branch
has the PR) to s3/tests/export_part/error_handling.py:171. The prose requirement in
s3/requirements/export_part.md:299 and its generated export_part.py:435, which both quote
"Tables have different partition key" verbatim, need the same treatment.

1.2 `iceberg_2` — 6 scenarios in `/iceberg/export partition/no catalog/replicated merge tree/partition compatibility/rejected`

All six failed the same way: the test expected exitcode 36 and got exitcode 0 — the export was
accepted. They split into two distinct root causes.

Group A — cases the PR intentionally accepts now (3 scenarios)

The test expectation itself is obsolete; these need to move from "rejected" to "accepted".

Scenario	Source `PARTITION BY`	Destination `PARTITION BY`	Why it is accepted
`rejected: compound field order reversed`	`(year, region)`	`(region, year)`	Matching is per column; the destination defines tuple order. Both columns structurally match.
`rejected: field-count mismatch (2 vs 1)`	`(year, region)`	`year`	The destination key is a subset of the source key, so each source partition holds exactly one `year`.
`rejected: partitioned source, unpartitioned destination`	`year`	(none)	An unpartitioned destination is a single partition, so nothing can split. `spec_size == 0` returns early.

Upstream made the same call in its own suite: this PR deletes test_rejected_field_count_mismatch
and test_rejected_compound_order_reversed from
tests/integration/test_storage_iceberg_with_spark/test_export_partition_iceberg.py, and adds
test_export_partition_multicolumn_subset_accepted / test_export_partition_subset_reversed_order_accepted
plus the new stateless 03572_export_part_hive_partition_subset.sh. Our tests should follow, and the
positive cases should also assert the Iceberg manifest partition tuple, not just a zero exit code.

Group B — cases where a single seed row cannot force a split (3 scenarios)

These schema pairs are genuinely unsafe in general, and the gate does reject them when the data
actually spans two destination partitions. But _run_rejected_case inserts one row:

    with And("insert a seed row so the partition exists"):
        insert_data(table_name=source_table, values=values)

With one row, min == max for every partition column, so the dynamic proof evaluates the destination
expression at two identical endpoints, gets one value, and correctly concludes the partition maps to a
single destination partition. The accept is the gate working as designed on trivially single-valued
data — not a product bug.

Scenario	Source	Destination	Seed value	Why one row is not enough
`rejected: transform vs identity on same column`	`toYearNumSinceEpoch(event_date)`	`event_date` (identity)	`(1, '2020-01-01')`	Needs ≥2 rows with different dates in the same year, e.g. `'2020-01-01'` and `'2020-06-15'`.
`rejected: truncate width mismatch ([4] vs [8])`	`icebergTruncate(4, category)`	`icebergTruncate(8, category)`	`(1, 'clickhouse')`	Needs ≥2 values sharing the first 4 bytes but differing in bytes 5–8, e.g. `'clickhouse'` and `'clickmouse'`.
`rejected: unsupported MergeTree expression (intDiv)`	`intDiv(year, 100)`	`year` (identity)	`(1, 2020)`	Needs ≥2 rows in the same century bucket, e.g. `year = 2020` and `year = 2021`.

Upstream reworked its equivalents the same way — test_rejected_truncate_width_mismatch was
"reworked around coarser source (truncate[4]) → finer destination (truncate[8]) with values that
split".

Positive control that the gate is not just rubber-stamping everything: two sibling scenarios in the
same feature still passed with the same single-row setup —
rejected: bucket width mismatch ([8] vs [16]) (icebergBucket is a hash, so the dynamic path always
rejects it; only a structural bucket match is accepted) and
rejected: different partition columns (year vs id) (the destination column is absent from the source
partition key). That is exactly the documented split between structural-only and dynamically-provable
transforms, so the gate is discriminating correctly.

arthurpassos · 2026-08-25T11:58:19Z

@Selfeer issue link for the bug you mentioned above #2258

arthurpassos · 2026-08-27T16:59:21Z

Dropped the port-antalya label since we have one for antalya-26.6 as well

arthurpassos added 2 commits July 17, 2026 16:59

vibe coded monotonicity check for mt -> iceberg exports

3f47047

add some more tests

343c2d9

make tests better

3f3af7c

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp Outdated

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp Outdated

arthurpassos added 3 commits July 20, 2026 09:48

more vibe coded changes

d385268

recompute dst key with correct set of parts, include tz

454e9fc

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-partitio…

abcba15

…n-monotonic-compat

arthurpassos changed the title ~~Allow non matching partition expressions iff monotonicity is proved and data is not repartitioned~~ Export partition - allow non matching partition expressions iff monotonicity is proved and data is not repartitioned Jul 21, 2026

arthurpassos added 6 commits July 22, 2026 09:31

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-partitio…

4e10ade

…n-monotonic-compat

some more vibe coded style changes

e4a69a0

add missing tests

c60c97d

code simplification

a9f51e1

some docs

12c5e5b

handle iceberg_timezone

ca43cf1

chatgpt-codex-connector Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp

arthurpassos added 10 commits July 24, 2026 10:23

missing arg restore

56fde58

docs

ee1993a

opsy

41fbe8f

possible simplification

450eddc

some more simplifications

6df47c7

add some comments

1501cc7

simplifications and docs

a89d2f0

fix more vibe coded issues

776be0b

further simplifications

7495ab6

one more simplification

e75220d

arthurpassos added 7 commits August 19, 2026 10:18

re-use existing code and make the structural match a bit more flexible

1822382

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-partitio…

378d9ca

…n-monotonic-compat

one fix for sure

9a6a4db

fix lc(nullable)

834a50b

fix tests

0743a3d

fix a few tests

7caa8d4

rmv corner case

35605c1

arthurpassos changed the title ~~Export partition - allow non matching partition expressions iff monotonicity is proved and data is not repartitioned~~ Export partition - allow non matching partition expressions in case we can prove the destination expression does not split the data Aug 20, 2026

k-morozov reviewed Aug 21, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp

rmv duplicated call

9a1cb91

k-morozov previously approved these changes Aug 21, 2026

View reviewed changes

arthurpassos dismissed k-morozov’s stale review via 9a1cb91 August 21, 2026 19:36

Selfeer added the verified Approved for release label Aug 25, 2026

arthurpassos merged commit d5e8e23 into antalya-26.3 Aug 25, 2026
258 of 274 checks passed

arthurpassos mentioned this pull request Aug 25, 2026

Export partition (Antalya 26.6) - allow non matching partition expressions in case we can prove the destination expression does not split the data #2253

Merged

29 tasks

arthurpassos removed the port-antalya PRs to be ported to all new Antalya releases label Aug 27, 2026

		)


		def test_export_part_unnamed_tuple_partition_key_owner_matching_named_destination_is_allowed(cluster):

Uh oh!

Conversation

arthurpassos commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthurpassos commented Jul 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

ilejn commented Jul 21, 2026

Uh oh!

arthurpassos commented Jul 21, 2026

Uh oh!

arthurpassos commented Jul 24, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Selfeer commented Aug 6, 2026

Uh oh!

arthurpassos commented Aug 20, 2026

Uh oh!

arthurpassos commented Aug 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Selfeer commented Aug 25, 2026

Uh oh!

arthurpassos commented Aug 25, 2026

Uh oh!

Selfeer commented Aug 25, 2026

1. Regression test failures — all expected fallout of the behaviour change

1.1 s3_export_part — 1 scenario — stale error-message assertion

1.2 iceberg_2 — 6 scenarios in /iceberg/export partition/no catalog/replicated merge tree/partition compatibility/rejected

Group A — cases the PR intentionally accepts now (3 scenarios)

Group B — cases where a single seed row cannot force a split (3 scenarios)

arthurpassos commented Jul 17, 2026 •

edited

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading

k-morozov Aug 21, 2026 •

edited

Loading

k-morozov Aug 21, 2026 •

edited

Loading

k-morozov Aug 21, 2026 •

edited

Loading

1.1 `s3_export_part` — 1 scenario — stale error-message assertion

1.2 `iceberg_2` — 6 scenarios in `/iceberg/export partition/no catalog/replicated merge tree/partition compatibility/rejected`