Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PG17 compatibility: Fix Test Failure in dml_recursive #7727

Open
wants to merge 5 commits into
base: naisila/pg17_support
Choose a base branch
from

Conversation

m3hm3t
Copy link
Contributor

@m3hm3t m3hm3t commented Nov 11, 2024

PostgreSQL 17 includes an enhancement that allows the optimizer to transform correlated IN subqueries into more efficient join operations.
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9f1337639

UPDATE
	second_distributed_table
SET
	dept = foo.tenant_id::int / 4
FROM
(
	SELECT baz.tenant_id FROM
	(
		SELECT
			second_distributed_table.dept, second_distributed_table.tenant_id
		FROM
			second_distributed_table, distributed_table as d1
		WHERE
			d1.tenant_id = second_distributed_table.tenant_id
		AND
			second_distributed_table.dept IN (3,4)
			AND
			second_distributed_table.tenant_id IN
			(
					SELECT s2.tenant_id
					FROM second_distributed_table as s2
					GROUP BY d1.tenant_id, s2.tenant_id
			)
	) as baz
	) as foo WHERE second_distributed_table.tenant_id = foo.tenant_id
RETURNING *;
-ERROR:  complex joins are only supported when all distributed tables are co-located and joined on their distribution columns
+ tenant_id | dept |          info          | tenant_id 
+-----------+------+------------------------+-----------
+ 14        |    3 | {"f1": 14, "f2": 196}  | 14
+ 23        |    5 | {"f1": 23, "f2": 529}  | 23
+ 24        |    6 | {"f1": 24, "f2": 576}  | 24
+ 3         |    0 | {"f1": 3, "f2": 9}     | 3
+ 33        |    8 | {"f1": 33, "f2": 1089} | 33
+ 34        |    8 | {"f1": 34, "f2": 1156} | 34
+ 4         |    1 | {"f1": 4, "f2": 16}    | 4
+ 43        |   10 | {"f1": 43, "f2": 1849} | 43
+ 44        |   11 | {"f1": 44, "f2": 1936} | 44
+ 53        |   13 | {"f1": 53, "f2": 2809} | 53
+ 54        |   13 | {"f1": 54, "f2": 2916} | 54
+ 63        |   15 | {"f1": 63, "f2": 3969} | 63
+ 64        |   16 | {"f1": 64, "f2": 4096} | 64
+ 73        |   18 | {"f1": 73, "f2": 5329} | 73
+ 74        |   18 | {"f1": 74, "f2": 5476} | 74
+ 83        |   20 | {"f1": 83, "f2": 6889} | 83
+ 84        |   21 | {"f1": 84, "f2": 7056} | 84
+ 93        |   23 | {"f1": 93, "f2": 8649} | 93
+ 94        |   23 | {"f1": 94, "f2": 8836} | 94
+(19 rows)
+

Copy link

codecov bot commented Nov 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.61%. Comparing base (b29c332) to head (ccf2920).

Additional details and impacted files
@@                  Coverage Diff                  @@
##           naisila/pg17_support    #7727   +/-   ##
=====================================================
  Coverage                 89.61%   89.61%           
=====================================================
  Files                       274      274           
  Lines                     59689    59689           
  Branches                   7446     7446           
=====================================================
+ Hits                      53490    53492    +2     
+ Misses                     4069     4067    -2     
  Partials                   2130     2130           

@m3hm3t m3hm3t self-assigned this Nov 11, 2024
@m3hm3t m3hm3t marked this pull request as ready for review November 11, 2024 13:09
Comment on lines +301 to +323
tenant_id | dept | info | tenant_id
---------------------------------------------------------------------
14 | 3 | {"f1": 14, "f2": 196} | 14
23 | 5 | {"f1": 23, "f2": 529} | 23
24 | 6 | {"f1": 24, "f2": 576} | 24
3 | 0 | {"f1": 3, "f2": 9} | 3
33 | 8 | {"f1": 33, "f2": 1089} | 33
34 | 8 | {"f1": 34, "f2": 1156} | 34
4 | 1 | {"f1": 4, "f2": 16} | 4
43 | 10 | {"f1": 43, "f2": 1849} | 43
44 | 11 | {"f1": 44, "f2": 1936} | 44
53 | 13 | {"f1": 53, "f2": 2809} | 53
54 | 13 | {"f1": 54, "f2": 2916} | 54
63 | 15 | {"f1": 63, "f2": 3969} | 63
64 | 16 | {"f1": 64, "f2": 4096} | 64
73 | 18 | {"f1": 73, "f2": 5329} | 73
74 | 18 | {"f1": 74, "f2": 5476} | 74
83 | 20 | {"f1": 83, "f2": 6889} | 83
84 | 21 | {"f1": 84, "f2": 7056} | 84
93 | 23 | {"f1": 93, "f2": 8649} | 93
94 | 23 | {"f1": 94, "f2": 8836} | 94
(19 rows)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to take a closer look into the new query plan to see how that optimization allowed Citus to run the query now, you can ping me tomorrow for us to quickly debug this together.

@naisila naisila changed the title Fix Test Failure in dml_recursive in PG17 PG17 compatibility: Fix Test Failure in dml_recursive Nov 12, 2024
naisila
naisila previously approved these changes Nov 12, 2024
Copy link
Member

@naisila naisila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix, thanks @m3hm3t and @onurctirtir
Always appreciate a fix that avoids a new output file :)
Let's merge directly to release-13.0 and then rebase naisila/pg17_support branch

EDIT: sorry, wrong PR

@naisila naisila dismissed their stale review November 12, 2024 11:01

accidentally approved the wrong PR

-- dml_recursive_0.out for PG16 and before
-- dml_recursive.out for PG17
-- related commit
-- PostgreSQL 17 includes an enhancement that allows the optimizer to transform correlated IN subqueries into more efficient join operations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's hold off a bit more before merging this, until we are sure that nothing needs to change in the Citus codebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants