PG17 compatibility: ruleutils #7725

naisila · 2024-11-10T21:43:39Z

PG17 compatibility - Part 2

#7699 was the first PG17 compatibility PR merged to main branch, which provided ONLY successful Citus compilation with PG17.0.

This PR, consider it as Part 2, provides ruleutils changes for PG17. Ruleutils changes is the first thing we should merge, after successful build. It's the core for deparsing logic in Citus.

Question: How do we add ruleutils changes?

Answer:

We add a new ruleutils file specific to PG17.
We keep track of the changes in Postgres's ruleutils file from here https://github.com/postgres/postgres/commits/REL_17_0/src/backend/utils/adt/ruleutils.c
Per each commit in that history that belongs only to 17.0, we add the relevant changes to static functions to our ruleutils file for PG17. It's like a manual commit copying.

Notes for reviewer

No need to review the logic here, changes are copy pasted from Postgres commits anyway. Just a general sanity review is needed to merge this.
You can use the wider pg17 support branch to check that the tests are passing Adds PG17.1 support - Regression tests sanity #7661, as this pull request doesn't have configure changes for PG17 or tests for PG17. This is intentional, as we don't want to merge such changes yet.
If this PR is nonsense to you 😆 check the PR for PG16 PG16 compatibility: ruleutils and successful CREATE EXTENSION #7087

(cherry picked from commit 728fe5c)

Relevant PG commit: 03734a7fed7d924679770adb78a7db8a37d14188 postgres/postgres@03734a7 (cherry picked from commit 978fdc2)

…CORD Vars. Relevant PG commit: e0e492e5a928e9c9eda01eeebadcfc36f9f8e7b7 postgres/postgres@e0e492e (cherry picked from commit 37ac804)

Relevant PG commit: 2075ba9dc9af8833b1147996c15246e44a1e4fab postgres/postgres@2075ba9 (cherry picked from commit 31def24)

Relevant PG commit: 97957fdbaa429c7c582d4753b108cb1e23e1b28a postgres/postgres@97957fd (cherry picked from commit 39ec86a)

Relevant PG commit: 29275b1d177096597675b5c6e7e7c9db2df8f4df postgres/postgres@29275b1 (cherry picked from commit fe3414f)

Relevant PG commit: dbbca2cf299b81299112ca6ada671a36235ec008 postgres/postgres@dbbca2c (cherry picked from commit 2e4bc1f)

Relevant PG commit: 33e729c5148c3a697abc552621b34bdc5fd497ed postgres/postgres@33e729c (cherry picked from commit 61b92ee)

Relevant PG commit: c649fa24a42ba89bf5460c7110e4fc8eeca65959 postgres/postgres@c649fa2 (cherry picked from commit c6c1ff2)

…rameters. Relevant PG commit: fd0398fcb099980fbedbb7750356ef234408c1c9 postgres/postgres@fd0398f (cherry picked from commit 6889367)

Relevant PG commit: 6185c9737cf48c9540782d88f12bd2912d6ca1cc postgres/postgres@6185c97 (cherry picked from commit d2648f4)

Relevant PG commit: 0294df2f1f842dfb0eed79007b21016f486a3c6c postgres/postgres@0294df2 (cherry picked from commit 6b4889f)

Relevant PG commit: de3600452b61d1bc3967e9e37e86db8956c8f577 postgres/postgres@de36004 (cherry picked from commit 662fc76)

Relevant PG commit: 87c21bb9412c8ba2727dec5ebcd74d44c2232d11 postgres/postgres@87c21bb (cherry picked from commit eefe0ae)

Relevant PG commit: f6a2529920cff76cb6e37ea840122574404dde8b postgres/postgres@f6a2529 (cherry picked from commit 661fcd6)

Relevant PG commit: bb766cde63b4f624d029b34c9cdd3d0a94fd5b46 postgres/postgres@bb766cd (cherry picked from commit 9c06754)

Relevant PG commit: 8461424fd717877ead0706984ef9b6440b2a97ad postgres/postgres@8461424 (cherry picked from commit c9f9d37)

Relevant PG commit: a42fc1c903b54ba94374e5e0c08905b9a1479f19 postgres/postgres@a42fc1c (cherry picked from commit 8e6dc39)

…ING. Relevant PG commit: 5c5bccef211cfc98e0d6c4bc1af40a33c8ac2488 postgres/postgres@5c5bcce (cherry picked from commit bd10f1a)

…LAIN. Relevant PG commit: postgres/postgres@81a12a4 81a12a4477533d7722bd6b6dac88b0e798f8e85b

…_var_field. Relevant PG commit: postgres/postgres@aed8813 aed881386aa6a6a542e46d14d3505e4e6f9310a0

…(S) commands Relevant PG commit: postgres/postgres@84f594d 84f594da358861cceeaeb7a97bb58f3765eeb284

… conflict. Relevant PG commit: postgres/postgres@a7eb633 a7eb633563c6ba8857cf59c7b43b593614537617

Relevant PG commit: postgres/postgres@c88ce38 c88ce386c4d7bfeb437ff31ec7c23c392c862e77

Relevant PG commit: postgres/postgres@5067c23 5067c230b8ee42a01cc77dc5745bc3a78f393af3

Reverts 8ad3592 def847f Relevant PG commit: postgres/postgres@eef5195 eef5195f300bb9cf2864d48761c0db2ad93842c1

Relevant PG commit: postgres/postgres@cd680b3 cd680b39211c5c3c88a143abcac576a22f996d7a

Relevant PG commit: postgres/postgres@446d5ad 446d5ad7ae7d3bf4fd08904ae54a6399cafb4e7d

codecov · 2024-11-10T21:46:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (release-13.0@da2624c). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff               @@
##             release-13.0    #7725   +/-   ##
===============================================
  Coverage                ?   89.64%           
===============================================
  Files                   ?      274           
  Lines                   ?    59583           
  Branches                ?     7436           
===============================================
  Hits                    ?    53414           
  Misses                  ?     4038           
  Partials                ?     2131

This is the final commit that adds PG17 compatibility with Citus's current capabilities. You can use Citus community, release-13.0 branch, with PG17.1. --------- Specifically, this commit: - Enables PG17 in the configure script. - Adds PG17 tests to CI using test images that have 17.1 - Fixes an upgrade test: see below for details In `citus_prepare_upgrade()`, don't drop any_value when upgrading from PG16+, because PG16+ has its own any_value function. Attempting to do so results in the error seen in [pg16-pg17 upgrade](https://github.com/citusdata/citus/actions/runs/11768444117/job/32778340003?pr=7661): ``` ERROR: cannot drop function any_value(anyelement) because it is required by the database system CONTEXT: SQL statement "DROP AGGREGATE IF EXISTS pg_catalog.any_value(anyelement)" ``` When 16 becomes the minimum supported Postgres version, the drop statements can be removed. --------- Several PG17 Compatibility commits have been merged before this final one. All these subtasks are done #7653 See the list below: Compilation PR: #7699 Ruleutils PR: #7725 Sister PR for tests: citusdata/the-process#159 Helpful smaller PRs: - #7714 - #7726 - #7731 - #7732 - #7733 - #7738 - #7745 - #7747 - #7748 - #7749 - #7752 - #7755 - #7757 - #7759 - #7760 - #7761 - #7762 - #7765 - #7766 - #7768 - #7769 - #7771 - #7774 - #7776 - #7780 - #7781 - #7785 - #7788 - #7793 - #7796 --------- Co-authored-by: Colm <[email protected]>

naisila added 28 commits November 11, 2024 00:35

Add ruleutils_17 as copy of ruleutils_16, define PG_VERSION_18

74d1f8b

(cherry picked from commit 728fe5c)

Ruleutils_17 Add more SQL/JSON constructor functions

657ab57

Relevant PG commit: 03734a7fed7d924679770adb78a7db8a37d14188 postgres/postgres@03734a7 (cherry picked from commit 978fdc2)

Ruleutils_17 Track nesting depth correctly when drilling down into RE…

fd20578

…CORD Vars. Relevant PG commit: e0e492e5a928e9c9eda01eeebadcfc36f9f8e7b7 postgres/postgres@e0e492e (cherry picked from commit 37ac804)

Ruleutils_17 Tidy-up some appendStringInfo*() usages

7145725

Relevant PG commit: 2075ba9dc9af8833b1147996c15246e44a1e4fab postgres/postgres@2075ba9 (cherry picked from commit 31def24)

Ruleutils_17 Add support for AT LOCAL

699fc48

Relevant PG commit: 97957fdbaa429c7c582d4753b108cb1e23e1b28a postgres/postgres@97957fd (cherry picked from commit 39ec86a)

Ruleutils_17 Update copyright for 2024

22adbdc

Relevant PG commit: 29275b1d177096597675b5c6e7e7c9db2df8f4df postgres/postgres@29275b1 (cherry picked from commit fe3414f)

Ruleutils_17 Remove unused #include's from backend .c files

c242858

Relevant PG commit: dbbca2cf299b81299112ca6ada671a36235ec008 postgres/postgres@dbbca2c (cherry picked from commit 2e4bc1f)

Ruleutils_17 Fix EXPLAIN output for subplans in MERGE.

f60a9a4

Relevant PG commit: 33e729c5148c3a697abc552621b34bdc5fd497ed postgres/postgres@33e729c (cherry picked from commit 61b92ee)

Ruleutils_17 Add RETURNING support to MERGE.

21d61c4

Relevant PG commit: c649fa24a42ba89bf5460c7110e4fc8eeca65959 postgres/postgres@c649fa2 (cherry picked from commit c6c1ff2)

Ruleutils_17 Improve EXPLAIN's display of SubPlan nodes and output pa…

e6062f0

…rameters. Relevant PG commit: fd0398fcb099980fbedbb7750356ef234408c1c9 postgres/postgres@fd0398f (cherry picked from commit 6889367)

Ruleutils_17 Add SQL/JSON query functions

f2da23b

Relevant PG commit: 6185c9737cf48c9540782d88f12bd2912d6ca1cc postgres/postgres@6185c97 (cherry picked from commit d2648f4)

Ruleutils_17 Add support for MERGE ... WHEN NOT MATCHED BY SOURCE

d9f5620

Relevant PG commit: 0294df2f1f842dfb0eed79007b21016f486a3c6c postgres/postgres@0294df2 (cherry picked from commit 6b4889f)

Ruleutils_17 Add basic JSON_TABLE() functionality

8e8f6d0

Relevant PG commit: de3600452b61d1bc3967e9e37e86db8956c8f577 postgres/postgres@de36004 (cherry picked from commit 662fc76)

Ruleutils_17 Implement ALTER TABLE ... SPLIT PARTITION ... command

77e8d98

Relevant PG commit: 87c21bb9412c8ba2727dec5ebcd74d44c2232d11 postgres/postgres@87c21bb (cherry picked from commit eefe0ae)

Ruleutils_17 Fix JsonExpr deparsing to emit QUOTES and WRAPPER correctly

d284ec5

Relevant PG commit: f6a2529920cff76cb6e37ea840122574404dde8b postgres/postgres@f6a2529 (cherry picked from commit 661fcd6)

Ruleutils_17 JSON_TABLE: Add support for NESTED paths and columns

ecd149f

Relevant PG commit: bb766cde63b4f624d029b34c9cdd3d0a94fd5b46 postgres/postgres@bb766cd (cherry picked from commit 9c06754)

Ruleutils_17 Fixup various StringInfo function usages

0cee806

Relevant PG commit: 8461424fd717877ead0706984ef9b6440b2a97ad postgres/postgres@8461424 (cherry picked from commit c9f9d37)

Ruleutils_17 Fix an assortment of typos

5f6c181

Relevant PG commit: a42fc1c903b54ba94374e5e0c08905b9a1479f19 postgres/postgres@a42fc1c (cherry picked from commit 8e6dc39)

Ruleutils_17 Fix another couple of outdated comments for MERGE RETURN…

b7d17f5

…ING. Relevant PG commit: 5c5bccef211cfc98e0d6c4bc1af40a33c8ac2488 postgres/postgres@5c5bcce (cherry picked from commit bd10f1a)

Ruleutils_17 Fix "failed to find plan for subquery/CTE" errors in EXP…

8b59620

…LAIN. Relevant PG commit: postgres/postgres@81a12a4 81a12a4477533d7722bd6b6dac88b0e798f8e85b

Ruleutils_17 Suppress Coverity warnings about Asserts in get_name_for…

f737292

…_var_field. Relevant PG commit: postgres/postgres@aed8813 aed881386aa6a6a542e46d14d3505e4e6f9310a0

Ruleutils_17 Revert support for ALTER TABLE ... MERGE/SPLIT PARTITION…

dd4c58e

…(S) commands Relevant PG commit: postgres/postgres@84f594d 84f594da358861cceeaeb7a97bb58f3765eeb284

Ruleutils_17 Fix mis-deparsing of ORDER BY lists when there is a name…

897d996

… conflict. Relevant PG commit: postgres/postgres@a7eb633 a7eb633563c6ba8857cf59c7b43b593614537617

Ruleutils_17 SQL/JSON: Fix JSON_TABLE() column deparsing

b988af3

Relevant PG commit: postgres/postgres@c88ce38 c88ce386c4d7bfeb437ff31ec7c23c392c862e77

Ruleutils_17 SQL/JSON: Fix default ON ERROR behavior for JSON_TABLE

330e886

Relevant PG commit: postgres/postgres@5067c23 5067c230b8ee42a01cc77dc5745bc3a78f393af3

Ruleutils_17 Revert recent SQL/JSON related commits

a07b59d

Reverts 8ad3592 def847f Relevant PG commit: postgres/postgres@eef5195 eef5195f300bb9cf2864d48761c0db2ad93842c1

Ruleutils_17 SQL/JSON: Fix JSON_TABLE() column deparsing

0dc10b6

Relevant PG commit: postgres/postgres@cd680b3 cd680b39211c5c3c88a143abcac576a22f996d7a

Ruleutils_17 SQL/JSON: Fix default ON ERROR behavior for JSON_TABLE

6c6d57f

Relevant PG commit: postgres/postgres@446d5ad 446d5ad7ae7d3bf4fd08904ae54a6399cafb4e7d

naisila added the pg17_support label Nov 10, 2024

naisila requested a review from onurctirtir November 10, 2024 21:45

naisila mentioned this pull request Nov 10, 2024

Adds PG17.1 support - Regression tests sanity #7661

Merged

onurctirtir approved these changes Nov 11, 2024

View reviewed changes

naisila merged commit c0a5f5c into release-13.0 Nov 11, 2024
120 checks passed

naisila deleted the pg17_support/ruleutils branch November 11, 2024 08:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PG17 compatibility: ruleutils #7725

PG17 compatibility: ruleutils #7725

naisila commented Nov 10, 2024 •

edited

Loading

codecov bot commented Nov 10, 2024 •

edited

Loading

PG17 compatibility: ruleutils #7725

PG17 compatibility: ruleutils #7725

Conversation

naisila commented Nov 10, 2024 • edited Loading

Question: How do we add ruleutils changes?

Notes for reviewer

codecov bot commented Nov 10, 2024 • edited Loading

Codecov Report

naisila commented Nov 10, 2024 •

edited

Loading

codecov bot commented Nov 10, 2024 •

edited

Loading