Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](mtmv) Fix generate hyper graph wrongly when has filter which can not push down #43539

Merged

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Nov 9, 2024

What problem does this PR solve?

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this leftExtendedNodes Shoule be contained both orders and lineitem, which should be 3, but now is 2
image

Issue Number: close #xxx

Related PR: #28006

Problem Summary:

This may cause associate wrongly when use hyper graph and materialized view which depends on it rewrite wrongly

Release note

Fix generate hyper graph wrongly when has filter which can not push down

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Nov 9, 2024

run buildall

@wm1581066 wm1581066 added usercase Important user case type label dev/2.1.x dev/3.0.x labels Nov 9, 2024
@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 52043 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0cecf414730a5235248f752ef16ed3c0d8ef709, data reload: false

------ Round 1 ----------------------------------
q1	17604	7529	7303	7303
q2	2583	1310	1305	1305
q3	9954	1155	1134	1134
q4	10228	870	861	861
q5	7604	3018	2971	2971
q6	228	144	145	144
q7	1031	606	604	604
q8	9394	2365	2378	2365
q9	12727	12515	12593	12515
q10	7151	2430	2425	2425
q11	457	259	257	257
q12	404	212	211	211
q13	17798	3036	3028	3028
q14	252	208	214	208
q15	590	531	508	508
q16	691	583	576	576
q17	997	520	577	520
q18	7242	6710	6792	6710
q19	1324	983	1170	983
q20	3148	2912	2826	2826
q21	4018	3210	3316	3210
q22	1409	1379	1380	1379
Total cold run time: 116834 ms
Total hot run time: 52043 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7516	7490	7436	7436
q2	339	244	250	244
q3	3112	2947	2976	2947
q4	2113	1850	1838	1838
q5	5691	5663	5716	5663
q6	223	138	137	137
q7	2224	1822	1831	1822
q8	3388	3551	3578	3551
q9	14774	14731	14607	14607
q10	3652	3614	3548	3548
q11	596	491	505	491
q12	857	607	605	605
q13	13696	3252	3282	3252
q14	315	259	278	259
q15	585	533	536	533
q16	665	624	639	624
q17	1833	1655	1623	1623
q18	8243	7806	7655	7655
q19	1690	1568	1485	1485
q20	2183	1932	1965	1932
q21	5293	5306	5174	5174
q22	637	558	550	550
Total cold run time: 79625 ms
Total hot run time: 65976 ms

@gavinchou gavinchou added the cir label Nov 11, 2024
@morrySnow
Copy link
Contributor

run performance

morrySnow
morrySnow previously approved these changes Nov 13, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@@ -801,6 +803,8 @@ protected SplitPredicate predicatesCompensate(
private boolean containsNullRejectSlot(Set<Set<Slot>> requireNoNullableViewSlot,
Set<Expression> queryPredicates,
SlotMapping queryToViewMapping,
StructInfo queryStructInfo,
StructInfo viwStructInfo,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

viwStructInfo -> viewStructInfo

@seawinde
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 13, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 182af8e into apache:master Nov 14, 2024
26 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 14, 2024
…n not push down (#43539)

### What problem does this PR solve?

Related PR: #28006

Problem Summary:

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this` leftExtendedNodes` Shoule be contained both orders and lineitem,
which should be 3, but now is 2

This may cause associate wrongly when use hyper graph and materialized
view which depends on it rewrite wrongly

### Release note

Fix generate hyper graph wrongly when has filter which can not push down
github-actions bot pushed a commit that referenced this pull request Nov 14, 2024
…n not push down (#43539)

### What problem does this PR solve?

Related PR: #28006

Problem Summary:

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this` leftExtendedNodes` Shoule be contained both orders and lineitem,
which should be 3, but now is 2

This may cause associate wrongly when use hyper graph and materialized
view which depends on it rewrite wrongly

### Release note

Fix generate hyper graph wrongly when has filter which can not push down
dataroaring pushed a commit that referenced this pull request Nov 14, 2024
…ter which can not push down #43539 (#43944)

Cherry-picked from #43539

Co-authored-by: seawinde <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. cir dev/2.1.x dev/3.0.3-merged reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants