HIVE-28549: Limit the maximum number of operators merged by SharedWorkOptimizer

[okumin]

What changes were proposed in this pull request?

This PR would limit the maximum number of table scan operators which SWO tries to merge.

https://issues.apache.org/jira/browse/HIVE-28549

Why are the changes needed?

We observed SWO makes a negative impact when it merges too many, e.g. 50, operators. If operators are memory intensive, they might throw OOM or might slow down.

I believe we can resolve OOM with the following patch, but we still want an upper limit so that we can tune concurrency or RAM per operator reasonably.
#5478

Does this PR introduce any user-facing change?

No.

Is the change a dependency upgrade?

No.

How was this patch tested?

I added a qtest

[sonarqubecloud]

Quality Gate passed

Issues
12 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[ngsg]

This puts all TS ops into a single List, regardless of their source. Maybe you intend the following code?

    if (batchSize == -1) {
      return sortedTables.stream()
          .map(entry -> tableNameToOps.get(entry.getKey()))
          .collect(Collectors.toList());
    }

[okumin]

You're right. I was just confused when I wrote the line. I modified it and ran some qtests which failed
e574161

[ngsg]

LGTM, +1
(The failed test seems just a flaky one. In my local environment, the test finished successfully.)

[okumin]

Retriggered CI just in case

[okumin]

I am checking why partition_explain_ddl always fails. In my impression, this PR is unrelated.
P.S. I found it failed on the latest master branch

[deniskuzZ]

hi @okumin, should we rebase?

[okumin]

@deniskuzZ
Hi, I've rebased the branch on the master branch

[deniskuzZ]

LGTM, minor comment

[deniskuzZ]

hi @okumin, not sure if you noticed the above comment, do you think it's legit?

[okumin]

@deniskuzZ Yes, I do. I'm using my machine power to review and test another PR. I will likely update this one tomorrow.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[okumin]

Thanks