Skip to content

ESQL: Makes ENRICH scope local to subqueries#153296

Draft
ncordon wants to merge 8 commits into
elastic:mainfrom
ncordon:enrich-with-subqueries
Draft

ESQL: Makes ENRICH scope local to subqueries#153296
ncordon wants to merge 8 commits into
elastic:mainfrom
ncordon:enrich-with-subqueries

Conversation

@ncordon

@ncordon ncordon commented Jul 8, 2026

Copy link
Copy Markdown
Member

Closes https://github.com/elastic/esql-planning/issues/997

Similar to what we did for LOOKUP JOIN, we scope ENRICH policy resolution to the part of the query that actually feeds that specific ENRICH (its own FROM-subquery branch or WHERE IN subquery), instead of resolving it against every cluster touched anywhere in the whole query.

For example, in this query:

  FROM remote-b:logs-*
  | WHERE v IN (
      FROM cluster-a:logs-*
      | WHERE v > 1 AND v < 7
      | ENRICH _remote:values_enrich_a ON v WITH enrich_name
      | KEEP v
    )
  | ENRICH _remote:values_enrich_b ON v WITH enrich_name
  | KEEP v

we only need values_enrich_a to exist on cluster-a and values_enrich_b to exist on remote-b — not on every cluster the query touches.

Implementation:

  • EsqlSession resolves all the relevant clusters for each ENRICH policy in the query, reusing the same tree walk we did for LOOKUP JOIN
  • EnrichPolicyResolver narrows each scope having into account the ENRICH mode.

@ncordon ncordon added >test Issues or PRs that are addressing/adding tests >test-mute Use for PR that only mute tests Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels Jul 8, 2026
@ncordon ncordon force-pushed the enrich-with-subqueries branch from 6980591 to 96fd217 Compare July 9, 2026 08:41
@ncordon ncordon changed the title ESQL: Adds tests for ENRICH interaction with subqueries ESQL: Makes ENRICH scope local to subqueries Jul 12, 2026
@ncordon ncordon force-pushed the enrich-with-subqueries branch from b63eb62 to d260e86 Compare July 12, 2026 15:54
import static org.hamcrest.Matchers.greaterThanOrEqualTo;

public abstract class AbstractEnrichBasedCrossClusterTestCase extends AbstractMultiClustersTestCase {
public abstract class AbstractEnrichBasedCrossClusterTestCase extends AbstractMultiClustersTestCase implements EnrichClusterPluginSupport {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file has been refactored moving relevant code for having ENRICH in a cluster setup into EnrichClusterPluginSupport so the CrossClusterInSubqueryIT and CrossClsuterSubqueryIT files don't have to repeat the same setup

@ncordon ncordon removed the >test-mute Use for PR that only mute tests label Jul 12, 2026
@ncordon ncordon requested a review from Copilot July 12, 2026 16:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates ES|QL cross-cluster ENRICH policy resolution so each ENRICH occurrence is validated only against the cluster(s) that actually feed rows into that ENRICH (including subquery branches and WHERE ... IN (subquery) shapes), similar to existing LOOKUP JOIN scoping.

Changes:

  • Add per-ENRICH “feeding cluster scope” computation in EsqlSession and pass that scope into EnrichPolicyResolver.
  • Key EnrichResolution by the originating Enrich node’s Source (instead of (policyName, mode)), enabling independent resolution of same-named policies across differently-scoped subqueries/views.
  • Extend unit and internal cluster tests to cover ENRICH scoping across FROM-subqueries and WHERE-IN subqueries, and add shared ENRICH test-cluster setup helpers.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/telemetry/PlanExecutorMetricsTests.java Updates resolver mocking to match new resolvePolicies signature.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/EsqlSessionTests.java Adds thorough unit tests for computeEnrichScope(s) across unions, ROW sources, FROM-subqueries, and WHERE-IN subqueries.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/AbstractLocalPhysicalPlanOptimizerTests.java Adapts test enrich resolution setup to Source-keyed EnrichResolution.
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/enrich/EnrichPolicyResolverTests.java Updates tests to supply per-occurrence Source+scope and validate Source-keyed resolution/errors.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Introduces ENRICH feeding-scope computation and wires scopes into enrich resolution.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/EnrichPolicyResolver.java Accepts per-ENRICH scopes, narrows target clusters per policy/mode, and stores results keyed by Source.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/EnrichResolution.java Switches resolution/error maps to be keyed by Source.
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java Consumes EnrichResolution by plan.source() instead of policy-name/mode.
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EnrichClusterPluginSupport.java Adds reusable ENRICH plugin + policy setup helpers for CCS ITs.
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossClusterSubqueryIT.java Enables ENRICH in CCS subquery ITs and adds coverage for ENRICH modes across FROM-subquery unions.
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossClusterInSubqueryIT.java Enables ENRICH in CCS WHERE-IN ITs and adds extensive mode/scope coverage, including view/source collisions.
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/AbstractEnrichBasedCrossClusterTestCase.java Refactors plugin/settings wiring to share ENRICH cluster setup logic.
x-pack/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/TestAnalyzer.java Adapts test fixtures to Source-keyed enrich resolution via deferred (pending) registrations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ncordon ncordon added >bug and removed >test Issues or PRs that are addressing/adding tests labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants