ES|QL support (#194)

* ESQL and DSL executors are introduced.  param can accept ES|QL query shape now.  is introduced for initial step but needs team's feedback. DSL logics moved into DSL executors.

* Apply suggestions from code review

Separate DSL and ESQL interface in the client.

Co-authored-by: Rye Biesemeyer <[email protected]>

* Rebase against upstream main after target support added. Separate unit test for DSL. Address comments: do not save ES version in client, add apply target method in executors, set to target if target is defined, docs update.

Co-authored-by: Rye Biesemeyer <[email protected]>

* Introduce query_type option which accepts dsl  or esql to define a query shape. Remove multi-depth nested named_params and keep only top-level query_params which aligns with placeholder structure in the ES|QL.

* Separate event referenced and static valued fields at initialization of the ESQL executor.

* query_params now supports both Array and Hash types.

* Add tech preview section under ESQL.

* Place the query results based on the target specified. If not specified, first result will be set to event's top level.

* Apply suggestions from code review

Doc corrections.

Co-authored-by: João Duarte <[email protected]>

* ES|QL result mapping to event doc correction.

* Integration tests to run with credentials enabled and SSL configs.

---------

Co-authored-by: Rye Biesemeyer <[email protected]>
Co-authored-by: João Duarte <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 4.3.0
+  - ES|QL support [#194](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/194)
+
 ## 4.2.0
   - Add `target` configuration option to store the result into it [#196](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/196)
 

docs/index.asciidoc

@@ -54,7 +54,7 @@ if [type] == "end" {
 
 The example below reproduces the above example but utilises the query_template.
 This query_template represents a full Elasticsearch query DSL and supports the
-standard Logstash field substitution syntax.  The example below issues
+standard {ls} field substitution syntax. The example below issues
 the same query as the first example but uses the template shown.
 
 [source,ruby]
@@ -118,6 +118,110 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
 Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
 The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.
 
+[id="plugins-{type}s-{plugin}-esql"]
+==== {esql} support
+
+.Technical Preview
+****
+The {esql} feature that allows using ES|QL queries with this plugin is in Technical Preview.
+Configuration options and implementation details are subject to change in minor releases without being preceded by deprecation warnings.
+****
+
+{es} Query Language ({esql}) provides a SQL-like interface for querying your {es} data.
+
+To use {esql}, this plugin needs to be installed in {ls} 8.17.4 or newer, and must be connected to {es} 8.11 or newer.
+
+To configure {esql} query in the plugin, set your {esql} query in the `query` parameter.
+
+IMPORTANT: We recommend understanding {ref}/esql-limitations.html[{esql} current limitations] before using it in production environments.
+
+The following is a basic {esql} query that sets the food name to transaction event based on upstream event's food ID:
+[source, ruby]
+    filter {
+      elasticsearch {
+        hosts => [ 'https://..']
+        api_key => '....'
+        query => '
+          FROM food-index
+            | WHERE id == ?food_id
+        '
+        query_params => {
+          "food_id" => "[food][id]"
+        }
+      }
+    }
+
+Set `config.support_escapes: true` in `logstash.yml` if you need to escape special chars in the query.
+
+In the result event, the plugin sets total result size in `[@metadata][total_values]` field.
+
+[id="plugins-{type}s-{plugin}-esql-event-mapping"]
+===== Mapping {esql} result to {ls} event
+{esql} returns query results in a structured tabular format, where data is organized into _columns_ (fields) and _values_ (entries).
+The plugin maps each value entry to an event, populating corresponding fields.
+For example, a query might produce a table like:
+
+[cols="2,1,1,1,2",options="header"]
+|===
+|`timestamp` |`user_id` | `action` | `status.code` | `status.desc`
+
+|2025-04-10T12:00:00 |123 |login |200 | Success
+|2025-04-10T12:05:00 |456 |purchase |403 | Forbidden (unauthorized user)
+|===
+
+For this case, the plugin creates two JSON look like objects as below and places them into the `target` field of the event if `target` is defined.
+If `target` is not defined, the plugin places the _only_ first result at the root of the event.
+[source, json]
+[
+  {
+    "timestamp": "2025-04-10T12:00:00",
+    "user_id": 123,
+    "action": "login",
+    "status": {
+        "code": 200,
+        "desc": "Success"
+    }
+  },
+  {
+    "timestamp": "2025-04-10T12:05:00",
+    "user_id": 456,
+    "action": "purchase",
+    "status": {
+        "code": 403,
+        "desc": "Forbidden (unauthorized user)"
+    }
+  }
+]
+
+NOTE: If your index has a mapping with sub-objects where `status.code` and `status.desc` actually dotted fields, they appear in {ls} events as a nested structure.
+
+[id="plugins-{type}s-{plugin}-esql-multifields"]
+===== Conflict on multi-fields
+
+{esql} query fetches all parent and sub-fields fields if your {es} index has https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/multi-fields[multi-fields] or https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/subobjects[subobjects].
+Since {ls} events cannot contain parent field's concrete value and sub-field values together, the plugin ignores sub-fields with warning and includes parent.
+We recommend using the `RENAME` (or `DROP` to avoid warning) keyword in your {esql} query explicitly rename the fields to include sub-fields into the event.
+
+This is a common occurrence if your template or mapping follows the pattern of always indexing strings as "text" (`field`) + " keyword" (`field.keyword`) multi-field.
+In this case it's recommended to do `KEEP field` if the string is identical and there is only one subfield as the engine will optimize and retrieve the keyword, otherwise you can do `KEEP field.keyword | RENAME field.keyword as field`.
+
+To illustrate the situation with example, assuming your mapping has a time `time` field with `time.min` and `time.max` sub-fields as following:
+[source, ruby]
+    "properties": {
+        "time": { "type": "long" },
+        "time.min": { "type": "long" },
+        "time.max": { "type": "long" }
+    }
+
+The {esql} result will contain all three fields but the plugin cannot map them into {ls} event.
+To avoid this, you can use the `RENAME` keyword to rename the `time` parent field to get all three fields with unique fields.
+[source, ruby]
+    ...
+    query => 'FROM my-index | RENAME time AS time.current'
+    ...
+
+For comprehensive ES|QL syntax reference and best practices, see the https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html[{esql} documentation].
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Filter Configuration Options
 
@@ -143,6 +247,8 @@ NOTE: As of version `4.0.0` of this plugin, a number of previously deprecated se
 | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-query>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-query_type>> |<<string,string>>, one of `["dsl", "esql"]`|No
+| <<plugins-{type}s-{plugin}-query_params>> |<<hash,hash>> or <<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-query_template>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-result_size>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retry_on_failure>> |<<number,number>>|No
@@ -339,11 +445,30 @@ environment variables e.g. `proxy => '${LS_PROXY:}'`.
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-Elasticsearch query string. More information is available in the
-{ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query
-string documentation].
-Use either `query` or `query_template`.
+The query to be executed.
+The accepted query shape is DSL query string or ES|QL.
+For the DSL query string, use either `query` or `query_template`.
+Read the {ref}/query-dsl-query-string-query.html[{es} query
+string documentation] or {ref}/esql.html[{es} ES|QL documentation] for more information.
+
+[id="plugins-{type}s-{plugin}-query_type"]
+===== `query_type`
+
+* Value can be `dsl` or `esql`
+* Default value is `dsl`
+
+Defines the <<plugins-{type}s-{plugin}-query>> shape.
+When `dsl`, the query shape must be valid {es} JSON-style string.
+When `esql`, the query shape must be a valid {esql} string and `index`, `query_template` and `sort` parameters are not allowed.
+
+[id="plugins-{type}s-{plugin}-query_params"]
+===== `query_params`
+
+* The value type is <<hash,hash>> or <<array,array>>. When an array provided, the array elements are pairs of `key` and `value`.
+* There is no default value for this setting
 
+Named parameters in {esql} to send to {es} together with <<plugins-{type}s-{plugin}-query>>.
+Visit {ref}/esql-rest.html#esql-rest-params[passing parameters to query page] for more information.
 
 [id="plugins-{type}s-{plugin}-query_template"]
 ===== `query_template` 
@@ -540,8 +665,9 @@ Tags the event on failure to look up previous log event information. This can be
 
 Define the target field for placing the result data.
 If this setting is omitted, the target will be the root (top level) of the event.
+It is highly recommended to set when using `query_type=>'esql'` to set all query results into the event.
 
-The destination fields specified in <<plugins-{type}s-{plugin}-fields>>, <<plugins-{type}s-{plugin}-aggregation_fields>>, and <<plugins-{type}s-{plugin}-docinfo_fields>> are relative to this target.
+When `query_type=>'dsl'`, the destination fields specified in <<plugins-{type}s-{plugin}-fields>>, <<plugins-{type}s-{plugin}-aggregation_fields>>, and <<plugins-{type}s-{plugin}-docinfo_fields>> are relative to this target.
 
 For example, if you want the data to be put in the `operation` field:
 [source,ruby]