[FEATURE] Support Patterns Command

[penghuo]

Is your feature request related to a problem?
Currently, the PPL Patterns command attaches detected patterns to each event in the input stream. In contrast, both CloudWatch Logs Insights and Splunk's cluster command provide summarized pattern detection, offering aggregated insights into log data.

What solution would you like?

1. Introduction

Enhances the patterns command in OpenSearch PPL to support summarized pattern detection, alongside backward-compatible label-mode and aggregation-mode output options. The goal is to:

• Retain existing patterns functionality (label-mode).
• Add aggregation-mode for grouped pattern statistics.

2. Expected behaviour

• with input dataset

_time	_raw
03-20-2018 09:37:33	ERROR: Unable to create directory /app/db1
03-20-2018 09:37:35	ERROR: Unable to create directory /app/db1
03-20-2018 09:38:01	DEBUG: Operation completed successfully
03-20-2018 09:38:05	DEBUG: Operation completed successfully

• label-mode

... | patterns message annotate input events with pattern and tokens

_time	message	pattern	tokens
03-20-2018 09:37:33	ERROR: Unable to create directory /app/db1	ERROR: Unable to create directory	{"token1": "/app/db1"}
03-20-2018 09:37:35	ERROR: Unable to create directory /app/db2	ERROR: Unable to create directory	{"token1": "/app/db2"}
03-20-2018 09:38:01	DEBUG: Operation completed successfully	DEBUG: Operation completed	{"token1": "successfully"}
03-20-2018 09:38:05	DEBUG: Operation completed error	DEBUG: Operation completed	{"token1": "error"}

• aggregation-mode

... | patterns mode=aggregation-mode message annotate input events with pattern and tokens

pattern	tokens	pattern_count
ERROR: Unable to create directory	[{"token1": ["/app/db1", "/app/db2"]}]	2
DEBUG: Operation completed	[{"token1": ["successfully", "error"}]	2

3. Proposed Changes

3.1 Command Syntax

patterns 
  [mode=(label-mode|aggregation-mode)]
  ... // existing paramaters

• mode
  • label-mode (default): Append per-event fields.
  • aggregation-mode: Group by pattern.

3.2 Output schema

• Label-mode

Column	Data Type	Description
label	int	Unique pattern identifier.
count	bigint	Frequency of the pattern.
pattern	string	Pattern template (e.g., <token1> server error, status: <token2>).
tokens	struct[]	Array of {tokenName: tokenValue} pairs (e.g., [{"token1": "ServiceA"}, ...]).

• Aggregation-mode

Column	Data Type	Description
pattern	string	Deduplicated pattern template.
count	bigint	Total occurrences of the pattern.
tokens	struct[]	Merged tokens with distinct values (e.g., [{"token1": ["ServiceA", "ServiceB"]}]).

[ashwin-pc]

Nice, i like this approach!

[ashwin-pc]

How do we want to address performance issues though? like sample size and limits for fast and slow pattern call?