[Feature branch] Add Neural Stats API #1208

q-andy · 2025-03-04T22:25:04Z

Description

Implementing Neural Stats API framework design proposed in #1196. This initial PR sets up the foundation for the framework to track event and state stats throughout the neural search plugin and exposed their values via API.

Event-based stats
- Event stats are recorded in code at a node level (processor executions, documents ingested, etc)
- When an API call is made, all node-level maps are fetched via transport action and returned in the response.
State stats
- State stats are defined by helper functions that populate state stat values
- When an API call is made, the functions are invoked and the information is added to the response on demand

See RFC for more details.

Initial implementation includes 3 stats:

Text embedding processor executions
Text embedding processors in pipelines
Cluster version

Related Issues

Resolves #1196 #1104
Related: #1146

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

Example requests

GET /_plugins/_neural/stats
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"cluster_version": "3.0.0",
	"processors": {
		"ingest": {
			"text_embedding_processors_in_pipelines": 0
		}
	},
	"all_nodes": {
		"processors": {
			"ingest": {
				"text_embedding_executions": 0
			}
		}
	},
	"nodes": {
		"r-hVPa7-Ra6FBbdpfVrHjg": {
			"processors": {
				"ingest": {
					"text_embedding_executions": 0
				}
			}
		}
	}
}

GET /_plugins/_neural/stats?include_metadata=true
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"cluster_version": {
		"value": "3.0.0",
		"stat_type": "settable"
	},
	"processors": {
		"ingest": {
			"text_embedding_processors_in_pipelines": {
				"value": 0,
				"stat_type": "countable"
			}
		}
	},
	"all_nodes": {
		"processors": {
			"ingest": {
				"text_embedding_executions": {
					"value": 0,
					"stat_type": "timestamped_counter",
					"trailing_interval_value": 0,
					"minutes_since_last_event": 29018783
				}
			}
		}
	},
	"nodes": {
		"r-hVPa7-Ra6FBbdpfVrHjg": {
			"processors": {
				"ingest": {
					"text_embedding_executions": {
						"value": 0,
						"stat_type": "timestamped_counter",
						"trailing_interval_value": 0,
						"minutes_since_last_event": 29018783
					}
				}
			}
		}
	}
}

GET _plugins/_neural/stats/text_embedding_executions?include_metadata=true&flat_keys=true

{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"all_nodes.processors.ingest.text_embedding_executions": {
		"value": 0,
		"stat_type": "timestamped_counter",
		"trailing_interval_value": 0,
		"minutes_since_last_event": 29018784
	},
	"nodes": {
		"r-hVPa7-Ra6FBbdpfVrHjg": {
			"processors.ingest.text_embedding_executions": {
				"value": 0,
				"stat_type": "timestamped_counter",
				"trailing_interval_value": 0,
				"minutes_since_last_event": 29018784
			}
		}
	}
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Andy Qin <[email protected]>

heemin32 · 2025-03-07T17:50:57Z

Then, shouldn't it be like this? (all_nodes unflattened)

Yes, that makes more sense, good catch

Also wondering if this could be better. (metadata flattened)

I considered that as well, what I'm thinking is the "stat metadata" object replaces the raw value so the flat key name is the same whether or not metadata is included or not. Like the format will always be at "path.to.stat: <data>", sometimes the data is the raw value, sometimes its a the metadata, but the stat key name is same.

It also keeps things organized when you parse recursively. For example. if a cluster manager is periodically calling the stats API and parsing the response without hardcoding stat names, if we keep metadata non-flat, for every stat you can foreach on the categories with flat_keys and parse each stat and all its metadata one by one in a single pass. Whereas if it was completely flat metadata, running a foreach would need additional logic to split, sort, and group the metadata keys by the stat name again. Also you would probably never be interested in the value of path.to.stat.stat_type independently of the other stat metadata.

Just some thoughts. I don't see the usecase right now but I could still include an option to flatten the stat metadata as well, it's a pretty small change.

Yes. One use case that I can think of is that, when operator want to grep stats containing processor, this whole flat option can be handy.

_plugin/neural/stats?flat=true | grep processor

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

src/main/java/org/opensearch/neuralsearch/settings/NeuralSearchSettingsAccessor.java

src/test/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandlerIT.java

Signed-off-by: Andy Qin <[email protected]>

q-andy

Addressed review comments, What's left is changing the format of the response according to the discussion above (breaking down high level categories as unflat like all_nodes), adding examples, and adding BWCs.

q-andy · 2025-03-10T17:19:01Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+        .map(String::toLowerCase)
+        .collect(Collectors.toSet());
+
+    private NeuralSearchSettingsAccessor settingsAccessor;


Don't static fields typically go before instance fields?

src/main/java/org/opensearch/neuralsearch/processor/util/RestActionUtils.java

src/main/java/org/opensearch/neuralsearch/plugin/NeuralSearch.java

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

src/main/java/org/opensearch/neuralsearch/stats/events/EventStatsManager.java

src/main/java/org/opensearch/neuralsearch/stats/events/TimestampedEventStat.java

src/test/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandlerIT.java

src/main/java/org/opensearch/neuralsearch/settings/NeuralSearchSettingsAccessor.java

Signed-off-by: Andy Qin <[email protected]>

q-andy · 2025-03-12T00:37:01Z

Added BWC tests. Currently they don't run since there's no backwards versions, but included them to have basis for future 3.x versions.

Updated response formatting:

Default

GET {{ _.base_url }}/_plugins/_neural/stats
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"info": {
		"cluster_version": "3.0.0",
		"processors": {
			"ingest": {
				"text_embedding_processors_in_pipelines": 0
			}
		}
	},
	"all_nodes": {
		"processors": {
			"ingest": {
				"text_embedding_executions": 0
			}
		}
	},
	"nodes": {
		"Qs3osnyfTz6AokNiPB2uRQ": {
			"processors": {
				"ingest": {
					"text_embedding_executions": 0
				}
			}
		}
	}
}

Flatten

GET {{ _.base_url }}/_plugins/_neural/stats?flat_stat_paths=true
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"info": {
		"cluster_version": "3.0.0",
		"processors.ingest.text_embedding_processors_in_pipelines": 0
	},
	"all_nodes": {
		"processors.ingest.text_embedding_executions": 0
	},
	"nodes": {
		"Qs3osnyfTz6AokNiPB2uRQ": {
			"processors.ingest.text_embedding_executions": 0
		}
	}
}

Flatten w/ metadata

GET {{ _.base_url }}/_plugins/_neural/stats?flat_stat_paths=true&include_metadata=true
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"info": {
		"cluster_version": {
			"value": "3.0.0",
			"stat_type": "info_string"
		},
		"processors.ingest.text_embedding_processors_in_pipelines": {
			"value": 0,
			"stat_type": "info_counter"
		}
	},
	"all_nodes": {
		"processors.ingest.text_embedding_executions": {
			"value": 0,
			"stat_type": "timestamped_event_counter",
			"trailing_interval_value": 0,
			"minutes_since_last_event": 29028938
		}
	},
	"nodes": {
		"Qs3osnyfTz6AokNiPB2uRQ": {
			"processors.ingest.text_embedding_executions": {
				"value": 0,
				"stat_type": "timestamped_event_counter",
				"trailing_interval_value": 0,
				"minutes_since_last_event": 29028938
			}
		}
	}
}

Signed-off-by: Andy Qin <[email protected]>

martin-gaievski

Looks good to me, thanks Andy. please address my comment regarding the rebasing on latest main

martin-gaievski · 2025-03-13T23:24:35Z

qa/build.gradle

    }
+    testRuntimeOnly group: 'net.minidev', name:'json-smart', version: "${versions.json_smart}"


you need to rebase on latest main and remove this

My fork branch is up to date with main with latest commits, I think this diff is from the feature branch on this repo being out of date. I don't have write permissions to sync it, could you try syncing it and this should disappear? thanks @junqiu-lei!

codecov · 2025-03-13T23:42:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.90%. Comparing base (57124dd) to head (82504ce).
Report is 1 commits behind head on feature/neural-stats-api.

Additional details and impacted files

@@                      Coverage Diff                       @@
##             feature/neural-stats-api    #1208      +/-   ##
==============================================================
- Coverage                       81.83%   80.90%   -0.93%     
+ Complexity                       2607     1423    -1184     
==============================================================
  Files                             190      115      -75     
  Lines                            8922     5001    -3921     
  Branches                         1520      803     -717     
==============================================================
- Hits                             7301     4046    -3255     
+ Misses                           1028      643     -385     
+ Partials                          593      312     -281

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

vibrantvarun

Looks good, Nice job Andy

vibrantvarun · 2025-03-14T17:40:54Z

@heemin32 if your all comments are resolved then can we merge this?

heemin32 · 2025-03-14T17:43:28Z

@heemin32 if your all comments are resolved then can we merge this?

I think my comments are all resolved. G2G.

q-andy added 19 commits February 26, 2025 09:32

Add neural stats

19480ae

Signed-off-by: Andy Qin <[email protected]>

Refactor event/state stats

a1f24f0

Signed-off-by: Andy Qin <[email protected]>

Extend state stats and refactor stat combination

064465b

Signed-off-by: Andy Qin <[email protected]>

Fix rest integ test

e4abeca

Signed-off-by: Andy Qin <[email protected]>

Add stats and nodes filtering

9274a71

Signed-off-by: Andy Qin <[email protected]>

Add flatten and include metadata query parameter

e1cb163

Signed-off-by: Andy Qin <[email protected]>

Add EventStatData to snapshot EventStat info

282db07

Signed-off-by: Andy Qin <[email protected]>

Refactor stat filtering

7d08433

Signed-off-by: Andy Qin <[email protected]>

Add stat aggregations, metadata

df81f11

Signed-off-by: Andy Qin <[email protected]>

Add trailing interval and minutes since last event metadata

4677199

Signed-off-by: Andy Qin <[email protected]>

Reorganize response processing logic

8cc6698

Signed-off-by: Andy Qin <[email protected]>

Add enable stats setting and reset

24760a5

Signed-off-by: Andy Qin <[email protected]>

Adding documentation for state and event stats

c15bcff

Signed-off-by: Andy Qin <[email protected]>

More documentation

a8c3d0a

Signed-off-by: Andy Qin <[email protected]>

Clean up unused state stats manager methods

9425021

Signed-off-by: Andy Qin <[email protected]>

Add unit tests and update ITs

7ba9dcb

Signed-off-by: Andy Qin <[email protected]>

Merge branch 'opensearch-project:main' into neural-stats

fe8a616

Add transport tests, refactor toXContent tests

56c2d19

Signed-off-by: Andy Qin <[email protected]>

Clean up files

d540f28

Signed-off-by: Andy Qin <[email protected]>

q-andy requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, sean-zheng-amazon, model-collapse and zane-neo as code owners March 4, 2025 22:25

martin-gaievski reviewed Mar 8, 2025

View reviewed changes

Address review comments

f6fc682

Signed-off-by: Andy Qin <[email protected]>

q-andy commented Mar 10, 2025

View reviewed changes

q-andy force-pushed the neural-stats branch from f07bb76 to 3adc2ca Compare March 11, 2025 23:35

Reformat response

c56a110

Signed-off-by: Andy Qin <[email protected]>

q-andy force-pushed the neural-stats branch from 3adc2ca to c56a110 Compare March 11, 2025 23:39

Add bwc tests for stats

73e2228

Signed-off-by: Andy Qin <[email protected]>

q-andy force-pushed the neural-stats branch from b7794fd to 73e2228 Compare March 12, 2025 00:32

q-andy added 2 commits March 12, 2025 14:42

Add parameter validation

0734a6a

Signed-off-by: Andy Qin <[email protected]>

Use readOptionalEnumSet from core

7b532a5

Signed-off-by: Andy Qin <[email protected]>

q-andy force-pushed the neural-stats branch 3 times, most recently from 720ea81 to 647d730 Compare March 13, 2025 18:15

q-andy requested review from heemin32, vibrantvarun and martin-gaievski March 13, 2025 18:17

Clean up files

2e0c119

Signed-off-by: Andy Qin <[email protected]>

q-andy force-pushed the neural-stats branch from 0304ef0 to 2e0c119 Compare March 13, 2025 18:35

Merge branch 'opensearch-project:main' into neural-stats

82504ce

martin-gaievski approved these changes Mar 13, 2025

View reviewed changes

q-andy force-pushed the neural-stats branch 2 times, most recently from 8648e13 to 82504ce Compare March 14, 2025 16:26

vibrantvarun approved these changes Mar 14, 2025

View reviewed changes

heemin32 approved these changes Mar 14, 2025

View reviewed changes

vibrantvarun merged commit 89e6932 into opensearch-project:feature/neural-stats-api Mar 14, 2025
90 of 96 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature branch] Add Neural Stats API #1208

[Feature branch] Add Neural Stats API #1208

q-andy commented Mar 4, 2025 •

edited

Loading

heemin32 commented Mar 7, 2025

q-andy left a comment •

edited

Loading

q-andy Mar 10, 2025

q-andy commented Mar 12, 2025 •

edited

Loading

martin-gaievski left a comment

martin-gaievski Mar 13, 2025

q-andy Mar 14, 2025 •

edited

Loading

codecov bot commented Mar 13, 2025 •

edited

Loading

vibrantvarun left a comment •

edited

Loading

vibrantvarun commented Mar 14, 2025

heemin32 commented Mar 14, 2025

		}
		testRuntimeOnly group: 'net.minidev', name:'json-smart', version: "${versions.json_smart}"

[Feature branch] Add Neural Stats API #1208

[Feature branch] Add Neural Stats API #1208

Conversation

q-andy commented Mar 4, 2025 • edited Loading

Description

Related Issues

Check List

Example requests

heemin32 commented Mar 7, 2025

q-andy left a comment • edited Loading

Choose a reason for hiding this comment

q-andy Mar 10, 2025

Choose a reason for hiding this comment

q-andy commented Mar 12, 2025 • edited Loading

martin-gaievski left a comment

Choose a reason for hiding this comment

martin-gaievski Mar 13, 2025

Choose a reason for hiding this comment

q-andy Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Mar 13, 2025 • edited Loading

Codecov Report

vibrantvarun left a comment • edited Loading

Choose a reason for hiding this comment

vibrantvarun commented Mar 14, 2025

heemin32 commented Mar 14, 2025

q-andy commented Mar 4, 2025 •

edited

Loading

q-andy left a comment •

edited

Loading

q-andy commented Mar 12, 2025 •

edited

Loading

q-andy Mar 14, 2025 •

edited

Loading

codecov bot commented Mar 13, 2025 •

edited

Loading

vibrantvarun left a comment •

edited

Loading