Skip to content

Update ml-commons metrics to add agent details #11163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 8, 2025
Merged

Update ml-commons metrics to add agent details #11163

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1dfa223
update metrics
pyek-bot Oct 1, 2025
b886eb4
fix: comments
pyek-bot Oct 6, 2025
87f8a5f
Apply suggestions from code review
kolchfa-aws Oct 7, 2025
0a58809
Apply suggestions from code review
natebower Oct 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion _monitoring-your-cluster/metrics/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,19 @@
Introduced 3.1
{: .label .label-purple }

OpenSearch provides enhanced observability for [machine learning (ML)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/) workflows. Metrics related to ML operations are pushed directly to the core metrics registry, giving you improved visibility into model usage and performance. Additionally, every 5 minutes, a periodic job collects and exports state data, helping you monitor the health and activity of your ML workloads over time.
OpenSearch provides enhanced observability for [machine learning (ML)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/) workflows. Metrics related to ML operations are pushed directly to the core metrics registry, giving you improved visibility into model usage and performance. Additionally, every 5 minutes, a periodic job collects and exports state data, helping you monitor the health and activity of your ML workloads over time.

The static collector job captures details about different types of models and agents created:

1. **Models** - Including deployment type (remote, pre-trained, custom), service provider (bedrock, sagemaker, azure, etc.), algorithm, model name, and model type (llm, embedding, image_generation, speech_audio).

Check failure on line 117 in _monitoring-your-cluster/metrics/getting-started.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: llm. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: llm. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_monitoring-your-cluster/metrics/getting-started.md", "range": {"start": {"line": 117, "column": 165}}}, "severity": "ERROR"}

Check warning on line 117 in _monitoring-your-cluster/metrics/getting-started.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.LatinismsElimination] Using 'etc.' is unnecessary. Remove.

Raw Output:
{"message": "[OpenSearch.LatinismsElimination] Using 'etc.' is unnecessary. Remove.", "location": {"path": "_monitoring-your-cluster/metrics/getting-started.md", "range": {"start": {"line": 117, "column": 119}}}, "severity": "WARNING"}

Check failure on line 117 in _monitoring-your-cluster/metrics/getting-started.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: sagemaker. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: sagemaker. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_monitoring-your-cluster/metrics/getting-started.md", "range": {"start": {"line": 117, "column": 101}}}, "severity": "ERROR"}

Check failure on line 117 in _monitoring-your-cluster/metrics/getting-started.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.SubstitutionsError] Use 'pretrained' instead of 'pre-trained'.

Raw Output:
{"message": "[OpenSearch.SubstitutionsError] Use 'pretrained' instead of 'pre-trained'.", "location": {"path": "_monitoring-your-cluster/metrics/getting-started.md", "range": {"start": {"line": 117, "column": 52}}}, "severity": "ERROR"}
2. **Agents** - Including LLM interface, model deployment, service provider, model type, memory type, and model identifier.

Additional metrics will be added in future releases.

Example agent metrics captured:
```
{_llm_interface=bedrock/converse/claude, model_deployment=remote, is_hidden=false, model_service_provider=bedrock, model_type=llm, memory_type=conversation_index, model=us.anthropic.claude-3-7-sonnet-20250219-v1:0, type=CONVERSATIONAL}
```

To enable ML observability, specify the following settings in `opensearch.yml`:

Expand Down
Loading