Skip to content

OpenSearch template does not match dated v11 log and alert indices #2576

Description

@osmontero

Summary

The OpenSearch composable template utmstack_indexes uses literal prefix strings rather than wildcard index patterns. It therefore does not apply to dated production indices such as v11-alert-2026-09-11 and v11-log-o365-2026-09-11.

Impact

Dated alert indices fall back to OpenSearch's default index.mapping.total_fields.limit of 1000. When a correlation alert includes an O365 event with additional dynamic lastEvent.log.* fields, alert indexing fails with HTTP 400 instead of creating an alert.

Observed on UTMStack v11.2.13-community:

Limit of total fields [1000] has been exceeded
process: plugin_com.utmstack.alerts
alert: RETEST: O365 Admin Role Assignment (RETEST)
failed to correlate alert

The real O365 event was ingested and matched the RETEST correlation rule; only persistence of the resulting alert failed.

Current template

{
  "index_patterns": [
    "v11-alert-",
    "v11-log-",
    ".utm-",
    ".utmstack-"
  ],
  "template": {
    "settings": {
      "index.mapping.total_fields.limit": "50000"
    }
  }
}

These patterns do not match dated indices. The setting is absent from v11-alert-2026-09-11.

Expected

The template should match all rolling indices, for example:

"index_patterns": [
  "v11-alert-*",
  "v11-log-*",
  ".utm-*",
  ".utmstack-*"
]

The deployment/init path should also apply the intended total-fields limit to an already-created current-day index, or ensure the corrected template is in place before index creation.

Reproduction

  1. Inspect GET /_index_template/utmstack_indexes.
  2. Inspect GET /v11-alert-YYYY-MM-DD/_settings?filter_path=*.settings.index.mapping.total_fields.limit; it returns {}.
  3. Correlate an event whose alert contains new dynamic fields under lastEvent.log.* (O365 AAD audit events reproduce this).
  4. The alerts plugin retries three times, then logs failed to correlate alert after OpenSearch returns HTTP 400.

Notes

  • The event processor and alerts plugin are otherwise running; GCP alerts continue to be indexed.
  • This blocks valid O365 rule validation because the rule can match but its alert cannot be stored.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions