[Performance] big5.cardinality-agg-high regressed in OS 3.1

[asimmahmood1]

Describe the bug

In OS 3 a new execution_hint was added to force using ordinals, which reduced p90 ST from 2402ms to 1905 with hint. With OS 3 concurrent search enabled by default, it dropped to 786ms.

In the nightly bechmark we can see similar numbers:

Big5 Term Aggregation Queries
	2.19.1		3.0.0		3.1.0
	May 26, 2025	May 27, 2025	May 26, 2025	May 27, 2025	May 26, 2025	May 27, 2025
Rows	p90_value	p90_value	p90_value	p90_value	p90_value	p90_value
cardinality-agg-high	2,285	2,315	648	630	874	889

https://us-east-1.quicksight.aws.amazon.com/sn/account/amazonbi/dashboards/fc187b76-79eb-4184-bf9d-98b495268244/sheets/fc187b76-79eb-4184-bf9d-98b495268244_2d999083-617b-4407-8af0-8bb727630529?#

After Lucence 10.1 is merged into 3.1, now the number has regressed. Nightly shows 630->889ms.

Personal testing shows the same:

Metric	Task	Value	Unit
Segment count		10
90th percentile service time	cardinality-agg-high	1096.88	ms

Related component

Search:Performance

To Reproduce

1. Set concurrent search to auto

curl -X PUT "http://localhost:9200/big5/_settings" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "search": {
      "concurrent_segment_search": {
        "mode": "auto"
      }
    }
  }
}
'

2. Index big5 workload using main branch

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=big5 --target-hosts=localhost:9200 --kill-running-processes   --workload-param "distribution_version:2.11.0"

4. Run opensearch-benchmark execute-test --pipeline=benchmark-only --workload=big5 --target-hosts=localhost:9200 --kill-running-processes --workload-param "max_num_segments:10,distribution_version:3.1.0" --include-tasks="cardinality-agg-high"

Expected behavior

No change in latency

Additional Details

Plugins
No plugins enabled

Screenshots
n/a

Host/Environment (please complete the following information):

• Tested using r5.xl host with EBS Gp3 16000 IOPS, 500Gb and 1000 throughput.
• al2023-ami-2023.6.20250303.0-kernel-6.1-x86_64

Additional context
No

[asimmahmood1]

OS 3.1

| 90th percentile service time | cardinality-agg-high | 1096.88 | ms |

Async shows 30% times spent looking up docValues vs 23% in OS 3.0, and 18% increasing the BitArray (no change in OS 3.0)

OS 3.0

| 90th percentile service time | cardinality-agg-high | 975.676 | ms |

Also going look at few alternatives:

1. binary field - [Feature Request] Use of Binary DocValue for high cardinality fields to improve aggregations performance #16837
2. sortedSetDocValues
3. global ordinals with Hloglog
4. Look for alternative for term dic

[kkewwei]

The term dictionary for BinaryDocValues was introduced prior to Lucene10, which likely isn’t the root cause. Based on my impression, when aggregations are present, concurrent search is automatically used, which seems minimally related to the concurrent_segment_search=auto.

As I’m preparing to optimize cardinality aggregation #18381, perhaps we can jointly investigate the underlying causes.

[asimmahmood1]

You're right, I don't think its term dictionary for 10.1. One additional observation is that in Lucene 10 it uses MatchAllDocsQuery scorer (which ads con, but Lucene 10.1 its DenseConjunctionBulkScorer.

Based on what you mentioned, I'm going to compare nextDocs in profile:true to see if there's any difference.

Another point is, now that RangeDocId.forEach in Lucene 10.1, I'll check if there's any shortcuts we can make there. I'm still coming up on speed on on Lucene 10.1 changes.

I'd be happy to work with you.

[asimmahmood1]

OS 3.0

[ec2-user@ip-172-31-61-197 ~]$ curl -X GET "http://localhost:9200/big5/_search?pretty=true&timeout=10m" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "agent": {
      "cardinality": {
        "execution_hint": "ordinals",
        "field": "agent.name"
      }
    }
  },"profile":true
}
'
{
  "took" : 7328,
  "timed_out" : false,
  "terminated_early" : true,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {                                                                                                                                  [227/1879]
    "agent" : {
      "value" : 26832
    }
  },
  "profile" : {
    "shards" : [
      {
        "id" : "[gfzUYKBxQe24ula3BR788A][big5][0]",
        "inbound_network_time_in_millis" : 0,
        "outbound_network_time_in_millis" : 0,
        "searches" : [
          {
            "query" : [
              {
                "type" : "ConstantScoreQuery",
                "description" : "ConstantScore(*:*)",
                "time_in_nanos" : 6918013914,
                "max_slice_time_in_nanos" : 6917770058,
                "min_slice_time_in_nanos" : 6581801553,
                "avg_slice_time_in_nanos" : 6749785805,
                "breakdown" : {
                  "max_match" : 0,
                  "set_min_competitive_score_count" : 0,
                  "match_count" : 0,
                  "avg_score_count" : 0,
                  "shallow_advance_count" : 0,
                  "next_doc" : 6917738901,
                  "min_build_scorer" : 4535940765,
                  "score_count" : 0,
                  "compute_max_score_count" : 0,
                  "advance" : 0,
                  "min_set_min_competitive_score" : 0,
                  "min_advance" : 0,
                  "score" : 0,
                  "avg_set_min_competitive_score_count" : 0,
                  "min_match_count" : 0,
                  "avg_score" : 0,
                  "max_next_doc_count" : 59392013,
                  "max_compute_max_score_count" : 0,
                 "max_compute_max_score_count" : 0,
                  "avg_shallow_advance" : 0,
                  "max_shallow_advance_count" : 0,
                  "set_min_competitive_score" : 0,
                  "min_build_scorer_count" : 14,
                  "next_doc_count" : 116000015,
                  "min_match" : 0,
                  "avg_next_doc" : 6749759357,
                  "compute_max_score" : 0,
                  "min_set_min_competitive_score_count" : 0,
                  "max_build_scorer" : 4714960645,
                  "avg_match_count" : 0,
                  "avg_advance" : 0,
                  "build_scorer_count" : 30,
                  "avg_build_scorer_count" : 15,
                  "min_next_doc_count" : 56608002,
                  "min_shallow_advance_count" : 0,
                  "max_score_count" : 0,
                  "avg_match" : 0,
                  "avg_compute_max_score" : 0,
                  "max_advance" : 0,
                  "avg_shallow_advance_count" : 0,
                  "avg_set_min_competitive_score" : 0,
                  "avg_compute_max_score_count" : 0,
                  "avg_build_scorer" : 4625450705,
                  "max_set_min_competitive_score_count" : 0,
                  "advance_count" : 0,
                  "max_build_scorer_count" : 16,
                  "shallow_advance" : 0,
                  "min_compute_max_score" : 0,
                  "max_match_count" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 4714960645,
                  "max_set_min_competitive_score" : 0,
                  "max_compute_max_score" : 0,
                  "min_shallow_advance" : 0,
                  "match" : 0,
                  "max_shallow_advance" : 0,
                  "avg_advance_count" : 0,
                  "min_next_doc" : 6581779814,
                  "max_advance_count" : 0,
                  "min_score" : 0,
                  "max_next_doc" : 6917738901,
                  "create_weight" : 14011,
                  "avg_next_doc_count" : 58000007,
                  "max_score" : 0,
                  "min_compute_max_score_count" : 0,
                  "min_score_count" : 0,
                  "min_advance_count" : 0
                },
                "children" : [
                  {
                    "type" : "MatchAllDocsQuery",
                    "description" : "*:*",
                    "time_in_nanos" : 5479277327,
                    "max_slice_time_in_nanos" : 5479037415,
                    "min_slice_time_in_nanos" : 5239380673,
                    "avg_slice_time_in_nanos" : 5359209044,
                    "breakdown" : {
                      "max_match" : 0,
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "avg_score_count" : 0,
                      "shallow_advance_count" : 0,
                      "next_doc" : 5479010358,
                      "min_build_scorer" : 4535935332,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "advance" : 0,
                      "min_set_min_competitive_score" : 0,
                      "min_advance" : 0,
                      "score" : 0,
                      "avg_set_min_competitive_score_count" : 0,
                      "min_match_count" : 0,
                      "avg_score" : 0,
                      "max_next_doc_count" : 59392013,
                      "max_compute_max_score_count" : 0,
                      "avg_shallow_advance" : 0,
                      "max_shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "min_build_scorer_count" : 14,
                      "next_doc_count" : 116000015,
                      "min_match" : 0,
                      "avg_next_doc" : 5359186248,
                      "compute_max_score" : 0,
                      "min_set_min_competitive_score_count" : 0,
                      "max_build_scorer" : 4714954042,
                      "avg_match_count" : 0,
                      "avg_advance" : 0,
                      "build_scorer_count" : 30,
                      "avg_build_scorer_count" : 15,
                      "min_next_doc_count" : 56608002,
                      "min_shallow_advance_count" : 0,
                      "max_score_count" : 0,
                      "avg_match" : 0,
                      "avg_compute_max_score" : 0,
                      "max_advance" : 0,
                      "avg_shallow_advance_count" : 0,
                      "avg_set_min_competitive_score" : 0,
                      "avg_compute_max_score_count" : 0,
                      "avg_build_scorer" : 4625444687,
                      "max_set_min_competitive_score_count" : 0,
                      "advance_count" : 0,
                      "max_build_scorer_count" : 16,
                      "shallow_advance" : 0,
                      "min_compute_max_score" : 0,
                      "max_match_count" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 4714954042,
                      "max_set_min_competitive_score" : 0,
                      "max_compute_max_score" : 0,
                      "min_shallow_advance" : 0,
                      "match" : 0,
                      "max_shallow_advance" : 0,
                      "avg_advance_count" : 0,
                      "min_next_doc" : 5239362138,
                      "max_advance_count" : 0,
                      "avg_next_doc_count" : 58000007,                                                                                                 [68/1879]
                      "max_score" : 0,
                      "min_compute_max_score_count" : 0,
                      "min_score_count" : 0,
                      "min_advance_count" : 0
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 10870,
            "collector" : [
              {
                "name" : "QueryCollectorManager",
                "reason" : "search_multi",
                "time_in_nanos" : 5534748287,
                "reduce_time_in_nanos" : 160004,
                "max_slice_time_in_nanos" : 5534748287,
                "min_slice_time_in_nanos" : 5297878760,
                "avg_slice_time_in_nanos" : 5416313523,
                "slice_count" : 2,
                "children" : [
                  {
                    "name" : "EarlyTerminatingCollectorManager",
                    "reason" : "search_count",
                    "time_in_nanos" : 186920,
                    "reduce_time_in_nanos" : 7973,
                    "max_slice_time_in_nanos" : 50974,
                    "min_slice_time_in_nanos" : 48976,
                    "avg_slice_time_in_nanos" : 49975,
                    "slice_count" : 2
                  },
                  {
                    "name" : "NonGlobalAggCollectorManager: [agent]",
                    "reason" : "aggregation",
                    "time_in_nanos" : 2677884105,
                    "reduce_time_in_nanos" : 135839,
                    "max_slice_time_in_nanos" : 2677884105,
                    "min_slice_time_in_nanos" : 2573204115,
                    "avg_slice_time_in_nanos" : 2625544110,
                    "slice_count" : 2
                  }
                ]
              }
            ]
          }
        ],
        "aggregations" : [
          {
            "type" : "CardinalityAggregator",
            "description" : "agent",
            "time_in_nanos" : 2755459131,
            "max_slice_time_in_nanos" : 2755459131,
            "min_slice_time_in_nanos" : 2672821099,
            "avg_slice_time_in_nanos" : 2714140115,
            "breakdown" : {
              "min_build_leaf_collector" : 1468599,
              "build_aggregation_count" : 2,
              "post_collection" : 314453943,
              "max_collect_count" : 59392005,
              "initialize_count" : 2,
              "reduce_count" : 0,
              "avg_collect" : 2712033783,
              "max_build_aggregation" : 166699,
              "avg_collect_count" : 58000000,
              "max_build_leaf_collector" : 1543617,
              "min_build_leaf_collector_count" : 7,
              "build_aggregation" : 314201913,
              "min_initialize" : 968,
              "max_reduce" : 0,
              "build_leaf_collector_count" : 15,
              "avg_reduce" : 0,
              "min_collect_count" : 56607995,
              "avg_build_leaf_collector_count" : 7,
              "avg_build_leaf_collector" : 1506108,
              "max_collect" : 2753374340,
              "reduce" : 0,
              "avg_build_aggregation" : 166129,
              "min_post_collection" : 417727,
              "collect_count" : 116000000,
              "avg_post_collection" : 432400,
              "avg_initialize" : 1693,
              "post_collection_count" : 2,
              "build_leaf_collector" : 1582000,
              "min_collect" : 2670693227,
              "min_build_aggregation" : 165560,
              "initialize" : 243173,
              "max_build_leaf_collector_count" : 8,
              "min_reduce" : 0,
              "collect" : 2753374340
            },
            "debug" : {
              "ordinals_collectors_used" : 7,
              "ordinals_collectors_overhead_too_high" : 0,
              "string_hashing_collectors_used" : 0,
              "dynamic_pruned_segments" : 0,
              "numeric_collectors_used" : 0,
              "empty_collectors_used" : 0
            }
          }
        ]
      }
    ]
  }
}

[asimmahmood1]

OS 3.1

{
  "took" : 7978,
  "timed_out" : false,
  "terminated_early" : true,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "agent" : {
      "value" : 26832
    }
  },
  "profile" : {
    "shards" : [
      {
        "id" : "[Hc_Omec3RW2kMXWsp_Wrow][big5][0]",
        "inbound_network_time_in_millis" : 0,
        "outbound_network_time_in_millis" : 0,
        "searches" : [
          {
            "query" : [
              {
                "type" : "ConstantScoreQuery",
                "description" : "ConstantScore(*:*)",
                "time_in_nanos" : 7457459421,
                "max_slice_time_in_nanos" : 7455527752,
                "min_slice_time_in_nanos" : 6443438145,
                "avg_slice_time_in_nanos" : 6949482948,
                "breakdown" : {
                  "max_match" : 0,
                  "set_min_competitive_score_count" : 0,
                  "match_count" : 0,
                  "avg_score_count" : 0,
                  "shallow_advance_count" : 0,
                  "next_doc" : 7452169464,
                  "min_build_scorer" : 4388105552,
                  "score_count" : 0,
                  "compute_max_score_count" : 0,
                  "advance" : 0,
                  "min_set_min_competitive_score" : 0,
                  "min_advance" : 0,
                  "score" : 0,
                  "avg_set_min_competitive_score_count" : 0,
                  "min_match_count" : 0,
                  "avg_score" : 0,
                  "max_next_doc_count" : 61798228,
                  "max_compute_max_score_count" : 0,
                  "avg_shallow_advance" : 0,
                  "max_shallow_advance_count" : 0,
                  "set_min_competitive_score" : 0,
                  "min_build_scorer_count" : 10,
                  "next_doc_count" : 116000010,
                  "min_match" : 0,
                  "avg_next_doc" : 6946118493,
                  "compute_max_score" : 0,
                  "min_set_min_competitive_score_count" : 0,
                  "max_build_scorer" : 5265411511,
                  "avg_match_count" : 0,
                  "avg_advance" : 0,
                  "build_scorer_count" : 20,
                  "avg_build_scorer_count" : 10,
                  "min_next_doc_count" : 54201782,
                  "min_shallow_advance_count" : 0,
                  "max_score_count" : 0,
                  "avg_match" : 0,
                  "avg_compute_max_score" : 0,
                  "max_advance" : 0,
                  "avg_shallow_advance_count" : 0,
                  "avg_set_min_competitive_score" : 0,
                  "avg_compute_max_score_count" : 0,
                  "avg_build_scorer" : 4826758531,
                  "max_set_min_competitive_score_count" : 0,
                  "advance_count" : 0,
                  "max_build_scorer_count" : 10,
                  "shallow_advance" : 0,
                  "min_compute_max_score" : 0,
                  "max_match_count" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 5265423818,
                  "max_set_min_competitive_score" : 0,
                  "max_compute_max_score" : 0,
                  "min_shallow_advance" : 0,
                  "match" : 0,
                  "max_shallow_advance" : 0,
                  "avg_advance_count" : 0,
                  "min_next_doc" : 6440067522,
                  "max_advance_count" : 0,
                  "min_score" : 0,
                  "max_next_doc" : 7452169464,
                  "create_weight" : 303736,
                  "avg_next_doc_count" : 58000005,
                  "max_score" : 0,
                  "min_compute_max_score_count" : 0,
                  "min_score_count" : 0,
                  "min_advance_count" : 0
                },
                "children" : [
                  {
                    "type" : "MatchAllDocsQuery",
                    "description" : "*:*",
                    "time_in_nanos" : 5919489943,
                    "max_slice_time_in_nanos" : 5917573375,
                    "min_slice_time_in_nanos" : 4988980712,
                    "avg_slice_time_in_nanos" : 5453277043,
                    "breakdown" : {
                      "max_match" : 0,
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "avg_score_count" : 0,
                      "shallow_advance_count" : 0,
                      "next_doc" : 5914231671,
                      "min_build_scorer" : 4388071103,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "advance" : 0,
                      "min_set_min_competitive_score" : 0,
                      "min_advance" : 0,
                      "score" : 0,
                      "avg_set_min_competitive_score_count" : 0,
                      "min_match_count" : 0,
                      "avg_score" : 0,
                      "max_next_doc_count" : 61798228,
                      "max_compute_max_score_count" : 0,
                      "avg_shallow_advance" : 0,
                      "max_shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "min_build_scorer_count" : 10,
                      "next_doc_count" : 116000010,
                      "min_match" : 0,
                      "avg_next_doc" : 5449935765,
                      "compute_max_score" : 0,
                      "min_set_min_competitive_score_count" : 0,
                      "max_build_scorer" : 5265390865,
                      "avg_match_count" : 0,
                      "avg_advance" : 0,
                      "build_scorer_count" : 20,
                      "avg_build_scorer_count" : 10,
                      "min_next_doc_count" : 54201782,
                      "min_shallow_advance_count" : 0,
                      "max_score_count" : 0,
                      "avg_match" : 0,
                      "avg_compute_max_score" : 0,
                      "max_advance" : 0,
                      "avg_shallow_advance_count" : 0,
                      "avg_set_min_competitive_score" : 0,
                      "avg_compute_max_score_count" : 0,
                      "avg_build_scorer" : 4826730984,
                      "max_set_min_competitive_score_count" : 0,
                      "advance_count" : 0,
                      "max_build_scorer_count" : 10,
                      "shallow_advance" : 0,
                      "min_compute_max_score" : 0,
                      "max_match_count" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 5265390865,
                      "max_set_min_competitive_score" : 0,
                      "max_compute_max_score" : 0,
                      "min_shallow_advance" : 0,
                      "match" : 0,
                      "max_shallow_advance" : 0,
                      "avg_advance_count" : 0,
                      "min_next_doc" : 4985639873,
                      "max_advance_count" : 0,
                      "min_score" : 0,
                      "max_next_doc" : 5914231657,
                      "create_weight" : 3440,
                      "avg_next_doc_count" : 58000005,
                      "max_score" : 0,
                      "min_compute_max_score_count" : 0,
                      "min_score_count" : 0,
                      "min_advance_count" : 0
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 21320,
            "collector" : [
              {
                "name" : "QueryCollectorManager",
                "reason" : "search_multi",
                "time_in_nanos" : 6012960654,
                "reduce_time_in_nanos" : 197009,
                "max_slice_time_in_nanos" : 6012960654,
                "min_slice_time_in_nanos" : 5217967286,
                "avg_slice_time_in_nanos" : 5615463970,
                "slice_count" : 2,
                "children" : [
                  {
                    "name" : "EarlyTerminatingCollectorManager",
                    "reason" : "search_count",
                    "time_in_nanos" : 1524426,
                    "reduce_time_in_nanos" : 11788,
                    "max_slice_time_in_nanos" : 255675,
                    "min_slice_time_in_nanos" : 244154,
                    "avg_slice_time_in_nanos" : 249914,
                    "slice_count" : 2
                  },
                  {
                    "name" : "NonGlobalAggCollectorManager: [agent]",
                    "reason" : "aggregation",
                    "time_in_nanos" : 3001740244,
                    "reduce_time_in_nanos" : 129539,
                    "max_slice_time_in_nanos" : 3001740244,
                    "min_slice_time_in_nanos" : 2582026165,
                    "avg_slice_time_in_nanos" : 2791883204,
                    "slice_count" : 2
                  }
                ]
              }
            ]
          }
        ],
        "aggregations" : [
          {
            "type" : "CardinalityAggregator",
            "description" : "agent",
            "time_in_nanos" : 2903810522,
            "max_slice_time_in_nanos" : 2903810522,
            "min_slice_time_in_nanos" : 2520049855,
            "avg_slice_time_in_nanos" : 2711930188,
            "breakdown" : {
              "min_build_leaf_collector" : 1355781,
              "build_aggregation_count" : 2,
              "post_collection" : 1053718125,
              "max_collect_count" : 61798223,
              "initialize_count" : 2,
              "reduce_count" : 0,
              "avg_collect" : 2709933033,
              "max_build_aggregation" : 120475,
              "avg_collect_count" : 58000000,
              "max_build_leaf_collector" : 1456572,
              "min_build_leaf_collector_count" : 5,
              "build_aggregation" : 1053305395,
              "min_initialize" : 2285,
              "max_reduce" : 0,
              "build_leaf_collector_count" : 10,
              "avg_reduce" : 0,
              "min_collect_count" : 54201777,
              "avg_build_leaf_collector_count" : 5,
              "avg_build_leaf_collector" : 1406176,
              "max_collect" : 2901808426,
              "reduce" : 0,
              "avg_build_aggregation" : 118159,
              "min_post_collection" : 424146,
              "max_initialize" : 5535,
              "max_post_collection" : 513674,
              "collect_count" : 116000000,
              "avg_post_collection" : 468910,
              "avg_initialize" : 3910,
              "post_collection_count" : 2,
              "build_leaf_collector" : 1469050,
              "min_collect" : 2518057640,
              "min_build_aggregation" : 115843,
              "initialize" : 2825158,
              "max_build_leaf_collector_count" : 5,
              "min_reduce" : 0,
              "collect" : 2901811005
            },
            "debug" : {
              "ordinals_collectors_used" : 5,
              "ordinals_collectors_overhead_too_high" : 0,
              "string_hashing_collectors_used" : 0,
              "dynamic_pruned_segments" : 0,
              "numeric_collectors_used" : 0,
              "empty_collectors_used" : 0
            }
          }
        ]
      }
    ]
  }
}

[asimmahmood1]

A few differences in OS3.0 vs OS3.1

• took: 7328 vs 7978 (+8.9%)
• next_doc: 6917738901 vs 7452169464 (+7%)
  • more docs, which makes sense OS3.1 takes longer, but why?
• ordinals_collectors_used: 7 vs 5
  • both have 10 segments so not sure should've used all 10