Skip to content

Fix aggregate reading a TestRun attribute that does not exist - #1097

Merged
OVI3D0 merged 1 commit into
opensearch-project:mainfrom
serhiy-bzhezytskyy:fix/aggregate-cluster-config-params-osb
Jul 28, 2026
Merged

Fix aggregate reading a TestRun attribute that does not exist#1097
OVI3D0 merged 1 commit into
opensearch-project:mainfrom
serhiy-bzhezytskyy:fix/aggregate-cluster-config-params-osb

Conversation

@serhiy-bzhezytskyy

Copy link
Copy Markdown
Contributor

Description

update_config_object read test_run.cluster_config_instance_params, which no test run has, so aggregate failed on every invocation:

[ERROR] ❌ Cannot aggregate. 'TestRun' object has no attribute 'cluster_config_instance_params'.

TestRun defines cluster_config_params (osbenchmark/metrics.py:1343), and the name being read occurs nowhere else in the repository — nothing assigns it. #882 (2c85d505) renamed the attribute on TestRun to cluster_config_params but renamed this read site to cluster_config_instance_params; #939 (e10342d1) later corrected the config key on that same line, from cluster_config_instance.params to cluster_config.params, and left the attribute name as it was. This one-word change reads the name TestRun actually defines.

Unlike the null-metrics crash (#1093), this does not depend on the data — it is on the plain aggregatebuild_aggregated_resultsupdate_config_object path. As far as I can tell it means aggregate cannot currently succeed for anyone on main.

A side effect of the fix: because the read never succeeded, the cluster-config params never reached the aggregated result. They now do.

Issues Resolved

Resolves #1094

Testing

  • New functionality includes testing

  • One new test, test_update_config_object_reads_attributes_that_test_runs_actually_have, which constructs a real metrics.TestRun and asserts the cluster_config.params value reaches config.add.

  • Verified red without the fix: with main's aggregator.py restored under the new test it fails with AttributeError: 'TestRun' object has no attribute 'cluster_config_instance_params'; with the fix it passes.

  • Full suite: 1423 passed, 5 skipped (baseline on main is 1422). pylint clean.

  • End-to-end through the real CLI: aggregate reports ✅ SUCCESS and the aggregated result now carries cluster-config-instance-params: {'heap_size': '6g'}, which was being silently lost.

The reason the existing suite doesn't catch this is worth flagging on its own: tests/aggregator_test.py never constructs a real TestRun — every test passes a bare Mock(), and a Mock returns a fresh Mock for any attribute name asked of it, so a wrong name reads fine. main sits at 1422 passed while aggregate cannot complete once. Mock(spec=TestRun) does not fix it either, because spec inspects the class, which knows nothing about attributes assigned in __init__; it raises on cluster_config first and points at the wrong line. A real TestRun is the only thing that catches it, which is why the new test builds one.

Notes

Found while using Apache solr-orbit, a Python port of OSB whose aggregator.py is byte-identical to this one apart from the import module name, to run a multi-configuration benchmark campaign. The same fix is proposed there as apache/solr-orbit#59.

This was the second of three defects blocking aggregate; it only became visible after the null-metrics crash was fixed, because that one happens earlier on the same path.

`aggregate` fails immediately for every invocation:

  [ERROR] Cannot aggregate. 'TestRun' object has no attribute
          'cluster_config_instance_params'.

update_config_object reads `test_run.cluster_config_instance_params`, but
TestRun defines `cluster_config_params`. The name being read occurs nowhere
else in the repository, so the aggregated results never carried the cluster
config params they were meant to.

The existing tests pass bare Mock() test runs, and a Mock returns a new Mock
for any attribute name asked of it, so they could not catch this. The
regression test constructs a real TestRun instead. Note that Mock(spec=TestRun)
would not help either: spec inspects the class and knows nothing about
attributes assigned in __init__.

Signed-off-by: Serhiy Bzhezytskyy <me@serhiy-bzhezytskyy.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ No major issues detected

@OVI3D0 OVI3D0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @serhiy-bzhezytskyy !

@OVI3D0
OVI3D0 merged commit 212e1e3 into opensearch-project:main Jul 28, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: aggregate always fails — update_config_object reads a TestRun attribute that does not exist

2 participants