Fix Flaky Test SpecificClusterManagerNodesIT.testElectOnlyBetweenClusterManagerNodes

[kkewwei]

Description

When we call the code, the cluster manager is being removed from cluster, it will happen in internalCluster().nonClusterManagerClient() or getClusterManagerNode():
https://github.com/opensearch-project/OpenSearch/blob/main/server/src/internalClusterTest/java/org/opensearch/cluster/SpecificClusterManagerNodesIT.java#L63C9-L79C10

• When it happens in internalCluster().nonClusterManagerClient(), which hit the case in Fix Flaky Test SpecificClusterManagerNodesIT.testElectOnlyBetweenClusterManagerNodes #16021.
• When it happens in getClusterManagerNode(), which hit the current case, and we will fix it in the pr. Details can be seen in [BUG] org.opensearch.cluster.SpecificClusterManagerNodesIT.testElectOnlyBetweenClusterManagerNodes is flaky #16015 (comment)

Related Issues

Resolves #15944 #16015

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for eb0a6a8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 91b5b6d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kkewwei]

@gaobinlon, could you review it in your spare time.

[github-actions]

❕ Gradle check result for 91b5b6d: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.45%. Comparing base (af5835f) to head (91b5b6d).
Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #17640   +/-   ##
=========================================
  Coverage     72.44%   72.45%           
+ Complexity    65732    65707   -25     
=========================================
  Files          5311     5311           
  Lines        305011   305011           
  Branches      44232    44232           
=========================================
+ Hits         220969   220983   +14     
+ Misses        65967    65905   -62     
- Partials      18075    18123   +48     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.