Fix flakiness in FieldDataLoadingIT.testIndicesFieldDataCacheSizeSetting()

[peteralfonsi]

Description

Fixes flakiness in FieldDataLoadingIT.testIndicesFieldDataCacheSizeSetting().

The issue here was that on index creation, OpenSearch temporarily creates and then closes a test/temp index with the same name as the real index, using IndicesService.withTempIndexService(). The close causes the field data cache to wipe keys for that index name. But, since FDC cleanup is asynchronous and the cleanup interval for this test was 1 second, occasionally the wipe could happen after we run the first query on the real index but after we check that the resulting memory size of the cache is >0. That wiped the actual keys so the memory size was incorrectly 0.

We fix this by, after index creation, ensuring the cache clear cycle has completed. This is done by adding a key and then clearing the cache and waiting until the stats return to 0. Afterwards the tests can continue without risk of any extra FDC wipes at unexpected times.

Related Issues

Resolves #19431

Check List

• Functionality includes testing.
• [N/A] API changes companion pull request created, if applicable.
• [N/A] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for a79c95a: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.08%. Comparing base (2bb5e33) to head (a79c95a).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19571      +/-   ##
============================================
+ Coverage     73.04%   73.08%   +0.04%     
+ Complexity    70587    70583       -4     
============================================
  Files          5719     5719              
  Lines        323288   323288              
  Branches      46818    46818              
============================================
+ Hits         236147   236279     +132     
+ Misses        68154    67964     -190     
- Partials      18987    19045      +58     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jainankitk]

Thanks @peteralfonsi for addressing the flaky test promptly. LGTM!