Onboard OpenSearch plugins to Smoke tests framework

[zelinh]

Sub-issue from opensearch-project/opensearch-build#5317

Background

With the smoke test framework (#5223) now established, we have successfully onboarded multiple OpenSearch plugins—including Index Management, Security, SQL, and K-NN—to the framework (PR #5255).

To maximize the effectiveness of smoke testing, we aim to onboard all OpenSearch plugins into the framework. This effort will enhance overall testing coverage, ensuring greater reliability and functionality validation for OpenSearch and its plugins.

Overview of Smoke Tests

Smoke tests verify the basic functionality of an OpenSearch distribution bundle with all plugins installed. Each plugin has its own test specification YAML file, stored in the OpenSearch Build repository and organized by major version.

These YAML files list API test cases, which the smoke test workflow executes. The results are then validated against the OpenSearch API Specification to ensure API correctness and expected behavior.

How to Onboard a Plugin

To onboard your plugin to the smoke test framework, follow the Onboarding Guide.

Steps to Onboard:

1. Add a YAML test file

   • Create a new test file for your plugin in the relevant major version directory under smoke_tests_spec.
   • Example: security.yml.

2. Define API tests in the YAML file

   • Specify:
     • API path
     • HTTP method
     • Request body (if applicable)
     • Custom Content-Type header if different from application/json.
   • See an example test definition here.

3. Ensure the APIs are onboarded to the OpenSearch API Specification repository

   • Before adding APIs to smoke tests, they must be present in the OpenSearch API Specification repository.
   • Refer to the Developer Guide for onboarding steps.

4. Reference the test file in the test manifest

   • Ensure the newly added YAML file is referenced in the test manifest for your plugin’s component.

Collaboration with Plugin Owners:

We are reaching out to all OpenSearch plugin owners to facilitate onboarding. We will provide:
✅ Documentation and guidelines for onboarding.
✅ Support in setting up test cases.
✅ Assistance in troubleshooting issues.

Expected Outcome:

• Stronger OpenSearch releases with better validation of plugins.
• Standardized, reusable testing framework for ongoing plugin validation.
• Faster detection of integration issues before release.

📢 Plugin owners: We encourage you to onboard your plugin to the smoke test framework! If you have any questions or need assistance, feel free to comment on this issue or reach out to EE team.

Let’s work together to strengthen OpenSearch’s testing and release process! 🚀

[dbwiddis]

These YAML files list API test cases, which the smoke test workflow executes. The results are then validated against the OpenSearch API Specification to ensure API correctness and expected behavior.

Hi @zelinh, we have already onboarded all our APIs to the api specification here which you correctly call out as a prerequisite.

However, we also have written yaml tests there, see https://github.com/opensearch-project/opensearch-api-specification/blob/main/tests/default/flow_framework/workflow.yaml which seem a lot more thorough, actually testing the functionality.

This effort seems duplicative (and not nearly as robust). Does this smoke test provide added value over those api spec tests?

[zelinh]

These YAML files list API test cases, which the smoke test workflow executes. The results are then validated against the OpenSearch API Specification to ensure API correctness and expected behavior.

Hi @zelinh, we have already onboarded all our APIs to the api specification here which you correctly call out as a prerequisite.

However, we also have written yaml tests there, see https://github.com/opensearch-project/opensearch-api-specification/blob/main/tests/default/flow_framework/workflow.yaml which seem a lot more thorough, actually testing the functionality.

This effort seems duplicative (and not nearly as robust). Does this smoke test provide added value over those api spec tests?

Hey @dbwiddis,

Yes, the smoke test framework is specifically designed to validate OpenSearch distribution bundles, including TAR, RPM, and DEB packages, as part of the release process. These tests enhance release quality by ensuring each distribution type is systematically verified.

To my knowledge, the tests in the api-spec repository primarily run against staging Docker images. Onboarding to this framework will provide more efficient and comprehensive sanity testing across all release artifacts, improving reliability and reducing manual effort in the release cycle.

[dbwiddis]

Onboarding to this framework will provide more efficient and comprehensive sanity testing across all release artifacts

Thanks, @zelinh. I understand the intent to have at least some basic API "sanity testing" and am fully on board.

What I'm trying to clarify is the scope of this effort. Specifically:

• is it the intent to add every API to the framework?
• Is it more important to add APIs that interact with other plugins?
• for Flow Framework in particular, we depend on ML Commons and many tests require waiting for initial ML Plugin configuration to process (cron job).

The API spec tests are pretty thorough covering every API and expected return codes for multiple path/parameter combinations, including expected 400 for invalid input. There are checklists in the PR template that indicate that we must add API spec tests for API changes, when necessary.

In comparison, the example tests in opensearch-project/opensearch-build#5255 just seem to check that the APIs are reachable. The original issue (opensearch-project/opensearch-build#5247) includes objectives like this for core:

Identify key OpenSearch APIs for inclusion in the smoke test framework.
Ensure comprehensive test coverage for critical functionalities.

So it doesn't test all APIs, just "key" ones.

And for the initial plugins, it seems limited with some exclusions:

We would not include this as part of k-NN smoke tests.

and to "few essential APIs"

Working on index-management plugins and successfully onboarded few essential APIs to our smoke tests.

Was this just to demonstrate the format and these plugins are expected to more completely fill out their testing framework?

TLDR:

Should every API be onboarded, testing all possible (valid and invalid) API return codes, essentially completely duplicating all the API spec tests in their entirety, but in a different format? Or is it intended only to check "key" functionality?

• if the former, is there any automation which can quickly convert the API spec testing format into the smoke test format to reduce manual effort?
• If the latter, what is "key" -- interaction with other plugins or some other criteria?

and

What is the process for making sure the smoke tests stay up to date over time with API changes? Will we be adding another line to the PR template to require a checkoff that API change PRs have both corresponding API spec and smoke test PRs?

[zelinh]

Hey @dbwiddis,

The intent of the smoke tests is not to fully duplicate the API spec tests but rather to validate "key" functionality that ensures the system is working as expected at a high level. The focus is on core interactions, integration points, and overall stability, rather than exhaustive validation of every possible API response.

The smoke test framework is using opensearch-api-spec as source of truth. For any API updates, you would update on the api-spec repo and smoke tests will be able to pick up those changes.

[dbwiddis]

Thanks for the clarification!