Functional testing workflow - Teams test plan and it variant's running schedule proposal

[nanda-katikaneni]

💡 Summary

The current Nightly Functional Testing workflow has following combination of tests for Teams:

• E5 tenant: teams.testplan.yaml and teams.e#.testplan.yaml
• G5,G3 and GCCHIGH tenants (all three tenants): teams.testplan.yaml and teams.gcc.testplan.yaml

In the above, the gcc test plan variant is not needed to run (the specific test in the plan does not apply to gcc tenants - test plan is more a place holder for any future gcc specific tests - none yet). Also, to reduce testing time without losing coverage, proposing the following combination runs:

Nightly Functional Testing (Mon-Fri):

• E5 tenant: teams.testplan.yaml and teams.e#.testplan.yaml
• G3 tenant: teams.testplan.yaml

Weekly Functional Testing (Sat or Sun):

• E5 tenant: teams.testplan.yaml and teams.e#.testplan.yaml
• G5,G3 and GCCHIGH tenants (all three tenants): teams.testplan.yaml

Motivation and context

The gcc test plan variant is a placeholder for any future gcc specific tests - the current test in that plan is not needed (it is a test designed for non-gcc tenants). Reducing the number of tenants for Nightly test is to ensure that we are not running out of runners. The weekly test plan runs covering all tenant types can uncover any platform specific issues.

Acceptance criteria

Nightly functional tests for teams run only on 2 tenant types (for one tenant 2 test plans and the other 1 test plan). Weekly run covering other tenant and test plan combinations.

[james-garriss]

While written for Teams, it's possible that the solutions that Nanda proposes are applicable to other products as well.