azure-pipelines: add nightly scheduled run for flakiness detection#607
Open
sigabrtv1-ui wants to merge 1 commit intosonic-net:masterfrom
Open
azure-pipelines: add nightly scheduled run for flakiness detection#607sigabrtv1-ui wants to merge 1 commit intosonic-net:masterfrom
sigabrtv1-ui wants to merge 1 commit intosonic-net:masterfrom
Conversation
Add a nightly scheduled trigger on master so test analytics accumulates clean data points independent of PR activity. Key details: - Runs at midnight UTC every day on master - always: true ensures the run fires even with no new commits, giving a consistent stream of data regardless of merge activity - RUN_TYPE variable tags test run titles as 'Nightly' vs 'CI' so scheduled and PR/push results can be filtered separately in the Azure DevOps test analytics dashboard Without regular master runs, flaky tests at 5-10% failure rates can go undetected for days. A nightly cadence gives 30 clean data points per month, sufficient to surface tests failing >10% of the time. Signed-off-by: Dawei Huang <daweihuang@microsoft.com>
Contributor
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a nightly scheduled Azure Pipelines run on master to generate consistent test analytics data for detecting flaky tests, independent of PR/merge activity.
Changes:
- Added a daily (midnight UTC) scheduled trigger on
masterwithalways: true. - Introduced a
RUN_TYPEvariable (Nightlyfor scheduled runs,CIotherwise) to label test run titles. - Prefixed published test run titles with
$(RUN_TYPE)for easier filtering in Azure DevOps test analytics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Test analytics in Azure DevOps accumulate results mostly from PR runs. PR run failures are ambiguous — a test might fail because the PR broke it, or because it's flaky. Without regular master runs, the analytics can't cleanly identify flaky tests.
With the current volume of master runs (only on merges, a few per day at peak), a test that flakes at 5–10% failure rate can go undetected for days or weeks.
Solution
Add a nightly scheduled trigger on
masterwithalways: true, so test analytics gets a consistent stream of clean data points independent of merge activity.always: true: fires even with no new commits — critical for consistent dataRUN_TYPEvariable: tags test run titles asNightlyvsCIso scheduled and PR/push results can be filtered separately in Azure DevOps test analyticsImpact
30 clean master data points per month, enough to reliably surface tests failing >10% of the time. Scheduled and PR results remain in the same analytics dashboard but are distinguishable by run title prefix.
Signed-off-by: Dawei Huang daweihuang@microsoft.com