[Autorevert] Separate signal columns for jobs with different failures rules outputs #7452
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jean Schmidt <[email protected]>
Signed-off-by: Jean Schmidt <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
meta-cla
bot
added
the
CLA Signed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, we ignore all job signals that are 'pytest failure's. This is done as we parse test signals independently in a separate track. This have been introduced to avoid reverts to get confused with signals that are noisy / flaky. A typical case would be a flakiness hapening right before a streak of test failures. Without the current behaviour autorevert would act on the flaky commit instead in the commit that introduced the failure.
With the test track, we noticed a few cases where for some reason we're not obtaining information for tests, and we're investigating all of them with the goal of improving the quality and fix all possible gaps. Having stated that, it is bad that in a few cases we could have reacted based on job signal but we failed to do so as we ignore job signals.
So, after discussion, we decided to re-include job signals that have pytest failures. But, to avoid problems and avoid flakiness, we opted to separate jobs in different columns for each classification rule into the loopback window.