Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Xfail and Mayfail attributes to city runner #686

Merged
merged 7 commits into from
Feb 25, 2025

Conversation

coldav
Copy link
Collaborator

@coldav coldav commented Feb 19, 2025

Overview

Support attribute Xfail and Mayfail for city runner and an add a --override option to override entries in the main csv file.

Reason for change

We want to be able to run tests that we know fail, but we don't want an overall fail.

Description of change

Added xfail and Mayfail as attributes to city runner.

Xfail means we expect it to fail and will give a failing return code if it unexpectedly pass, otherwise it does not contribute to an overall fail.

Mayfail means it sometimes fails and will not contribute to a failing return code.

Unlike some of the other attributes, these cannot be set with additional csv files, as it was felt this leads to overcomplexity. Instead it is expected that some form of csv combination script could be used alongside this.

Added --override option. This provides an additional csv file which can override entries in the main csv file if it matches the first two values in an entry. This is useful for overriding attributes such as Xfail.

Anything else we should know?

The following is example output

dump.txt

Checklist

  • Read and follow the project Code of Conduct.
  • Make sure the project builds successfully with your changes.
  • Run relevant testing locally to avoid regressions.
  • Run clang-format-19 on all modified code.

@coldav coldav requested a review from a team as a code owner February 21, 2025 10:07
@coldav coldav force-pushed the colin/add_xfail_city_runner branch from 8142c1d to be7fd1f Compare February 21, 2025 10:15
@coldav coldav force-pushed the colin/add_xfail_city_runner branch 4 times, most recently from 7029900 to 3b259da Compare February 21, 2025 16:29
@coldav coldav force-pushed the colin/add_xfail_city_runner branch from 3b259da to c78de58 Compare February 25, 2025 09:01
Copy link
Collaborator

@hvdijk hvdijk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good! Some odd whitespace and perhaps slightly more verbose naming and complicated logic than needed, left some suggestions for that, but the logic for the results looks right as far as I can see

Support attribute Xfail and Mayfail for city runner. These can be set
as additional elements in the csv file.

Xfail means we expect it to fail and will give a failing return code
if it unexpectedly pass, otherwise it does not contribute to an overall
fail.

Mayfail means it sometimes fails and will not contribute to a failing
return code.

Unlike some of the other attributes, these cannot be set with additional
csv files, as it was felt this leads to overcomplexity. Instead it is
expected that some form of csv combination script could be used alongside
this.
@coldav coldav force-pushed the colin/add_xfail_city_runner branch from 27ce583 to 9e3a667 Compare February 25, 2025 15:55
@coldav coldav merged commit 0ac47ba into uxlfoundation:main Feb 25, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants