Skip to content

[CI] Added Test Suite Verification to CI #3121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

AlexandreSinger
Copy link
Contributor

Found that we were regressing on many features in VTR due to tasks being
added to the appropriate test suite directory, but not being included in
the necessary task list. As such, it appeared as though the tests were
being run, but in reality they were not.

Added a script which will be run by the CI which will verify that all of
the test suites that we care about have all their tasks in the
appropriate task list.

From this tool, found many tasks which were not in the task lists.
Marked these tasks as "ignored" for now. These should be handled in a
separate PR.

@github-actions github-actions bot added infra Project Infrastructure lang-python Python code labels Jun 7, 2025
@AlexandreSinger AlexandreSinger force-pushed the feature-ci-task-list-checker branch 3 times, most recently from 7286cd2 to 7604811 Compare June 7, 2025 18:57
@AlexandreSinger AlexandreSinger changed the title [WIP][CI] Added Test Suite Verification to CI [CI] Added Test Suite Verification to CI Jun 7, 2025
@AlexandreSinger
Copy link
Contributor Author

This PR adds a new CI test which will check that all of the test suites that we use in the CI have all of the tasks they are supposed to have in their task lists.

The following test suites have tasks that are not in their task list:

  • vtr_reg_strong
  • vtr_reg_strong_odin
  • vtr_reg_nightly_test1
  • vtr_reg_nightly_test2
  • vtr_reg_nightly_test3
  • vtr_reg_nightly_test5
  • vtr_reg_nightly_test7

Added the test in these suites as ignored for now and will try and add as many of them as I can in a future PR.

Copy link
Contributor

@vaughnbetz vaughnbetz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I suggest a couple of commenting upgrades.

Found that we were regressing on many features in VTR due to tasks being
added to the appropriate test suite directory, but not being included in
the necessary task list. As such, it appeared as though the tests were
being run, but in reality they were not.

Added a script which will be run by the CI which will verify that all of
the test suites that we care about have all their tasks in the
appropriate task list.

From this tool, found many tasks which were not in the task lists.
Marked these tasks as "ignored" for now. These should be handled in a
separate PR.
@AlexandreSinger AlexandreSinger force-pushed the feature-ci-task-list-checker branch from 7604811 to 93fc0fc Compare June 10, 2025 01:01
@AlexandreSinger AlexandreSinger merged commit c1fb065 into verilog-to-routing:master Jun 10, 2025
33 checks passed
@AlexandreSinger AlexandreSinger deleted the feature-ci-task-list-checker branch June 10, 2025 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra Project Infrastructure lang-python Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants