feat: Add 'casp reproduce project' command #5040
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the new command, allowing users to bulk-reproduce testcases for a specified project. Key changes include: - **New Command:** Added to implement the new command. - **CLI Refactoring:** was refactored to a to accommodate the new subcommand. - **Docker Utilities Enhancement:** was updated to support silent execution, environment variable passing, and log callbacks, essential for the new command's clean and efficient operation. - **Reproduction Logic:** Implemented robust logic to accurately detect successful testcase reproduction (even when fuzzers exit with non-zero codes) and ensured all verbose Docker/gcloud logs are redirected to specific log files for a clean console experience. - **Code Quality:** Removed unused imports, translated comments, and applied consistent formatting.
hunsche
force-pushed
the
fix/casp-reproduce-tool
branch
from
a99cf5f to
3d220bd
Compare
This commit introduces the flag to the command. Users can now specify the desired operating system version for the Docker environment during testcase reproduction. Supported options are , , and . - The option uses the existing immutable image. - The and options temporarily utilize their respective ClusterFuzz *base* images. This allows for OS-version-specific testing now, with the understanding that dedicated *immutable* images for these OS versions will be created and integrated in a subsequent PR.
PauloVLB
reviewed
Nov 19, 2025
Member
There was a problem hiding this comment.
Thanks for contributing to casp! I really appreciate the effort here. There are some great improvements in code quality that we should definitely look into adopting, such as the silent mode, file logging, and the structure using Click groups.
However, I don't think this feature fits into the CLI in its current form. The design goal for casp is to be a standalone package that doesn't directly depend on the ClusterFuzz source code. I noticed this implementation imports from clusterfuzz._internal.config and clusterfuzz._internal.datastore, which introduces a direct dependency we want to avoid.
Recommendation:
I suggest implementing this feature within butler.py instead. Since butler operates inside the repository, it has native access to those internal modules. If we decide to migrate this to casp later, the preferred approach would be to execute it via the immutable Docker image (similar to how the reproduce command works) rather than importing the code directly.
- Update in to use the correct repository for all OS versions. - Add support for and tags. - Improve logging in to show the full Docker image being used.
- Introduce `--project-type` arg to reproduce project command. - Replace `OS_TO_IMAGE` map with `get_image_name` function in `docker_utils.py`. - Dynamically construct image tags for legacy and ubuntu versions based on project type (external/internal/dev) and a shared base tag.
- The reproduce project command will now always execute its full logic. - Removed the --non-dry-run option definition and its conditional check.
- Rename --project-type to --project (-p) to match the instance type argument pattern. - Make --config-dir (-c) optional and default to the container config path. - Use docker_utils.prepare_docker_volumes for consistent volume management. - Ensure local Datastore access logic respects custom config paths.
- Renamed the `--project` argument to `--environment` in both `reproduce testcase` and `reproduce project` commands to clarify that it refers to the ClusterFuzz instance type (e.g., external, internal, dev). - Updated variable names and help text to reflect this change.
- Appended 'OS' to the OS version in the start reproduction log message for better clarity (e.g., 'legacy OS').
- Changed the default value of the option to 10 in . - Added a comprehensive summary section after testcase reproduction, detailing success/failure counts and percentages.
- Filter out testcases already marked as 'Unreproducible' in Datastore. - Update the summary output to include the count and percentage of skipped testcases.
- Combined 'Found X open testcases' and 'Skipped X testcases' into a single line for clarity. - Removed the redundant skipped testcases message. - Ensured 'Starting reproduction' message accurately reflects the number of testcases to be run.
This change refines the summary output of the command. - Success and failure rates are now calculated based solely on the number of *attempted* testcases, excluding those that were skipped (unreproducible or one-time crashers). - The percentage for skipped testcases has been removed, with only their count being displayed. - Added Apache 2.0 license header and module docstring. - Removed unused 'run_command_success' variable assignment.
…reproduce_project command.
…, robust symlink management, and reduced reproduction iterations.
…rdize output to logger, and ensure consistent run ID generation.
…ssing, and async test case counting, and update existing migration tools.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Try to find PR URL in output | ||
| pr_url = None | ||
| for line in output.splitlines(): | ||
| if 'github.com' in line and '/pull/' in line: |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the new
casp reproduce projectcommand, allowing users to bulk-reproduce testcases for a specified project.Key changes include:
cli/casp/src/casp/commands/reproduce_project.pyto implement the new command.cli/casp/src/casp/commands/reproduce.pywas refactored to aclick.groupto accommodate the newprojectsubcommand.cli/casp/src/casp/utils/docker_utils.pywas updated to support silent execution, environment variable passing, and log callbacks, essential for the new command's clean and efficient operation.