refactor: improve tasks code structure and efficiency #280
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To help track Celery task in debug mode
This commit introduces significant improvements to the scan task management and execution within reNgine-ng. Key changes include refactoring command execution for better streaming and error handling, optimizing Celery task queues for specific scan types. These changes aim to improve performance, reliability, and resource management during scans. Additionally, the commit refines logging and debugging capabilities, and streamlines certain code sections for better maintainability.
This change improves the logic of the directory scan. The update associates discovered endpoints with the correct subdomain for more accurate results. Additionally, debugpy now waits for a client with a timeout to prevent indefinite blocking. Warnings are also now suppressed to reduce clutter in the logs.
This commit introduces several refactoring improvements to enhance code structure, logging, and functionality: - Modularization: Common functions related to database operations, logging, and utilities are moved to dedicated modules within reNgine.utils. This improves code organization and maintainability. - Logging Enhancement: The logging system is improved by introducing a custom Logger class for better control and formatting of log messages. - Command Execution: The run_command function is replaced with run_command_line, which provides better handling of command execution - Code Cleanup: Several code sections are simplified and cleaned up for better readability and efficiency. For example, string concatenation in loops is replaced with more efficient join operations. - Bug Fixes: Minor bug fixes and improvements are included, such as handling missing keys in dictionaries and correcting conditional checks. - Testing Improvements: Adjusted tests to reflect the refactoring changes and ensure continued functionality. - API Changes: Minor changes to API endpoints and serializers for consistency and clarity. Removed unused serializers and updated API documentation. - UI/UX Improvements: Minor updates to UI elements and forms for better user experience. Simplified form handling and improved error messages.
This commit introduces several refactoring improvements to enhance code structure, logging, and functionality: - Modularization: Common functions related to database operations, logging, and utilities are moved to dedicated modules within reNgine.utils. This improves code organization and maintainability. - Logging Enhancement: The logging system is improved by introducing a custom Logger class for better control and formatting of log messages. - Command Execution: The run_command function is replaced with run_command_line, which provides better handling of command execution - Code Cleanup: Several code sections are simplified and cleaned up for better readability and efficiency. For example, string concatenation in loops is replaced with more efficient join operations. - Bug Fixes: Minor bug fixes and improvements are included, such as handling missing keys in dictionaries and correcting conditional checks. - Testing Improvements: Adjusted tests to reflect the refactoring changes and ensure continued functionality. - API Changes: Minor changes to API endpoints and serializers for consistency and clarity. Removed unused serializers and updated API documentation. - UI/UX Improvements: Minor updates to UI elements and forms for better user experience. Simplified form handling and improved error messages.
Improved the handling of task configurations by allowing YAML configuration to be passed as a string or dictionary, and refactored custom header handling for better organization and flexibility. Simplified some error handling and debug logging. Updated several tasks to use the new configuration methods. Minor updates were also made to amass and OneForAll command building and shell usage in port scanning.
Updated config retrieval throughout the codebase to use the more versatile get() method instead of get_value(). Additionally, minor improvements were made to directory/file fuzzing, S3 scanner configuration, and debug functionality.
This commit enhances logging messages with descriptive emojis and prefixes, improves error handling in task execution, and fixes a bug in CMS detection. Additionally, it removes unnecessary logging of available scan engines and cleans up temporary files more reliably. Finally, it ensures that input files exist before processing.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Reviewer's Guide by SourceryThis pull request refactors the codebase to improve code structure, efficiency, and security. It includes reorganizing utility functions, optimizing string building and file handling, using parameterized commands for enhanced security, and streamlining Celery worker management and task execution. Minor updates to form handling and conditional checks further improve code clarity and maintainability. Sequence diagram for deleting a scansequenceDiagram
participant User
participant View
participant ScanHistory
participant CommandBuilder
participant CeleryTask
User->>View: Initiates delete scan
View->>ScanHistory: Retrieves ScanHistory object
View->>CommandBuilder: Creates CommandBuilder with 'rm'
CommandBuilder->>CommandBuilder: Adds '-rf' option with directory
View->>CeleryTask: run_command_line(command)
CeleryTask->>ScanHistory: Deletes ScanHistory object
ScanHistory-->>View: Success message
View-->>User: Displays success message
Updated class diagram for CommandBuilderclassDiagram
class CommandBuilder {
-command: string
-options: list
+CommandBuilder(command: string)
+add_option(option: string, value: string, condition: bool)
+add_raw_option(option: string)
+add_pipe_command(pipe_command: string)
+add_redirection(symbol: string, file: string)
+build_list(): list
+build_string(): string
}
note for CommandBuilder "This class is used to build commands in a secure way, avoiding shell injection vulnerabilities."
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This change enhances the vulnerability scanning pipeline with improved logging, more informative messages, and better task management. It also refactors command execution for increased security and flexibility, and updates the task queue for nuclei_scan to io_queue. Additionally, it includes minor updates to port scanning and task dependencies.
This change introduces a new feature to visualize the scan workflow. It generates a text-based representation of the engine's scan process, including task dependencies and parallelism, using emojis and a tree-like structure. This visualization is displayed in the logs when a scan is initiated. Additionally, minor logging improvements were made in other tasks for better clarity.
This change enhances the command execution module with improved JSON handling, streaming output support, and more robust dry run capabilities. It also refines the command builder for greater flexibility and updates task logging for better clarity. Additionally, it fixes a model import issue in the notification utility and ensures unsupported subdomain tools are handled gracefully.
psyray
changed the title
This commit refactors the command building process for various security tools, introduces a new CommandBuilder class, and enhances the mocking utilities for dry run testing. The changes improve code organization, security, and testability. Specifically, the command building logic is now more centralized and uses a safer approach for constructing commands, reducing the risk of command injection vulnerabilities. The mocking utilities are expanded to cover more tools and provide more realistic mock data, improving the effectiveness of dry run testing. Additionally, some unused imports and minor code style issues are addressed.
This commit introduces several improvements to logging, security, and dependencies: - Enhanced Logging: Improved color handling in logs, making them more readable and informative. Logs now include task names and IDs for better tracking. - Security Enhancements: The Netlas API key is now handled more securely using ephemeral environment variables. - Dependency Updates: Updated Python to 3.10.16 and several Python packages to their latest versions. Install flower with poetry and added colorama. - Standardized JSON Serialization: Implemented a utility function for consistent JSON serialization across the project. - Minor Refactoring: Updated YAML configuration and default scan engine settings for consistency and clarity. Improved handling of null values in JSON serialization. Simplified command building with a new set_env function. Corrected a few minor issues in scan and target views. - Dockerfile Improvements: Updated the Python installation process in the Dockerfile for efficiency. Removed the flower dependency installation via pipx.
This commit refactors the logging system and task categorization within the application. The changes improve code organization, readability, and provide more context in log messages. Specifically, the ANSI color codes are moved to a dedicated Colors class, and task logging now includes color-coded task categories for better visual distinction. Additionally, several log messages have been adjusted to provide more relevant information and use more appropriate log levels. Finally, the docker-compose file is updated to improve container behavior.
This commit introduces a DRY_RUN mode to reNgine-ng, allowing users to simulate scans and task execution without making actual changes or sending real requests. This is achieved by generating mock data for various tasks, enabling users to test workflows and configurations safely. The implementation includes a new get_mock_for_task function in utils/mock.py to handle mock data creation for different tasks, and modifications to the RengineTask in celery_custom_task.py to manage DRY_RUN mode execution. Several utility functions for generating mock data for specific tasks like subdomain discovery, URL fetching, OSINT, screenshots, WAF detection, directory fuzzing, and vulnerability scanning have also been added. Additionally, the nmap and scan_http_ports tasks have been updated to support DRY_RUN mode and use the new mock data. Finally, the delete_scan and delete_all_screenshots views have been improved to handle cases where results directories do not exist.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several refactoring improvements to enhance code structure, efficiency, and security. Key changes include reorganizing utility functions, optimizing string building and file handling, and using parameterized commands for enhanced security. Additionally, Celery worker management and task execution have been streamlined for better performance and debugging. Minor updates to form handling and conditional checks further improve code clarity and maintainability.