fix(scan): first endpoint detection for IP

[psyray]

Fix #250
Fix #217

Fix first endpoint detection by IPs

• Scan by hostname
• Scan by IP

Summary by Sourcery

Improve handling of IP address scans by limiting the available tasks and prioritizing the creation of endpoints for the IP itself, even when rDNS is available.

New Features:

• Display the number of subdomains associated with each IP and port in their respective badges.

Bug Fixes:

• Fix an issue where IP addresses were not being properly scanned.

Enhancements:

• Limit the tasks performed during an IP address scan to only essential ones.
• Prioritize creating an endpoint for the IP address itself during scans.
• Improve port information gathering during scans by including service details and uncommon web ports.
• Update port service information with nmap service detection results.
• Standardize port information by ensuring only one entry exists per port number.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request enhances the handling of IP address scans by limiting the tasks and correctly creating the first endpoint.

Sequence diagram for IP scan endpoint detection

sequenceDiagram
    participant Scanner
    participant TaskManager
    participant EndpointManager
    participant Database

    Scanner->>TaskManager: initiate_scan(domain)
    TaskManager->>TaskManager: Check if IP scan
    alt is IP scan
        TaskManager->>TaskManager: Filter allowed tasks
        Note over TaskManager: Only allow: port_scan, fetch_url,<br/>dir_file_fuzz, vulnerability_scan,<br/>screenshot, waf_detection
    end

    TaskManager->>EndpointManager: create_first_endpoint_from_nmap_data()
    alt is IP scan
        EndpointManager->>EndpointManager: Create IP endpoint
        EndpointManager->>Database: Save IP endpoint
        opt rDNS hostname exists
            EndpointManager->>EndpointManager: Create rDNS endpoint
            EndpointManager->>Database: Save rDNS endpoint
        end
    else is domain scan
        EndpointManager->>EndpointManager: Create domain endpoint
        EndpointManager->>Database: Save domain endpoint
    end

Loading

Class diagram for endpoint and domain relationships

classDiagram
    class EndPoint {
        +scan_history: ScanHistory
        +target_domain: Domain
        +http_url: String
        +is_default: Boolean
        +discovered_date: DateTime
        +http_status: Integer
    }

    class Domain {
        +name: String
        +last_scan_date: DateTime
    }

    class Subdomain {
        +scan_history: ScanHistory
        +target_domain: Domain
        +name: String
        +discovered_date: DateTime
    }

    Domain "1" -- "*" EndPoint
    Domain "1" -- "*" Subdomain
    Subdomain "1" -- "*" EndPoint

    note for EndPoint "Added http_status field"
    note for Domain "Enhanced IP validation"

Loading

Flow diagram for endpoint creation logic

flowchart TD
    A[Start Endpoint Creation] --> B{Is IP Scan?}
    B -->|Yes| C[Filter Allowed Tasks]
    B -->|No| D[Regular Domain Tasks]

    C --> E{Process Hosts Data}
    E -->|IP Address| F[Create IP Endpoint]
    E -->|rDNS Exists| G[Create rDNS Endpoint]

    D --> H[Create Domain Endpoint]
    H --> I[Try Crawling]
    I -->|Success| J[Save with Crawl Data]
    I -->|Failure| K[Save without Crawling]

    F --> L[Save Metadata]
    G --> L
    J --> L
    K --> L
    L --> M[End]

Loading

File-Level Changes

Change	Details	Files
Limit tasks for IP scans	Added a check to identify if the scan target is an IP address. Filtered the list of tasks to be executed for IP scans, allowing only 'port_scan', 'fetch_url', 'dir_file_fuzz', 'vulnerability_scan', 'screenshot', and 'waf_detection'.	web/reNgine/tasks.py
Improve endpoint creation for IP scans	Modified the save_endpoint function to handle IP scans correctly, skipping domain validation for IP addresses. Added http_status parameter to save_endpoint function. Modified create_first_endpoint_from_nmap_data to create endpoints for both the IP and rDNS hostnames when scanning an IP address. Added logic to create an endpoint for the IP itself if it's not present in the Nmap data, using rDNS data if available. Modified save_subdomain to validate both domain and IP formats. Modified save_subdomain to not validate subdomain against domain for IP scans.	web/reNgine/tasks.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#250	First endpoint should be correctly detected for hostname or IP	✅
#250	Fix scan by IP not working well	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[AnonymousWP]

Please solve all comments, then I'll review.

[psyray]

Please solve all comments, then I'll review.

Which ones ?
They are all solved for me

[AnonymousWP]

Please solve all comments, then I'll review.

Which ones ? They are all solved for me

Hmm, on phone it said they weren't resolved, perhaps a bug. On PC it does indeed show they're solved. Will review later.

[0b3ud]

I have finished testing and reviewing this PR and it works as expected
No bugs detected