fix(tools): update configuration file paths and directory structure (#…

…224)

* refactor: update configuration file paths and directory structure

- Modified the directory structure and file paths for configuration files in the Docker setup and application code.
- Updated the amass configuration file path to be more organized under a dedicated directory.
- Adjusted symbolic links and file handling logic to align with the new directory structure.
- Enhanced the update_config function to accommodate the new file naming conventions.

* fix: update GAU configuration file path and usage

- Updated the path and usage of the GAU configuration file across multiple files to ensure consistency and correct file referencing.
- Modified the command map in tasks.py to include the GAU configuration file path.
- Adjusted the Dockerfile to copy the GAU configuration file to the correct location and removed unnecessary symbolic link creation.
- Updated file path references in views.py and scanEngine/views.py to align with the new GAU configuration file path.

* refactor: standardize theHarvester configuration element IDs

The changes standardize the casing of the ID and name attributes for the theHarvester configuration text area across JavaScript, HTML, and Python files. This involves updating the ID from theharvester_config_text_area to theHarvester_config_text_area to ensure consistency and improve code readability.

* chore: update dependencies & fix theHarvester

- Updated several Go-based tool dependencies to their latest versions in the Celery Dockerfile.
- Modified the command in tasks.py to remove github-code sources for theHarvester due to this bug.
laramies/theHarvester#1857
- Adjusted the column layout in the form_engine.html template to improve UI responsiveness.

* fix(docker): correct s3scanner installation reference in Dockerfile

Corrected the reference for the s3scanner installation in the Celery Dockerfile to fix a typo in the commit hash.

docker/celery/Dockerfile

@@ -112,17 +112,17 @@ RUN ARCH=$(dpkg --print-architecture) \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/tomnomnom/gf@dcd4c361f9f5ba302294ed38b8ce278e8ba69006 \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/tomnomnom/[email protected] \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/tomnomnom/[email protected] \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/httpx/cmd/[email protected].0 \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/subfinder/v2/cmd/[email protected].6 \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.2.6 \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/naabu/v2/cmd/[email protected].0 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/httpx/cmd/[email protected].9 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/subfinder/v2/cmd/[email protected].7 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.3.5 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/naabu/v2/cmd/[email protected].2 \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/hakluke/hakrawler@latest \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/lc/gau/v2/cmd/[email protected].1 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/lc/gau/v2/cmd/[email protected].4 \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/owasp-amass/amass/v4/[email protected] \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/ffuf/ffuf/[email protected] \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/tlsx/cmd/[email protected].6 \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/hahwul/dalfox/[email protected].2 \
-        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/katana/cmd/[email protected].0 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/tlsx/cmd/[email protected].8 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/hahwul/dalfox/[email protected].3 \
+        && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/katana/cmd/[email protected].1 \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/dwisiswant0/crlfuzz/cmd/[email protected] \
         && GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/sa7mon/s3scanner@c544f1cf00f70cae3f2155b24d336f515b7c598b \
         && chmod 700 -R $GOPATH/pkg/* \
@@ -177,12 +177,14 @@ RUN cd $TOOLPATH/.github/OneForAll && mv /home/$USERNAME/oneforall-pyproject.tom
     cd /home/$USERNAME && poetry install
 
 # Create tools config files
-RUN nuclei -silent && naabu -version && subfinder -version && mkdir -p /home/$USERNAME/.config/theHarvester
+RUN nuclei -silent && naabu -version && subfinder -version && \
+    mkdir -p /home/$USERNAME/.config/theHarvester && \
+    mkdir -p /home/$USERNAME/.config/amass && \
+    mkdir -p /home/$USERNAME/.config/gau
 COPY --chown=$USERNAME:$USERNAME ./config/the-harvester-api-keys.yaml /home/$USERNAME/.config/theHarvester/api-keys.yaml
-COPY --chown=$USERNAME:$USERNAME ./config/amass.ini /home/$USERNAME/.config/amass.ini
-COPY --chown=$USERNAME:$USERNAME ./config/.gau.toml /home/$USERNAME/.config/.gau.toml
-RUN ln -s /home/$USERNAME/.config/.gau.toml /home/$USERNAME/.gau.toml
-
+COPY --chown=$USERNAME:$USERNAME ./config/amass.ini /home/$USERNAME/.config/amass/config.ini
+COPY --chown=$USERNAME:$USERNAME ./config/gau.toml /home/$USERNAME/.config/gau/config.toml
+RUN ln -s /home/$USERNAME/.config/theHarvester /home/$USERNAME/.theHarvester
 COPY ./entrypoint.sh /entrypoint.sh
 RUN mkdir -p /home/$USERNAME/rengine /home/$USERNAME/scan_results \
     && chown -R $USERNAME:$USERNAME /home/$USERNAME/rengine \

web/api/views.py

@@ -1324,7 +1324,7 @@ def get(self, request, format=None):
 			return Response(response)
 
 		if 'amass_config' in req.query_params:
-			path = str(Path.home() / ".config" / "amass.ini")
+			path = str(Path.home() / ".config" / "amass" / "config.ini")
 			if not os.path.exists(path):
 				run_command(f'touch {path}')
 				response['message'] = 'File Created!'
@@ -1361,7 +1361,7 @@ def get(self, request, format=None):
 			return Response(response)
 
 		if 'gau_config' in req.query_params:
-			path = str(Path.home() / ".config" / '.gau.toml')
+			path = str(Path.home() / ".config" / 'gau' / 'config.toml')
 			if not os.path.exists(path):
 				run_command(f'touch {path}')
 				response['message'] = 'File Created!'

web/reNgine/tasks.py

@@ -422,14 +422,14 @@ def subdomain_discovery(
             if tool == 'amass-passive':
                 use_amass_config = config.get(USE_AMASS_CONFIG, False)
                 cmd = f'amass enum -passive -d {host} -o ' + str(Path(self.results_dir) / 'subdomains_amass.txt')
-                cmd += (' -config ' + str(Path.home() / '.config' / 'amass.ini')) if use_amass_config else ''
+                cmd += (' -config ' + str(Path.home() / '.config' / 'amass' / 'config.ini')) if use_amass_config else ''
 
             elif tool == 'amass-active':
                 use_amass_config = config.get(USE_AMASS_CONFIG, False)
                 amass_wordlist_name = config.get(AMASS_WORDLIST, AMASS_DEFAULT_WORDLIST_NAME)
                 wordlist_path = str(Path(AMASS_DEFAULT_WORDLIST_PATH) / f'{amass_wordlist_name}.txt')
                 cmd = f'amass enum -active -d {host} -o ' + str(Path(self.results_dir) / 'subdomains_amass_active.txt')
-                cmd += (' -config ' + str(Path.home() / '.config' / 'amass.ini')) if use_amass_config else ''
+                cmd += (' -config ' + str(Path.home() / '.config' / 'amass' / 'config.ini')) if use_amass_config else ''
                 cmd += f' -brute -w {wordlist_path}'
 
             elif tool == 'sublist3r':
@@ -1027,7 +1027,7 @@ def theHarvester(config, host, scan_history_id, activity_id, results_dir, ctx={}
     output_path_json = str(Path(results_dir) / 'theHarvester.json')
     theHarvester_dir = str(Path.home() / ".config"  / 'theHarvester')
     history_file = str(Path(results_dir) / 'commands.txt')
-    cmd  = f'theHarvester -d {host} -b all -f {output_path_json}'
+    cmd  = f'theHarvester -d {host} -f {output_path_json} -b anubis,baidu,bevigil,binaryedge,bing,bingapi,bufferoverun,brave,censys,certspotter,criminalip,crtsh,dnsdumpster,duckduckgo,fullhunt,hackertarget,hunter,hunterhow,intelx,netlas,onyphe,otx,pentesttools,projectdiscovery,rapiddns,rocketreach,securityTrails,sitedossier,subdomaincenter,subdomainfinderc99,threatminer,tomba,urlscan,virustotal,yahoo,zoomeye'
 
     # Update proxies.yaml
     proxy_query = Proxy.objects.all()
@@ -1828,11 +1828,11 @@ def fetch_url(self, urls=[], ctx={}, description=None):
 
     # Initialize command map for tools
     cmd_map = {
-        'gau': f'gau',
+        'gau': 'gau --config ' + str(Path.home() / '.config' / 'gau' / 'config.toml'),
         'hakrawler': 'hakrawler -subs -u',
         'waybackurls': 'waybackurls',
-        'gospider': f'gospider --js -d 2 --sitemap --robots -w -r -a',
-        'katana': f'katana -silent -jc -kf all -d 3 -fs rdn',
+        'gospider': 'gospider --js -d 2 --sitemap --robots -w -r -a',
+        'katana': 'katana -silent -jc -kf all -d 3 -fs rdn',
     }
     if proxy:
         cmd_map['gau'] += f' --proxy "{proxy}"'

web/scanEngine/static/scanEngine/js/custom_tools.js

@@ -132,17 +132,17 @@ $("#amass_config_text_area").dblclick(function() {
 
 // get theharvester config
 $.getJSON(`/api/getFileContents?theharvester_config&format=json`, function(data) {
-  $("#theharvester_config_text_area").attr("rows", 14);
-  $("textarea#theharvester_config_text_area").html(htmlEncode(data['content']));
+  $("#theHarvester_config_text_area").attr("rows", 14);
+  $("textarea#theHarvester_config_text_area").html(htmlEncode(data['content']));
 }).fail(function(){
-  $("#theharvester_config_text_area").removeAttr("readonly");
-  $("textarea#theharvester_config_text_area").html(`# Your the Harvester configuration here.`);
+  $("#theHarvester_config_text_area").removeAttr("readonly");
+  $("textarea#theHarvester_config_text_area").html(`# Your the Harvester configuration here.`);
   $("#theHarvester-config-form").append('<input type="submit" class="btn btn-primary mt-2 float-right" value="Save Changes" id="theharvester-config-submit">');
 });
 
-$("#theharvester_config_text_area").dblclick(function() {
+$("#theHarvester_config_text_area").dblclick(function() {
   if (!document.getElementById('theharvester-config-submit')) {
-    $("#theharvester_config_text_area").removeAttr("readonly");
+    $("#theHarvester_config_text_area").removeAttr("readonly");
     $("#theharvester-config-form").append('<input type="submit" class="btn btn-primary mt-2 float-end" value="Save Changes" id="theharvester-config-submit">');
   }
 });

web/scanEngine/templates/scanEngine/_items/form_engine.html

@@ -110,7 +110,7 @@ <h5 class="text-secondary">Vulnerability Scanner&nbsp;&nbsp;&nbsp;<span class="b
       </tr>
     </table>
   </div>-->
-  <div class="col-xl-8 col-md-12 col-sm-12 col-12 mt-4">
+  <div class="col-xxl-10 col-xl-12 col-md-12 col-sm-12 col-12 mt-4">
     <h4 class="text-secondary">YAML Configurations</h4>
     <p>
       reNgine supports YAML based configurations for customizing the scan. The default settings are given below, you may choose to proceed with the default settings or configure it according to your choice.

web/scanEngine/templates/scanEngine/settings/tool.html

@@ -65,7 +65,7 @@ <h4 class="header-title">Current Subfinder Configuration</h4>
         <h4 class="header-title">Amass</h4>
         <p>
           This section lets you modify <a href="https://github.com/owasp-amass/amass/" class="text-primary" target="_blank">Amass</a> config file. You can find more information about the <a href="https://github.com/owasp-amass/amass/blob/master/examples/config.yaml" class="text-primary" target="_blank">Amass config here.</a><br>
-          <span class="text-danger">Please note that only YAML config is supported.</span>
+          <span class="text-danger">Please note that only INI config is supported.</span>
         </p>
         <form method="post" enctype="multipart/form-data" id="amass-config-form">
           {% csrf_token %}
@@ -159,7 +159,7 @@ <h4 class="header-title">theHarvester</h4>
           {% csrf_token %}
           <h4 class="header-title">Current theHarvester Configuration</h4>
           <label for="naabu_config_text_area">Double click the configuration to edit.</label>
-          <textarea class="form-control" id="theharvester_config_text_area" spellcheck="false" readonly="true" name="theharvester_config_text_area"></textarea>
+          <textarea class="form-control" id="theHarvester_config_text_area" spellcheck="false" readonly="true" name="theHarvester_config_text_area"></textarea>
         </form>
       </div>
     </div>

web/scanEngine/views.py

@@ -223,9 +223,9 @@ def handle_post_request(request):
         'nuclei_config_text_area': lambda r: update_config(r, 'nuclei', 'Nuclei'),
         'subfinder_config_text_area': lambda r: update_config(r, 'subfinder', 'Subfinder'),
         'naabu_config_text_area': lambda r: update_config(r, 'naabu', 'Naabu'),
-        'amass_config_text_area': lambda r: update_config(r, 'amass', 'Amass', '.ini'),
-        'theharvester_config_text_area': lambda r: update_config(r, 'theHarvester/api-keys', 'theHarvester'),
-        'gau_config_text_area': lambda r: update_config(r, '.gau', 'GAU', '.toml'),
+        'amass_config_text_area': lambda r: update_config(r, 'amass', 'Amass', 'config', '.ini'),
+        'theHarvester_config_text_area': lambda r: update_config(r, 'theHarvester', 'theHarvester', 'api-keys', '.yaml'),
+        'gau_config_text_area': lambda r: update_config(r, 'gau', 'GAU', 'config', '.toml'),
     }
     for key, handler in handlers.items():
         if key in request.FILES or key in request.POST:
@@ -250,8 +250,8 @@ def handle_file_upload(request, file_key, directory, expected_extension, pattern
             file.write(uploaded_file.read().decode("utf-8"))
         messages.info(request, f'{pattern_name} {uploaded_file.name[:4]} successfully uploaded')
 
-def update_config(request, tool_name, display_name, file_extension='.yaml'):
-    config_path = Path.home() / '.config' / tool_name / f'config{file_extension}'
+def update_config(request, tool_name, display_name, file_name='config', file_extension='.yaml'):
+    config_path = Path.home() / '.config' / tool_name / f'{file_name}{file_extension}'
     with open(config_path, "w", encoding='utf-8') as fhandle:
         fhandle.write(request.POST.get(f'{tool_name}_config_text_area'))
     messages.info(request, f'{display_name} config updated!')