Skip to content

Commit e6bf5c5

Browse files
psyraypsyray
authored
fix(tools): update configuration file paths and directory structure (#224)
* refactor: update configuration file paths and directory structure - Modified the directory structure and file paths for configuration files in the Docker setup and application code. - Updated the amass configuration file path to be more organized under a dedicated directory. - Adjusted symbolic links and file handling logic to align with the new directory structure. - Enhanced the update_config function to accommodate the new file naming conventions. * fix: update GAU configuration file path and usage - Updated the path and usage of the GAU configuration file across multiple files to ensure consistency and correct file referencing. - Modified the command map in tasks.py to include the GAU configuration file path. - Adjusted the Dockerfile to copy the GAU configuration file to the correct location and removed unnecessary symbolic link creation. - Updated file path references in views.py and scanEngine/views.py to align with the new GAU configuration file path. * refactor: standardize theHarvester configuration element IDs The changes standardize the casing of the ID and name attributes for the theHarvester configuration text area across JavaScript, HTML, and Python files. This involves updating the ID from theharvester_config_text_area to theHarvester_config_text_area to ensure consistency and improve code readability. * chore: update dependencies & fix theHarvester - Updated several Go-based tool dependencies to their latest versions in the Celery Dockerfile. - Modified the command in tasks.py to remove github-code sources for theHarvester due to this bug. laramies/theHarvester#1857 - Adjusted the column layout in the form_engine.html template to improve UI responsiveness. * fix(docker): correct s3scanner installation reference in Dockerfile Corrected the reference for the s3scanner installation in the Celery Dockerfile to fix a typo in the commit hash.
1 parent 660ad44 commit e6bf5c5

File tree

8 files changed

+37
-35
lines changed

8 files changed

+37
-35
lines changed

‎docker/celery/Dockerfile

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -112,17 +112,17 @@ RUN ARCH=$(dpkg --print-architecture) \
112112
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/tomnomnom/gf@dcd4c361f9f5ba302294ed38b8ce278e8ba69006 \
113113
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/tomnomnom/[email protected] \
114114
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/tomnomnom/[email protected] \
115-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/httpx/cmd/[email protected].0 \
116-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/subfinder/v2/cmd/[email protected].6 \
117-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.2.6 \
118-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/naabu/v2/cmd/[email protected].0 \
115+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/httpx/cmd/[email protected].9 \
116+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/subfinder/v2/cmd/[email protected].7 \
117+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.3.5 \
118+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/naabu/v2/cmd/[email protected].2 \
119119
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/hakluke/hakrawler@latest \
120-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/lc/gau/v2/cmd/[email protected].1 \
120+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/lc/gau/v2/cmd/[email protected].4 \
121121
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/owasp-amass/amass/v4/[email protected] \
122122
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/ffuf/ffuf/[email protected] \
123-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/tlsx/cmd/[email protected].6 \
124-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/hahwul/dalfox/[email protected].2 \
125-
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/katana/cmd/[email protected].0 \
123+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/tlsx/cmd/[email protected].8 \
124+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/hahwul/dalfox/[email protected].3 \
125+
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/projectdiscovery/katana/cmd/[email protected].1 \
126126
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/dwisiswant0/crlfuzz/cmd/[email protected] \
127127
&& GOARCH=$ARCH go install -ldflags="-s -w" -v github.com/sa7mon/s3scanner@c544f1cf00f70cae3f2155b24d336f515b7c598b \
128128
&& chmod 700 -R $GOPATH/pkg/* \
@@ -177,12 +177,14 @@ RUN cd $TOOLPATH/.github/OneForAll && mv /home/$USERNAME/oneforall-pyproject.tom
177177
cd /home/$USERNAME && poetry install
178178

179179
# Create tools config files
180-
RUN nuclei -silent && naabu -version && subfinder -version && mkdir -p /home/$USERNAME/.config/theHarvester
180+
RUN nuclei -silent && naabu -version && subfinder -version && \
181+
mkdir -p /home/$USERNAME/.config/theHarvester && \
182+
mkdir -p /home/$USERNAME/.config/amass && \
183+
mkdir -p /home/$USERNAME/.config/gau
181184
COPY --chown=$USERNAME:$USERNAME ./config/the-harvester-api-keys.yaml /home/$USERNAME/.config/theHarvester/api-keys.yaml
182-
COPY --chown=$USERNAME:$USERNAME ./config/amass.ini /home/$USERNAME/.config/amass.ini
183-
COPY --chown=$USERNAME:$USERNAME ./config/.gau.toml /home/$USERNAME/.config/.gau.toml
184-
RUN ln -s /home/$USERNAME/.config/.gau.toml /home/$USERNAME/.gau.toml
185-
185+
COPY --chown=$USERNAME:$USERNAME ./config/amass.ini /home/$USERNAME/.config/amass/config.ini
186+
COPY --chown=$USERNAME:$USERNAME ./config/gau.toml /home/$USERNAME/.config/gau/config.toml
187+
RUN ln -s /home/$USERNAME/.config/theHarvester /home/$USERNAME/.theHarvester
186188
COPY ./entrypoint.sh /entrypoint.sh
187189
RUN mkdir -p /home/$USERNAME/rengine /home/$USERNAME/scan_results \
188190
&& chown -R $USERNAME:$USERNAME /home/$USERNAME/rengine \

‎docker/celery/config/.gau.toml renamed to ‎docker/celery/config/gau.toml

File renamed without changes.

‎web/api/views.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1324,7 +1324,7 @@ def get(self, request, format=None):
13241324
return Response(response)
13251325

13261326
if 'amass_config' in req.query_params:
1327-
path = str(Path.home() / ".config" / "amass.ini")
1327+
path = str(Path.home() / ".config" / "amass" / "config.ini")
13281328
if not os.path.exists(path):
13291329
run_command(f'touch {path}')
13301330
response['message'] = 'File Created!'
@@ -1361,7 +1361,7 @@ def get(self, request, format=None):
13611361
return Response(response)
13621362

13631363
if 'gau_config' in req.query_params:
1364-
path = str(Path.home() / ".config" / '.gau.toml')
1364+
path = str(Path.home() / ".config" / 'gau' / 'config.toml')
13651365
if not os.path.exists(path):
13661366
run_command(f'touch {path}')
13671367
response['message'] = 'File Created!'

‎web/reNgine/tasks.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -422,14 +422,14 @@ def subdomain_discovery(
422422
if tool == 'amass-passive':
423423
use_amass_config = config.get(USE_AMASS_CONFIG, False)
424424
cmd = f'amass enum -passive -d {host} -o ' + str(Path(self.results_dir) / 'subdomains_amass.txt')
425-
cmd += (' -config ' + str(Path.home() / '.config' / 'amass.ini')) if use_amass_config else ''
425+
cmd += (' -config ' + str(Path.home() / '.config' / 'amass' / 'config.ini')) if use_amass_config else ''
426426

427427
elif tool == 'amass-active':
428428
use_amass_config = config.get(USE_AMASS_CONFIG, False)
429429
amass_wordlist_name = config.get(AMASS_WORDLIST, AMASS_DEFAULT_WORDLIST_NAME)
430430
wordlist_path = str(Path(AMASS_DEFAULT_WORDLIST_PATH) / f'{amass_wordlist_name}.txt')
431431
cmd = f'amass enum -active -d {host} -o ' + str(Path(self.results_dir) / 'subdomains_amass_active.txt')
432-
cmd += (' -config ' + str(Path.home() / '.config' / 'amass.ini')) if use_amass_config else ''
432+
cmd += (' -config ' + str(Path.home() / '.config' / 'amass' / 'config.ini')) if use_amass_config else ''
433433
cmd += f' -brute -w {wordlist_path}'
434434

435435
elif tool == 'sublist3r':
@@ -1027,7 +1027,7 @@ def theHarvester(config, host, scan_history_id, activity_id, results_dir, ctx={}
10271027
output_path_json = str(Path(results_dir) / 'theHarvester.json')
10281028
theHarvester_dir = str(Path.home() / ".config" / 'theHarvester')
10291029
history_file = str(Path(results_dir) / 'commands.txt')
1030-
cmd = f'theHarvester -d {host} -b all -f {output_path_json}'
1030+
cmd = f'theHarvester -d {host} -f {output_path_json} -b anubis,baidu,bevigil,binaryedge,bing,bingapi,bufferoverun,brave,censys,certspotter,criminalip,crtsh,dnsdumpster,duckduckgo,fullhunt,hackertarget,hunter,hunterhow,intelx,netlas,onyphe,otx,pentesttools,projectdiscovery,rapiddns,rocketreach,securityTrails,sitedossier,subdomaincenter,subdomainfinderc99,threatminer,tomba,urlscan,virustotal,yahoo,zoomeye'
10311031

10321032
# Update proxies.yaml
10331033
proxy_query = Proxy.objects.all()
@@ -1828,11 +1828,11 @@ def fetch_url(self, urls=[], ctx={}, description=None):
18281828

18291829
# Initialize command map for tools
18301830
cmd_map = {
1831-
'gau': f'gau',
1831+
'gau': 'gau --config ' + str(Path.home() / '.config' / 'gau' / 'config.toml'),
18321832
'hakrawler': 'hakrawler -subs -u',
18331833
'waybackurls': 'waybackurls',
1834-
'gospider': f'gospider --js -d 2 --sitemap --robots -w -r -a',
1835-
'katana': f'katana -silent -jc -kf all -d 3 -fs rdn',
1834+
'gospider': 'gospider --js -d 2 --sitemap --robots -w -r -a',
1835+
'katana': 'katana -silent -jc -kf all -d 3 -fs rdn',
18361836
}
18371837
if proxy:
18381838
cmd_map['gau'] += f' --proxy "{proxy}"'

‎web/scanEngine/static/scanEngine/js/custom_tools.js

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -132,17 +132,17 @@ $("#amass_config_text_area").dblclick(function() {
132132

133133
// get theharvester config
134134
$.getJSON(`/api/getFileContents?theharvester_config&format=json`, function(data) {
135-
$("#theharvester_config_text_area").attr("rows", 14);
136-
$("textarea#theharvester_config_text_area").html(htmlEncode(data['content']));
135+
$("#theHarvester_config_text_area").attr("rows", 14);
136+
$("textarea#theHarvester_config_text_area").html(htmlEncode(data['content']));
137137
}).fail(function(){
138-
$("#theharvester_config_text_area").removeAttr("readonly");
139-
$("textarea#theharvester_config_text_area").html(`# Your the Harvester configuration here.`);
138+
$("#theHarvester_config_text_area").removeAttr("readonly");
139+
$("textarea#theHarvester_config_text_area").html(`# Your the Harvester configuration here.`);
140140
$("#theHarvester-config-form").append('<input type="submit" class="btn btn-primary mt-2 float-right" value="Save Changes" id="theharvester-config-submit">');
141141
});
142142

143-
$("#theharvester_config_text_area").dblclick(function() {
143+
$("#theHarvester_config_text_area").dblclick(function() {
144144
if (!document.getElementById('theharvester-config-submit')) {
145-
$("#theharvester_config_text_area").removeAttr("readonly");
145+
$("#theHarvester_config_text_area").removeAttr("readonly");
146146
$("#theharvester-config-form").append('<input type="submit" class="btn btn-primary mt-2 float-end" value="Save Changes" id="theharvester-config-submit">');
147147
}
148148
});

‎web/scanEngine/templates/scanEngine/_items/form_engine.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ <h5 class="text-secondary">Vulnerability Scanner&nbsp;&nbsp;&nbsp;<span class="b
110110
</tr>
111111
</table>
112112
</div>-->
113-
<div class="col-xl-8 col-md-12 col-sm-12 col-12 mt-4">
113+
<div class="col-xxl-10 col-xl-12 col-md-12 col-sm-12 col-12 mt-4">
114114
<h4 class="text-secondary">YAML Configurations</h4>
115115
<p>
116116
reNgine supports YAML based configurations for customizing the scan. The default settings are given below, you may choose to proceed with the default settings or configure it according to your choice.

‎web/scanEngine/templates/scanEngine/settings/tool.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ <h4 class="header-title">Current Subfinder Configuration</h4>
6565
<h4 class="header-title">Amass</h4>
6666
<p>
6767
This section lets you modify <a href="https://github.com/owasp-amass/amass/" class="text-primary" target="_blank">Amass</a> config file. You can find more information about the <a href="https://github.com/owasp-amass/amass/blob/master/examples/config.yaml" class="text-primary" target="_blank">Amass config here.</a><br>
68-
<span class="text-danger">Please note that only YAML config is supported.</span>
68+
<span class="text-danger">Please note that only INI config is supported.</span>
6969
</p>
7070
<form method="post" enctype="multipart/form-data" id="amass-config-form">
7171
{% csrf_token %}
@@ -159,7 +159,7 @@ <h4 class="header-title">theHarvester</h4>
159159
{% csrf_token %}
160160
<h4 class="header-title">Current theHarvester Configuration</h4>
161161
<label for="naabu_config_text_area">Double click the configuration to edit.</label>
162-
<textarea class="form-control" id="theharvester_config_text_area" spellcheck="false" readonly="true" name="theharvester_config_text_area"></textarea>
162+
<textarea class="form-control" id="theHarvester_config_text_area" spellcheck="false" readonly="true" name="theHarvester_config_text_area"></textarea>
163163
</form>
164164
</div>
165165
</div>

‎web/scanEngine/views.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -223,9 +223,9 @@ def handle_post_request(request):
223223
'nuclei_config_text_area': lambda r: update_config(r, 'nuclei', 'Nuclei'),
224224
'subfinder_config_text_area': lambda r: update_config(r, 'subfinder', 'Subfinder'),
225225
'naabu_config_text_area': lambda r: update_config(r, 'naabu', 'Naabu'),
226-
'amass_config_text_area': lambda r: update_config(r, 'amass', 'Amass', '.ini'),
227-
'theharvester_config_text_area': lambda r: update_config(r, 'theHarvester/api-keys', 'theHarvester'),
228-
'gau_config_text_area': lambda r: update_config(r, '.gau', 'GAU', '.toml'),
226+
'amass_config_text_area': lambda r: update_config(r, 'amass', 'Amass', 'config', '.ini'),
227+
'theHarvester_config_text_area': lambda r: update_config(r, 'theHarvester', 'theHarvester', 'api-keys', '.yaml'),
228+
'gau_config_text_area': lambda r: update_config(r, 'gau', 'GAU', 'config', '.toml'),
229229
}
230230
for key, handler in handlers.items():
231231
if key in request.FILES or key in request.POST:
@@ -250,8 +250,8 @@ def handle_file_upload(request, file_key, directory, expected_extension, pattern
250250
file.write(uploaded_file.read().decode("utf-8"))
251251
messages.info(request, f'{pattern_name} {uploaded_file.name[:4]} successfully uploaded')
252252

253-
def update_config(request, tool_name, display_name, file_extension='.yaml'):
254-
config_path = Path.home() / '.config' / tool_name / f'config{file_extension}'
253+
def update_config(request, tool_name, display_name, file_name='config', file_extension='.yaml'):
254+
config_path = Path.home() / '.config' / tool_name / f'{file_name}{file_extension}'
255255
with open(config_path, "w", encoding='utf-8') as fhandle:
256256
fhandle.write(request.POST.get(f'{tool_name}_config_text_area'))
257257
messages.info(request, f'{display_name} config updated!')

0 commit comments

Comments
 (0)