Harden Docker Compose: credential security, sqlite support, profiles, and AMASS_LOGLEVEL#13
Open
Dude4Linux wants to merge 12 commits into
Open
Harden Docker Compose: credential security, sqlite support, profiles, and AMASS_LOGLEVEL#13Dude4Linux wants to merge 12 commits into
Dude4Linux wants to merge 12 commits into
Conversation
- Replace tracked .env with .env.template and .gitignore to keep credentials, config files, and data directories out of version control - Add config-init one-shot service using Docker secrets to source .env credentials and prepare config.yaml (DB selection + placeholder substitution) before any client service starts - Add config-init dependency to all 5 client services (enum, viz, subs, assoc, track) - Add Neo4j initial database name env var and DB_SERVER switching between PostgreSQL and Neo4j - Rewrite README: streamlined setup instructions, Neo4j Browser section, Usage section with examples for all subcommands, shell function tips for Linux/macOS/Windows, and simplified update steps Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pin image versions, replace no-op healthchecks with real checks, bind database ports to localhost, drop all capabilities (adding back only what each service needs), add log rotation, enforce resource limits on all services, and document port binding in README. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add sqlite as a zero-dependency database option alongside postgres and
neo4j. Propagate AMASS_LOGLEVEL from .env into all amass containers so
users can control log verbosity (INFO/WARN/ERROR/DEBUG) with a single
uncommented line.
config/config-init.sh
- Resolve DB_SERVER once upfront; default to sqlite when unset
- Add explicit sqlite branch that comments out all network DB lines
(session.go falls through to its built-in SQLite when GraphDBs is nil)
- Gate credential substitution on non-sqlite backends to avoid set -u
failures when AMASS_USER/PASSWORD/DB are absent from the environment
compose.yaml
- Add AMASS_LOGLEVEL=${AMASS_LOGLEVEL:-} to engine, enum, viz, subs,
assoc, and track so the variable flows from .env into each container
.env.template
- Document sqlite/postgres/neo4j options; postgres remains the default
- Add commented-out AMASS_LOGLEVEL=INFO with level choices
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds Docker Compose profiles so the heavyweight database services only
start when explicitly requested, making sqlite-only deployments (such as
on a Raspberry Pi) work with a plain `docker compose up -d`.
compose.yaml
- assetdb assigned profile 'postgres'
- neo4j assigned profile 'neo4j'
- engine, viz, subs, assoc, track: remove assetdb/neo4j from depends_on
(engine depends only on postal + syslog; clients depend only on
syslog + config-init — the user is responsible for activating the
right profile before running DB-backed tools)
.env.template
- Document the three start commands alongside each DB_SERVER option
- Add commented-out COMPOSE_PROFILES=postgres as a reminder to keep
DB_SERVER and COMPOSE_PROFILES in sync
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
COMPOSE_PROFILES already controls which database service starts; having a
separate DB_SERVER variable that must be kept in sync was error-prone.
config-init.sh now reads COMPOSE_PROFILES directly, making .env a single
control point for the entire stack.
config/config-init.sh
- Replace if/elif/else on DB_SERVER with case match on COMPOSE_PROFILES
- Wrap value in commas (,${COMPOSE_PROFILES:-},) to match whole profile
names — unrelated profiles (tor, mcp, …) are safely ignored
- Credential substitution co-located with each database branch
.env.template
- Remove DB_SERVER; COMPOSE_PROFILES is the only database selector
- Sectioned layout with whitespace for readability
- Credentials moved to bottom under their own header
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Hardcoded /home/john/Docker/amass-test caused all benchmark scripts to fail when run from any other host or user. Use dirname-based resolution so scripts work wherever the repo is cloned. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runs only Mode C' (NLP only) and Mode C (brute + NLP), reusing existing Mode A/B results and ground_truth.txt from a previous benchmark.sh run. Regenerates the full report at the end. Usage: benchmark/run_nlp_modes.sh -d <domain> [-t <timeout_min>] Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…VERRIDE from clients - config-init.sh: add escape_sed() to escape backslash/pipe/ampersand in sed replacement strings, preventing password injection into config.yaml - config-init.sh: fail fast if AMASS_PASSWORD or POSTGRES_PASSWORD equals the .env.template default 'ChangeMe!' before any config files are touched - config-init.sh: replace /tmp/config.yaml with mktemp + EXIT trap to avoid predictable temp filename collision - compose.yaml: add x-client-hardened anchor (cap_drop ALL, no cap_add) for stateless client services (enum, viz, subs, assoc, track) that don't need DAC_OVERRIDE; engine, postal, config-init retain the full anchor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Client containers default to localhost:4000 for the engine GraphQL endpoint, which resolves to the container's own loopback and fails. Set AMASS_ENGINE_HOST=engine on enum, viz, subs, assoc, and track so they resolve the engine container by name via Docker's internal DNS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two config-init.sh fixes for upstream amass compatibility: 1. Strip the /graphql suffix from the engine URL: upstream amass migrated from GraphQL to a REST API at /api/v1/*. The template may carry the legacy /graphql path; normalize it to the base URL. 2. chmod 644 config.yaml after writing: config-init runs as root so the file is created 0600 root:root. Client containers run as uid=1000 and silently fall back to SQLite defaults when they cannot read it. 644 lets the service user read without write. Also fix the config.yaml template engine URL from the legacy GraphQL path to the current REST base URL. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two neo4j-related fixes:
1. Remove NEO4J_initial_dbms_default__database from the neo4j service
environment. Setting this to ${AMASS_DB} (= assetdb) prevents the
neo4j-go-driver's VerifyConnectivity from working because the driver
checks for the default "neo4j" database during routing table resolution.
Using neo4j's standard default database avoids this startup failure.
2. Hardcode "neo4j" as the database name in the bolt:// URL template.
The neo4j database is always named "neo4j" with this setup; ${AMASS_DB}
is only meaningful for postgres.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a bats-based unit test suite (23 tests) covering all config-init.sh logic — credential guards, DB backend selection, credential substitution, special-character escaping, profile exactness, and /graphql stripping — with no Docker required. Adds integration smoke tests for all three DB backends (SQLite, postgres, neo4j) and flag tests for -brute, -alts, and -active, each verified by querying the resulting asset.db or network DB. Also patches config-init.sh with SECRETS_FILE and CONFIG_FILE env overrides to enable host-side unit testing, removes stale benchmark/ scripts and config/proxy.env artifacts, and updates .gitignore to exclude runtime DB files (config/asset.db*). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
config/config.yaml. A newconfig-initservice (Alpine one-shot container) substitutes${AMASS_USER},${AMASS_PASSWORD}, and${AMASS_DB}from.envat startup via Docker secrets, so secrets never live in the config file.postgresnorneo4jprofile is active, the engine falls back to its built-in SQLite. No database server required for simple deployments.assetdb(PostgreSQL) andneo4jservices now useprofiles: [postgres]andprofiles: [neo4j]respectively, so they only start when explicitly requested. Default (docker compose up -d) runs engine + postal + syslog only.DB_SERVERenv var — settingCOMPOSE_PROFILES=postgresboth starts theassetdbservice and configuresconfig.yamlto use PostgreSQL. No two variables to keep in sync.AMASS_LOGLEVELenvironment variable (INFO/WARN/ERROR/DEBUG) threaded into all six Amass containers viacompose.yaml. Replaces a binary debug flag with fine-grained control.cap_drop: [ALL],security_opt: [no-new-privileges:true], capped resource limits, and structured JSON logging with rotation via YAML anchors..env.template: Ships a documented template with sensible defaults;.envis gitignored.Rebase note
This PR supersedes #12 and is rebased onto the current
developtip (commit784d0c9), incorporating upstream changes to build refs (#develop), the new libpostal repo URL, and the engine URL fix.Test plan
docker compose up -dstarts engine + postal + syslog with SQLite (no DB server needed)docker compose --profile postgres up -dstarts assetdb,config-initsubstitutes postgres credentials intoconfig.yamldocker compose --profile neo4j up -dstarts neo4j,config-initsubstitutes bolt credentialsdocker compose run --rm enum -d owasp.orgdiscovers subdomainsAMASS_LOGLEVEL=DEBUG docker compose up -dproduces debug output in container logs🤖 Generated with Claude Code