Full Docker support

.dockerignore

@@ -0,0 +1,3 @@
+__pycache__
+Dockerfile
+data

.gitignore

@@ -1 +1,2 @@
 __pycache__
+data

Dockerfile

@@ -0,0 +1,53 @@
+ARG PYTHON_VERSION=3.7
+ARG ALPINE_VERSION=3.13
+
+FROM python:${PYTHON_VERSION}-alpine${ALPINE_VERSION}
+
+WORKDIR /app
+VOLUME [ "/data" ]
+
+ENV GRAB_SITE_INTERFACE=0.0.0.0
+ENV GRAB_SITE_PORT=29000
+ENV GRAB_SITE_HOST=127.0.0.1
+EXPOSE 29000
+
+RUN apk add --no-cache \
+		git \
+		gcc \
+		libxml2-dev \
+		musl-dev \
+		libxslt-dev \
+		g++ \
+		re2-dev \
+		libffi-dev \
+		openssl-dev \
+		patch \
+		cargo \
+	&& ln -s /usr/include/libxml2/libxml /usr/include/libxml \
+	&& addgroup -S grab-site \
+	&& adduser -S -G grab-site grab-site \
+	&& chown -R grab-site:grab-site $(pwd) \
+	&& mkdir -p /data \
+	&& chown -R grab-site:grab-site /data
+
+USER grab-site:grab-site
+ENV PATH="/app:$PATH"
+ENTRYPOINT [ "entrypoint.sh" ]
+CMD [ "gs-server" ]
+
+# TODO: resolve dependencies before loading library code to take advantage of build caching
+# 	setup.py requires libgrabsite/__init__.py (__version__ property) to work
+
+COPY --chown=grab-site:grab-site . .
+
+RUN pip install . \
+	&& chmod +x entrypoint.sh
+
+WORKDIR /data
+
+# docker build -t grab-site:latest .
+# docker run --rm -it --entrypoint sh grab-site:latest
+# docker network create -d bridge grab-network
+# docker run --net=grab-network --name=gs-server -d -p 29000:29000 --restart=unless-stopped grab-site:latest
+# docker run --net=grab-network --rm -d -e GRAB_SITE_HOST=gs-server -v ./data:/data:rw grab-site:latest grab-site https://www.example.com/
+# docker run --net=grab-network --rm -d -e GRAB_SITE_HOST=gs-server -v C:\projects\grab-site\data:/data:rw grab-site:latest grab-site https://www.example.com/

README.md

@@ -42,6 +42,7 @@ please [file an issue](https://github.com/ArchiveTeam/grab-site/issues) - thank
 - [Upgrade an existing install](#upgrade-an-existing-install)
 - [Usage](#usage)
   - [`grab-site` options, ordered by importance](#grab-site-options-ordered-by-importance)
+  - [Docker](#docker)
   - [Warnings](#warnings)
   - [Tips for specific websites](#tips-for-specific-websites)
 - [Changing ignores during the crawl](#changing-ignores-during-the-crawl)
@@ -372,6 +373,155 @@ Options can come before or after the URL.
 
 *	`--help`: print help text.
 
+### Docker
+
+grab-site and gs-server can be called from Docker! Please see: [Get Docker](https://docs.docker.com/get-docker/) for more information
+
+#### Quick Start
+
+```
+# Build application & environment
+docker build -t grab-site:latest .
+
+# Create a network for inter-node communication and name resolution on linux hosts
+docker network create -d bridge grab-network
+
+# Start gs-server dashboard
+docker run --net=grab-network --name=gs-server -d -p 29000:29000 --restart=unless-stopped grab-site:latest
+
+# Create a grab-site instance that crawls a website and stores it in ./data/example.com
+# Ensure that ./data exists and container has write privileges on volume
+docker run --net=grab-network --rm -d -e GRAB_SITE_HOST=gs-server -v ./data:/data:rw grab-site:latest grab-site https://www.example.com/ --delay=100-250 --concurrency=4 --no-offsite-links
+```
+
+#### Docker Build
+
+To build the application, including all dependencies, run:
+
+```
+docker build -t grab-site:latest .
+```
+
+##### Simple container access
+
+You can use the system's shell to inspect files and run the programs:
+
+```
+docker run --rm -it --entrypoint sh grab-site:latest
+```
+
+#### Docker Network
+
+grab-site and gs-server communicate to eachother on a network that is isolated from any other container outside of a grab-site related container.
+
+The following will create a docker network called "grab-network" for our gs-server and grab-site instances to talk to (and find eachother) on.
+
+```
+docker network create -d bridge grab-network
+```
+
+We will use this "grab-network" later in `docker run` commands with the `--net="grab-network"` flag.
+
+#### Run gs-server on Docker
+
+Run a container named "gs-server" to host the dashboard and for grab-site instances to connect to:
+
+```
+docker run --net=grab-network --name=gs-server -d -p 29000:29000 --restart=unless-stopped grab-site:latest
+```
+
+The server will be running with the port forwarded (the -p parameter) from the host port 29000 -> container port 29000. You can access it via [http://localhost:29000](http://localhost:29000)
+
+##### View logs
+
+To tail the gs-server instance:
+
+```
+docker logs -f gs-server
+```
+
+##### Attach to process
+
+You can attach local STDIN/STDOUT/STDERR to your running gs-server instance:
+
+```
+docker attach gs-server
+```
+
+You can exit by using CTRL-p + CTRL+q, as documented further [here](https://docs.docker.com/engine/reference/commandline/attach/).
+
+##### Access container
+
+To enter a container with a running process (either gs-server or grab-site):
+
+```
+docker exec -it gs-server sh
+```
+
+#### Run grab-site on Docker
+
+The following commands will download example.com to a local directory, "data". This will vary slightly in the example usage, so please review your paths before executing any scripts!
+
+##### Linux host
+
+From any common shell:
+```
+docker run --net=grab-network --rm -d -e GRAB_SITE_HOST=gs-server -v ./data:/data:rw / grab-site:latest grab-site https://www.example.com/
+```
+
+##### Windows host
+
+From a PowerShell window:
+```
+docker run --net=grab-network --rm -d -e GRAB_SITE_HOST=gs-server -v C:\projects\grab-site\data:/data:rw grab-site:latest grab-site https://www.example.com/
+```
+
+Note: Windows file shares can be done several ways, this is using the legacy Windows full path volume sharing which can be "slower" if you are using WSL2.
+
+##### grab-site processes
+
+When you run a docker run with the -d flag you will get returned to you the unique ID of the container. You can use this to follow the logs.
+
+If you'd like to name your container, please specify a --name to the grab-site container you are trying to run. If you do not specify it, it will give it a funny name which you can find from here:
+
+Show all containers:
+
+```
+docker ps -a
+```
+
+##### View logs
+
+You can then use the container ID or the name here:
+
+```
+docker logs -f ead9034470ed
+```
+
+##### Attach to process
+
+You can attach local STDIN/STDOUT/STDERR to your running gs-server instance:
+
+```
+docker attach ead9034470ed
+```
+
+You can exit by using CTRL-p + CTRL+q, as documented further [here](https://docs.docker.com/engine/reference/commandline/attach/).
+
+##### Pause a crawl
+
+You can pause a crawl by using [docker pause](https://docs.docker.com/engine/reference/commandline/pause/):
+
+```
+docker pause ead9034470ed
+```
+
+Resume it using [docker unpause](https://docs.docker.com/engine/reference/commandline/unpause/):
+
+```
+docker unpause ead9034470ed
+```
+
 ### Warnings
 
 If you pay no attention to your crawls, a crawl may head down some infinite bot

entrypoint.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env sh
+set -eax
+
+# TODO set docker default parameters (if not set)
+
+exec "$@"