Backing up a database via a docker command and piping result

[OTheNonE]

Hello!

I have this bash script that backups a SurrealDB database:

#!/usr/bin/env bash

# Some safety features
set -euo pipefail

# Move to and get current directory.
CURRENT_DIRECTORY="$(dirname "$(realpath "$0")")"
cd "$CURRENT_DIRECTORY" || exit 1

# Load environment variables into the current scope.
set -a
source .env
set +a

# Backup database and pipe the content into restic backup.
docker run \
        --rm  \
        --env-file .env \
        surrealdb/surrealdb:v2 \
        export --endpoint "$DATABASE_ENDPOINT" \
| restic backup \
        --stdin \
        --stdin-filename kristnastovafc-backup.surql

# Forget certain backups.
restic forget \
        --keep-daily $RESTIC_KEEP_DAILY \
        --keep-weekly $RESTIC_KEEP_WEEKLY \
        --keep-monthly $RESTIC_KEEP_MONTHLY \
        --prune

echo "Backup completed successfully"

This script backups the database via the surreal export command, and pipes the data to the restic backup command via the --stdin and --stdin-filename flags. The surreal export commands requires the surreal cli, and is therefore run with docker.

How would i migrate this to Backrest? Is it even possible? I'll surely need a CONDITION_SNAPSHOT_START hook, but the two problems i see, are:

• How do i run a docker command in another docker container?
• and how do i pipe the result to the Restic commands used by Backrest? (I might have to store the file temporarily in the Backrest container).

[edoardo-c]

Hi @OTheNonE ,

Would the solution not simply be, to write to file rather than pipe? That way you can share the file between containers. I get how piping to restic allows you to not have to write to the filesystem but doing so would also allow you to add checks to make sure the export was successful.

Hope this makes sense.

[OTheNonE]

Yeah, SurrealDB would run in another container, although it would be a "start container -> run -> stop container"-thing.

Are you suggesting that i run the Surrealdb container separately, therefore not using the docker socket mount? How do i trigger the start of the database backup? The only way i currently see restic backup and the database backup running simultaneously in Backrest is if Backrest can access the docker socket and start the database backup container.

[edoardo-c]

Well, if Backrest is also a container, it's going to be hard for it to start the SurrealDB export container. I would have the underlying OS start the script that will dump the database and Backrest will back it up later in the day.

Underlying OS starts script at 1AM
Backrest runs a backup at 1:30AM

The setup would be that both containers mount the underlying OS directory "/surrealdb/dump" in the respective containers so that surrealdb can export the data to the mounted volume and Backrest can then back it up.

[edoardo-c]

you could also use CONDITION_SNAPSHOT_START in Backrest to check for the existence of a flag that would validate that the dump was successful.

[OTheNonE]

This already complicates matters to a degree that i'll rather stay with the script outside of Backrest. I'd prefer that either all configuration stays within Backrest, or within the OS in a simple cron job, otherwise one big point of using Backrest is gone.

[edoardo-c]

Sounds reasonable to me. Note however that I am not an expert. Someone might have another idea, but your script + cronjob seems to have you covered.