Do we have a rustic forget or rustic prune option to turn the repository to "append-only" mode?

[s-leroux]

From https://rustic.cli.rs/docs/commands/forget/remove_by_policy.html:

With rustic forget, I can use --keep-last n to keep the most recent snapshots, potentially leaving some of the oldest data packs orphaned and subject to pruning.

But, do we have a rustic forget or rustic prune option to mostly turn the repository to "append-only" mode?

Motivation

I have the interesting use case of backing up large amounts of AVCHD footage.

Data files are never updated, rarely deleted, but occasionally moved around—for example, to transfer old footage to the archive/ subdirectory. Here comes the bit rot problem. Imagine the following scenario:

1. We recorded videos for a customer. The project is alive, and the footage is stored in the project's directory.
2. This is backed up as part of our scheduled rustic backup
3. Bit rot happens. The upcoming rustic backup will not detect this since the file's metadata remains unaffected.
4. The problem goes unnoticed, and the project is closed. The footage is then moved to the archive directory.
5. In the next backup, rustic backup will detect a new file and back up its data, including the rotten bits. It also detects the removal of the data file from its old location.
6. In my understanding, in an upcoming rustic prune command, the original, unaltered data will definitely be removed from the storage, and only the altered data will remain.

It would be great to have an option to ensure we never remove the initial (hopefully intact) backup of a data file. If we detect the rotten bit issue at a later point, we will still have the original content to restore the damaged file.

Ideas?

This may be as simple as a config option that forbids pruning a repository, or as complicated as keeping all snapshots that added a file (so the original version of each file will never be forgotten).

Was something like that already discussed?

[kapitainsky]

It would be great to have an option to ensure we never remove the initial (hopefully intact) backup of a data file. If we detect the rotten bit issue at a later point, we will still have the original content to restore the damaged file.

$ rustic forget --help
...
      --keep-tags <TAG[,TAG,..]>
          Keep snapshots with this taglist (can be specified multiple times)

      --keep-id <ID>
          Keep snapshots ids that start with ID (can be specified multiple times)
...

For append only solution what you really need is:

https://github.com/rustic-rs/rustic_server

It is in "early development" though.

Restic is more battle tested here: https://github.com/restic/rest-server

[s-leroux]

For append only solution what you really need is:

https://github.com/rustic-rs/rustic_server

It is in "early development" though.

Restic is more battle tested here: https://github.com/restic/rest-server

Thank you for these links. From the doc (https://rustic.cli.rs/docs/commands/forget/security_considerations_append_only.html) I also learned about rclone. All isn't clear in my mind, but from my understanding, this level of indirection would be beneficial in mitigating the risk of compromised backup clients, albeit at the cost of increased complexity.

[s-leroux]

Oh, yes. After writing the initial message, I thought about using a tag to mark the first backup after we upload the videos for a new project, and add the corresponding keep-tags in the config.

[aawsome]

In addition to using some of the --keep-* options to keep snapshots, rustic has the possibility to mark snapshots to be either never deleted or after a given time. This can be also set in rustic backup (--delete-never or --delete-after) or for existing snapshots by e.g. rustic tag:

      --set-delete-never             Mark snapshot as uneraseable
      --set-delete-after <DURATION>  Mark snapshot to be deleted after given duration (e.g. 10d)

IIRC the delete-never mark can be also set in the interactive mode rustic snapshots -i.

These marks are automatically taken into account during rustic forget. (Be aware: restic doesn't support this, if you run restic forget on a repository with such marked snapshots, restic will forget them!)

[s-leroux]

--set-delete-never Mark snapshot as uneraseable

Indeed, that is an interesting alternative to tagging "important" backup as I considered in an above reply. Thanks for pointing that out.

Be aware: restic doesn't support this, if you run restic forget on a repository with such marked snapshots, restic will forget them!

Yes, that is something to consider. We didn't use restic (nor any other backup solution besides rsync :/ ), so that shouldn't be an issue for us.

[aawsome]

About having a true append-only repository: As @kapitainsky correctly said, this should be supported by the backend. Also, rustic has the option to configure a repo as append-only: rustic config --set-append-only true). In this case, commands like forget or prune won't do anything but print an error.

[s-leroux]

rustic has the option to configure a repo as append-only: rustic config --set-append-only true).

I wasn't aware of that. Is it documented somewhere? I might have overlooked it.

[s-leroux]

About having a true append-only repository: As @kapitainsky correctly said, this should be supported by the backend

{
	"Effect": "Allow",
	"Action": [
		"s3:DeleteObject",
		"s3:GetObject",
		"s3:RestoreObject"
	],
	"Resource": [
		"arn:aws:s3:::backup-hot/*",
		"arn:aws:s3:::backup-cold/*"
	]
},

Does simply removing the s3:DeleteObject option turn the repository effectively append-only, or would that compromise proper rustic operations?

[aawsome]

A remark about the bit-rot: rustic has rustic diff --only-identical which allows to compare files referenced by a snapshot with the files on your local disc, but only for the files that should be identical by metadata information.
So, this is able to detect bit rot (that's the use case this option was implemented for)

[s-leroux]

rustic diff --only-identical

Can this be used with cold storage?