Summary
lifecycle.Destroy deletes the instance databases before its files, and runs the
whole purge synchronously inside the HTTP handler. On a large instance this
leaks the file contents permanently and times out the caller.
This is not specific to a storage backend: Swift and S3 both go through the same
call site.
Where
web/instances/instances.go:296 — deleteHandler calls lifecycle.Destroy(domain) inline.
model/instance/lifecycle/destroy.go:95 — couchdb.DeleteAllDBs(inst).
model/instance/lifecycle/destroy.go:100 — inst.VFS().Delete().
model/instance/lifecycle/reset.go:29 — same VFS().Delete() on the reset path.
Orphaned objects
DeleteAllDBs runs before VFS().Delete(). If the stack restarts, the storage
is unreachable, or the request is cancelled between the two, the instance
databases are gone while the file contents remain.
Those contents cannot be recovered by any later pass: both backends derive their
storage location from the instance DBPrefix, which no longer exists anywhere.
The objects are orphaned silently, with no error and no way to enumerate them.
Deleting the file contents before the databases would make the operation
restartable instead.
Synchronous purge
Neither backend delegates the purge to the object store; both enumerate every
object of the instance first:
vfsswift.DeleteContainer (model/vfs/vfsswift/swift.go:24) calls
ObjectNamesAll and then bulk-deletes.
s3util.DeletePrefixObjects lists the instance prefix and deletes in batches.
For an instance holding a few million objects that is thousands of round trips
inside a single HTTP request. The reverse proxy times the request out, the
caller records a failure while the stack is in fact still deleting, and nothing
retries or reports progress if the process goes away mid-purge.
Suggested direction
- Delete the file contents before the databases, so an interrupted destroy can
be resumed.
- Move the purge to a job, so
DELETE /instances/:domain returns immediately
and the work is retried on failure.
- Once a job exists, the purge can be delegated to the object store where it is
supported — an S3 lifecycle rule scoped to the instance prefix removes the
round trips entirely. Note the tradeoffs before relying on it: expiration is
asynchronous (up to ~48h, which conflicts with treating Destroy as a
deletion guarantee), a bucket accepts at most 1000 lifecycle rules, and the
configuration is bucket-wide so concurrent updates must be serialised.
Context
Raised while reviewing the S3 VFS backend (feat/s3-vfs-backend). The S3
implementation makes the cost more visible, but the ordering bug and the
synchronous purge both predate it.
Summary
lifecycle.Destroydeletes the instance databases before its files, and runs thewhole purge synchronously inside the HTTP handler. On a large instance this
leaks the file contents permanently and times out the caller.
This is not specific to a storage backend: Swift and S3 both go through the same
call site.
Where
web/instances/instances.go:296—deleteHandlercallslifecycle.Destroy(domain)inline.model/instance/lifecycle/destroy.go:95—couchdb.DeleteAllDBs(inst).model/instance/lifecycle/destroy.go:100—inst.VFS().Delete().model/instance/lifecycle/reset.go:29— sameVFS().Delete()on the reset path.Orphaned objects
DeleteAllDBsruns beforeVFS().Delete(). If the stack restarts, the storageis unreachable, or the request is cancelled between the two, the instance
databases are gone while the file contents remain.
Those contents cannot be recovered by any later pass: both backends derive their
storage location from the instance
DBPrefix, which no longer exists anywhere.The objects are orphaned silently, with no error and no way to enumerate them.
Deleting the file contents before the databases would make the operation
restartable instead.
Synchronous purge
Neither backend delegates the purge to the object store; both enumerate every
object of the instance first:
vfsswift.DeleteContainer(model/vfs/vfsswift/swift.go:24) callsObjectNamesAlland then bulk-deletes.s3util.DeletePrefixObjectslists the instance prefix and deletes in batches.For an instance holding a few million objects that is thousands of round trips
inside a single HTTP request. The reverse proxy times the request out, the
caller records a failure while the stack is in fact still deleting, and nothing
retries or reports progress if the process goes away mid-purge.
Suggested direction
be resumed.
DELETE /instances/:domainreturns immediatelyand the work is retried on failure.
supported — an S3 lifecycle rule scoped to the instance prefix removes the
round trips entirely. Note the tradeoffs before relying on it: expiration is
asynchronous (up to ~48h, which conflicts with treating
Destroyas adeletion guarantee), a bucket accepts at most 1000 lifecycle rules, and the
configuration is bucket-wide so concurrent updates must be serialised.
Context
Raised while reviewing the S3 VFS backend (
feat/s3-vfs-backend). The S3implementation makes the cost more visible, but the ordering bug and the
synchronous purge both predate it.