Skip to content

Instance destroy orphans file contents and blocks the HTTP handler #4880

Description

@Crash--

Summary

lifecycle.Destroy deletes the instance databases before its files, and runs the
whole purge synchronously inside the HTTP handler. On a large instance this
leaks the file contents permanently and times out the caller.

This is not specific to a storage backend: Swift and S3 both go through the same
call site.

Where

  • web/instances/instances.go:296deleteHandler calls lifecycle.Destroy(domain) inline.
  • model/instance/lifecycle/destroy.go:95couchdb.DeleteAllDBs(inst).
  • model/instance/lifecycle/destroy.go:100inst.VFS().Delete().
  • model/instance/lifecycle/reset.go:29 — same VFS().Delete() on the reset path.

Orphaned objects

DeleteAllDBs runs before VFS().Delete(). If the stack restarts, the storage
is unreachable, or the request is cancelled between the two, the instance
databases are gone while the file contents remain.

Those contents cannot be recovered by any later pass: both backends derive their
storage location from the instance DBPrefix, which no longer exists anywhere.
The objects are orphaned silently, with no error and no way to enumerate them.

Deleting the file contents before the databases would make the operation
restartable instead.

Synchronous purge

Neither backend delegates the purge to the object store; both enumerate every
object of the instance first:

  • vfsswift.DeleteContainer (model/vfs/vfsswift/swift.go:24) calls
    ObjectNamesAll and then bulk-deletes.
  • s3util.DeletePrefixObjects lists the instance prefix and deletes in batches.

For an instance holding a few million objects that is thousands of round trips
inside a single HTTP request. The reverse proxy times the request out, the
caller records a failure while the stack is in fact still deleting, and nothing
retries or reports progress if the process goes away mid-purge.

Suggested direction

  • Delete the file contents before the databases, so an interrupted destroy can
    be resumed.
  • Move the purge to a job, so DELETE /instances/:domain returns immediately
    and the work is retried on failure.
  • Once a job exists, the purge can be delegated to the object store where it is
    supported — an S3 lifecycle rule scoped to the instance prefix removes the
    round trips entirely. Note the tradeoffs before relying on it: expiration is
    asynchronous (up to ~48h, which conflicts with treating Destroy as a
    deletion guarantee), a bucket accepts at most 1000 lifecycle rules, and the
    configuration is bucket-wide so concurrent updates must be serialised.

Context

Raised while reviewing the S3 VFS backend (feat/s3-vfs-backend). The S3
implementation makes the cost more visible, but the ordering bug and the
synchronous purge both predate it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions