Doc pruning

[fredcarle]

We can truncate a full collection but we may want to completely remove some documents from the database.

Add a prune command that can prune docs based on the provided filter

mutation {
  prune_User( filter: { name: { _eq: "John" } } ) {
    _docID
  }
}

Note: Pruning does not mutate the document. It doesn't delete it from the network's perspective. It only removes documents from the local node.

[edjroz]

I've been implementing this and wanted to surface a design constraint before going further, since it affects the API shape proposed in the issue.

Approach

Prune is close to a filtered truncate. truncate does a local-only physical key wipe with no CRDT block and no network event, which is the right model here.

So prune reuses truncate's per-document teardown primitives, and borrows per-document secondary-index removal from the delete path (truncate wipes indexes by whole-collection prefix, which can't work per-doc since index keys put the docID last).

flowchart TD
    CLI["CLI / HTTP<br/>client collection prune --filter"]
    API["Collection.PruneDocumentsWithFilter"]
    A["PHASE A - COLLECT<br/>read-only txn runs the selection plan<br/>collect docIDs and primary keys<br/>txn discarded, never committed"]
    B["PHASE B - TEARDOWN, no txn live<br/>loop each matched doc into prune"]
    P1["1. secondary-index entries<br/>in prune's own txn, committed first"]
    P2["2. DAG blocks and headstore keys"]
    P3["3. docID short-ID mappings"]
    P4["4. datastore field keys and primary key<br/>out-of-band via rootstore"]
    OUT["Document gone locally<br/>no CRDT delta, no update event<br/>nothing propagates to peers"]
    CLI --> API --> A --> B --> P1 --> P2 --> P3 --> P4 --> OUT

Loading

Constraints

• The physical deletes must be non-transactional — store-level txn size limits, and leveldb permits only one open txn store-wide (the same reasons truncate works this way).
• Index removal must commit before the physical deletes. It reads the doc's field values transactionally; if those keys are then deleted out-of-band and that txn is committed afterward, Badger's SSI raises a conflict and the index entry is left orphaned.

Blocker

ExecRequest wraps every request in a transaction, unconditionally — internal/db/store.go:44 calls ensureContextTxn(ctx, db, false) for every query and mutation, regardless of whether the caller used db.WithTxn(...). That txn is bound onto the ctx and flows into the planner, so a pruneNode would always see a live txn:

flowchart TD
    GQL["mutation { prune_User(filter: ...) { _docID } }"]
    EXEC["ExecRequest — store.go:44<br/>ensureContextTxn(ctx, db, false)<br/><b>always opens a txn</b>"]
    CTX["txn bound onto ctx<br/>flows into planner.New"]
    NODE["pruneNode.Next()"]
    CALL["Collection.PruneDocument(p.ctx, ...)"]
    ERR["❌ ErrPruneWithExplicitTxn<br/>every call · every backend"]
    GQL --> EXEC --> CTX --> NODE --> CALL --> ERR

Loading

Options

• Ship prune interface-only, mirroring truncate (client.Collection + CLI/HTTP/cbindings/js). Safe, consistent, no core-path risk, however not as proposed above.
• Special-case prune in ExecRequest so its per-request txn wrapping is skipped, letting PruneDocumentsWithFilter manage its own transactions. Delivers the mutation, but touches a path shared by every query and mutation.
• Reframing prune as an admin action rather than a mutation.

[AndrewSisley]

@edjroz Be aware that there is an ongoing design discussion about this on discord at the moment.