insert, update: opt-in dumboDocHashes — return each written document's content hash

[christopherreay-claudecode]

Clients that maintain caches or live state over documents need per-document
change detection, and today the wire carries none: no write acknowledgment
or read result includes per-document version metadata, so the only shape is
head-query-then-read (dumboStatus / dumboLog {limit: 1}) — whole-database
granularity, and racy against a concurrent commit.

This PR adds an opt-in: pass dumboDocHashes: true on insert or update,
and the ack carries each written document's content hash. Without the flag,
replies are byte-identical to today — nothing opts in by accident (the
parity harness stays clean).

insert ack:  dumboDocHashes: [ { index, _id, hash }, ... ]   // request order
update ack:  dumboDocHashes: [ { _id, hash }, ... ]          // upsert included

Why hash-at-write is the mechanism (what we learned reading the storage)

We first assumed the storage engine already held a per-document address to
surface. It doesn't, for the normal case: the document body is one
adaptive-bytes field, and below DefaultTupleLengthTarget (2048) the bytes
live inline in the leaf tuple and never become a chunk
(val/tuple_builder.go — only larger documents spill to an addressed blob).
So there is no existing per-document content address to expose, and the
right identity is a hash of the canonical stored bytes —
[bsonFormatVersion][BSON, keys lex-sorted at every level] — computed at
the write points where those bytes are already in memory
(collection.go insert/update paths). Cost: one digest over an in-memory
buffer; no extra tree read, no extra chunk, no storage change, no
dependency added.

The digest is SHA-512/20 rendered base32 — the same family and width as the
hashes already on the wire — and because the input is the canonical form, a
client can recompute it independently: identical content gives an identical
hash across commits, branches, servers, and restarts.

Tests

• internal/backends/dolt/doc_hash_test.go — backend level: hash present
  and stable, changes iff bytes change, canonical (key order does not
  matter), upsert covered.
• tests/doc_hash_test.go — through the wire: flag on/off, insert order,
  update and upsert shapes, reply byte-parity without the flag.

go test ./... -count=1 -p 1 green (the -p 1 is for the pre-existing
.runtime/bin race between tests and tests/verify, not this change).

Try it

// npm i mongodb
// DUMBO_URL=mongodb://127.0.0.1:27017 DUMBO_DB=example node example.mjs
import { MongoClient } from "mongodb";

const client = await new MongoClient(process.env.DUMBO_URL, { directConnection: true }).connect();
const db = client.db(process.env.DUMBO_DB);

const ins = await db.command({
  insert: "docs",
  documents: [{ _id: 1, status: "draft" }, { _id: 2, status: "live" }],
  dumboDocHashes: true,
});
console.log(ins.dumboDocHashes);   // [ { index: 0, _id: 1, hash: "..." }, { index: 1, _id: 2, hash: "..." } ]

const upd = await db.command({
  update: "docs",
  updates: [{ q: { _id: 1 }, u: { $set: { status: "review" } } }],
  dumboDocHashes: true,
});
console.log(upd.dumboDocHashes);   // [ { _id: 1, hash: "..." } ] — different bytes, different hash

// a no-op update returns the SAME hash: the hash is content, not history

Scope, deliberately narrow

insert, update, upsert. Not covered, and open for discussion before
code: findAndModify, bulkWrite, reads — and the merge path, which
is where we want this most ("which of my thousand held documents did that
merge touch") and where the ack shape deserves agreement first rather than
a guess.

Happy to adjust any of the shape.

[macneale4]

I love this idea. I'll take a look today

[macneale4]

I think there needs to be some deeper thought on product questions here. For starters, having it opt-in puts a lot of expectations on the application code to get this right. And in that case, there is already a mechanism for applications to do this with sequences embedded in the document:

db.docs.updateOne(
  { _id: 1, v: 7 },                       // CAS condition
  { $set: { status: "review" }, $inc: { v: 1 } }
)
// n: 1 -> we won.  n: 0 -> someone else moved it; re-read and retry.

But.... this actually doesn't work. There is a deeper issue which is that currently out merge algorithm is forgiving when two writers change a field to the same value. This is kind of a dolt thing, which I have reservations about.

I'm working through the idea of having a collection level merge resolution configuration that specifically prevents two writers from updating the same doc. This would harden the 3 way merge behavior of the given collection. I think the default level should prohibit two writers from setting a field to the same value, but they could edit the same document if the fields don't overlap. There would be another level which is document level.

But all of that is more at the transactional level. And it's different than reading a document, taking "a lot" of time to decide how to update it, then using the an optimistic lock to update.

Optimistic updates of documents has an established pattern in Mongo, and I think our first priority is to make it so that pattern works. Disabling the current fuzzy merge behavior needs to take priority over building new mechanisms to do similar things. So this PR will probably need to be revisited in the future. Maybe better to make it a feature request.