|
| 1 | +# `custom-trace.anyenc` — incremental editing trace |
| 2 | + |
| 3 | +A replayable, per-edit sequence of Yjs updates: one real document plus the individual |
| 4 | +edits that produced it, in order, each tagged with who made it and when. `Y7` in this |
| 5 | +benchmark suite replays it to measure apply-update throughput, incremental sync, |
| 6 | +propagation and compaction against a workload nobody designed to be convenient. |
| 7 | + |
| 8 | +Drop a trace at `benchmarks/custom-trace.anyenc` and `Y7` runs; without one it skips |
| 9 | +itself and the rest of the suite is unaffected. |
| 10 | + |
| 11 | +> **The trace file is gitignored and must stay that way — it contains real document |
| 12 | +> content.** This format document is checked in so a trace can be produced without one |
| 13 | +> to copy from. |
| 14 | +
|
| 15 | +- **Encoding:** a single [`lib0`](https://github.com/dmonad/lib0) `encoding.writeAny` value |
| 16 | +- **Updates:** Yjs **v1** (`Y.applyUpdate` / `Y.encodeStateAsUpdate`, not V2) |
| 17 | +- **Versions:** produce it with the versions yhub uses — `@y/y@14`, `lib0@1`. |
| 18 | + `@y/y` (Yjs v14) is **not** wire-compatible with `yjs@13` for these updates. |
| 19 | + |
| 20 | +## Decoding |
| 21 | + |
| 22 | +```js |
| 23 | +import * as fs from 'node:fs' |
| 24 | +import * as decoding from 'lib0/decoding' |
| 25 | +import * as Y from '@y/y' |
| 26 | + |
| 27 | +const trace = decoding.readAny(decoding.createDecoder(new Uint8Array(fs.readFileSync('custom-trace.anyenc')))) |
| 28 | + |
| 29 | +const ydoc = new Y.Doc({ gc: trace.gc }) // gc: false — the trace preserves deleted content |
| 30 | +for (const u of trace.updates) { |
| 31 | + Y.applyUpdate(ydoc, u.update) |
| 32 | +} |
| 33 | +``` |
| 34 | + |
| 35 | +`benchmarks/src/trace.js` is the suite's loader; `splitTrace()` returns the baseline |
| 36 | +update and the incremental edits separately, which is usually what you want. |
| 37 | + |
| 38 | +## Structure |
| 39 | + |
| 40 | +``` |
| 41 | +{ |
| 42 | + type: 'yhub:editing-trace:v1', |
| 43 | + updateFormat: 'yjs-v1', // Y.applyUpdate / Y.encodeStateAsUpdate (not V2) |
| 44 | + gc: false, // updates were carved from the non-gc doc |
| 45 | + source: { |
| 46 | + documentId, org, docid, branch, |
| 47 | + exportedAt // ISO string, when the trace was built |
| 48 | + }, |
| 49 | + users: [ string ], // distinct authors, in first-edit order |
| 50 | + totalBytes: number, // sum of all update byte lengths |
| 51 | + updates: [ // chronological, apply in array order |
| 52 | + { |
| 53 | + update: Uint8Array, // a Yjs v1 update |
| 54 | + time: number, // ms epoch, when the edit happened |
| 55 | + user: string, // whatever your auth plugin puts in `userid` |
| 56 | + kind: 'insert' | 'delete' | 'mixed', |
| 57 | + ranges: number // how many attributed id-ranges this step covers |
| 58 | + } |
| 59 | + ] |
| 60 | +} |
| 61 | +``` |
| 62 | + |
| 63 | +A document that was bulk-imported and then edited by hand typically has a very large |
| 64 | +update `0` (the import) and many tiny ones after it. For benchmarks that care about |
| 65 | +incremental work, load update `0` as the baseline and treat `updates.slice(1)` as the |
| 66 | +workload — that is exactly what `splitTrace()` gives you. |
| 67 | + |
| 68 | +--- |
| 69 | + |
| 70 | +# Creating your own trace |
| 71 | + |
| 72 | +## Where the edit history comes from |
| 73 | + |
| 74 | +You do not need to have recorded anything in advance. **yhub already stores the edit |
| 75 | +history as attributions**, and a trace is that history turned back into updates. |
| 76 | + |
| 77 | +For every inbound update, the server derives the id-ranges it touched and writes an |
| 78 | +attribution for each (`createContentMapFromParams`, `src/server.js:23-31`): |
| 79 | + |
| 80 | +| side | attributes yhub writes | |
| 81 | +|---|---| |
| 82 | +| inserted ranges | `insert` = userid, `insertAt` = ms epoch, plus `insert:<k>` for each custom attribution | |
| 83 | +| deleted ranges | `delete` = userid, `deleteAt` = ms epoch, plus `delete:<k>` for each custom attribution | |
| 84 | + |
| 85 | +These live in the **contentmap**, persisted next to the document. Decoded, it is: |
| 86 | + |
| 87 | +```js |
| 88 | +const contentMap = Y.decodeContentMap(bin) // { inserts: IdMap, deletes: IdMap } |
| 89 | + |
| 90 | +contentMap.inserts.forEach((range, client) => { |
| 91 | + // range = { clock: number, len: number, attrs: Array<{ name: string, val: any }> } |
| 92 | + // e.g. attrs = [ { name: 'insert', val: 'user_42' }, { name: 'insertAt', val: 1700000001000 } ] |
| 93 | +}) |
| 94 | +``` |
| 95 | + |
| 96 | +So the document knows, for every struct in it, **who wrote it and at what millisecond**. |
| 97 | +Grouping by `(user, timestamp)` recovers the individual editing steps, and each group can |
| 98 | +be carved back out of the full update. |
| 99 | + |
| 100 | +## The algorithm |
| 101 | + |
| 102 | +1. **Read the non-gc document and its contentmap.** It must be the *non-gc* document — |
| 103 | + the trace has to reproduce deleted content, and a gc'd document no longer contains it. |
| 104 | +2. **Enumerate the steps.** Walk both `IdMap`s and collect the distinct `(user, time)` |
| 105 | + pairs from `insert`/`insertAt` and `delete`/`deleteAt`. One pair is one editing step; |
| 106 | + a step that both inserts and deletes at the same instant was a single transaction — |
| 107 | + overwriting a cell, for example. |
| 108 | +3. **Carve each step.** Filter the contentmap down to just that step's ranges, convert to |
| 109 | + an `IdSet` pair, and intersect the full update with it: |
| 110 | + |
| 111 | + ```js |
| 112 | + const stepMap = Y.filterContentMap( |
| 113 | + contentMap, |
| 114 | + attrs => attr(attrs, 'insert') === user && attr(attrs, 'insertAt') === time, |
| 115 | + attrs => attr(attrs, 'delete') === user && attr(attrs, 'deleteAt') === time |
| 116 | + ) |
| 117 | + const contentIds = Y.createContentIdsFromContentMap(stepMap) |
| 118 | + const update = Y.intersectUpdateWithContentIds(fullUpdate, contentIds) |
| 119 | + ``` |
| 120 | + |
| 121 | + `intersectUpdateWithContentIds` extracts exactly those structs and delete-set entries |
| 122 | + from the full document update — no document is rebuilt and no content is invented. |
| 123 | +4. **Sort by timestamp**, then **verify**: replaying every carved update into a fresh |
| 124 | + `Y.Doc({ gc: false })` must produce a byte-identical `encodeStateAsUpdate` to the |
| 125 | + original. If it does not, the trace is a plausible-looking fiction — throw it away |
| 126 | + rather than benchmark against it. |
| 127 | + |
| 128 | +## Option A — from a live yhub deployment (easiest) |
| 129 | + |
| 130 | +[`tools/build-trace.js`](./tools/build-trace.js) does all of the above: |
| 131 | + |
| 132 | +```sh |
| 133 | +cd benchmarks |
| 134 | +node --max-old-space-size=8192 \ |
| 135 | + --env-file-if-exists=../.env \ |
| 136 | + tools/build-trace.js --org <org> --docid <docid> [--branch main] \ |
| 137 | + [--bucket <s3-bucket>] [--redis-prefix <prefix>] \ |
| 138 | + [--out custom-trace.anyenc] |
| 139 | +``` |
| 140 | + |
| 141 | +It connects with `server: null, worker: null` — a read-only hub that touches nothing — |
| 142 | +calls `getDoc(room, { nongc: true, contentmap: true })`, reconstructs, verifies, and |
| 143 | +writes. It refuses to write an unverified trace, and it tells you if the document is |
| 144 | +empty or unattributed rather than emitting a silently useless file. |
| 145 | + |
| 146 | +Point `--bucket` and `--redis-prefix` at the deployment you mean. Reading the wrong |
| 147 | +bucket looks exactly like an empty document. |
| 148 | + |
| 149 | +## Option B — from a yhub export |
| 150 | + |
| 151 | +If you have an export rather than database access, the same two inputs are in it: |
| 152 | + |
| 153 | +| file | any-decoded value | |
| 154 | +|---|---| |
| 155 | +| `storage_assets/id:ydoc:v1_…_0_…` | `{ type: 'asset:ydoc:v1', update }` — **non-gc** document (the `0` is the gc flag) | |
| 156 | +| `storage_assets/id:ydoc:v1_…_1_…` | the gc'd document — *not* what you want | |
| 157 | +| `storage_assets/id:contentmap:v1_…` | `{ type: 'asset:contentmap:v1', contentmap }` → `Y.decodeContentMap` | |
| 158 | +| `postgres_gc.bin`, `postgres_non_gc.bin` | `{ type: 'asset:retrievable:v1', plugin: … }` — pointers only; the payload is in `storage_assets/` | |
| 159 | + |
| 160 | +`assetIdToString` encodes the gc flag as `${gc ? 1 : 0}`, hence `…/main/0/…` for non-gc. |
| 161 | +Decode both with `lib0/decoding.readAny`, then follow the algorithm above from step 2 — |
| 162 | +the rest of `build-trace.js` applies unchanged. |
| 163 | + |
| 164 | +## Things that will bite you |
| 165 | + |
| 166 | +- **No attributions, no trace.** yhub only attributes writes it received itself. Content |
| 167 | + loaded with `unsafePersistDoc`, or written before attribution existed, carries none — |
| 168 | + `build-trace.js` will tell you the document has content but no history. |
| 169 | +- **Clock ordering.** A step must not depend on structs that arrive later. Sorting by |
| 170 | + timestamp is normally enough; the verification step in 4 is what actually proves it. |
| 171 | +- **Timestamp granularity.** Two edits by the same user in the same millisecond collapse |
| 172 | + into one step. That is usually correct — it was one transaction — but it does mean the |
| 173 | + step count is a lower bound on the number of user actions. |
| 174 | +- **Privacy.** The trace contains the document's real content, and `user` values are |
| 175 | + whatever your auth plugin puts in `userid`. Treat it as production data: it is |
| 176 | + gitignored here, and it should not leave wherever you are entitled to keep it. |
0 commit comments