Hi Andrew! 👋 Bobsch again — second find from the same session.
Same disclaimer as the other issue: no idea if this is really a bug — Fable 5 (my AI assistant) isolated it while stabilizing our test suite and thought you'd want to know. Maybe it's helpful, maybe it's known/expected.
Setup: omnigraph-server v0.7.0, cluster mode, local filesystem (macOS, temp dir), Cedar policy enabled.
Symptom: under a burst of parallel writes to different branches of one freshly-booted graph, the server occasionally returns:
500 {"error":"storage read failed for '<cluster>/graphs/<g>.omni/__recovery/01KW….json':
Object at location <...>/__recovery/01KW….json not found: No such file or directory (os error 2)",
"code":"internal"}
…and then works again immediately — an instant retry of the same request succeeds. It looks like a read of a recovery artifact racing its own cleanup/rotation.
Frequency/repro: roughly once per ~10 runs of a test suite that boots an ephemeral cluster and concurrently seeds ~10 branches (branch create from main + a couple of stored-query mutations each, several suites in parallel worker threads). Our workaround: retry 5xx briefly on the client — succeeds on first retry every time we've seen it.
Can share the exact repro harness shape and full error lines if that helps. Thanks!
Hi Andrew! 👋 Bobsch again — second find from the same session.
Same disclaimer as the other issue: no idea if this is really a bug — Fable 5 (my AI assistant) isolated it while stabilizing our test suite and thought you'd want to know. Maybe it's helpful, maybe it's known/expected.
Setup: omnigraph-server v0.7.0, cluster mode, local filesystem (macOS, temp dir), Cedar policy enabled.
Symptom: under a burst of parallel writes to different branches of one freshly-booted graph, the server occasionally returns:
…and then works again immediately — an instant retry of the same request succeeds. It looks like a read of a recovery artifact racing its own cleanup/rotation.
Frequency/repro: roughly once per ~10 runs of a test suite that boots an ephemeral cluster and concurrently seeds ~10 branches (branch create from main + a couple of stored-query mutations each, several suites in parallel worker threads). Our workaround: retry 5xx briefly on the client — succeeds on first retry every time we've seen it.
Can share the exact repro harness shape and full error lines if that helps. Thanks!