Skip to content

Claims persistence is O(table) per create and globally serialized, capping claim throughput at high pod density #2

Description

@CMGS

Problem

Every CreatePod commit (commitClaimwrite() in provider.go) marshals the entire claims table to JSON and rewrites the whole state file under the global saveMu. The cost of persisting one new claim is O(total claims held), and all creates/deletes serialize on the same lock for the duration of the disk write.

Impact

At the advertised 2000-pod capacity the table is ~500 KB of JSON, so a single commit costs ~2–4 ms — larger than the 0.2–0.7 ms sandboxd handover itself, i.e. persistence dominates the per-create cost on a full node. Serialization caps sustained claim throughput at roughly 250–500 claims/s, and a full-node churn storm does O(N²) total serialization work (~5–8 s of pure persistence across 2000 creates).

Assessment

Acceptable under the current contract (2000 pods/node); this is the first thing that breaks if node density grows ~10×. Not worth changing now — filing so the scaling limit is on record.

Direction if/when needed

Coalesce concurrent saves (batch multiple commits into one write) or move to per-key files / an append log. Either is a design change to the durability path (the tentative/commit rollback semantics depend on write ordering), so it should come with its own review round.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions