spec has no way to set cluster private_backend, so partsync never runs on the file backend

[tdakkota]

The operator's default topology is shared-nothing — a StatefulSet whose pods each get their own
ReadWriteOnce PVC, with spec.storage.backend: file — but the rendered config never sets
storage.cluster.private_backend, and there is no way for a user to set it either.

Consequence (this is oteldb/oteldb#1262, which the operator still reproduces): cluster/partsync
never runs, so only the unflushed head is replicated. Every flushed part exists in exactly one
copy regardless of spec.cluster.replicationFactor.

Measured on kind (oteldb/e2e, 3 nodes, backend: file, replicationFactor: 2, flushInterval: 10s)

Flushed parts, per node, after ingesting all three signals:

oteldb-0  /var/lib/oteldb/default/_s1/{metrics,traces}, /default/_s3/metrics   35 files
oteldb-1  (nothing)                                                             0 files
oteldb-2  /var/lib/oteldb/default/_s0/logs                                     13 files

Every shard's parts sit on one node only. Deleting a node together with its PVC and letting the
StatefulSet return it under the same identity, the node never recovers — for 3 minutes, and
structurally forever:

{"msg":"Joining storage cluster", ... ,"private_backend":false}
{"msg":"bootstrap: gained shard with no local engine","shard":"default/_s3"}
{"msg":"bootstrap: gained shard with no local engine","shard":"default/_s3"}
...   (repeats once per maintenance tick, indefinitely; data dir stays empty)

bootstrapShard calls syncParts, which is a no-op when the node believes its backend is shared,
so no bucket index appears locally, so no engine is created, so hasAnyEngine stays false and the
next tick rediscovers the same shard.

With storage.cluster.private_backend: true in the same cluster, same chaos:

{"msg":"Joining storage cluster", ... ,"private_backend":true}
{"msg":"bootstrap: gained shard with no local engine","shard":"default/_s0"}
{"msg":"bootstrap: shard signal loaded","shard":"default/_s0","signal":"log"}
part_sync: {"passes":8,"mirrored":1,"copied":9,"copied_bytes":557}

— the wiped node has its 9 part objects back ~20s (two maintenance ticks) after becoming Ready, and
flushed parts land on 2 nodes as replicationFactor: 2 implies.

Why spec.extraConfig is not a workaround

storage.cluster is in reservedConfigPaths (internal/controller/extraconfig.go), and a reserved
path covers everything below it, so

spec:
  extraConfig:
    storage:
      cluster:
        private_backend: true

is rejected: spec.extraConfig sets reserved config path storage.cluster (use spec.cluster and spec.etcd.endpoints).

Suggested fix

A spec.cluster.privateBackend field rendered into cluster["private_backend"]. I verified this is
sufficient — a two-hunk patch (the API field plus one if in renderConfig) produced the
private_backend: true config and the recovery behaviour above.

Defaulting is worth a decision: backend: file on a per-pod PVC is always shared-nothing, so
deriving true for the file backend (and leaving s3 explicit, since an S3 bucket may or may not be
shared) would fix existing clusters rather than only new ones. Note the flag is not inferable from
the backend type in general, which is why oteldb made it explicit.

Blocks oteldb/e2e#, which asserts the mode via the node admin API's part_sync stats.