controller: status-write conflicts churn the reconcile loop while a rollout settles

[tdakkota]

Observed during an end-to-end run on kind with the role-pool branches (claude/role-query-group + claude/16-private-backend), deploying a 3-node OtelDBCluster with an ingest pool and a query pool.

While the role-pool merge patch was settling, the controller emitted a steady stream of:

Operation cannot be fulfilled on oteldbclusters.db.oteldb.io "oteldb": the object has been modified;
please apply your changes to the latest version and try again

It self-corrected — the cluster reached Ready, all 11 e2e specs passed, and nothing was left in a bad state. So this is noise rather than breakage. But it is the signature of a stale-object write, usually one of:

• the reconciler mutating and writing status from an object it fetched earlier in the same pass, after having patched spec-adjacent resources in between;
• status being written unconditionally rather than only when it actually changed, so every pass races the previous one's write;
• separate Update calls for the object and its status where a single Status().Patch would do.

Worth looking at because it gets worse with more moving parts, and the role stanzas add two Deployments plus their Services and ConfigMaps to every reconcile. A conflict storm during a rollout also makes real errors hard to see in the logs at exactly the moment an operator is watching them.

Suggested direction:

• Re-read the object (or use the client's cached copy) immediately before the status write, and write via Status().Patch with optimistic concurrency rather than Update.
• Compute the desired status first and skip the write entirely when it is unchanged — that alone removes most of the churn, since a steady-state cluster reconciles repeatedly with identical status.
• If the churn is inherent to the resync cadence, consider whether a conflict on the status write should be logged at debug rather than error; it is an expected outcome of optimistic concurrency, not a failure.

Reproduction: bring up a cluster with both spec.ingest and spec.query set and watch the manager logs through the initial rollout. It was most visible while the pools' Deployments were still scaling up.

Context: the branches involved are #14, #15 and #17, none merged at the time of observation. The same pattern may exist on main with just the StatefulSet — worth checking whether the role pools cause it or merely make it easier to see.