CI: track `object has been modified` instances in Controller and Sidecar

[BlaineEXE]

Enhancement

Why is this needed?:

Instances of object has been modified errors when updating CRs/statuses indicate that mulitple actors are modifying the resource at once. Kubernetes and controller-runtime can easily handle these intances, and occasional occurrences are not reason for alarm. However, we should be on guard for cases where these errors pop up regularly. Frequently encountering these messages would mean that COSI controller/sidecar processes are interfering with each others' operations often, and COSI should limit this as much as reasonably possible.

Describe the solution you'd like in detail:

In order to help us gauge this, I would suggest that we create a CI test to look for this. These errors are more likely to occur when multiple CRs are created at once with references to each other. I would thus propose a test that creates a number of BucketClaims and referring BucketAccesses at once. In the test, we can wait for objects to reconcile, and then scan controller and sidecar logs for instances of object has been modified errors. If the number of instances passes a threshold (to be determined), fail the test.

Initially, we may want to set a somewhat high failure threshold. As we optimize the controllers to interfere less with each other, we can lower the failure threshold.

As a starting point, if the test creates 5 BucketClaims and 3 BucketAccesses, I would suggest 8 (5+3) total system occurrences of the error to be a failure. To me, this would indicate obvious interference that needs work.

As an end-goal for a well-optimized system, I think more than 2 instances would be a good failure threshold.

Describe alternatives you've considered:

We could aim for high optimization as a starting goal, but this would slow down development -- IMO, unnecessarily.

Additional context:

In my experience, errors like these are often raised by end users when they become too frequent or too predicable. Different users will have different ideas of what frequency/predictability is too much.