mechanism for drivers to indicate that an access needs re-reconciled at a given date/time

[BlaineEXE]

Enhancement

Is your feature request related to a problem?/Why is this needed

Some object storage backends have automatic credential rotation for accesses. After an automatic rotation, COSI's stored credentials will fail to work. COSI should update the secret containing credentials after the rotation so that user apps can continue operating healthily.

Describe the solution you'd like in detail

COSI should provide an interface that allows drivers to indicate a desire to re-reconcile certain accesses at/after a given time.

Alternatively, (possibly preferably) COSI could implement a more generic API that would allow a driver to request the controller re-reconcile any access(es) by name/ID. This would allow drivers to re-reconcile after internal updates of any kind, even those not tied to a known timeframe.

Describe alternatives you've considered

A possible workaround could be to have drivers reboot their own pod when any access (or bucket) needs updated. This is an unfortunately blunt mechanism but a viable workaround while COSI works on an API and implementation details for this.

Additional context

n/a

[BlaineEXE]

TBH, I think a generic mechanism for requesting that the COSI sidecar re-reconcile a BucketAccess will allow for a broader range of driver needs.

I think this could also be beneficial for Buckets themselves, though less critical. Rook with the lib-bucket-provisioner OBCs has seen instances of users changing S3 endpoints after initial deployment. For examples, converting an endpoint from HTTP to HTTPS, or switching company branding to a different web address.

I would imagine that the driver would call the sidecar and ask it to requeue one or more Accesses by the accountID. The sidecar would look up and reconcile the access with the given ID. I would guess this would require the sidecar to run a gRPC server, which would add complexity.

We could instead put any burden onto the drivers themselves. Driver pods need read access to BucketAccess objects in order for the sidecar to function. The driver itself could use those permissions to List() the BucketAccesses and then modify an annotation on the access(es) that it wishes to re-reconcile. With that, COSI wouldn't need to add any new API. A document explaining the niche case and recommended approach could be useful.