v1alpha2 KEP update: Allow BucketAccess to reference multiple BucketClaims

[BlaineEXE]

Enhancement

Why is this needed?:

Allow users to reuse the same BucketAccess (user) for multiple buckets.

Because cross-namespace sharing isn't natively supported, limit this to BA/BC in the same namespace

Describe the solution you'd like in detail:

Before COSI, a user self-service library for object buckets was created called lib-bucket-provisioner/ObjectBucketClaims. Rook has used the predecessor library for many years, and the most requested feature is to use a single user for access to multiple buckets.

Some off-the-shelf applications make an assumption that a single object storage user can access all of the application's needed buckets. GitLab was provided as one example a number of years ago.

Describe alternatives you've considered:

No reasonable alternative is known. Most users cannot make custom changes to applications (due to closed source software or a lack of user ability) if their assumptions don't match COSI's assumptions. Without COSI support, these users would be without a good solution.

Additional context:

[BlaineEXE]

I made an initial proposal for the KEP, and an issue was found here: kubernetes/enhancements#4599 (comment)

In short, there was no way for COSI's secret to encode the different info on a per-bucket basis. While the user credentials should be the same for all buckets, each bucket will likely have unique info.

In S3, each bucket has a unique name. It's possible (though unlikely) that each bucket could have a unique realm or endpoint as well.

@satoru-takeuchi proposed rendering the various bucket info in the user-facing secret something like this:

COSI_S3_BUCKET_ID_0: foo
COSI_S3_BUCKET_ID_2: bar
...
COSI_S3_BUCKET_ID_N: cux

In working on the KEP, I am finding that the user-facing experience with the Secret as well as the driver developer experience with the gRPC calls is non-ideal when bunched together like this.

This is the fairly naive prototype gRPC response I drafted

DriverGrantBucketAccessResponse {
  "accountID": "<ID returned by driver>", // will be applied to BucketAccess.status.accountID
  "bucketsInfo": [
      {
        "S3": { // must match input protocol, and if required fields are missing, will be treated as provisioning failure
          "bucketName": "<bucket ID in S3 backend>",
          "region": "<bucket region>",
          "endpoint": "<endpoint where bucket can be accessed>"
        }
      }
      // ... additional buckets
    ]
  },
  "credentials": { // bucket access credentials
    "S3": { // must match input protocol, and if required fields are missing, will be treated as provisioning failure
      "accessKeyID": "<s3 access key id>", // e.g., "AKIAODNN7EXAMPLE"
      "accessSecretKey": "<s3 access secret key>" // e.g., "wJaUtnFEMI/K..."
    }
  }
}

This structure is workable but is pretty confusing. I wonder if we can do better.

End users would see this rendered into secrets something like this:

// access creds shared between all buckets
COSI_S3_ACCESS_KEY_ID: "AKIAODNN7EXAMPLE"
COSI_S3_ACCESS_SECRET_KEY: "wJaUtnFEMI/K..."

// individual bucket info
COSI_S3_PROBABLY_REALLY_LONG_UUID_BUCKET_NAME: "ABC"
...
COSI_S3_OTHER_REALLY_LONG_UUID_BUCKET_NAME: "DEF"
...

This naive draft doesn't retain portability because of the UUIDs. We could replace this with the namespaced-name of the BucketClaim, which would preserve portability. However, it would result in worst-case string lengths on the order of 150 chars long. I don't believe this exceeds a k8s limitation, but would be unwieldy for users.

After this draft, I am wondering if it might make more sense for each BucketClaim reference in the BucketAccess to have its own Secret output. This would make the it much easier for COSI to output consistent secrets, and users would hopefully have an easier time parsing the content of the secrets as well. This might also add a little more flexibility for users, allowing them to mount individual secrets to applications that only need some buckets.

A possible downside is that COSI would have to consider whether it's important to ensure that each secret only has the same credential info. We don't want driver implementers to create multiple users in the backend when the end user indicates they want a single user. Maybe this means the proposed gRPC structure above is a necessary complexity.

I'd appreciate other thoughts on this from users/vendors.

Which alternative is more preferable, focused on the end-user experience?

1. A single output Secret with possibly complex data field names like COSI_S3_BUCKCLAIMNAMESPACE_BUCKETCLAIMNAME_BUCKET_NAME: "ABC"
2. Multiple output Secrets for each referenced BucketClaim, each with simpler data field names like COSI_S3_BUCKET_NAME: "ABC"

[jhoblitt]

I have multiple use cases for configuring object storage. I believe all of them require a single user to be able to access multiple buckets. For example, a basic mimir deployment needs access to 4 buckets and a basic loki deployment needs access to 3 buckets.

My most important use case, for bespoke internal software, requires multiple independent users (applications) accessing a collection of buckets with policy that allows read and/or write depending on key prefixes. The access is all external to the k8s storage cluster and an external source of truth is needed for access credentials to coordinated with the external application(s). Currently, this is done using the external-secrets operator to pull secrets into k8s. I actually tried using external-secrets to push obc automatically generated s3 credentials out of the cluster but this was sub-optimal for boot strapping and key rotation as most applications require a manual restart to change credentials, and it was difficult to allow credentials to propagate organically.

Of the options listed for output secrets, I have a strong preference for option 2. When using automatically generated s3 credentials, the most common use case would be to either reference those credentials in a k8s resource which is secret away with a secretRef or to mount that secret as env vars in a pod. For the later case, it is preferable to minimize the number of created env vars.

However, my mission critical applications need to input credentials. They actually need to be able to input multiple s3 credentials in parallel per usef in order to support key rotation. Automatic output is nice to have for some applications (e.g. mimir) but I could live without it (as long as credentials input is supported).

[BlaineEXE]

Thanks for the feedback, Josh.

Also, for your notes, COSI does have a backlog item for input secrets and credential rotation that didn't make it in to the v1a2 possibles list. I would probably not expect COSI API support for until after v1 GA, but anything past v1a2 is flexible if we get more user requests. #4

(The issue is currently closed by the auto-bot. I'll proabably leave it closed since it's a longterm backlog, but it is in the kanban board so it's not lost.)

Addendum:

Something I'm reading into Josh's usecase that I have had on my mind, I think bears clarifying: COSI should assume support for complex arrangments between multiple accesses and multiple buckets including sharing.

This is a contrived example based on nothing, but illustrative of a complex arrangement:

Buckets: A, B, C
Accesses: X, Y
RW modes in parens

A ------(RO)----┐
│               │
└---(WO)-----┐  │
             X  │
┌---(RO)-----┘  │ 
│               │
B ------(RO)--- Y
                │
C ------(RW)----┘

[gattytto]

instead of having a single secret for a multiple-buckets bucketaccess, another way could be creating separate secrets for each buckeAccess=>bucketClaim pair and append the bucketclaim name to the end of the secret name.

this way it would still honor the "secretName" property for single bucketClaim bucketAccess and also covers the 1 to many requirement.

[BlaineEXE]

Thanks all for the inputs so far! 🙂

It is seeming like multiple secrets are likely better as the end-user API. I updated the KEP to reflect this. The current proposal (still under review) is to allow a different user-defined secret name for each referenced BucketClaim, configured by end users like so:

# ...
kind: BucketAccess
spec:
  bucketClaims:
    - bucketClaimName: claim-1
      accessSecretName: secret-1
      accessMode: ReadWrite
    - bucketClaimName: claim-2
      accessSecretName: secret-2
      accessMode: ReadOnly
  # ...

Continued feedback is welcome and appreciated

[satoru-takeuchi]

It might be nice if we can create a ConfigMap or Secret per BucketClaim which contains bucket specific info and create a Secret for each BucketAccess. It would be like this (envvars are from S3 object storage's ones in COSI alpha1v2 KEP):

• ConfigMap/Secret for each BucketClaim:
  • COSI_S3_ENDPOINT: S3 endpoint URL, e.g., https://s3.amazonaws.com
  • COSI_S3_BUCKET_ID: S3 bucket ID (must be client-facing OSP bucket ID)
  • COSI_S3_REGION: S3 region, e.g., us-west-1
  • COSI_S3_SIGNATURE_VERSION: signature version for signing all S3 requests
• Secret for each BucketAccess:
  • COSI_S3_ACCESS_KEY_ID: S3 access key ID, e.g., AKIAIOSFODNN7EXAMPLE
  • COSI_S3_ACCESS_SECRET_KEY: S3 access secret key, e.g., wJalrXUtnFEMI/K...

It's the similar way as OB/OBC. This way has pros/cons.

• Pros:
  • We can separate secret information and non-secret information.
  • It's not necessary to introduce accessSecretName for each bucketClaimRef.
    • NOTE: If there are (BucketAccess,BucketClaim)-pair specific secret information, we still need Secret for each pair. IIUC, at least S3 doesn't have such information. We can revisit this problem after v1alpha2.
  • We can avoid redundancy. If 10 BucketClaims are referred from one BucketAccess, we can avoid the duplicated definition of two envvars for credential information.
• Cons:
  • Users are forced to specify both ConfigMap and Secret in their Pod definition.

Any thought?

[BlaineEXE]

I think the most compelling reason to keep a Secret-only design stems from security/risk planning. It's safer if COSI assumes that any bucket info might be considered sensitive. For most drivers and users, I don't expect that to be true, but it doesn't seem unreasonable that some storage backends might prefer to keep some of the information protected. By keeping info only in Secrets, COSI can avoid a certain amount of risk of making the wrong decision about what info an admin or driver wants to protect.

[jhoblitt]

kind: BucketAccess
spec:
bucketClaims:
- bucketClaimName: claim-1
accessSecretName: secret-1
accessMode: ReadWrite

I think it is important to allow the field names in the secret to be specified. I commonly need to use s3 credentials for multiple deployments in the same namespace and it is an irritation to have to either duplicate the secret or duplicate fields within the secret in order to match hard-coded secret field names. I feel it is a general principal that applications shouldn't require a fixed format for secret field names.

[satoru-takeuchi]

I think it is important to allow the field names in the secret to be specified. I commonly need to use s3 credentials for multiple deployments in the same namespace and it is an irritation to have to either duplicate the secret or duplicate fields within the secret in order to match hard-coded secret field names. I feel it is a general principal that applications shouldn't require a fixed format for secret field names.

@jhoblitt It's like this?

kind: BucketAccess
spec:
  bucketClaims:
    - bucketClaimName: claim-1
      accessSecretName: secret-1
      envVarPrefix: SECRET_1_
      accessMode: ReadWrite
    - bucketClaimName: claim-2
      accessSecretName: secret-2
      envVarPrefix: SECRET_2
      accessMode: ReadOnly