Dynamic provisioning ignores securityContext

[benethon]

/kind bug

What happened?

I have two clusters, one v1.27 named eks-poc and another new one created recently on v1.28 called eks-dev. On eks-dev, I noticed a problem that despite setting a securityContext to force the fsGroup user of the mounted volume to be 1035 [1], it doesn't respect this and instead sets it at 1999 (one off the upper limit set in the storage class [2]).

We didn't have this problem on eks-poc, but we updated it this morning to v1.28 and the problem appeared, so to me it seems like the issue is related to Kubernetes 1.28.

I installed the EFS driver manually on eks-poc and using the EKS Add-on on eks-dev. The image of the efs-plugin container in the efs-csi-controller pod on eks-poc is 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-efs-csi-driver:v1.5.7 and the image on eks-dev is 602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/aws-efs-csi-driver:v1.7.1 - so different versions, but the common thing that makes it stop working is Kubernetes 1.28.

Some other thing's we've tried: rolling back the eks-dev EFS driver version to 1.5.7 - still happens, but the POSIX user is now 1000 rather than 199x. I haven't checked the change-log but I assume it was switched around to count down from the maximum gId, as evidenced by the log line "Allocator found GID which is already in use: -1 - trying next one."

What you expected to happen?

mounted volume to be owned by user 1035 from the securityContext, not the one set by the provisioner

How to reproduce it (as minimally and precisely as possible)?

[1] Statefulset YAML (also happening on other deployments, also note the command to override the entrypoint):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  minReadySeconds: 10
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: postgres
  serviceName: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - command:
        - sleep
        - infinity
        env:
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        image: timescale/timescaledb:2.12.1-pg14
        imagePullPolicy: IfNotPresent
        name: postgres
        ports:
        - containerPort: 5432
          protocol: TCP
        resources: {}
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgresdb
          subPath: pgdata
      restartPolicy: Always
      securityContext:
        fsGroup: 1035
        fsGroupChangePolicy: Always
        runAsNonRoot: true
        runAsUser: 1035
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: postgresdb
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: efs-sc
      volumeMode: Filesystem

[2] StorageClass:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: efs-sc
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: "700"
  fileSystemId: fs-redacted
  gidRangeEnd: "2000"
  gidRangeStart: "1000"
  provisionerID: efs.csi.aws.com
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Retain
volumeBindingMode: Immediate

This outputs (with 1.5.7) - note the 1000 user, not 1035

postgres-0:/$ ls -lah /var/lib/postgresql/
total 4K
drwxr-xr-x    1 postgres postgres      18 Oct  6 01:04 .
drwxr-xr-x    1 root     root          24 Oct  6 01:04 ..
drwx------    2 1000     1000        6.0K Nov 29 13:23 data

Environment

• Kubernetes version (use kubectl version): see above
• Driver version: see above

Please also attach debug logs to help us better diagnose

Attached
csi-provisioner.txt
efs-plugin.txt

[benethon]

Some other things we've tried today:

• Replacing the statefulset with a minimal pod definition with no security context - same result as above running ls.
• Removing the gidRangeEnd and gidRangeStart params from the storage class, and setting the directoryPerms to 777
• Downgrading CSI driver version to 1.6

[seanzatzdev-amazon]

Hi @benethon , thank you for bringing this to our attention, apologies for the delayed response. I have reproduced this behavior on v1.7.1 of the driver, with k8s v1.28. We will have to push out a PR to address this so that the securityContext is not ignored.

postgres-0:/$ ls -lah /var/lib/postgresql/
total 4K     
drwxr-xr-x    1 postgres postgres      18 Oct  6 01:04 .
drwxr-xr-x    1 root     root          24 Oct  6 01:04 ..
drwx------    2 2000     2000        6.0K Dec  8 20:32 data
postgres-0:/$ exit
exit
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 csi-driver-1202]$ cat storageclass.yaml 
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: efs-sc
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: "700"
  fileSystemId: fs-redacted
  gidRangeEnd: "2000"
  gidRangeStart: "1000"
  provisionerID: efs.csi.aws.com
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 csi-driver-1202]$ cat statefulset.yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  minReadySeconds: 10
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: postgres
  serviceName: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - command:
        - sleep
        - infinity
        env:
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        image: timescale/timescaledb:2.12.1-pg14
        imagePullPolicy: IfNotPresent
        name: postgres
        ports:
        - containerPort: 5432
          protocol: TCP
        resources: {}
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgresdb
          subPath: pgdata
      restartPolicy: Always
      securityContext:
        fsGroup: 1035
        fsGroupChangePolicy: Always
        runAsNonRoot: true
        runAsUser: 1035
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: postgresdb
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: efs-sc
      volumeMode: Filesystem

[snowmanstark]

@seanzatzdev-amazon Do we have any progress on this? I am facing this issue in k8s 1.29 too.

@benethon Did you face the issue in k8s version < 1.27? if yes, then how did you work around it (if you managed to by any chance)?

[benethon]

@snowmanstark no, we didn't try anything less than 1.27. Worked around it temporarily by using an EBS volume rather than EFS

[snowmanstark]

@nishant221 does your PR #1152 fix for this issue? if yes, then did 1.7.6 release ship #1152?

[snowmanstark]

@seanzatzdev-amazon Is the fix for this issue in 1.7.6 release?

[gbsingh1993]

@seanzatzdev-amazon Is there any update on this fix. My stateful set is completely useless without this being fixed

[jseiser]

Same problem here.

Mounted as 1002

❯ kubectl exec -it pod/atlantis-0 -n atlantis -c atlantis -- bash
atlantis@atlantis-0:/$ ls -lah
total 4.0K
drwxr-xr-x.   1 root root   72 May 15 13:23 .
drwxr-xr-x.   1 root root   72 May 15 13:23 ..
drwxrwxr-x.   4 1002 1002 6.0K May 15 13:23 atlantis-data

context is 1000

  securityContext:                                                                                                                                                                                                                            
    fsGroup: 1000                                                                                                                                                                                                                             
    fsGroupChangePolicy: Always                                                                                                                                                                                                               
    runAsUser: 1000 

pvc

❯ kubectl get pvc -n atlantis atlantis-data -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: atlantis
    meta.helm.sh/release-namespace: atlantis
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
    volume.kubernetes.io/storage-provisioner: efs.csi.aws.com
  creationTimestamp: "2024-05-15T13:13:15Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: atlantis
    app.kubernetes.io/managed-by: Helm
    chart: atlantis-5.0.2
    helm.sh/chart: atlantis-5.0.2
    heritage: Helm
    release: atlantis
  name: atlantis-data
  namespace: atlantis
  resourceVersion: "565062419"
  uid: 058bb4a6-8151-4eed-bff6-94e1e3de065c
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: efs-sc
  volumeMode: Filesystem
  volumeName: pvc-058bb4a6-8151-4eed-bff6-94e1e3de065c
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound

SC

❯ kubectl get storageclass efs-sc -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: aws-efs-csi-driver
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2022-03-30T18:58:11Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: efs-sc
  resourceVersion: "72583228"
  uid: 481c4ac5-b2e5-4c25-bbfb-a07c5104532b
mountOptions:
- tls
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: "775"
  fileSystemId: fs-00a080564b9d87ff4
  gidRangeEnd: "2000"
  gidRangeStart: "1000"
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

[gbsingh1993]

Same problem here.

Mounted as 1002

❯ kubectl exec -it pod/atlantis-0 -n atlantis -c atlantis -- bash

atlantis@atlantis-0:/$ ls -lah

total 4.0K

drwxr-xr-x.   1 root root   72 May 15 13:23 .

drwxr-xr-x.   1 root root   72 May 15 13:23 ..

drwxrwxr-x.   4 1002 1002 6.0K May 15 13:23 atlantis-data

context is 1000

  securityContext:                                                                                                                                                                                                                            

    fsGroup: 1000                                                                                                                                                                                                                             

    fsGroupChangePolicy: Always                                                                                                                                                                                                               

    runAsUser: 1000 

pvc

❯ kubectl get pvc -n atlantis atlantis-data -o yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  annotations:

    meta.helm.sh/release-name: atlantis

    meta.helm.sh/release-namespace: atlantis

    pv.kubernetes.io/bind-completed: "yes"

    pv.kubernetes.io/bound-by-controller: "yes"

    volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com

    volume.kubernetes.io/storage-provisioner: efs.csi.aws.com

  creationTimestamp: "2024-05-15T13:13:15Z"

  finalizers:

  - kubernetes.io/pvc-protection

  labels:

    app: atlantis

    app.kubernetes.io/managed-by: Helm

    chart: atlantis-5.0.2

    helm.sh/chart: atlantis-5.0.2

    heritage: Helm

    release: atlantis

  name: atlantis-data

  namespace: atlantis

  resourceVersion: "565062419"

  uid: 058bb4a6-8151-4eed-bff6-94e1e3de065c

spec:

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: 50Gi

  storageClassName: efs-sc

  volumeMode: Filesystem

  volumeName: pvc-058bb4a6-8151-4eed-bff6-94e1e3de065c

status:

  accessModes:

  - ReadWriteOnce

  capacity:

    storage: 50Gi

  phase: Bound

SC

❯ kubectl get storageclass efs-sc -o yaml

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  annotations:

    meta.helm.sh/release-name: aws-efs-csi-driver

    meta.helm.sh/release-namespace: kube-system

  creationTimestamp: "2022-03-30T18:58:11Z"

  labels:

    app.kubernetes.io/managed-by: Helm

  name: efs-sc

  resourceVersion: "72583228"

  uid: 481c4ac5-b2e5-4c25-bbfb-a07c5104532b

mountOptions:

- tls

parameters:

  basePath: /dynamic_provisioning

  directoryPerms: "775"

  fileSystemId: fs-00a080564b9d87ff4

  gidRangeEnd: "2000"

  gidRangeStart: "1000"

  provisioningMode: efs-ap

provisioner: efs.csi.aws.com

reclaimPolicy: Delete

volumeBindingMode: Immediate

I had to use a work around and make a new storage class on which I set the gid and uid to what I need. And then in my statefulset I use this new storage class. Hope that helps. If you would like to see the manifest let me know and I can share that I get home.

[jseiser]

@gbsingh1993

Thats what I had to do as well, create a specific SC just for this app. Which is fine, but its def. not ideal.

[armhart]

Create a storage-class with a fix groupid 1000 works for me:

> cat storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: efs-csi
    meta.helm.sh/release-namespace: core-efs-csi
  labels:
    app.kubernetes.io/managed-by: Helm
  name: efs-sc-gid1000
parameters:
  directoryPerms: "700"
  fileSystemId: fs-[YOUR_FLE_SYSTEM_ID]
  gid: "1000"
  uid: "1000"
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

> kubectl apply -f storageclass.yaml

And use:

storageClassName: efs-sc-gid1000

And use:

      securityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsGroup: 1000

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jean-francoismougnot]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dankova22]

Hi all,

EFS CSI Driver uses access points for dynamic provisioning. When you create a volume, we dynamically select a GID from range for this PV, create an access point with this GID, and later mount the volume using this access point.

When you use access points, access control is done on the EFS side and identity will be enforced for all access through that AP. Even with mismatch of directory ownership and pod identity, the pod should still be able to access the volume.

Changing the ownership of the access point root dir would mean that we can no longer access the volume through the access point. This means for dynamic provisioning with efs-ap mode, we do not support the fsGroup feature.

If you require the volume to be owned by a specific GID, we do support setting a static gid in the storage class which will be applied at access point creation time.

Please let me know if any follow up questions/concerns