Enable readOnlyRootFilesystem to complete restricted Pod Security Standard compliance

[roman1887]

Problem

#241 added pod and container security contexts to all seven deployments in .helm-charts/dashboard:

runAsNonRoot: true
runAsUser: <10001 | 101 | 1000>
seccompProfile: { type: RuntimeDefault }
allowPrivilegeEscalation: false
privileged: false
capabilities: { drop: [ALL] }

This meets the Kubernetes restricted Pod Security Standard on every control except readOnlyRootFilesystem, which is wired as a per-service toggle but defaults to false.

It was left off deliberately: several services write scratch data at runtime and would crashloop with a read-only root, and shipping that untested would have been worse than shipping it off. But it means a cluster enforcing the full restricted profile will still reject these pods, which is exactly the kind of gate enterprise customers apply.

Known write paths

Service	Writes to
schema-service	model cache (/app/model_cache, already a volume), HuggingFace cache
nexus	litellm cache, matplotlib config, /tmp
prediction workers	matplotlib config, /tmp, model scratch
frontend	nginx temp paths (/tmp, /var/cache/nginx)
trino	JVM scratch, /tmp

Unverified — enable per service and observe what actually fails.

Recommended fix

Per service, one at a time:

1. Add an emptyDir for each write path:

   volumes:
     - name: tmp
       emptyDir: {}
   volumeMounts:
     - name: tmp
       mountPath: /tmp

2. Set the relevant cache env vars so libraries write inside the mount rather than $HOME:

   env:
     - name: MPLCONFIGDIR
       value: /tmp/matplotlib
     - name: HF_HOME
       value: /tmp/huggingface

3. Flip the toggle — the helper already accepts it:

   {{- include "dashboard.containerSecurityContext" (dict "readOnlyRootFilesystem" true) | nindent 10 }}

4. Deploy and confirm the pod reaches Ready and stays there under load. A read-only filesystem failure often appears only on the first write, which may be well after startup — exercise the service, do not just check that it boots.

Suggested order, easiest first: frontend (nginx-unprivileged is designed for this), then the prediction workers, then nexus, then schema-service and trino.

Definition of done

• All seven deployments run with readOnlyRootFilesystem: true
• Pods pass admission under a namespace labelled pod-security.kubernetes.io/enforce=restricted
• security/CONTAINER_SECURITY.md section 6 updated to drop the caveat

Related: #241