Some K8s Questions #481

Activity

1. Barre commented on Jun 29, 2026

   

   Barre

   on Jun 29, 2026

   Last edited by Barre

   Owner

   More actions

   Hi,

   Cache (emptyDir):

   • It is safe, and there is no data-loss risk. The on-disk cache is a read-through cache of decrypted SST blocks; all durable state is in S3 and unflushed writes live in memory, not in the cache dir. Losing it just means a cold cache.

   Gateway topology:

   • Two single-replica StatefulSets are used because the two nodes need asymmetric config (node_id, peers, replication_listen, separate Secrets). A single STS works too but needs an entrypoint/init container to template that config from the pod ordinal, plus anti-affinity. I believe the static two-STS form is just easier to read as a reference.

   Compactor:

   • Yes, a separate Deployment is supported and is a good pattern for write-heavy setups: run gateways with zerofs run --no-compactor and a Deployment of stateless zerofs compactor workers (no identity/volume needed, safe to run many).
   • Autoscaling: workers are stateless, so an HPA on CPU/network could work.
2. andrewheberle commented on Jun 29, 2026

   

   andrewheberle

   on Jun 29, 2026

   ContributorAuthor

   More actions

   Thanks for the quick response.

   My concern with the two seperate StatefulSet's was mainly that there is no coordination between them for changes that might affect both when used with a GitOps solution like Fleet or Flux (eg updating the image version). We use Fleet in our case, so I have the gateway running as a two replica StatefulSet but with two different configurations named based on the seperate pod names and those are referenced via an environment variable in the pod args as follows:

   spec:
     template:
       spec:
         containers:
           - name: zerofs
             args: ["run", "-c", "/etc/zerofs/$(POD_NAME).toml", "--no-compactor"]
             env:
               - name: POD_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.name
   

   Having said that as an example of deploying the Gateway the current manifest is probably easier to understand as you said, and simple enough to adapt for someone that wants to do this.

   Thanks again.
3. andrewheberle commented on Jun 29, 2026

   

   andrewheberle

   on Jun 29, 2026

   Last edited by andrewheberle

   ContributorAuthor

   More actions

   Another question I wasn’t sure about was relating to the WAL feature and if both gateways in the HA pair need access to the same WAL storage?

   The documentation says the WAL and HA are compatible but does this need to be the same storage or could it be a local PVC?

   If it needs to be the “same” storage could it be a ReadWriteMany PVC or is S3 the only option here?

   Finally if shared object storage is required for the WAL is there any advantage to configuring a WAL to the same S3 storage provider/tier as the standard storage used for ZeroFS or does it only help if WAL storage is faster/lower latency?

   Also if there is a better way to ask these sort of questions without cluttering up your GitHub Issues please let me know.
4. Barre commented on Jun 29, 2026

   

   Barre

   on Jun 29, 2026

   Owner

   More actions

   Another question I wasn’t sure about was relating to the WAL feature and if both gateways in the HA pair need access to the same WAL storage?

   The documentation says the WAL and HA are compatible but does this need to be the same storage or could it be a local PVC?

   Yes, they do need to access the same storage. The WAL holds what hasn't yet been materialised into L0 SSTs. Essentially, it only exists to absorb fsync latency rather than creating an L0 per fsync.

   If it needs to be the “same” storage could it be a ReadWriteMany PVC or is S3 the only option here?

   This would work!

   Finally if shared object storage is required for the WAL is there any advantage to configuring a WAL to the same S3 storage provider/tier as the standard storage used for ZeroFS or does it only help if WAL storage is faster/lower latency?

   If your goal is more performance, local NVMe or something like S3 Express would work great. Otherwise, using the same bucket or a different bucket won't make any difference (assuming latency to both buckets is the same).

   Also if there is a better way to ask these sort of questions without cluttering up your GitHub Issues please let me know.

   There's a github discussions tab on this repo. I just migrated this issue there as well.
5. Repository owner locked and limited conversation to collaborators on Jun 29, 2026
6. Barre

   converted this issue into a discussion #484 on Jun 29, 2026