[Draft] feat : support tmpfs provisioned volumes

[pranav767]

Pull Request

Closes #11446

What? (description)

• Add tmpfs support for ephemeral & STATE volumes

Why? (reasoning)

Acceptance

Please use the following checklist:

• you linked an issue (if applicable)
• you included tests (if applicable)
• you ran conformance (make conformance)
• you formatted your code (make fmt)
• you linted your code (make lint)
• you generated documentation (make docs)
• you ran unit-tests (make unit-tests)

See make help for a description of the available targets.

[pranav767]

Hey @smira,
I tried creating a cluster with my changes, and add following UserVolumeConfig:

---
apiVersion: v1alpha1
kind: UserVolumeConfig
name: testmem
volumeType: memory

This works just fine,

_out/talosctl-linux-amd64 get volumestatus --nodes 172.20.0.3 u-testmem

NODE         NAMESPACE   TYPE           ID          VERSION   TYPE     PHASE   LOCATION   SIZE
172.20.0.3   runtime     VolumeStatus   u-testmem   2         memory   ready

_out/talosctl-linux-amd64 read /proc/mounts --nodes 172.20.0.3 | grep testmem

tmpfs /var/mnt/testmem tmpfs rw,seclabel,relatime,size=102400k 0 0

But to do a test on ephemeral volume to be setup as tmpfs, we'd have to add that config before boot , i.e. during cluster creation,
When I set this up the cluster creation hangs with failures from kubelet being stuck in initializing , Controllers do pass initialization phase(the ephemeral volume does get a memory volume status)

Any suggestions where should I look in does ephemeral volume have any role with kubelet being setup during boot 🤔

[smira]

I don't think user volumes backed by tmpfs makes a lot of sense in general? As one could simply use emptyDir mounts backed by tmpfs.

The reason why EPHEMERAL doesn't work would require some investigation

[shanduur]

What's the reasoning about adding memory instead reusing tmpfs? Is there functionality that will be hard to split?

[pranav767]

To keep validation and lifecycle semantics separate for different kinds of volumes. See #11446 (comment)

[pranav767]

There was an issue with kubelet waiting on the runtime mount status. I’ve fixed that, and with the changes in this PR, I’m now able to successfully start a tmpfs-provisioned EPHEMERAL volume.
With the use of new volume type memory we can move directly to ready with few validations (no encryption, no disk selector, but size is expected)

I'm worried that the memory volume mounts tmpfs via a direct unix.Mount and only uses size option, while disk/partition go through mount.Manager. If we later add more mount options like nodev, nosuid, noatime, it might get easy to forget updating other options,
should we maybe route tmpfs through manager

For now, sticking in working logs for tmpfs provisioned EPHEMERAL volumes:

Starting the cluster with a simple config patch as:

apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
volumeType: memory
provisioning:
  minSize: "4GiB"

sudo --preserve-env=HOME _out/talosctl-linux-amd64 -n $WORKER_IP get volumestatuses.block.talos.dev EPHEMERAL -o yaml
node: 172.20.0.3
metadata:
    namespace: runtime
    type: VolumeStatuses.block.talos.dev
    id: EPHEMERAL
   ...
spec:
    phase: ready
    type: memory
    size: 4294967296
    prettySize: 4.3 GB
    mountSpec:
        targetPath: /var
        selinuxLabel: system_u:object_r:ephemeral_t:s0
        projectQuotaSupport: false
        fileMode: 493
        parameters:
            - type: string
              name: size
              string: "4294967296"

sudo --preserve-env=HOME _out/talosctl-linux-amd64 -n $WORKER_IP get volumemountstatuses.block.talos.dev EPHEMERAL -o yaml
node: 172.20.0.3
metadata:
    namespace: runtime
    type: VolumeMountStatuses.block.talos.dev
    id: EPHEMERAL
    labels:
        mount-status-id: EPHEMERAL
spec:
    volumeID: EPHEMERAL
    requester: sequencer
    target: /var
    readOnly: false
    detached: false

jinx@jinx:~/siderolabs/talos$ sudo --preserve-env=HOME _out/talosctl-linux-amd64 -n $WORKER_IP read /proc/mounts | grep -E ' /var '
tmpfs /var tmpfs rw,seclabel,relatime,size=4194304k 0 0

Let me know if this approach is correct or needs changes as an initial draft , before proceeding with STATE volumes.

[github-actions]

This PR is stale because it has been open 45 days with no activity.

[smira]

I guess if we want to be closer to real usecase, we should have tmpfs only for workers

I think we should still test a cluster (that is more than a single worker, and at least a single controlplane)

[smira]

is this related?

[pranav767]

Irrelevant changes, that was me trying to remove some warning/error noise during boot, can be removed.

[smira]

do we re-create tmpfs every time we mount? this sounds wrong, e.g. for STATE: we should create tmpfs probably as part of Volume provisioning, and only mount to the tree/unmount for the lifetime of the volume

[pranav767]

Yes that's true.
A bit confused atm, @shanduur have you taken over the PR? I don’t want to step on your toes. Am I expected to make this change?

[shanduur]

@pranav767, thank you for your contribution!

We had some internal discussions about the direction where should we take it, and I had some spare time to check your branch and experiment on it a bit. It had drifted quite a bit, so I thought it might be easier if I push rebased PR with some fixes.

Do you want to continue work on this? Or do you want me to fully take it over and finish the implementation?

[pranav767]

I'd love to continue the work here, if explaining the internal discussions is not too much trouble for you.

[shanduur]

Sure! So there are two goals:

1. Tests
   We need to make sure the ephemeral node is working correctly. There are two main use-cases that need to be tested. We need to go through the existing tests and find fix (or skip) tests that should not work on tmpfs-backed EPHEMERAL and STATE partitions.

   1. Ephemeral worker nodes in clusters - this is not tested yet - we need to add new integration/misc suite;
   2. Ephemeral single node cluster - this has some basic tests already added - we need to move them to integration/misc suite;

2. In-memory Talos
   Right now there is still some space for improvement that should be deferred to next PR. Right now Talos will attempt to install itself when there is no META partition. This means we will still require some form of storage. But the end goal is to run Talos on devices with no storage whatsoever. This means that talos performs network boot, and if the machine.install config (or new v1alpha1 equivalent) is missing, it proceeds to start up as a fully ephemeral in memory node acquiring it's configuration from e.g. a HTTP server. This should work only for worker nodes.

[github-actions]

This PR is stale because it has been open 45 days with no activity.

[shanduur]

Hi @pranav767 ! Any updates on this PR? Should we take it over?

[pranav767]

Apologies for the delay @shanduur , you can take over.

[dsseng]

Rebasing it step by step to accommodate newer features (#13655 will enable more useful configs, such as put META/STATE/ETCD on the persistent storage, but EPHEMERAL in RAM)

[dsseng]

The fixup commit I've added migrates to the current Kubernetes config documents