Deleting nodes

[james-callahan]

Feature Request

When a node is deleted, e.g. from an EC2 ASG via terminate-instance-in-autoscaling-group, what should inform kubernetes and run delete node?

Description

Normally using the aws-cloud-controller-manager it would do that job; but this project replaces that.

I would expect

talos-cloud-controller-manager/pkg/talos/instances.go

Line 49 in 5a0623f

func (i *instances) InstanceExists(_ context.Context, node *v1.Node) (bool, error) {

to reach out and check with AWS for the existence of the node. Currently it only supports some sort of timeout based GCP spot instance flow.

I understand that for Karpenter uses, this is taken care of? But otherwise it seems like an unfulfilled role that normatively is meant to be done by the cloud provider.
I also note that aws-node-termination-handler doesn't currently offer this functionality: aws/aws-node-termination-handler#719

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request

[sergelogvinov]

Talos CCM is responsible only for the node initialization stage https://github.com/siderolabs/talos-cloud-controller-manager/blob/main/docs/controllers.md

Because in a hybrid environment, it is not possible to use the official CCM plugins. They are designed to work only with their own specific cloud provider and will remove any other resources that do not belong to that provider.

To support the node-lifecycle controller (for the removal process), the CCM must either: know how to interact with all cloud providers or provide a way to extend it with plugins.

Better to use two or more CCMs in one cloud.

• Talos CCM with cloud-node controller
• Othe CCMs with node-lifecycle controller

PS. The GCP implementation is essentially a workaround to fix spot (preemptible) instances. This is because GCP changes the VM’s IP address after a restart, which affects the proper operation of Kubernetes addons.

Thanks.

[james-callahan]

Othe CCMs with node-lifecycle controller

We're trying this; but there appears to be some sort of race with the AWS cloud provider where it will sometimes delete the node:

E0908 21:05:25.431404       1 node_lifecycle_controller.go:156] error checking if node i-0ff35ea0889246039 exists: getInstanceByNodeName failed for "i-0ff35ea0889246039" with "error listing AWS instances: \"InvalidInstanceID.NotFound: The instance ID 'i-0ff35ea0889246039' does not exist\\n\\tstatus code: 400, request id: adb6dd3f-b80f-428b-b778-baf03e242d26\""
E0908 21:05:30.529149       1 node_lifecycle_controller.go:156] error checking if node i-0ff35ea0889246039 exists: getInstanceByNodeName failed for "i-0ff35ea0889246039" with "error listing AWS instances: \"InvalidInstanceID.NotFound: The instance ID 'i-0ff35ea0889246039' does not exist\\n\\tstatus code: 400, request id: 5d25e56a-e9a0-4614-a97c-2e1d7ebed1d2\""
E0908 21:05:35.627604       1 node_lifecycle_controller.go:156] error checking if node i-0ff35ea0889246039 exists: getInstanceByNodeName failed for "i-0ff35ea0889246039" with "error listing AWS instances: \"InvalidInstanceID.NotFound: The instance ID 'i-0ff35ea0889246039' does not exist\\n\\tstatus code: 400, request id: ec381106-97be-411f-a3c2-1a078f990259\""
E0908 21:05:40.723064       1 node_lifecycle_controller.go:156] error checking if node i-0ff35ea0889246039 exists: getInstanceByNodeName failed for "i-0ff35ea0889246039" with "error listing AWS instances: \"InvalidInstanceID.NotFound: The instance ID 'i-0ff35ea0889246039' does not exist\\n\\tstatus code: 400, request id: 551aa61a-efb1-42ad-b44f-b1bbc840c842\""
I0908 21:05:45.822636       1 node_lifecycle_controller.go:163] deleting node since it is no longer present in cloud provider: i-0ff35ea0889246039
I0908 21:05:45.822930       1 event.go:389] "Event occurred" object="i-0ff35ea0889246039" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="DeletingNode" message="Deleting node i-0ff35ea0889246039 because it does not exist in the cloud provider"

Other times it won't delete it; but it does continuously log:

E0908 21:10:42.204598       1 node_lifecycle_controller.go:156] error checking if node i-0c7d8b13101f076cf exists: getInstanceByNodeName failed for "i-0c7d8b13101f076cf" with "error listing AWS instances: \"InvalidInstanceID.NotFound: The instance ID 'i-0c7d8b13101f076cf' does not exist\\n\\tstatus code: 400, request id: 5ea6f461-ac89-453b-bd0a-2df0a535bf5b\""

Upstream has a stale issue: kubernetes/cloud-provider-aws#939

We're still trying to use this dual-cloud-controller setup for now, but it doesn't feel stable.

[TomyLobo]

I'm trying to get Proxmox, Talos, Cilium and proxmox-csi-plugin to work together.
proxmox-csi-plugin needs labels, set by proxmox-ccm, but I can make do with setting the labels manually if necessary.

Here's what I found out, through experimentation and looking at the source:

• The cloud-node controller of proxmox-ccm sets .spec.providerID to proxmox://<region>/<VM ID> (or proxmox://<UUID> in the case of CAPMOX)
• The cloud-node controller of talos-ccm sets .spec.providerID to talos://<platform>/<node IP>
• The cloud-node-lifecycle controller of proxmox-ccm expects .spec.providerID to match ^proxmox://([^/]*)/([^/]+)$
• If I use the cloud-node controller of talos-ccm with the cloud-node-lifecycle controller of proxmox-ccm, the later does nothing, because none of the nodes' providerIDs match.
• If I use both the cloud-node and cloud-node-lifecycle controllers of proxmox-ccm together with Cilium, the nodes won't initialize because proxmox-ccm needs a working CNI in order to work and Cilium needs initialized nodes in order to work.
• If I use both the cloud-node and cloud-node-lifecycle controllers of talos-ccm, Node objects aren't destroyed, because InstanceExists isn't implemented to really do anything except on GCP.

So that leaves me 3 options:

1. Ignore the fact that the Node objects aren't removed
2. Not use Cilium
3. Use GCP

The first one is not ideal, the others are incompatible with the requirements I got.

Can talos-ccm look at the Talos API (through the talos.default service) to get the list of cluster members?
If I run talosctl get members on any single node, it returns a list of all nodes.
So once the API equivalent of talosctl get members no longer returns a response containing that node, talos-ccm's cloud-node-lifecycle controller can remove the Node resource.

[sergelogvinov]

Other times it won't delete it; but it does continuously log:

E0908 21:10:42.204598       1 node_lifecycle_controller.go:156] error checking if node i-0c7d8b13101f076cf exists: getInstanceByNodeName failed for "i-0c7d8b13101f076cf" with "error listing AWS instances: \"InvalidInstanceID.NotFound: The instance ID 'i-0c7d8b13101f076cf' does not exist\\n\\tstatus code: 400, request id: 5ea6f461-ac89-453b-bd0a-2df0a535bf5b\""

Upstream has a stale issue: kubernetes/cloud-provider-aws#939

We're still trying to use this dual-cloud-controller setup for now, but it doesn't feel stable.

Hello, I don’t have experience with Talos + AWS, but I know that running multiple CCMs in one cluster (Azure, Oracle, GCP, etc.) usually works. Could you try running the AWS CCM alongside and check if it works?
It seems like it might require additional flags or permissions. Perhaps the region name needs to be set correctly, or the CCM might be missing permissions to retrieve metadata from the EC2 network.

[TomyLobo]

@sergelogvinov should I create a new issue for what I detailed in my comment or does it fit into this one?

[sergelogvinov]

Hello,

Everything is achievable, but you need to clarify a few points first:

• How will you add a new node? (e.g., using Terraform or manually)
• How will you pass the provider ID (or VM-ID) to Talos?
  • via cloud-init user-data (Talos machine config)
  • via cloud-init metadata
  • via an SMBIOS argument

user-data: see https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

machine:
  type: worker
  kubelet:
    extraArgs:
      provider-id: proxmox://<region>/<VMID>

metadata:

hostname: ${hostname}
instance-id: ${id}
provider-id: ${providerID}
region: ${region}
zone: ${zone}

smbios: see https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html#dmi-specific-kernel-command-line

serial: h=<host-name>;i=<VM-ID>

Depending on your choice, you may also need to apply Talos transformation rules:

transformations:
  - name: nocloud
    nodeSelector:
      - matchExpressions:
          - key: platform
            operator: In
            values:
              - nocloud
    platformMetadata:
      InstanceType: "{{ coalesce .InstanceType .SKUNumber }}"
      ProviderID: '{{ if .SerialNumber }}proxmox://region-1/{{ getValue .SerialNumber "i" }}{{ else }}{{ .ProviderID }}{{ end }}'

PS, CCMs can work without CNI https://github.com/siderolabs/talos-cloud-controller-manager/blob/main/charts/talos-cloud-controller-manager/values.yaml

# -- Deploy CCM  in Daemonset mode.
# CCM will use hostNetwork and connect to the Kubernetes API server on the current node by default.
# Optionally you can specify the Kubernetes API server host and port.
# You can run it without CNI plugin.
daemonSet:
  enabled: false
  k8s:
    # -- Kubernetes API server host. Default is the current node IP.
    serviceHost: ""
    # -- Kubernetes API server port. Default is 6443.
    servicePort: 6443

PS2, TalosCCM + ProxmoxCCM + ProxmoxCSI works well, I have a few production clusters like this.
May be it will helps too

• https://github.com/sergelogvinov/terraform-talos/tree/main/proxmox
• https://github.com/sergelogvinov/karpenter-provider-proxmox

[james-callahan]

Hello, I don’t have experience with Talos + AWS, but I know that running multiple CCMs in one cluster (Azure, Oracle, GCP, etc.) usually works. Could you try running the AWS CCM alongside and check if it works?

@sergelogvinov that's exactly what my post was showing.
It works most of the time; but on two occasions I've seen the AWS CCM decide to delete a node it shouldn't have; and it sometimes gets in a logging loop.

[sergelogvinov]

Hello, I don’t have experience with Talos + AWS, but I know that running multiple CCMs in one cluster (Azure, Oracle, GCP, etc.) usually works. Could you try running the AWS CCM alongside and check if it works?

@sergelogvinov that's exactly what my post was showing. It works most of the time; but on two occasions I've seen the AWS CCM decide to delete a node it shouldn't have; and it sometimes gets in a logging loop.

I’ve reread the issue, and we are discussing the situation where TalosCCM (init) and AwsCCM (lifecycle) together cause a problem during scale-up/down events. Is that correct? @james-callahan

[james-callahan]

I’ve reread the issue, and we are discussing the situation where TalosCCM (init) and AwsCCM (lifecycle) together cause a problem during scale-up/down events. Is that correct? @james-callahan

Yes (though I'm not sure it's limited to problems during scale up/scale down)

[james-callahan]

Today we saw the kubernetes deletion behaviour again from the AWS cloud controller:

$ kubectl -n aws-cloud-controller-manager logs pod/aws-cloud-controller-manager-wl54c | grep i-0ace0851f6aa38476
I1002 22:16:37.178927       1 describe_instance_batch.go:85] Batched describe instances &{<nil> [] [i-0ace0851f6aa38476] <nil> <nil> {}}
E1002 22:16:37.299341       1 describe_instance_batch.go:88] Error occurred trying to batch describe instance, trying individually, error listing AWS instances: "operation error EC2: DescribeInstances, https response error StatusCode: 400, RequestID: 80b66403-41cf-4f44-bee5-c9ed33f074ec, api error InvalidInstanceID.NotFound: The instance ID 'i-0ace0851f6aa38476' does not exist"
E1002 22:16:37.393705       1 node_lifecycle_controller.go:156] error checking if node i-0ace0851f6aa38476 exists: getInstanceByNodeName failed for "i-0ace0851f6aa38476" with "error listing AWS instances: \"operation error EC2: DescribeInstances, https response error StatusCode: 400, RequestID: e2b03ec6-520c-4b18-8cbb-32974c052155, api error InvalidInstanceID.NotFound: The instance ID 'i-0ace0851f6aa38476' does not exist\""
I1002 22:16:42.494038       1 describe_instance_batch.go:85] Batched describe instances &{<nil> [] [i-0ace0851f6aa38476] <nil> <nil> {}}
E1002 22:16:42.593751       1 describe_instance_batch.go:88] Error occurred trying to batch describe instance, trying individually, error listing AWS instances: "operation error EC2: DescribeInstances, https response error StatusCode: 400, RequestID: 33670ab1-0a4c-4b78-9b09-989816d25a2f, api error InvalidInstanceID.NotFound: The instance ID 'i-0ace0851f6aa38476' does not exist"
E1002 22:16:42.711635       1 node_lifecycle_controller.go:156] error checking if node i-0ace0851f6aa38476 exists: getInstanceByNodeName failed for "i-0ace0851f6aa38476" with "error listing AWS instances: \"operation error EC2: DescribeInstances, https response error StatusCode: 400, RequestID: 4c64f380-ff95-4289-ad38-dc4ed90686c5, api error InvalidInstanceID.NotFound: The instance ID 'i-0ace0851f6aa38476' does not exist\""
I1002 22:16:49.016283       1 describe_instance_batch.go:85] Batched describe instances &{<nil> [] [i-0ace0851f6aa38476] <nil> <nil> {}}
E1002 22:16:49.111520       1 describe_instance_batch.go:88] Error occurred trying to batch describe instance, trying individually, error listing AWS instances: "operation error EC2: DescribeInstances, https response error StatusCode: 400, RequestID: ddefa5ee-badf-4495-8192-ad2de913167c, api error InvalidInstanceID.NotFound: The instance ID 'i-0ace0851f6aa38476' does not exist"
I1002 22:16:49.202892       1 node_lifecycle_controller.go:163] deleting node since it is no longer present in cloud provider: i-0ace0851f6aa38476
I1002 22:16:49.203307       1 event.go:389] "Event occurred" object="i-0ace0851f6aa38476" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="DeletingNode" message="Deleting node i-0ace0851f6aa38476 because it does not exist in the cloud provider"

[samuelebstein]

@sergelogvinov we are seeing this race condition happen very often with aws CCM deleting nodes. Did you come to any conclusion on how to handle this and is it being prioritized?

I’ve reread the issue, and we are discussing the situation where TalosCCM (init) and AwsCCM (lifecycle) together cause a problem during scale-up/down events.

[pashtet04]

I can confirm that this issue is also reproducible with talos-cloud-controller-manager and hcloud-cloud-controller-manager running together on Hetzner Cloud.

• Talos CCM controllers: cloud-node, node-ipam-controller
• Hetzner CCM controllers: service-lb-controller, cloud-node-lifecycle-controller

The Hetzner CCM logs the following error:

hcloud/instancesv2.InstanceExists: failed to convert provider id to server id: providerID does not have one of the the expected prefixes (hcloud://, hrobot://, hcloud://bm-): talos://hcloud/10.0.0.78

Talos CCM sets spec.providerID to talos://hcloud/, so Hetzner's cloud-node-lifecycle-controller cannot resolve the corresponding server and check whether it still exists.

[sergelogvinov]

Hello, If your node has a ProviderID with the talos:// prefix, it means the node was installed using a talos metal image.

Try hetzner image from image factory.

[pashtet04]

In our case, this is consistently reproducible when Cluster Autoscaler is actively scaling the cluster up and down.

It looks like a race condition during node provisioning:

1. A new VM is created in Hetzner.
2. The VM is not fully ready yet and Talos has not finished booting/joining the cluster.
3. The node is treated as unavailable, and the controller requests another host.
4. The replacement host starts successfully and becomes Ready.
5. The original abandoned Node object remains in Kubernetes as NotReady indefinitely and has to be deleted manually.

This happens quite frequently under active autoscaling. At the moment, we have accumulated the following stale nodes for the latest 38h:

talos-23b-d1k   NotReady
talos-8lf-u3k   NotReady
talos-9cq-jto   NotReady
talos-aas-vxh   NotReady
talos-ezo-ux2   NotReady
talos-fpn-rjv   NotReady
talos-g0l-za6   NotReady
talos-m9m-rpo   NotReady
talos-ni7-q4d   NotReady
talos-ntc-qw4   NotReady
talos-otu-k3b   NotReady
talos-r8h-vzx   NotReady
talos-slk-xyj   NotReady
talos-vqz-tf9   NotReady
talos-zoi-f8n   NotReady
talos-zue-m6d   NotReady

I will check the Hetzner image from Image Factory as suggested.
that image is expected to change the Node providerID from talos://... to hcloud://...?

Even with a slow or failed provisioning attempt, the abandoned Node object should ideally be cleaned up automatically instead of remaining permanently in the cluster.

[sergelogvinov]

Hello, @pashtet04 please share the details

• Talos CCM version:
• Talos version: [talosctl version --nodes <control plane nodes>]
• Talos metadata: [talosctl --nodes <node> get PlatformMetadatas.talos.dev -oyaml]
• Kubernetes version: [kubectl version --short]
• Node describe: [kubectl describe node <node>]
• Talos CCM logs