DNS Hostnames for LoadBalancer services

[nightkr]

Currently, cloud-provider-kind will only provision IP addresses for LoadBalancer services. It would be nice if it also supported DNS hostnames (probably optionally, since it would require clients to configure DNS resolution separately).

[aojea]

It would be nice if it also supported DNS hostnames (probably optionally, since it would require clients to configure DNS resolution separately).

How it will look like?

type LoadBalancerIngress struct {
	// IP is set for load-balancer ingress points that are IP based
	// (typically GCE or OpenStack load-balancers)
	// +optional
	IP string

	// Hostname is set for load-balancer ingress points that are DNS based
	// (typically AWS load-balancers)
	// +optional
	Hostname string

	// IPMode specifies how the load-balancer IP behaves, and may only be specified when the ip field is specified.
	// Setting this to "VIP" indicates that traffic is delivered to the node with
	// the destination set to the load-balancer's IP and port.
	// Setting this to "Proxy" indicates that traffic is delivered to the node or pod with
	// the destination set to the node's IP and node port or the pod's IP and port.
	// Service implementations may use this information to adjust traffic routing.
	// +optional
	IPMode *LoadBalancerIPMode

	// Ports is a list of records of service ports
	// If used, every port defined in the service should have an entry in it
	// +optional
	Ports []PortStatus
}

which hostname should go there? it will not work in all platforms if we set the container names, ....

[nightkr]

Yeah.. I don't have a full design for it. My current thinking is something like "we bring our own DNS server, you set which zone it should serve on, it's your responsibility to configure your system DNS resolver to use it".

Maybe it would be a better fit to do this in a separate controller/project instead, but the current API doesn't really lean towards aggregating LoadBalancerIngress data from multiple sources.

[aojea]

I mean, I'm open to do it, but it seems it requires a bit of thinking, we can easily set the hostname on that fields, but if users need to do a lot of manual work on their systems for it to work is going to be complex ... not thinking about desktop docker, wsl2, mac, windows, ...

[marcv81]

Maybe cloud-provider-kind could call an optional hook script after it assigns an external IP address to a LoadBalancer? This would provide an easy way to register the IP address to a DNS server, or maybe rewrite /etc/hosts.

Such a solution would delegate the problem of determining the host name to the script. And it would keep cloud-provider-kind simple and less coupled to any particular DNS server.

[BenTheElder]

Maybe cloud-provider-kind could call an optional hook script after it assigns an external IP address to a LoadBalancer? This would provide an easy way to register the IP address to a DNS server, or maybe rewrite /etc/hosts.

Such a solution would delegate the problem of determining the host name to the script. And it would keep cloud-provider-kind simple and less coupled to any particular DNS server.

That sort of thing is a big source of security issues and bugs though.

What's the use case where we need to set hostname on sevices that can't be tested either via the IP or via out of band DNS <> IP?

[marcv81]

What's the use case where we need to set hostname on sevices that can't be tested either via the IP or via out of band DNS <> IP?

Here is my use case. I'm new to Kind, so I could be doing this all wrong.

• I run a test cluster on a machine that I reboot sometimes. The external IPs that cloud-provider-kind assigns may change when I reboot.
• I set up TLS with real certificates. Nginx routes requests based on their host name. For both these reasons I need custom host names, not just IP addresses.
• I wrote a custom script to generate /etc/hosts. I run it when the system reboots, or after adding a LoadBalancer to the cluster. It would be nicer to run it automatically when required.

[aojea]

cloud-provider-kind goal is to emulate real deployments, so I want to think about this problem the same way as someone has to solve it in GKE or EKS or AKS, ... how people is solving this problem today on those environments?

[BenTheElder]

There are non-standard and complex features to obtain certificates (eg GKE manager certs) or sometimes cert manager is used.

I don't think hostname is generally assigned by the LB at the Kubernetes level.

Your script only needs to run on restart right? You could configure this to run after docker starts in your host's init.

Otherwise I'd say a separate "controller" could watch the LB IPs, but I don't think that's even necessary since IPs don't change arbitrarily, only on restart

[marcv81]

I want to think about this problem the same way as someone has to solve it in GKE or EKS or AKS

I think you have a very fair point. I'm not talking from experience, but after some research I think people either handle this manually (generally speaking IPs are not expected to change), or they use something like https://github.com/kubernetes-sigs/external-dns.

ExternalDNS sounds like something I could use.

[BenTheElder]

I think you have a very fair point. I'm not talking from experience, but after some research I think people either handle this manually (generally speaking IPs are not expected to change), or they use something like https://github.com/kubernetes-sigs/external-dns.

FWIW The Kubernetes project itself currently uses static IPs + semi-manually configured DNS entries (automated reconcile, manual config file: https://github.com/kubernetes/k8s.io/tree/main/dns).

Static-IPs is kind of mess for docker/podman/docker desktop/nerdctl/ ..., so far we've avoided this due to not being able to give good portability / guarantees around it.

They are typically "static" until restart though.

ExternalDNS sounds like something I could use.

Please let us know here if that works for you, I do think we want to offer answers, just not sure what the best answer is :-)

[aojea]

in addition, if you find something that does not exist in Kubernetes today that will simplify this problem , that is also other venue we can explore and add that functionality

[sebastian-philipp]

In my use case, I have two kind clusters:

• kind-management
• kind-workload

I have a minio + Ingress on the workload cluster, but knowing the LB IP from within the management cluster is a hassle. Would be great to have an internal DNS name, similar to the already working control-plane hostname: curl --insecure https://workload-control-plane:6443

[BenTheElder]

I have a minio + Ingress on the workload cluster, but knowing the LB IP from within the management cluster is a hassle. Would be great to have an internal DNS name, similar to the already working control-plane hostname: curl --insecure https://workload-control-plane:6443

That hostname is provided by docker, for the container name, if and only if you're on the kind network (docker does not support this on the default bridge network, but it does by default on any non-default bridge network)

This isn't necessarily provided by other runtimes though (podman, nerdctl/containerd) and it won't work from your host.

https://docs.docker.com/engine/network/#dns-services

[sebastian-philipp]

That hostname is provided by docker, for the container name, if and only if you're on the kind network (docker does not support this on the default bridge network, but it does by default on any non-default bridge network)

This isn't necessarily provided by other runtimes though (podman, nerdctl/containerd) and it won't work from your host.

Right. With kind running outside of the docker network, none of my ideas really worked out:

• using --add-host is a hassle as I need to pass it before I know the IP of the externalIP. I don't think adding --add-host to kind would help me here.
• Podman has a --dns-add adding new DNS resolvers to an existing network, but I couldn't find this for e.g. Docker.