Skip to content

add controller-driven kube-vip load balancer for K3S_BASE clusters#5683

Open
naiming-zededa wants to merge 2 commits intolf-edge:masterfrom
naiming-zededa:naiming-loadbalancing
Open

add controller-driven kube-vip load balancer for K3S_BASE clusters#5683
naiming-zededa wants to merge 2 commits intolf-edge:masterfrom
naiming-zededa:naiming-loadbalancing

Conversation

@naiming-zededa
Copy link
Contributor

Description

This PR implements controller-driven Kubernetes LoadBalancer services
for CLUSTER_TYPE_K3S_BASE in eve-k.

 - pkg/pillar/types/clustertypes.go: Add LBInterfaceConfig (interface +
   CIDR string) and LBInterfaces []LBInterfaceConfig to both
   EdgeNodeClusterConfig and EdgeNodeClusterStatus.
  - pkg/pillar/cmd/zedagent/parseconfig.go: Parse LoadBalancerService
    from the controller proto and populate
    EdgeNodeClusterConfig.LBInterfaces (K3S_BASE only; first
    interface/CIDR entry applied).
  - pkg/pillar/cmd/zedkube/clusterstatus.go: Relay LBInterfaces from
    EdgeNodeClusterConfig into EdgeNodeClusterStatus on the bootstrap
    node only; non-bootstrap nodes publish an empty list so they do not
    trigger kube-vip setup.
  - pkg/pillar/dpcmanager/dns.go: Filter kube-vip VIPs out of
    DeviceNetworkStatus.AddrInfoList using the LBInterfaces CIDR range,
    preventing VIPs from being used as source addresses for
    controller-bound traffic.
  - pkg/kube/cluster-init.sh: Add check_kubevip_lb loop that reads
    EdgeNodeClusterStatus JSON each iteration and calls kubevip-apply.sh
    or kubevip-delete.sh when the LB config changes. Persists
    last-applied state to avoid redundant re-applies across restarts.
  - pkg/kube/kubevip-apply.sh / kubevip-delete.sh: Scripts to
    install/remove the kube-vip DaemonSet and kube-vip-cloud-provider
    Deployment, configuring the IP pool via a kubevip ConfigMap.
  - pkg/kube/kubevip-ds.yaml: kube-vip DaemonSet manifest (ARP mode,
    control-plane nodes).
  - pkg/kube/config.yaml: Disable k3s built-in ServiceLB (servicelb) and
    Traefik for K3S_BASE — kube-vip replaces ServiceLB; users bring
    their own ingress.
  - pkg/pillar/docs/zedkube.md: Document the feature with an overview
    diagram, data-flow, EVE-API proto, and DeviceNetworkStatus filtering
    notes.

PR dependencies

How to test and validate this PR

Create a native orchestration type cluster on EVE devices, and in the controller config the enabling of 'loadbalancing'
for the cluster. Specify the interface and IP prefix of the LB.

In the kubernetes side, user specify the helm/yaml definition of App and Service with type of 'loadbalancer'. Verify
the service has the IP address allocated on the interface for this service, and endpoint created for the App.

User the specify client to access that ip address and port. and verify when one device is down, the IP is reallocated
to another device of the cluster, and the app is still reachable.

there are many different ways to use the LB service, see examples in pkg/pillar/docs/zedkube.md for detail

Changelog notes

add controller-driven kube-vip load balancer for K3S_BASE clusters

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

// collectLBPoolStatus reads the kubevip ConfigMap from kube-system to get the configured
// load balancer pool, and gathers IPs currently allocated to LoadBalancer-type services.
// Returns nil if the kubevip ConfigMap does not exist (kubevip not yet deployed).
func collectLBPoolStatus(clientset *kubernetes.Clientset, services []types.KubeServiceInfo) *types.KubeLBPoolStatus {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we publish this LBPoolStatus in any eve pub/sub calls for collect info ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this LBPoolStatus is published by zedkube in KubeUserServices, so it is part of the collectinfo.

This PR implements controller-driven Kubernetes LoadBalancer services
for CLUSTER_TYPE_K3S_BASE in eve-k.

  - pkg/pillar/types/clustertypes.go: Add LBInterfaceConfig (interface +
    CIDR string) and LBInterfaces []LBInterfaceConfig to both
    EdgeNodeClusterConfig and EdgeNodeClusterStatus.
  - pkg/pillar/cmd/zedagent/parseconfig.go: Parse LoadBalancerService
    from the controller proto and populate
    EdgeNodeClusterConfig.LBInterfaces (K3S_BASE only; first
    interface/CIDR entry applied).
  - pkg/pillar/cmd/zedkube/clusterstatus.go: Relay LBInterfaces from
    EdgeNodeClusterConfig into EdgeNodeClusterStatus on the bootstrap
    node only; non-bootstrap nodes publish an empty list so they do not
    trigger kube-vip setup.
  - pkg/pillar/dpcmanager/dns.go: Filter kube-vip VIPs out of
    DeviceNetworkStatus.AddrInfoList using the LBInterfaces CIDR range,
    preventing VIPs from being used as source addresses for
    controller-bound traffic.
  - pkg/kube/cluster-init.sh: Add check_kubevip_lb loop that reads
    EdgeNodeClusterStatus JSON each iteration and calls kubevip-apply.sh
    or kubevip-delete.sh when the LB config changes. Persists
    last-applied state to avoid redundant re-applies across restarts.
  - pkg/kube/kubevip-apply.sh / kubevip-delete.sh: Scripts to
    install/remove the kube-vip DaemonSet and kube-vip-cloud-provider
    Deployment, configuring the IP pool via a kubevip ConfigMap.
  - pkg/kube/kubevip-ds.yaml: kube-vip DaemonSet manifest (ARP mode,
    control-plane nodes).
  - pkg/kube/config.yaml: Disable k3s built-in ServiceLB (servicelb) and
    Traefik for K3S_BASE — kube-vip replaces ServiceLB; users bring
    their own ingress.
  - pkg/pillar/docs/zedkube.md: Document the feature with an overview
    diagram, data-flow, EVE-API proto, and DeviceNetworkStatus filtering
    notes.

Signed-off-by: naiming-zededa <naiming@zededa.com>
… K3S_BASE

 - add the pillar vendor files updated eve-api

Signed-off-by: naiming-zededa <naiming@zededa.com>
@naiming-zededa naiming-zededa force-pushed the naiming-loadbalancing branch from 9c5b309 to 7952e18 Compare March 23, 2026 19:05
@github-actions github-actions bot requested a review from zedi-pramodh March 23, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants