Skip to content

Commit 539245d

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request kubernetes-sigs#4109 from M00nF1sh/subnet-reachability
Enhance subnet discovery
2 parents 49a4f10 + c183219 commit 539245d

18 files changed

+2136
-1165
lines changed

‎docs/deploy/subnet_discovery.md

+62-29
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,71 @@
1-
# Subnet auto-discovery
2-
By default, the AWS Load Balancer Controller (LBC) auto-discovers network subnets that it can create AWS Network Load Balancers (NLB) and AWS Application Load Balancers (ALB) in. ALBs require at least two subnets across Availability Zones by default,
3-
set [Feature Gate ALBSingleSubnet](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/configurations/#feature-gates) to "true" allows using only 1 subnet for provisioning ALB. NLBs require one subnet.
4-
The subnets must be tagged appropriately for auto-discovery to work. The controller chooses one subnet from each Availability Zone. During auto-discovery, the controller
5-
considers subnets with at least eight available IP addresses. In the case of multiple qualified tagged subnets in an Availability Zone, the controller chooses the first one in lexicographical
6-
order by the subnet IDs.
7-
For more information about the subnets for the LBC, see [Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html)
8-
and [Network Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html).
9-
If you used `eksctl` or an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For
10-
more information about the Amazon EKS AWS CloudFormation VPC templates, see [Creating a VPC for your Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html).
1+
# Subnet Auto-Discovery
2+
The AWS Load Balancer Controller (LBC) automatically discovers subnets for creating AWS Network Load Balancers (NLB) and AWS Application Load Balancers (ALB). This auto-discovery process follows three main steps:
113

12-
## Public subnets
13-
Public subnets are used for internet-facing load balancers. These subnets must have the following tags:
4+
1. **Candidate Subnet Determination**: The controller identifies potential candidate subnets
5+
2. **Subnet Filtering**: The controller filters these candidates based on eligibility criteria
6+
3. **Final Selection**: The controller selects one subnet per availability zone
147

15-
| Key | Value |
16-
| --------------------------------------- | --------------------- |
17-
| `kubernetes.io/role/elb` | `1` or `` |
8+
## Candidate Subnet Determination
9+
The controller determines candidate subnets using the following process:
1810

19-
## Private subnets
20-
Private subnets are used for internal load balancers. These subnets must have the following tags:
11+
1. **If tag filters are specified**: Only subnets matching these filters become candidates
12+
13+
!!!tip
14+
You can only specify subnet tag filters for Ingress via IngressClassParams
2115

22-
| Key | Value |
23-
| --------------------------------------- | --------------------- |
24-
| `kubernetes.io/role/internal-elb` | `1` or `` |
16+
2. **If no tag filters are specified**:
17+
* If subnets with matching [role tag](#subnet-role-tag) exists: Only these become candidates
18+
* [**For LBC version >= 2.12.1**] If no subnets have role tags: Candidates are subnets whose [reachability](#subnet-reachability) (public/private) matches the LoadBalancer's schema
2519

2620

27-
## Common tag
28-
In version v2.1.1 and older of the LBC, both the public and private subnets must be tagged with the cluster name as follows:
21+
### Subnet Role Tag
22+
Subnets can be tagged appropriately for auto-discovery selection:
2923

30-
| Key | Value |
31-
| --------------------------------------- | --------------------- |
32-
| `kubernetes.io/cluster/${cluster-name}` | `owned` or `shared` |
24+
* **For internet-facing load balancers**, the controller looks for public subnets with following tags:
3325

34-
`${cluster-name}` is the name of the Kubernetes cluster.
35-
36-
The cluster tag is not required in versions v2.1.2 to v2.4.1, unless a cluster tag for another cluster is present.
26+
| Key | Value |
27+
| --------------------------------------- | --------------------- |
28+
| `kubernetes.io/role/elb` | `1` or `` |
3729

38-
With versions v2.4.2 and later, you can disable the cluster tag check completely by specifying the feature gate `SubnetsClusterTagCheck=false`
30+
* **For internal load balancers**, the controller looks for private subnets with following tags:
31+
32+
| Key | Value |
33+
| --------------------------------------- | --------------------- |
34+
| `kubernetes.io/role/internal-elb` | `1` or `` |
35+
36+
### Subnet reachability
37+
The controller automatically discovers all subnets in your VPC and determines whether each is a public or private subnet based on its associated route table configuration.
38+
A subnet is classified as public if its route table contains a route to an Internet Gateway.
39+
40+
!!!tip
41+
You can disable this behavior via SubnetDiscoveryByReachability feature flag.
42+
43+
## Subnet Filtering
44+
45+
1. **Cluster Tag Check**: The controller checks for cluster tags on subnets. Subnets with ineligible cluster tags will be filtered out.
46+
47+
| Key | Value |
48+
| --------------------------------------- | --------------------- |
49+
| `kubernetes.io/cluster/${cluster-name}` | `owned` or `shared` |
50+
51+
* If such cluster tag exists but no `<clusterName>` matches the current cluster, those subnets will be filtered out.
52+
* [**For LBC version < 2.1.1**] subnets without a cluster tag matching cluster name will be filtered out.
53+
54+
!!! tip
55+
You can disable this behavior via the `SubnetsClusterTagCheck` feature flag. When disabled, no cluster tag check will be performed against subnets.
56+
57+
2. **IP Address Availability**: Subnets with insufficient available IP addresses(**<8**) are filtered out.
58+
59+
## Final Selection
60+
61+
The controller selects one subnet per availability zone. When multiple subnets exist per Availability Zone, the following priority order applies:
62+
63+
1. Subnets with cluster tag for the current cluster (`kubernetes.io/cluster/<clusterName>`) are prioritized
64+
2. Subnets with lower lexicographical order of subnet ID are prioritized
65+
66+
## Minimum Subnet Requirements
67+
68+
* **ALBs**: Require at least two subnets across different Availability Zones by default
69+
70+
!!! tip
71+
For customers allowlisted by the AWS Elastic Load Balancing team, you can enable the [ALBSingleSubnet feature gate](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/configurations/#feature-gates). This allows provisioning an ALB with just one subnet instead of the standard requirement of two subnets.

‎docs/install/iam_policy.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎docs/install/iam_policy_cn.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎docs/install/iam_policy_iso.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎docs/install/iam_policy_isob.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎docs/install/iam_policy_isoe.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎docs/install/iam_policy_isof.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎docs/install/iam_policy_us-gov.json

+1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"ec2:DescribeCoipPools",
3232
"ec2:GetSecurityGroupsForVpc",
3333
"ec2:DescribeIpamPools",
34+
"ec2:DescribeRouteTables",
3435
"elasticloadbalancing:DescribeLoadBalancers",
3536
"elasticloadbalancing:DescribeLoadBalancerAttributes",
3637
"elasticloadbalancing:DescribeListeners",

‎main.go

+5-1
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,11 @@ func main() {
123123
sgReconciler := networking.NewDefaultSecurityGroupReconciler(sgManager, ctrl.Log)
124124
azInfoProvider := networking.NewDefaultAZInfoProvider(cloud.EC2(), ctrl.Log.WithName("az-info-provider"))
125125
vpcInfoProvider := networking.NewDefaultVPCInfoProvider(cloud.EC2(), ctrl.Log.WithName("vpc-info-provider"))
126-
subnetResolver := networking.NewDefaultSubnetsResolver(azInfoProvider, cloud.EC2(), cloud.VpcID(), controllerCFG.ClusterName, ctrl.Log.WithName("subnets-resolver"))
126+
subnetResolver := networking.NewDefaultSubnetsResolver(azInfoProvider, cloud.EC2(), cloud.VpcID(), controllerCFG.ClusterName,
127+
controllerCFG.FeatureGates.Enabled(config.SubnetsClusterTagCheck),
128+
controllerCFG.FeatureGates.Enabled(config.ALBSingleSubnet),
129+
controllerCFG.FeatureGates.Enabled(config.SubnetDiscoveryByReachability),
130+
ctrl.Log.WithName("subnets-resolver"))
127131
multiClusterManager := targetgroupbinding.NewMultiClusterManager(mgr.GetClient(), mgr.GetAPIReader(), ctrl.Log)
128132
tgbResManager := targetgroupbinding.NewDefaultResourceManager(mgr.GetClient(), cloud.ELBV2(), cloud.EC2(),
129133
podInfoRepo, sgManager, sgReconciler, vpcInfoProvider, multiClusterManager, lbcMetricsCollector,

‎pkg/aws/services/ec2.go

+21
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package services
22

33
import (
44
"context"
5+
56
"github.com/aws/aws-sdk-go-v2/service/ec2"
67
"github.com/aws/aws-sdk-go-v2/service/ec2/types"
78
"sigs.k8s.io/aws-load-balancer-controller/pkg/aws/provider"
@@ -24,6 +25,9 @@ type EC2 interface {
2425
// DescribeVPCsAsList wraps the DescribeVpcsPagesWithContext API, which aggregates paged results into list.
2526
DescribeVPCsAsList(ctx context.Context, input *ec2.DescribeVpcsInput) ([]types.Vpc, error)
2627

28+
// DescribeRouteTablesAsList wraps the DescribeRouteTablesWithContext API, which aggregates paged results into list.
29+
DescribeRouteTablesAsList(ctx context.Context, input *ec2.DescribeRouteTablesInput) ([]types.RouteTable, error)
30+
2731
CreateTagsWithContext(ctx context.Context, input *ec2.CreateTagsInput) (*ec2.CreateTagsOutput, error)
2832
DeleteTagsWithContext(ctx context.Context, input *ec2.DeleteTagsInput) (*ec2.DeleteTagsOutput, error)
2933
CreateSecurityGroupWithContext(ctx context.Context, input *ec2.CreateSecurityGroupInput) (*ec2.CreateSecurityGroupOutput, error)
@@ -141,6 +145,23 @@ func (c *ec2Client) DescribeVPCsAsList(ctx context.Context, input *ec2.DescribeV
141145
return result, nil
142146
}
143147

148+
func (c *ec2Client) DescribeRouteTablesAsList(ctx context.Context, input *ec2.DescribeRouteTablesInput) ([]types.RouteTable, error) {
149+
var result []types.RouteTable
150+
client, err := c.awsClientsProvider.GetEC2Client(ctx, "DescribeRouteTables")
151+
if err != nil {
152+
return nil, err
153+
}
154+
paginator := ec2.NewDescribeRouteTablesPaginator(client, input)
155+
for paginator.HasMorePages() {
156+
output, err := paginator.NextPage(ctx)
157+
if err != nil {
158+
return nil, err
159+
}
160+
result = append(result, output.RouteTables...)
161+
}
162+
return result, nil
163+
}
164+
144165
func (c *ec2Client) CreateTagsWithContext(ctx context.Context, input *ec2.CreateTagsInput) (*ec2.CreateTagsOutput, error) {
145166
client, err := c.awsClientsProvider.GetEC2Client(ctx, "CreateTags")
146167
if err != nil {

‎pkg/aws/services/ec2_mocks.go

+15
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/config/feature_gates.go

+26-24
Original file line numberDiff line numberDiff line change
@@ -11,18 +11,19 @@ import (
1111
type Feature string
1212

1313
const (
14-
ListenerRulesTagging Feature = "ListenerRulesTagging"
15-
WeightedTargetGroups Feature = "WeightedTargetGroups"
16-
ServiceTypeLoadBalancerOnly Feature = "ServiceTypeLoadBalancerOnly"
17-
EndpointsFailOpen Feature = "EndpointsFailOpen"
18-
EnableServiceController Feature = "EnableServiceController"
19-
EnableIPTargetType Feature = "EnableIPTargetType"
20-
EnableRGTAPI Feature = "EnableRGTAPI"
21-
SubnetsClusterTagCheck Feature = "SubnetsClusterTagCheck"
22-
NLBHealthCheckAdvancedConfig Feature = "NLBHealthCheckAdvancedConfig"
23-
NLBSecurityGroup Feature = "NLBSecurityGroup"
24-
ALBSingleSubnet Feature = "ALBSingleSubnet"
25-
LBCapacityReservation Feature = "LBCapacityReservation"
14+
ListenerRulesTagging Feature = "ListenerRulesTagging"
15+
WeightedTargetGroups Feature = "WeightedTargetGroups"
16+
ServiceTypeLoadBalancerOnly Feature = "ServiceTypeLoadBalancerOnly"
17+
EndpointsFailOpen Feature = "EndpointsFailOpen"
18+
EnableServiceController Feature = "EnableServiceController"
19+
EnableIPTargetType Feature = "EnableIPTargetType"
20+
EnableRGTAPI Feature = "EnableRGTAPI"
21+
SubnetsClusterTagCheck Feature = "SubnetsClusterTagCheck"
22+
NLBHealthCheckAdvancedConfig Feature = "NLBHealthCheckAdvancedConfig"
23+
NLBSecurityGroup Feature = "NLBSecurityGroup"
24+
ALBSingleSubnet Feature = "ALBSingleSubnet"
25+
SubnetDiscoveryByReachability Feature = "SubnetDiscoveryByReachability"
26+
LBCapacityReservation Feature = "LBCapacityReservation"
2627
)
2728

2829
type FeatureGates interface {
@@ -50,18 +51,19 @@ type defaultFeatureGates struct {
5051
func NewFeatureGates() FeatureGates {
5152
return &defaultFeatureGates{
5253
featureState: map[Feature]bool{
53-
ListenerRulesTagging: true,
54-
WeightedTargetGroups: true,
55-
ServiceTypeLoadBalancerOnly: false,
56-
EndpointsFailOpen: true,
57-
EnableServiceController: true,
58-
EnableIPTargetType: true,
59-
EnableRGTAPI: false,
60-
SubnetsClusterTagCheck: true,
61-
NLBHealthCheckAdvancedConfig: true,
62-
NLBSecurityGroup: true,
63-
ALBSingleSubnet: false,
64-
LBCapacityReservation: true,
54+
ListenerRulesTagging: true,
55+
WeightedTargetGroups: true,
56+
ServiceTypeLoadBalancerOnly: false,
57+
EndpointsFailOpen: true,
58+
EnableServiceController: true,
59+
EnableIPTargetType: true,
60+
EnableRGTAPI: false,
61+
SubnetsClusterTagCheck: true,
62+
NLBHealthCheckAdvancedConfig: true,
63+
NLBSecurityGroup: true,
64+
ALBSingleSubnet: false,
65+
SubnetDiscoveryByReachability: true,
66+
LBCapacityReservation: true,
6567
},
6668
}
6769
}

0 commit comments

Comments
 (0)