Skip to content

fix: additional_ip_ranges_config - again #2458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

fix: additional_ip_ranges_config - again #2458

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7d50b79
fix ranges variable
DrFaust92 Sep 26, 2025
2853d0a
CR comments
DrFaust92 Sep 28, 2025
177378f
add tests
DrFaust92 Oct 3, 2025
faed61b
add tests
DrFaust92 Oct 3, 2025
b5d6784
add tests
DrFaust92 Oct 8, 2025
6b7692a
align tests
DrFaust92 Oct 8, 2025
98b64fa
add tests
DrFaust92 Oct 9, 2025
e5035eb
add tests
DrFaust92 Oct 9, 2025
825d36a
Merge branch 'main' into fix-pods-ip-again
apeabody Oct 14, 2025
b2e23bb
Merge branch 'main' into fix-pods-ip-again
apeabody Oct 15, 2025
bb17c81
Merge branch 'main' into fix-pods-ip-again
apeabody Oct 16, 2025
1aece4e
Merge branch 'main' into fix-pods-ip-again
apeabody Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions autogen/main/cluster.tf.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -557,8 +557,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,8 +424,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
2 changes: 2 additions & 0 deletions examples/node_pool/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ This example illustrates how to create a cluster with multiple custom node-pool

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| additional\_ip\_pod\_range | The secondary ip range to use for pods in the additional range | `any` | n/a | yes |
| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br> enabled = bool<br> autoscaling_profile = string<br> min_cpu_cores = number<br> max_cpu_cores = number<br> min_memory_gb = number<br> max_memory_gb = number<br> gpu_resources = list(object({<br> resource_type = string<br> minimum = number<br> maximum = number<br> }))<br> auto_repair = bool<br> auto_upgrade = bool<br> })</pre> | <pre>{<br> "auto_repair": true,<br> "auto_upgrade": true,<br> "autoscaling_profile": "BALANCED",<br> "enabled": false,<br> "gpu_resources": [],<br> "max_cpu_cores": 0,<br> "max_memory_gb": 0,<br> "min_cpu_cores": 0,<br> "min_memory_gb": 0<br>}</pre> | no |
| cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
| compute\_engine\_service\_account | Service account to associate to the nodes in the cluster | `any` | n/a | yes |
Expand All @@ -22,6 +23,7 @@ This example illustrates how to create a cluster with multiple custom node-pool

| Name | Description |
|------|-------------|
| additional\_ip\_range\_pods | The secondary IP range used for pods in the additional range |
| ca\_certificate | n/a |
| client\_token | n/a |
| cluster\_name | Cluster name |
Expand Down
7 changes: 7 additions & 0 deletions examples/node_pool/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@ module "gke" {
logging_variant = "MAX_THROUGHPUT"
dns_allow_external_traffic = true

additional_ip_ranges_config = [
{
subnetwork = "projects/${var.project_id}/regions/${var.region}/subnetworks/${var.subnetwork}"
Copy link
Contributor Author

@DrFaust92 DrFaust92 Oct 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apeabody I think this was the issue, plan passes either way. but error is strange

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

additional_ip_ranges_config is not a list/array as defined in the google_container_cluster documentation. See https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#additional_ip_ranges_config-1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it is, according to provider code and google container api
https://github.com/hashicorp/terraform-provider-google/blob/4003211203eb953bab681ae9a2972eff1a90ef09/google/services/container/resource_container_cluster.go#L1695

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well that's interesting. So as a data point, I was able to update the module locally so that it was an 'object' and NOT a 'list' and it then the terraform plan handled this section for me correctly. As a list it complained, as detailed in the comment I referenced above.

I'm not sure why the discrepancy..?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like in the Google console UI it is a List so I can try again and see what comes out of the changes here

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @DrFaust92 - Looks like we are close

Error: Error updating cluster for []: googleapi: Error 400: The cluster's default subnetwork cannot be used as an additional subnetwork.

If needed, this is where the current one is defined: https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/main/test/fixtures/node_pool/network.tf

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets try once more @apeabody , thanks for bearing with me on this

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @DrFaust92!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DrFaust92 - Looks like the docs need to be regenerated after the change. Thanks!

Checking submodule's files generation
diff -r '--exclude=.terraform' '--exclude=.kitchen' '--exclude=.git' /workspace/examples/node_pool/README.md /tmp/tmp.VdIZFvHmj1/workspace/examples/node_pool/README.md
10a11
> | additional\_ip\_pod\_range\_subnetwork | The subnetwork to host the additional pod range in | `any` | n/a | yes |
Error: submodule's files generation has not been run, please run the
'make build' command and commit changes
Checking for documentation generation
diff -r '--exclude=.terraform' '--exclude=.kitchen' '--exclude=autogen' '--exclude=*.tfvars' '--exclude=*metadata.yaml' /workspace/examples/node_pool/README.md /tmp/tmp.QlBKcRfkkq/generate_docs/workspace/examples/node_pool/README.md
10a11
> | additional\_ip\_pod\_range\_subnetwork | The subnetwork to host the additional pod range in | `any` | n/a | yes |
Error: Documentation generation has not been run, please run the
'make docker_generate_docs' command and commit the above changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regenerated, thanks

pod_ipv4_range_names = [var.additional_ip_pod_range]
}
]

resource_manager_tags = {
"${var.project_id}/${google_tags_tag_key.key.short_name}" = google_tags_tag_value.value.short_name
}
Expand Down
5 changes: 5 additions & 0 deletions examples/node_pool/test_outputs.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ output "ip_range_services" {
value = var.ip_range_services
}

output "additional_ip_range_pods" {
description = "The secondary IP range used for pods in the additional range"
value = var.additional_ip_pod_range
}

output "zones" {
description = "List of zones in which the cluster resides"
value = module.gke.zones
Expand Down
4 changes: 4 additions & 0 deletions examples/node_pool/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ variable "ip_range_services" {
description = "The secondary ip range to use for services"
}

variable "additional_ip_pod_range" {
description = "The secondary ip range to use for pods in the additional range"
}

variable "compute_engine_service_account" {
description = "Service account to associate to the nodes in the cluster"
}
Expand Down
4 changes: 2 additions & 2 deletions modules/beta-autopilot-private-cluster/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,8 +325,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/beta-autopilot-public-cluster/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,8 +325,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/beta-private-cluster-update-variant/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,8 +461,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/beta-private-cluster/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,8 +461,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/beta-public-cluster-update-variant/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,8 +461,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/beta-public-cluster/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,8 +461,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/private-cluster-update-variant/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,8 +424,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
4 changes: 2 additions & 2 deletions modules/private-cluster/cluster.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,8 +424,8 @@ resource "google_container_cluster" "primary" {
dynamic "additional_ip_ranges_config" {
for_each = var.additional_ip_ranges_config
content {
subnetwork = var.additional_ip_ranges_config.subnetwork
pod_ipv4_range_names = var.additional_ip_ranges_config.pod_ipv4_range_names
subnetwork = additional_ip_ranges_config.value.subnetwork
pod_ipv4_range_names = additional_ip_ranges_config.value.pod_ipv4_range_names
}
}
stack_type = var.stack_type
Expand Down
1 change: 1 addition & 0 deletions test/fixtures/node_pool/example.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ module "example" {
subnetwork = google_compute_subnetwork.main.name
ip_range_pods = google_compute_subnetwork.main.secondary_ip_range[0].range_name
ip_range_services = google_compute_subnetwork.main.secondary_ip_range[1].range_name
additional_ip_pod_range = google_compute_subnetwork.main.secondary_ip_range[2].range_name
compute_engine_service_account = local.compute_engine_service_account

cluster_autoscaling = {
Expand Down
5 changes: 5 additions & 0 deletions test/fixtures/node_pool/outputs.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ output "ip_range_services" {
value = google_compute_subnetwork.main.secondary_ip_range[1].range_name
}

output "additional_ip_range_pods" {
description = "The secondary IP range used for pods in the additional range"
value = google_compute_subnetwork.main.secondary_ip_range[2].range_name
}

output "zones" {
description = "List of zones in which the cluster resides"
value = module.example.zones
Expand Down
8 changes: 7 additions & 1 deletion test/integration/node_pool/testdata/TestNodePool.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,13 @@
"servicesIpv4CidrBlock": "192.168.64.0/18",
"servicesSecondaryRangeName": "cft-gke-test-services-RANDOM_STRING",
"stackType": "IPV4",
"useIpAliases": true
"useIpAliases": true,
"additionalIpRangesConfigs": [
{
"subnetwork": "projects/PROJECT_ID/regions/europe-west4/subnetworks/cft-gke-test-RANDOM_STRING",
"podIpv4RangeNames": ["test"]
}
]
},
"labelFingerprint": "78cdf2f6",
"legacyAbac": {},
Expand Down