[v1.34] Improved Error handling #869

swastik959 · 2025-06-21T10:19:34Z

Improves error handling and timeout so that do Identify when the test is actually failing due to unavailability of loadbalancer and nor the fault in code
solves #131863

k8s-ci-robot · 2025-06-21T10:19:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: swastik959
Once this PR has been reviewed and has the lgtm label, please assign bentheelder for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-06-21T10:19:42Z

This issue is currently awaiting triage.

If the repository mantainers determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-06-21T10:19:42Z

Welcome @swastik959!

It looks like this is your first PR to kubernetes/cloud-provider-gcp 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-gcp has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-06-21T10:19:43Z

Hi @swastik959. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

YifeiZhuang · 2025-06-23T17:04:32Z

/ok-to-test

k8s-ci-robot · 2025-06-23T17:12:09Z

@swastik959: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
cloud-provider-gcp-verify-up-to-date	`f94d5a6`	link	true	`/test cloud-provider-gcp-verify-up-to-date`
cloud-provider-gcp-verify-all	`f94d5a6`	link	true	`/test cloud-provider-gcp-verify-all`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

BenTheElder · 2025-06-24T23:18:59Z

/uncc
#875

bowei · 2025-06-25T16:54:58Z

test/e2e/loadbalancer.go

-		By("waiting for the UDP service to have a load balancer")
-		// 2nd one should be faster since they ran in parallel.
-		udpService, err = udpJig.WaitForLoadBalancer(ctx, loadBalancerCreateTimeout)
+		By("waiting for the UDP service to have a load balancer with enhanced error handling")


You don't need to add this to the log message.

bowei · 2025-06-25T16:55:12Z

test/e2e/loadbalancer.go

-		// 2nd one should be faster since they ran in parallel.
-		udpService, err = udpJig.WaitForLoadBalancer(ctx, loadBalancerCreateTimeout)
+		By("waiting for the UDP service to have a load balancer with enhanced error handling")
+		// Enhanced load balancer creation with better error reporting


Same here, delete this.

bowei · 2025-06-25T16:55:23Z

test/e2e/loadbalancer.go

 	})
 })

+// Enhanced helper functions with GCP-specific retry logic


remove this

bowei · 2025-06-25T16:56:14Z

test/e2e/loadbalancer.go

+	// Check for known GCP infrastructure errors
+	retriableErrors := []string{
+		"INTERNAL_ERROR",
+		"503",


If you are checking for error code, you should check the code explicitly vs doing a string match.

bowei · 2025-06-25T16:56:23Z

test/e2e/loadbalancer.go

+	}
+
+	for _, retriableErr := range retriableErrors {
+		if strings.Contains(errStr, retriableErr) {


See comment above

bowei · 2025-06-25T16:57:25Z

test/e2e/loadbalancer.go

+				continue
+			}
+		} else {
+			// Non-retriable error, fail immediately


it would be good to track start and end time and print it in the logs (it makes it easier to debug what happened when reading from the logs)

bowei · 2025-06-25T16:57:49Z

test/e2e/loadbalancer.go

+		}
+	}
+
+	// If we get here, all retries failed with retriable errors


bowei · 2025-06-25T16:58:54Z

test/e2e/loadbalancer.go

+}
+
+// waitForLoadBalancerDestroyWithRetry waits for load balancer destruction with retry
+func waitForLoadBalancerDestroyWithRetry(ctx context.Context, jig *e2eservice.TestJig, ip string, port int, timeout time.Duration) (*v1.Service, error) {


seems like this is exactly the same function as above but the jig.WaitForxxx is different.

you should consolidate this into one function taking the jig.WaitForXXX as an argument instead of duplicating the code

k8s-triage-robot · 2025-09-23T17:46:51Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2025-10-23T18:36:30Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Improved Error handling

f94d5a6

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 21, 2025

k8s-ci-robot requested a review from BenTheElder June 21, 2025 10:19

k8s-ci-robot requested a review from cheftako June 21, 2025 10:19

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jun 21, 2025

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 21, 2025

swastik959 changed the title ~~Improved Error handling~~ [v1.34] Improved Error handling Jun 21, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 23, 2025

k8s-ci-robot removed the request for review from BenTheElder June 24, 2025 23:19

bowei reviewed Jun 25, 2025

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 23, 2025

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 23, 2025

[v1.34] Improved Error handling #869

Are you sure you want to change the base?

[v1.34] Improved Error handling #869

Conversation

swastik959 commented Jun 21, 2025

Uh oh!

k8s-ci-robot commented Jun 21, 2025

Uh oh!

k8s-ci-robot commented Jun 21, 2025

Uh oh!

k8s-ci-robot commented Jun 21, 2025

Uh oh!

k8s-ci-robot commented Jun 21, 2025

Uh oh!

YifeiZhuang commented Jun 23, 2025

Uh oh!

k8s-ci-robot commented Jun 23, 2025

Uh oh!

BenTheElder commented Jun 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-triage-robot commented Sep 23, 2025

Uh oh!

k8s-triage-robot commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants