start running the access tokens and huge services test #17671

upodroid · 2025-10-14T11:37:53Z

We need to run the access tokens and hugepage services perf tests for scale jobs to achieve parity with the current GCE 5k test.

https://github.com/kubernetes/test-infra/blob/7c5dc4ca148503409199df1945856845730b94bb/config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml#L162

hakman · 2025-10-14T11:43:05Z

Scenario is shared by both AWS and GCP.
/hold for feedback from @kubernetes/sig-scalability @dims

upodroid · 2025-10-14T11:54:17Z

I can scope the change to just GCE, but let's benchmark the presubmits.

/test presubmit-kops-aws-scale-amazonvpc-using-cl2
/test presubmit-kops-gce-scale-ipalias-using-cl2

hakman · 2025-10-14T11:57:31Z

I can scope the change to just GCE, but let's benchmark the presubmits.

I don't have anything against enabling those, just would be nice to be all on the same page.
If these diverge too much, we might as well split the script.

hakman · 2025-10-14T16:18:24Z

/retest

hakman · 2025-10-14T16:19:16Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

upodroid · 2025-10-14T16:24:58Z

Something isn't right, the test suite shrunk

hakman · 2025-10-15T08:45:11Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2025-10-15T10:24:56Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

upodroid · 2025-10-15T10:37:13Z

/test presubmit-kops-gce-small-scale-ipalias-using-cl2

upodroid · 2025-10-15T10:44:54Z

/test presubmit-kops-gce-small-scale-ipalias-using-cl2

upodroid · 2025-10-15T10:58:30Z

/test presubmit-kops-gce-small-scale-ipalias-using-cl2

hakman · 2025-10-15T12:03:55Z

/test presubmit-kops-aws-small-scale-using-cl2

hakman · 2025-10-15T12:04:46Z

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2

tests/e2e/scenarios/scalability/run-test.sh

upodroid · 2025-10-20T18:52:34Z

...3_object_additionalobjects.example.com-addons-kops-controller.addons.k8s.io-k8s-1.16_content

      - args:
        - --v=2
        - --conf=/etc/kubernetes/kops-controller/config/config.yaml
+        - --log_file=/var/log/kops-controller.log


found this

kubernetes/klog#60

A blast from the past :-)

upodroid · 2025-10-20T18:53:02Z

/test presubmit-kops-gce-scale-ipalias-using-cl2
/test presubmit-kops-aws-scale-amazonvpc-using-cl2

alaypatel07 · 2025-10-22T14:27:43Z

@upodroid, looks like the scale jobs are failing due to latency. Should we also test DRA the config in this pre-submit job? https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kops/17671/presubmit-kops-gce-scale-ipalias-using-cl2/1980346550060060672

upodroid · 2025-10-22T17:16:24Z

We don't have a DRA presubmit against this repo. Also, I need this merged first, then we can soak it and then address the other performance issues.

upodroid · 2025-10-22T17:16:45Z

tests/e2e/kubetest2-kops/deployer/common.go

 			d.GCPProject = resource.Name
 			klog.V(1).Infof("Got project %s from boskos", d.GCPProject)

+			if os.Getenv("SCALE_SCENARIO") == "performance" {


Not required after kubernetes/perf-tests#3653

I do think this is a good idea though.... kubetest2 assumes that we will build everything into kubetest2, and I think there's a lot of power in more modular tests for performance and specific scenarios. In #17636 I am trying out exporting the variable so that the test can source it: 7d339a6

upodroid · 2025-10-24T10:27:55Z

/test presubmit-kops-gce-scale-ipalias-using-cl2
/test presubmit-kops-aws-scale-amazonvpc-using-cl2

upodroid · 2025-10-24T12:45:01Z

/test presubmit-kops-gce-scale-ipalias-using-cl2
/test presubmit-kops-aws-scale-amazonvpc-using-cl2

justinsb · 2025-10-24T13:11:26Z

nodeup/pkg/model/logrotate.go

 	}

 	b.addLogRotate(c, "docker", "/var/log/docker.log", logRotateOptions{})
+	b.addLogRotate(c, "kops-controller", "/var/log/kops-controller.log", logRotateOptions{})


I think kops-controller is "just a pod" and doesn't write logs here. But it is relatively important, so I think we could make the case for doing so...

justinsb · 2025-10-24T13:13:27Z

tests/e2e/kubetest2-kops/deployer/common.go

 			d.GCPProject = resource.Name
 			klog.V(1).Infof("Got project %s from boskos", d.GCPProject)

+			if os.Getenv("SCALE_SCENARIO") == "performance" {


I do think this is a good idea though.... kubetest2 assumes that we will build everything into kubetest2, and I think there's a lot of power in more modular tests for performance and specific scenarios. In #17636 I am trying out exporting the variable so that the test can source it: 7d339a6

justinsb · 2025-10-24T13:14:53Z

tests/e2e/kubetest2-kops/deployer/up.go

-	if version > "1.29" {
-		// Requires https://github.com/kubernetes/kops/pull/16128
-		args = append(args, "--set", `spec.containerd.configAdditions=plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler.runtime_type=io.containerd.runc.v2`)
+		"--set", `spec.containerd.configAdditions=plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler.runtime_type=io.containerd.runc.v2`,


This looks like a cleanup for handling of old kube versions that we no longer test? (I.e. could be split into its own PR, though that would be a very short PR!)

justinsb · 2025-10-24T13:15:51Z

tests/e2e/kubetest2-kops/deployer/up.go


 	exec.InheritOutput(cmd)
-	err = cmd.Run()
+	err := cmd.Run()


👍

I like if err := cmd.Run(); err != nil { because then you don't influence the next single-error return. But this is an improvement so 👍

justinsb · 2025-10-24T13:17:35Z

...3_object_additionalobjects.example.com-addons-kops-controller.addons.k8s.io-k8s-1.16_content

      - args:
        - --v=2
        - --conf=/etc/kubernetes/kops-controller/config/config.yaml
+        - --log_file=/var/log/kops-controller.log


A blast from the past :-)

justinsb · 2025-10-24T13:20:22Z

...3_object_additionalobjects.example.com-addons-kops-controller.addons.k8s.io-k8s-1.16_content

          requests:
            cpu: 50m
            memory: 50Mi
-        securityContext:


I think there's a case that kops-controller is important so should have a log in /var/log, but the components that are there today are the ones that we can't get with kubectl logs (when they fail). Is there e.g. a perf issue with getting kops-controller logs from kubectl logs (or some other reason)? Because now we will have to deal with the scanners flagging this one as running as root...

There isn't a performance issue and assuming the control plane is bootstrapped, we can see the logs from kops toolbox dump.

Happy to remove it if it's too tricky

justinsb · 2025-10-24T13:21:32Z

.ko.yaml

@@ -0,0 +1 @@
+defaultBaseImage: gcr.io/distroless/static-debian12


👍 to introducing this file so we can more easily configure some of these ko options, for example I realized we aren't passing the Version properly to ko

justinsb · 2025-10-24T13:23:35Z

pkg/model/gcemodel/external_access.go

 		if err != nil {
 			return err
 		}
+		b.AddFirewallRulesTasks(c, "ssh-external-to-master", &gcetasks.FirewallRule{


So this hits the "main code path" of kOps. Can we SSH through the bastion? Or if not, can we somehow make this not change the configuration for "everyone else" - e.g. with a feature flag or by adding something in the cluster or instancegroup? (The feature flag is normally easiest)

This specific change won't be in this PR, just need the patch to get the scale tests passing.

It is being worked in a different PR

upodroid · 2025-10-24T21:23:41Z

/test presubmit-kops-gce-scale-ipalias-using-cl2
/test presubmit-kops-aws-scale-amazonvpc-using-cl2

k8s-ci-robot · 2025-10-24T21:28:52Z

@upodroid: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
presubmit-kops-gce-scale-ipalias-using-cl2	`11f922b`	link	true	`/test presubmit-kops-gce-scale-ipalias-using-cl2`
presubmit-kops-aws-scale-amazonvpc-using-cl2	`11f922b`	link	false	`/test presubmit-kops-aws-scale-amazonvpc-using-cl2`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

upodroid · 2025-10-24T21:59:21Z

/test presubmit-kops-gce-scale-ipalias-using-cl2
/test presubmit-kops-aws-scale-amazonvpc-using-cl2

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 14, 2025

k8s-ci-robot requested review from dims and hakman October 14, 2025 11:38

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 14, 2025

This was referenced Oct 14, 2025

allow repeated --test-overrides & --test-configs to work for cl2 kubernetes-sigs/kubetest2#303

Merged

create new 5k dra job kubernetes/test-infra#35700

Open

upodroid force-pushed the scale-tweaks-one branch from 1933886 to cab9990 Compare October 15, 2025 10:35

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 15, 2025

upodroid mentioned this pull request Oct 15, 2025

add KOPS_CL2_TEST_CONFIG to scalibility/run-test.sh #17673

Closed

upodroid force-pushed the scale-tweaks-one branch from cab9990 to 82d80c9 Compare October 15, 2025 10:44

upodroid force-pushed the scale-tweaks-one branch from 82d80c9 to cb4e4e6 Compare October 15, 2025 10:58

kubernetes deleted a comment from k8s-ci-robot Oct 15, 2025

alaypatel07 reviewed Oct 15, 2025

View reviewed changes

tests/e2e/scenarios/scalability/run-test.sh Outdated Show resolved Hide resolved

upodroid mentioned this pull request Oct 15, 2025

bump yq in kubekins-e2e kubernetes/test-infra#35705

Merged

upodroid force-pushed the scale-tweaks-one branch from cb4e4e6 to 70506ac Compare October 15, 2025 13:40

upodroid force-pushed the scale-tweaks-one branch from 08f6cf3 to 480a3c0 Compare October 20, 2025 07:01

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 20, 2025

upodroid force-pushed the scale-tweaks-one branch 2 times, most recently from 3756f25 to f5468c8 Compare October 20, 2025 10:35

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 20, 2025

upodroid force-pushed the scale-tweaks-one branch from f5468c8 to 1f31040 Compare October 20, 2025 11:38

upodroid commented Oct 20, 2025

View reviewed changes

upodroid commented Oct 22, 2025

View reviewed changes

k8s-ci-robot added the area/provider/gcp Issues or PRs related to gcp provider label Oct 24, 2025

upodroid force-pushed the scale-tweaks-one branch from 04f620d to 6ebf4a0 Compare October 24, 2025 11:23

justinsb reviewed Oct 24, 2025

View reviewed changes

upodroid force-pushed the scale-tweaks-one branch from 6ebf4a0 to 69116e7 Compare October 24, 2025 21:19

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 24, 2025

upodroid added 3 commits October 25, 2025 00:20

start running the access tokens and huge services test

c9afa00

change base image to distroless where uid is 0

4add0d9

temporary ssh patch

11f922b

upodroid force-pushed the scale-tweaks-one branch from 69116e7 to 11f922b Compare October 24, 2025 21:22

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 24, 2025

fix clusterloader2 arg

c311481

		@@ -0,0 +1 @@
		defaultBaseImage: gcr.io/distroless/static-debian12

start running the access tokens and huge services test #17671

Are you sure you want to change the base?

start running the access tokens and huge services test #17671

Conversation

upodroid commented Oct 14, 2025

Uh oh!

hakman commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

upodroid commented Oct 14, 2025

Uh oh!

hakman commented Oct 14, 2025

Uh oh!

hakman commented Oct 14, 2025

Uh oh!

hakman commented Oct 14, 2025

Uh oh!

upodroid commented Oct 14, 2025

Uh oh!

hakman commented Oct 15, 2025

Uh oh!

hakman commented Oct 15, 2025

Uh oh!

upodroid commented Oct 15, 2025

Uh oh!

upodroid commented Oct 15, 2025

Uh oh!

upodroid commented Oct 15, 2025

Uh oh!

hakman commented Oct 15, 2025

Uh oh!

hakman commented Oct 15, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

upodroid commented Oct 20, 2025

Uh oh!

alaypatel07 commented Oct 22, 2025

Uh oh!

upodroid commented Oct 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

upodroid commented Oct 24, 2025

Uh oh!

upodroid commented Oct 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

upodroid commented Oct 24, 2025

Uh oh!

k8s-ci-robot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

upodroid commented Oct 24, 2025

Uh oh!

Reviewers

hakman commented Oct 14, 2025 •

edited

Loading

k8s-ci-robot commented Oct 24, 2025 •

edited

Loading