Very high CPU usage

[Vormillion]

We are planning to run test with 7500 users but I'm struggling with running even 100.

We are testing ecommerce site, just browsing scenario so page is almost fully cached, our infra utilization is < 10%.

So far I reach the moment when I have parallelism set to 20, so each pod takes 5 VUs. Runner requests are set to 40 cores, so each vUser in theory will get 8 CPU.

Now, first thing which I observed is that - doesn't matter how much CPU requests I will set, vUS will consume it all. I'm getting after some time processes killed (but not OOM, memory is barely 5% ) + timeout 20 seconds errors. When I'm browsing site during this time manually everything is perfect.

Another strange thing is a RPm pattern - you can see big drops and spikes in NewRelic.

On my local mac I'm able to run 10 vUsers without any issues, so why 40 CPU node can't handle 5 vUsers? It's impossible that 8 cores is not enough... (I'm pretty sure 1 user shouldn't consume more than 1-2 cores).

I'm discarding response body.

[Vormillion]

Another thing indicating issue with K6 itself is page rendering. It's crazy high, like chromium can't cope with content.

[yorugac]

Hi @Vormillion, it sounds like you're describing a browser test? If so, k6-operator does not "officially" support the browser yet: #403

Otherwise, I suggest to share a script and how to repeat this case. Also, the cluster setup is important here too.

[Vormillion]

Hi,

I can't share exactly whole test, but stripped version looks like this. It's simple navigation on page.

import { browser } from 'k6/browser';

import { ProductPage } from '../pages/product-page.js';
import { ProductListPage } from '../pages/product-list-page.js';
import { HomePage } from '../pages/home-page.js';

import { clickAndNavigateStep, simulateUserInteractionDelayStep } from '../steps/navigation-steps.js';
import { checkMinNumberOfItemsStep } from '../steps/check-steps.js';

export const options = {
  scenarios: {
    browser: {
      exec: 'browse',
      executor: 'ramping-vus',
      startVUs: 1,
      stages: [
          { duration: '3m', target: 100 },
          { duration: "5m", target: 100 }
      ],
      gracefulRampDown: '0s',
      options: {
        browser: {
          type: 'chromium',
        },
      },
    },
  },
  thresholds: {
    checks: [
            { threshold: 'rate==1.0', abortOnFail: false },
        ],
  },
};

export async function browse() {

  // ARRANGE
  const userInteractionDelayInMilliSeconds = 1000;
  const context = await browser.newContext();
  context.setDefaultTimeout(20000); // 20s
  const page = await context.newPage();

  const homePage = new HomePage(page, __ENV.BASE_URL);
  const productListPage = new ProductListPage(page, __ENV.BASE_URL);
  const productPage = new ProductPage(page, __ENV.BASE_URL);

  const productIdToPDP = '31';
  const minAllProducts = 10;

  try {

    /***** HOMEPAGE ******/
    /*********************/

    // ACT: GO TO HOMEPAGE
    await homePage.goto();
    await homePage.waitForPageReady();

    // ACT: GO TO PDP
    await productListPage.goto();
    await productListPage.waitForPageReady();

    // ASSERT
    checkMinNumberOfItemsStep(
      page,
      'Check Products In All Drinks',
      productListPage.addToCartButtonLocatorString,
      minAllProducts);

    // Slow the test down to mimic a user looking for the element on the page.
    await simulateUserInteractionDelayStep(page, userInteractionDelayInMilliSeconds);

    // ACT: GO TO PRODUCT
    await clickAndNavigateStep(page, productListPage.getSpecificProductAhref(productIdToPDP));
    await productPage.waitForPageReady();

    await clickAndNavigateStep(page, homePage.logo);
    await homePage.waitForPageReady();

  }
  finally {
    await page.close();
  }
}

Then TestRun looks like this:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: k6-sample-test
  namespace: k6
spec:
  parallelism: 20
  arguments: "--out output-statsd"
  script:
    localFile: /test/scripts/1-browse.js
  runner:
    image: xxx/k6-browse:latest
    resources:
      requests:
        cpu: 40
    nodeSelector:
      karpenter.sh/nodepool: k6
    env:
      - name: BASE_URL
        value: "https://xxx"
      - name: K6_STATSD_ENABLE_TAGS
        value: "true"
      - name: K6_STATSD_ADDR
        value: newrelic-statsd.k6.svc.cluster.local:8125
      - name: K6_BROWSER_ARGS
        value: "disable-web-security,enable-site-per-process,no-sandbox,disable-setuid-sandbox,disable-dev-shm-usage,disable-accelerated-2d-canvas,no-first-run,no-zygote,single-process,noerrdialogs,disable-gpu,disable-gl-drawing-for-tests,disable-renderer-backgrounding,disable-background-timer-throttling,disable-backgrounding-occluded-windows,disable-client-side-phishing-detection,disable-crash-reporter,disable-oopr-debug-crash-dump,no-crash-upload,disable-extensions,disable-low-res-tiling"
    volumeMounts:
    - mountPath: /test
      name: k6-scripts
    volumes:
    - name: k6-scripts
      persistentVolumeClaim:
        claimName: k6-efs-vol-claim
    tolerations:
    - effect: NoSchedule
      key: k6-taint
      operator: Exists

This test will create 2x 192 cores EC2 instances (c7i.48xlarge)

[yorugac]

Hi @Vormillion,
ATM, I see no reason to think that your observations around Chromium are caused by k6-operator itself. But perhaps, these are caused by running k6 with browser in Kubernetes in general -- that'd be very useful to confirm. But as mentioned in grafana/k6#4628, we need a website we can test against, which "reliably" exhibits these issues. Could you help with that?

Also, this line:

    image: xxx/k6-browse:latest

Are you running the official k6-browser image built in k6 repository or something custom?

[max-tremblay]

I'm not saying this is the solution, but you may want to use K6_BROWSER_HEADLESS=true and I would also put headless=new in K6_BROWSER_ARGS.

I had issues with no-zygote and you should remove single-process since it's not supported and recommended

In our case we're using a custom image, but nothing too fancy other than building k6 with some plugins + adding chromium-swiftshader in the alpine base image.

We're able to scale to thousands on AWS

[yorugac]

@max-tremblay, thanks for sharing! (and sorry for the delay!)
AFAIK, the official k6 image has headless mode enabled too, as well as swiftshader, here. Did that image not work for you at scale? Could you elaborate on that and / or share your Dockerfile for browser image? That would be very helpful for us, as we've been low-key looking into improving our browser-in-Kubernetes UX.

[max-tremblay]

@yorugac It's really not too far from what we have. From the declared UID, it looks like one of our devs probably took the recipe from your repo.

FROM golang:1.24 AS builder
ARG K6_VERSION

RUN go install go.k6.io/xk6/cmd/xk6@latest
RUN xk6 build \
    -v \
    --with github.com/grafana/xk6-faker@latest \
    --with github.com/grafana/xk6-exec@latest \
    --output /k6

FROM alpine:3.22
RUN apk update && apk --no-cache add chromium-swiftshader
COPY --from=builder /k6 /usr/bin/k6
RUN adduser -D -u 12345 -g 12345 k6
USER k6
ENV CHROME_BIN=/usr/lib/chromium/chromium
ENV CHROME_PATH=/usr/lib/chromium/
ENV K6_BROWSER_HEADLESS=true
ENTRYPOINT ["k6"]

Note that this is a stripped-down Dockerfile. We’ve added some internal assets to the image.

So this works for us with scaling, my latest benchmark was around 16k instances using 4 browsers per pod (parallelism == 4000).

To achieve that, we used a custom starter, otherwise you’ll run into #408. We also encountered another issue with EnabledServiceLinks, which became excessively long past a certain threshold. We moved execution-segment-sequence to its own config.json file, as I suspected it would eventually hit the ARG_MAX kernel limit.

I'm about to send a PR for this.

[yorugac]

@max-tremblay, thanks for sharing!

4 browsers per pod

I'm a bit confused here: how exactly does that work? E.g. this image above would have only one browser per pod, and k6-operator is not "aware" of browsers, yet. (issue)

To achieve that, we used a custom starter, otherwise you’ll run into #408.

Btw, as mentioned in this comment, removal of starter pod is planned soon-ish. So starter pod will probably become deprecated. I'll try to see how we can test this 1000 pods use case with the new approach...

We also encountered another issue with EnabledServiceLinks, which became excessively long past a certain threshold. We moved execution-segment-sequence to its own config.json file, as I suspected it would eventually hit the ARG_MAX kernel limit.

Only runners get the list of segments so I assume this is about runners. enableServiceLinks seems to be about env vars injection though: do you have a lot of env vars (services) and compounded with a long list of segments, it became too large on runners too? IIRC, when I tried the runner with ~ 500 segments it worked, because argument list is not that large. And in that issue #408, we never figured out what the problem with runners actually was.

[max-tremblay]

@yorugac

I'm a bit confused here: how exactly does that work? E.g. this image above would have only one browser per pod, and k6-operator is not "aware" of browsers, yet. (issue)

Sorry, I used our internal terminology here. What I meant is that I have 4 virtual users, therefore 4 parallel chromium processes, running under the same pod.

Only runners get the list of segments so I assume this is about runners. enableServiceLinks seems to be about env vars injection though: do you have a lot of env vars (services) and compounded with a long list of segments, it became too large on runners too? IIRC, when I tried the runner with ~ 500 segments it worked, because argument list is not that large. And in that issue #408, we never figured out what the problem with runners actually was.

For the enableServiceLinks issue, this is indeed for runners. It was found around 5000 pods. 5k is my next milestone, but I have issues that are not related to k6 at the moment.

For #408, the issue is mostly on the starter, but I can confirm this.

[yorugac]

Thank you for the clarifications, @max-tremblay!

The runners issue: that's interesting; I admit we didn't have a test of that scale, AFAIK. Of course, if you could open an issue with details / PR for this case, that'd be amazing 🙂