Skip to content

Conversation

@omriKaltura
Copy link

Problem

When running k6 tests with a large number of hostnames (more than 500 IPs), the starter job fails with a 'Linux argument list too long' error. This is a limitation of the Linux command line length.

Solution

This PR implements a solution to handle large hostname lists by:

  1. Adding a constant maxHostnamesPerBatch = 500 to define the maximum number of hostnames per batch
  2. Implementing batching logic to split the hostnames list into smaller chunks
  3. Creating a loop to generate a starter job for each batch
  4. Adding unique naming for each starter job when multiple batches are needed
  5. Enhancing error handling to track successful job creation
  6. Improving logging to show batch information and provide a summary when multiple jobs are created

Testing

The solution has been tested with more than 500 hostnames and successfully creates multiple starter jobs to handle the large list.

…too long' error

- Added constant maxHostnamesPerBatch = 500 to define the maximum number of hostnames per batch
- Implemented batching logic to split the hostnames list into smaller chunks
- Created a loop to generate a starter job for each batch
- Added unique naming for each starter job when multiple batches are needed
- Enhanced error handling to track successful job creation
- Improved logging to show batch information and provide a summary when multiple jobs are created
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link

@adi-ziv adi-ziv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yorugac
Copy link
Collaborator

yorugac commented Sep 9, 2025

Hi @omriKaltura, I believe this problem first came up in this issue:
#408

But I personally think we should solve this by removing starter pod instead of creating multiple starter pods. Btw, multiple starter pods also add complexity: these pods are no longer synchronized.
FYI, we're likely to prioritize that work with starter removal rather soon. Please watch for updates in that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants