Increase Hadoop container startup timeout and add a retry - #31007
Open
tbaeg wants to merge 2 commits into
Open
Conversation
findepi
reviewed
Sep 4, 2026
tbaeg
force-pushed
the
fix/hadoop-container-startup
branch
from
September 4, 2026 20:59
a4ad45d to
7b7ee7b
Compare
findepi
reviewed
Sep 4, 2026
Raise the default to six minutes and drop the GcsEnvironment override that now duplicates it.
The container occasionally misses its readiness deadline on a loaded runner and fails before any test runs, while rerunning the same job passes. Testcontainers stops the failed container before retrying, so a second attempt starts clean and recovers without a manual rerun.
tbaeg
changed the title
tbaeg
force-pushed
the
fix/hadoop-container-startup
branch
from
September 5, 2026 04:23
7b7ee7b to
a55a181
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Kerberos environments intermittently fail before any test runs, because the Hadoop
container never reaches readiness:
Allow more time.
HadoopContainergave supervisord three minutes.GcsEnvironmentandAzureEnvironmentalready overrode that same wait for the same image with six and twelveminutes, so three has been judged too short twice. This raises the default to six, matching
GcsEnvironment, whose override then duplicates it and is removed.Retry once. Rerunning the failed job passes, so the failure is transient — #30611
records four occurrences since August, each cleared by a rerun.
withStartupAttempts(2)gets that recovery within the run:
tryStart()stops the failed container beforerethrowing, so the second attempt starts clean. Verified locally that the second attempt
runs and leaves nothing behind.
pt (hive-kerberos): Kerberos Hive container startup times out (waiting forsocks-proxy entered RUNNING) #30611Additional context and related issues
HadoopContaineris constructed in 25 places, so this affects every Hadoop-based producttest. Neither change alters behaviour when the container starts in time.
Two costs: a rerun lands on a fresh runner whereas an attempt retries on the same host, so
the retry will not cover every case the rerun evidence does; and a genuinely broken
environment now takes two full waits to report.
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.