Skip to content

bug: Intermittent test failure on slow CI servers #2292

Description

@farkasmate

Describe the Bug

Bats tests are intermittently failing on slow CI servers with an obscure error message:

--- FAIL: TestBatsTests (79.29s)
    --- FAIL: TestBatsTests/plugin_list_all_command (1.57s)
        main_test.go:57: stdout: 1..5
            ok 1 plugin_list_all should exit before syncing the plugin repo if disabled
            ok 2 plugin_list_all should sync repo when check_duration set to 0
            not ok 3 plugin_list_all no immediate repo sync expected because check_duration is greater than 0
            # (in test file /builds/farkasmate/aports/testing/asdf/src/asdf-0.18.1/test/plugin_list_all_command.bats, line 48)
            #   `[ "$status" -eq 0 ]' failed
            # error loading plugin index: unable to update plugin index:  * [new branch] master     -> origin/master
            ok 4 plugin_list_all skips repo sync because check_duration is set to never
            ok 5 plugin_list_all list all plugins in the repository
            
        main_test.go:57: stderr: 
        main_test.go:57: bats command failed to run test file successfully
FAIL

I noticed it with the 0.18.1 release when packaging it for Alpine Linux:
https://gitlab.alpinelinux.org/farkasmate/aports/-/jobs/2419328#L155

After a lot of tries I reproduced the issue with 0.18.0. This version was successfully released for Alpine Linux, I suspect this is a latent bug triggered by the load on the Alpine Linux CI servers.

I added delay to setup_repo() to reproduce the issue, see in the Steps to Reproduce section.

I narrowed down the issue to git init calls producing different hashes for plugin-index and plugin-index-2 repos if they are executing on different seconds.

I'm working on a patch for the Alpine Linux packaging and I'm happy to open a PR here.
If you have a preferred way to fix this, please let me know.

Steps to Reproduce

  1. To make failure persistent add delay into setup_repo() in test/test_helpers.bash:
    --- a/test/test_helpers.bash
    +++ b/test/test_helpers.bash
    @@ -131,6 +131,7 @@ setup_repo() {
       cp -r "$BATS_TEST_DIRNAME/fixtures/dummy_plugins_repo" "$ASDF_DIR/plugin-index"
       cp -r "$BATS_TEST_DIRNAME/fixtures/dummy_plugins_repo" "$ASDF_DIR/plugin-index-2"
       init_git_repo "$ASDF_DIR/plugin-index-2"
    +  sleep 1
       init_git_repo "$ASDF_DIR/plugin-index" "$ASDF_DIR/plugin-index-2"
       touch "$(asdf_dir)/tmp/repo-updated"
     }
    
  2. Run:
    bats --verbose-run --show-output-of-passing-tests ../../test/plugin_list_all_command.bats -f 'should sync repo when check_duration set to 0'
    

Expected Behaviour

1..1
ok 1 plugin_list_all should sync repo when check_duration set to 0
# bar                           http://example.com/bar
# dummy                         http://example.com/dummy
# foo                           http://example.com/foo

Actual Behaviour

1..1
not ok 1 plugin_list_all should sync repo when check_duration set to 0
# (in test file /workdir/test/plugin_list_all_command.bats, line 35)
#   `[ "$status" -eq 0 ]' failed
# error loading plugin index: unable to update plugin index:  * [new branch]      master     -> origin/master

Environment

OS:
Linux 738c13566009 7.0.11-orbstack-00360-gc9bc4d96ac70 #1 SMP PREEMPT Thu Jun  4 16:40:25 UTC 2026 x86_64 Linux

SHELL:
GNU bash, version 5.3.3(1)-release (x86_64-alpine-linux-musl)
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

BASH VERSION:
5.3.3(1)-release

ASDF VERSION:
0.18.0 (revision unknown)

ASDF INTERNAL VARIABLES:
ASDF_TOOL_VERSIONS_FILENAME=.tool-versions
ASDF_DATA_DIR=/root/.asdf
ASDF_CONFIG_FILE=/root/.asdfrc

ASDF INSTALLED PLUGINS:

asdf plugins affected (if relevant)

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions