Skip to content

Some CI tests are flaky or real failures #2824

@ubaidsk

Description

@ubaidsk

There are some CI tests that seem to be flaky and fail.

  1. Failure in ctest.
Test project /home/runner/work/lpython/lpython
    Start 1: test_stacktrace
1/2 Test #1: test_stacktrace ..................   Passed    0.00 sec
    Start 2: test_lpython
2/2 Test #2: test_lpython .....................Subprocess aborted***Exception:   0.86 sec
[doctest] doctest version is "2.4.8"
[doctest] run with "--help" for options
0 0 0 0 0 0 0 0 0 0 �
test_lpython: /home/runner/micromamba/envs/lp/include/llvm/IR/DataLayout.h:656: uint64_t llvm::StructLayout::getElementOffset(unsigned int) const: Assertion `Idx < NumElements && "Invalid element idx!"' failed.
===============================================================================
/home/runner/work/lpython/lpython/src/lpython/tests/test_llvm.cpp:[15](https://github.com/lcompilers/lpython/actions/runs/13378363981/job/37362241953?pr=2823#step:6:16)56:
TEST CASE:  PythonCompiler classes

/home/runner/work/lpython/lpython/src/lpython/tests/test_llvm.cpp:1556: FATAL ERROR: test case CRASHED: SIGABRT - Abort (abnormal termination) signal

===============================================================================
[doctest] test cases:  55 |  54 passed | 1 failed | [17](https://github.com/lcompilers/lpython/actions/runs/13378363981/job/37362241953?pr=2823#step:6:18) skipped
[doctest] assertions: 486 | 486 passed | 0 failed |
[doctest] Status: FAILURE!


50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.87 sec

The following tests FAILED:
	  2 - test_lpython (Subprocess aborted)
Errors while running CTest
Error: Process completed with exit code 8.
  1. Failure in reference tests
compiler_tester.tester.RunException: Testing with reference output failed.
runtime_errors/test_assert_01.py * run_dbg
The JSON metadata differs against reference results
Reference JSON: tests/reference/run_dbg-test_assert_01-2f34744.json
Output JSON:    tests/output/run_dbg-test_assert_01-2f34744.json
Omitting 9 identical items
Differing items:
{'stderr_hash': '32b0a24f111e577fe4fc5b3f4a5994b951e34dde7986b3fb750c5f5e'} != {'stderr_hash': '4811af471c73572b285e9ea01c8689abdd3cb32c717b3cd4876d2669'}
{'returncode': 134} != {'returncode': 1}
Diff against: tests/reference/run_dbg-test_assert_01-2f34744.stderr
1,7c1,2
<   File "tests/runtime_errors/test_assert_01.py", line 1
<     def test():
<   File "tests/runtime_errors/test_assert_01.py", line 4
<     test()
<   File "tests/runtime_errors/test_assert_01.py", line 2
<     assert False
< AssertionError
---
> *** buffer overflow detected ***: terminated
> Aborted (core dumped)

Error: Process completed with exit code 1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions