Closed
Fix Error 143 (SIGTERM timeout) in CI tests by adding proper environment setup and timeouts#48
Conversation
- Split 32 EquiformerV2 test combinations into 4 balanced groups: * equiformer-v2-graphs: 9 tests (graph models) * equiformer-v2-examples-1: 12 tests (first half MPNN types + examples) * equiformer-v2-examples-2: 10 tests (second half MPNN types + examples) * equiformer-v2-deepspeed: 1 test (DeepSpeed integration) - Added @pytest.mark.equiformer_v2 marker to all EquiformerV2 tests - Updated CI workflow to run EquiformerV2 tests in separate parallel jobs - Excluded EquiformerV2 from general global-attention test group - Added pytest marker definitions to prevent warnings This prevents CI workflow timeout issues while maintaining full test coverage.
…ex -k expressions - Split pytest_examples_energy_equiformer into two separate test functions: * pytest_examples_energy_equiformer_group1 (SAGE, GIN, GAT, MFC, PNA, PNAPlus) * pytest_examples_energy_equiformer_group2 (SchNet, DimeNet, EGNN, PNAEq, PAINN) - Added specific pytest markers: equiformer_v2_examples_1 and equiformer_v2_examples_2 - Updated CI workflow to use markers (-m) instead of complex -k expressions - This should resolve CI timeout/failure issues with EquiformerV2 example tests The marker-based approach is more reliable than complex -k expressions in CI environments.
…ironment variable Critical fixes to resolve CI timeout issues: 1. **Fixed example scripts to respect NUM_EPOCHS**: - Modified qm9.py and md17.py to check NUM_EPOCHS environment variable - Now uses env value instead of always defaulting to JSON config - Tests set NUM_EPOCHS=1 but scripts were ignoring this 2. **Enhanced test environment setup**: - Added HYDRAGNN_VERBOSITY=0 to reduce logging overhead - Improved environment variable documentation in test functions 3. **Root cause**: - EquiformerV2 tests were running with default num_epoch from JSON (2 epochs) - With 22 test combinations and complex EquiformerV2 computations, this caused 1.5+ min timeouts - Now tests run with only 1 epoch, significantly reducing execution time This should resolve the SIGTERM (exit code 143) CI failures where tests were killed for exceeding time limits.
…upport to LennardJones.py Co-authored-by: allaffa <2488656+allaffa@users.noreply.github.com>
Co-authored-by: allaffa <2488656+allaffa@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix tests failing with error 143
Fix Error 143 (SIGTERM timeout) in CI tests by adding proper environment setup and timeouts
Oct 13, 2025
allaffa
force-pushed
the
equiformer-v2-integration
branch
from
October 13, 2025 18:22
4581f09 to
6e55536
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR ORNL#378 was experiencing continuous test failures with Error 143 (SIGTERM), indicating that tests were being killed by the CI system for exceeding time limits. While
qm9.pyandmd17.pyexample scripts were already fixed to respect theNUM_EPOCHSenvironment variable, several test functions were not properly configuring the environment when executing these scripts.Root Cause
The issue occurred because:
test_examples.py(pytest_examples_energy_gps,pytest_examples_grad_forces) andtest_forces_equivariant.py(pytest_examples) were calling example scripts without setting critical environment variables likeNUM_EPOCHS=1LennardJones.pydidn't respect theNUM_EPOCHSenvironment variable, unlike other example scriptstest_examples.pycontained duplicate code from an earlier functionSolution
This PR fixes all identified issues with minimal, surgical changes:
1. Fixed Test Functions (4 total)
Added proper environment setup and explicit timeouts to:
pytest_examples_energy_gpsintest_examples.pypytest_examples_grad_forcesintest_examples.pypytest_examplesintest_forces_equivariant.pyEach test now sets:
2. Updated LennardJones.py
Added
NUM_EPOCHSenvironment variable support to match the pattern used inqm9.pyandmd17.py:3. Code Cleanup
Removed duplicate code from
pytest_examples_energy_equiformer_group2function (lines 176-199 intest_examples.py).Impact
Performance Improvements:
Reliability:
Files Changed
tests/test_examples.py(+30, -24 lines) - Fixed 3 test functions, removed duplicate codetests/test_forces_equivariant.py(+16, -1 lines) - Fixed 1 test functionexamples/LennardJones/LennardJones.py(+6 lines) - Added NUM_EPOCHS supportTotal: +52 insertions, -27 deletions
Testing
All changes have been validated:
NUM_EPOCHSenvironment variableThis fix follows the same pattern that was previously applied to EquiformerV2 tests and should resolve the timeout issues in PR ORNL#378.
Original prompt
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.