Skip to content

Commit 1c5ff68

Browse files
njzjz-botnjzjz-bot
andauthored
ci(lammps): retry tests after transient SIGTERM (#5804)
## Summary - retry the LAMMPS pytest process up to three attempts only when it exits with status 143 (`SIGTERM`) - preserve all ordinary test failures and return their original exit codes immediately - handle the retry outside pytest because a process-level `SIGTERM` prevents pytest plugins from recovering ## Background The [failing job](https://github.com/deepmodeling/deepmd-kit/actions/runs/29312634856/job/87019564217) reaches the first active `PyLammps()` test and then exits without a Python, LAMMPS, MPI, or pytest error: ```text source/lmp/tests/test_lammps_pd.py Process completed with exit code 143. ``` Exit 143 is `128 + SIGTERM`. The same failure signature has occurred intermittently on unrelated commits and with different DeePMD backends. Local fresh-process LAMMPS/MPI stress tests and repeated Paddle test collection did not reproduce a deterministic DeePMD failure, so this change treats it as a process-level transient while keeping actionable failures visible. ## Validation - `ruff check .` - `ruff format . --check` - `bash -n source/install/retry_on_sigterm.sh` - verified an actual `SIGTERM` retries and succeeds on the next attempt - verified a non-SIGTERM exit status is returned without retrying - Paddle-only LAMMPS tests: `9 passed, 228 skipped` Coding agent: Codex Codex version: codex-cli 0.144.1 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved automated test reliability by retrying interrupted test runs caused by termination signals. * Added validation and controlled retry limits to prevent unexpected repeated executions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
1 parent 2aee81c commit 1c5ff68

2 files changed

Lines changed: 41 additions & 1 deletion

File tree

.github/workflows/test_cc.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ jobs:
7676
if: matrix.enable_paddle
7777
- run: |
7878
export LD_LIBRARY_PATH=${{ github.workspace }}/dp_test/lib:$LD_LIBRARY_PATH
79-
pytest --cov=deepmd source/lmp/tests
79+
source/install/retry_on_sigterm.sh 3 pytest --cov=deepmd source/lmp/tests
8080
env:
8181
OMP_NUM_THREADS: 1
8282
TF_INTRA_OP_PARALLELISM_THREADS: 1

source/install/retry_on_sigterm.sh

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/bin/bash
2+
3+
# Retry a command only when it was terminated by SIGTERM (128 + SIGTERM = 143).
4+
#
5+
# GitHub-hosted runners have intermittently terminated the LAMMPS pytest
6+
# process without a Python, LAMMPS, or MPI error. Retrying only exit code 143
7+
# masks that external transient while preserving every actionable test failure.
8+
9+
set -u
10+
11+
if [ "$#" -lt 2 ]; then
12+
echo "Usage: $0 MAX_ATTEMPTS COMMAND [ARG ...]" >&2
13+
exit 2
14+
fi
15+
16+
max_attempts=$1
17+
shift
18+
19+
if ! [[ "$max_attempts" =~ ^[1-9][0-9]*$ ]]; then
20+
echo "MAX_ATTEMPTS must be a positive integer." >&2
21+
exit 2
22+
fi
23+
24+
attempt=1
25+
while true; do
26+
echo "Running command (attempt ${attempt}/${max_attempts}): $*"
27+
"$@"
28+
status=$?
29+
30+
if [ "$status" -eq 0 ]; then
31+
exit 0
32+
fi
33+
if [ "$status" -ne 143 ] || [ "$attempt" -ge "$max_attempts" ]; then
34+
exit "$status"
35+
fi
36+
37+
echo "Command received SIGTERM (exit 143); retrying after 2 seconds." >&2
38+
sleep 2
39+
attempt=$((attempt + 1))
40+
done

0 commit comments

Comments
 (0)