Skip to content

Commit 14564e7

Browse files
committed
ci(deploy): pause watchdog timer during container swap
The systemd watchdog (/etc/systemd/system/ocaml-lob-watchdog.timer) races the deploy script's stop/rm/run sequence: the timer fires every 30s, and if it ticks between [docker stop] and [docker rm] it sees an unhealthy container and does [docker restart ocaml_lob], which brings the container back up. The subsequent [docker rm] then either fails (without -f) or leaves a stale container, and [docker run --name ocaml_lob] hits a name conflict. Saw this fire on the SSE deploy: build succeeded, deploy died with "container name already in use" while watchdog logs showed a restart firing in the same minute. Fix: 1. [systemctl stop ocaml-lob-watchdog.timer] before the swap, [systemctl start] via trap on EXIT so liveness coverage resumes even if the deploy fails mid-way. 2. [docker rm -f] for defense in depth — handles a running container regardless of how it got there.
1 parent adfeb77 commit 14564e7

1 file changed

Lines changed: 14 additions & 1 deletion

File tree

.github/workflows/deploy.yml

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,9 +138,22 @@ jobs:
138138
echo "→ pulling $IMAGE"
139139
docker pull "$IMAGE"
140140
141+
# Pause the host watchdog (see /etc/systemd/system/ocaml-lob-watchdog.timer)
142+
# for the duration of the swap. Otherwise it can fire between
143+
# `docker stop` and `docker rm`, see the new container as down,
144+
# and `docker restart` it back to running — at which point our
145+
# subsequent `docker rm` either fails or leaves a stale container,
146+
# and `docker run --name ocaml_lob` hits a name conflict. We
147+
# restart the timer unconditionally after, even on failure, so
148+
# liveness coverage resumes regardless of how the deploy exited.
149+
trap 'sudo systemctl start ocaml-lob-watchdog.timer 2>/dev/null || true' EXIT
150+
sudo systemctl stop ocaml-lob-watchdog.timer 2>/dev/null || true
151+
141152
echo "→ replacing container"
153+
# `-f` on rm forces removal of a running container — defense in
154+
# depth in case anything (manual or otherwise) restarted it.
142155
docker stop ocaml_lob 2>/dev/null || true
143-
docker rm ocaml_lob 2>/dev/null || true
156+
docker rm -f ocaml_lob 2>/dev/null || true
144157
docker run -d \
145158
--name ocaml_lob \
146159
--restart unless-stopped \

0 commit comments

Comments
 (0)