fix(server): drain fully before the self-update/restore restart handoff (#1380)

* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
J3vb
2026-08-16 08:25:40 +02:00
committed by GitHub
co-authored by Claude Fable 5
parent a366160dc8
commit 6a26f2a839
29 changed files with 1540 additions and 171 deletions
+2
View File
@@ -42,6 +42,7 @@ the server automatically when a startup-only value changed. Note that
| `server.waf_enabled` | bool | `false` | Enable the Coraza WAF middleware (inline rules + OWASP Core Rule Set) |
| `server.waf_paranoia_level` | int | `2` | OWASP CRS paranoia level 14; values outside that range fall back to 2 |
| `server.waf_crs_mode` | string | `"detect"` | CRS layer mode: `off` (inline rules only), `detect` (matches logged, never blocks), `block` (anomaly-scoring blocking). Unknown values fall back to `detect`. |
| `server.restart_mode` | string | `"auto"` | How self-restarts (update apply, backup restore, setup wizard) hand off after the server drains: `supervised` exits cleanly and relies on systemd/NSSM/Docker to relaunch; `spawn` starts the replacement binary directly; `auto` picks `supervised` when a supervisor or container is detected, else `spawn`. NSSM deployments must set `supervised` explicitly (see [Deployment](deployment.md)). |
### TLS (`tls`)
@@ -190,6 +191,7 @@ absent from the table below (it is a representative subset, not the full list).
| `OWNCORD_SERVER_PORT` | `server.port` |
| `OWNCORD_SERVER_NAME` | `server.name` |
| `OWNCORD_SERVER_DATA_DIR` | `server.data_dir` |
| `OWNCORD_SERVER_RESTART_MODE` | `server.restart_mode` |
| `OWNCORD_DATABASE_PATH` | `database.path` |
| `OWNCORD_TLS_MODE` | `tls.mode` |
| `OWNCORD_TLS_CERT_FILE` | `tls.cert_file` |