fix(server): drain fully before the self-update/restore restart handoff (#1380)

* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>

docs/deployment.md

@@ -104,6 +104,13 @@ image sets `OWNCORD_CONTAINER=1` to mark this; operators who bind-mount the
 server binary into a container and genuinely want in-place self-update can
 set `OWNCORD_CONTAINER=0` to opt back in.
 
+The admin panel's backup **restore** (and a setup-wizard restart) does work
+in containers: the server drains and exits cleanly, relying on the
+container's restart policy to relaunch it. The shipped `docker-compose.yml`
+sets `restart: unless-stopped`, which covers this; if you run the container
+by hand, pass `--restart unless-stopped` or the restore leaves the container
+stopped.
+
 ### LiveKit in Docker
 
 LiveKit runs as its own container (`livekit/livekit-server:v1`) and is **not** managed by OwnCord's companion-process system. Leave `voice.livekit_binary` unset. See [LiveKit Setup — Docker](livekit-setup.md#docker) for details.
@@ -148,9 +155,17 @@ supervisor. A ready-made unit template ships in the repo at
 [`deploy/owncord.service`](../deploy/owncord.service); installation steps are
 in its header comments. The important choices it encodes:
 
-- `Restart=on-failure` — the server deliberately exits (rather than limping
-  along) when its WebSocket dispatch loop dies; the supervisor is what turns
-  that into a recovery.
+- `Restart=always` — two deliberate exits rely on it: the server exits
+  nonzero (rather than limping along) when its WebSocket dispatch loop dies,
+  and it exits **cleanly** after an admin-panel self-update, backup restore,
+  or setup-wizard restart, expecting systemd to relaunch it running the
+  swapped binary (the server auto-detects systemd via `INVOCATION_ID` and
+  hands off this way instead of spawning a child that the unit's cgroup
+  cleanup would kill). `systemctl stop` still stops it — systemd never
+  auto-restarts an explicitly stopped unit. **Update the unit file before
+  applying server updates from the admin panel** — it also repairs the
+  update handoff when updating from older OwnCord releases, whose spawned
+  replacement gets reaped by the cgroup cleanup.
 - `TimeoutStopSec=35` — the server drains gracefully on SIGTERM with a 30s
   budget; systemd waits it out before escalating.
 - `ReadWritePaths=/opt/owncord` under `ProtectSystem=strict` — the install
@@ -178,6 +193,12 @@ nssm set OwnCord AppDirectory "C:\OwnCord"
 nssm set OwnCord DisplayName "OwnCord Chat Server"
 nssm set OwnCord Start SERVICE_AUTO_START
 
+# REQUIRED: tell the server NSSM supervises it. On a self-update/restore the
+# server then exits cleanly and NSSM's default AppExit=Restart relaunches it
+# with the new binary. (NSSM 2.24 is not auto-detectable, so without this the
+# server spawns its own replacement, which races NSSM's relaunch.)
+nssm set OwnCord AppEnvironmentExtra OWNCORD_SERVER_RESTART_MODE=supervised
+
 # Manage
 nssm start OwnCord
 nssm stop OwnCord
@@ -193,6 +214,10 @@ nssm restart OwnCord
 5. Check "Run whether user is logged on or not"
 6. Check "Run with highest privileges"
 
+Task Scheduler starts the process but does not supervise it, so leave
+`server.restart_mode` on its default (`auto` resolves to `spawn` here): on a
+self-update or restore the server starts its own replacement after draining.
+
 ## TLS Setup
 
 ### Self-Signed (default)
@@ -430,7 +455,17 @@ The server checks GitHub Releases for updates:
 - Downloads `chatserver.exe` with detached Ed25519/minisign signature verification
 - Verifies a signed `server-update-manifest.json` that binds the binary hash to the release version
 - Cross-checks the binary SHA256 against `checksums.sha256`
-- On restart, the current binary is rotated to `chatserver.exe.old` before the new binary takes its place
+
+Applying an update then runs in this order: the current binary is rotated to
+`chatserver.exe.old` and the verified download takes its place; connected
+clients get a "restarting in 5s" notice; the server drains completely
+(HTTP listeners, WebSocket hub, the companion `livekit-server`, queued
+event/audit writes, the database and its process lock); and only then does
+the handoff happen — the server either starts the new binary itself or, under
+a supervisor (systemd/NSSM/Docker, see `server.restart_mode` in
+[Server Configuration](server-configuration.md)), exits cleanly so the
+supervisor relaunches it. Because the old process is fully gone before the
+new one starts, the successor boots with no port or database-lock contention.
 
 Set `github.token` in config for higher API rate limits (5000/hr vs 60/hr unauthenticated).
 

docs/server-configuration.md

@@ -42,6 +42,7 @@ the server automatically when a startup-only value changed. Note that
 | `server.waf_enabled` | bool | `false` | Enable the Coraza WAF middleware (inline rules + OWASP Core Rule Set) |
 | `server.waf_paranoia_level` | int | `2` | OWASP CRS paranoia level 1–4; values outside that range fall back to 2 |
 | `server.waf_crs_mode` | string | `"detect"` | CRS layer mode: `off` (inline rules only), `detect` (matches logged, never blocks), `block` (anomaly-scoring blocking). Unknown values fall back to `detect`. |
+| `server.restart_mode` | string | `"auto"` | How self-restarts (update apply, backup restore, setup wizard) hand off after the server drains: `supervised` exits cleanly and relies on systemd/NSSM/Docker to relaunch; `spawn` starts the replacement binary directly; `auto` picks `supervised` when a supervisor or container is detected, else `spawn`. NSSM deployments must set `supervised` explicitly (see [Deployment](deployment.md)). |
 
 ### TLS (`tls`)
 
@@ -190,6 +191,7 @@ absent from the table below (it is a representative subset, not the full list).
 | `OWNCORD_SERVER_PORT` | `server.port` |
 | `OWNCORD_SERVER_NAME` | `server.name` |
 | `OWNCORD_SERVER_DATA_DIR` | `server.data_dir` |
+| `OWNCORD_SERVER_RESTART_MODE` | `server.restart_mode` |
 | `OWNCORD_DATABASE_PATH` | `database.path` |
 | `OWNCORD_TLS_MODE` | `tls.mode` |
 | `OWNCORD_TLS_CERT_FILE` | `tls.cert_file` |