Files
J3vbandClaude Fable 5 6a26f2a839 fix(server): drain fully before the self-update/restore restart handoff (#1380)
* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 08:25:40 +02:00

113 lines
4.3 KiB
Go

package admin
import (
"log/slog"
"net/http"
"sync"
"sync/atomic"
)
// The admin panel has three actions that end in a process restart: applying a
// server update, restoring a database backup, and finishing the setup wizard
// with listener-affecting changes. None of them stop or spawn processes
// directly — they request a restart through the hook below, and the main
// package performs the actual work: trigger the same graceful drain a SIGTERM
// would (which also works on Windows, where a process cannot signal itself),
// and only after run() has fully torn down either spawn the replacement
// binary or exit for the process supervisor to relaunch, per
// server.restart_mode (see Server/restart.go).
// restartSelf is the process-restart request hook. Swapped by
// SetRestartHandoff at startup and by tests (StubRestart); the default logs
// loudly instead of exiting, so a mis-wired binary degrades to "restart
// manually" rather than to a silent no-op or a test-killing os.Exit.
var (
restartMu sync.Mutex
restartSelf = restartUnwired
)
func restartUnwired(reason string) {
slog.Error("restart requested but no restart coordinator is wired — restart the server manually",
"reason", reason)
}
// SetRestartHandoff wires restart requests to the main package's restart
// coordinator. Call once at startup, before the router serves (main.go, next
// to SetDatabasePath).
func SetRestartHandoff(fn func(reason string)) {
restartMu.Lock()
restartSelf = fn
restartMu.Unlock()
}
// requestRestart invokes the current restart hook.
func requestRestart(reason string) {
restartMu.Lock()
fn := restartSelf
restartMu.Unlock()
fn(reason)
}
// restartState serializes the restart-ending admin actions against each
// other. Two problems it closes: concurrent update applies raced the same
// staged .new file (each download deletes the other's staged file, and the
// loser broadcast a spurious update_aborted to every client while a restart
// was actually happening), and a restore could tear the database down under
// an apply that had already responded 200.
//
// idle ──beginRestartSensitiveOp──▶ busy ──commitRestartPending──▶ pending
// ▲ │
// └────abortRestartSensitiveOp─────┘
//
// pending is terminal: it means a restart request has been (or is about to
// be) issued and only the process replacement clears it. Ownership of the
// busy state transfers to the background goroutine that finishes the work —
// the HTTP handler responds while the state is still busy, so it must NOT
// defer a reset.
const (
restartStateIdle int32 = 0
restartStateBusy int32 = 1 // an update apply or restore is in flight
restartStatePending int32 = 2 // a restart has been requested
)
var restartState atomic.Int32
// beginRestartSensitiveOp claims the exclusive restart-sensitive slot.
// Callers that fail any later step must release it with
// abortRestartSensitiveOp; callers that reach the point of no return promote
// it with commitRestartPending.
func beginRestartSensitiveOp() bool {
return restartState.CompareAndSwap(restartStateIdle, restartStateBusy)
}
// abortRestartSensitiveOp releases the slot after a failed operation. CAS,
// not a blind store: an abort must never demote an already-pending restart.
func abortRestartSensitiveOp() {
restartState.CompareAndSwap(restartStateBusy, restartStateIdle)
}
// commitRestartPending marks the process as committed to restarting.
func commitRestartPending() {
restartState.Store(restartStatePending)
}
// tryDirectRestartPending is commitRestartPending for paths with no failable
// work between claiming the slot and requesting the restart (setup wizard):
// idle → pending in one step. Reports whether the claim won.
func tryDirectRestartPending() bool {
return restartState.CompareAndSwap(restartStateIdle, restartStatePending)
}
// writeRestartConflict answers a request that lost to an in-flight
// restart-sensitive operation, distinguishing "busy, try again shortly" from
// "the process is about to be replaced".
func writeRestartConflict(w http.ResponseWriter) {
if restartState.Load() == restartStatePending {
writeErr(w, http.StatusConflict, "RESTART_PENDING",
"the server is restarting — retry after it comes back")
return
}
writeErr(w, http.StatusConflict, "UPDATE_IN_PROGRESS",
"another update or restore is already in progress")
}