mirror of
https://github.com/J3vb/OwnCord.git
synced 2026-09-03 03:50:00 +03:00
* feat(server): supervisor detection and server.restart_mode config key RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort, NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set the mode explicitly). server.restart_mode (auto|spawn|supervised, default auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands off after the server drains: exit for the supervisor to relaunch, or spawn the replacement directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp * fix(server): make the self-restart handoff drain fully before starting the successor The update/restore/wizard restart previously spawned the replacement while the old server was still serving, then SIGTERMed itself and hard-exited after 10s. That design failed in every documented deployment mode: under the shipped systemd unit the spawned child (same cgroup) was killed when the old main process exited and Restart=on-failure never relaunched a clean exit; on Windows the self-SIGTERM is unsupported and silently dropped, so graceful shutdown never ran — hub.GracefulStop (the only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and NSSM's relaunch raced the self-spawned replacement for the database lock. Admin handlers now perform only the on-disk swap and request a restart through an injected hook (admin.SetRestartHandoff). The main package's restart coordinator cancels the parent of run()'s signal.NotifyContext — the exact drain a SIGTERM triggers, on every platform — and after run() has fully torn down (listeners closed, hub and LiveKit stopped, queues flushed, DB closed and its lock released) main() performs the handoff: spawn the replacement in spawn mode, or exit 0 for the supervisor in supervised mode. A 90s backstop force-exits a wedged teardown; the DB-lock and bind retries demote to safety nets. A three-state guard (idle/busy/restart-pending) serializes update apply, backup restore, and setup-wizard restarts against each other: concurrent applies no longer race the same staged .new file or broadcast a spurious update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS / RESTART_PENDING. The swap being free of process side effects also makes the apply success path unit-testable for the first time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp * fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE 10048 on Windows) with the English strings kept only as fallback — the string-only match never fired on localized Windows, silently disabling the bind retry. The retry loop is extracted into serveWithBindRetry and now also covers the ACME :80 challenge server, which previously gave up on first conflict and stayed dead (breaking HTTP-01 renewals) until the next restart. The .old-binary boot cleanup retries briefly for the window where a spawn-mode predecessor has not fully exited. The companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent killed without teardown (kill -9, OOM, backstop exit) cannot orphan it with the voice ports held. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp * docs(deploy): Restart=always unit and per-supervisor restart-mode guidance Restart=always is what lets the deliberate clean exit after a self-update/restore relaunch under systemd (systemctl stop is never auto-restarted; failure exits behave as before). Deployment docs gain the required NSSM AppEnvironmentExtra line, the Task Scheduler and Docker restart-policy notes, and the new drain-then-handoff update flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp --------- Co-authored-by: Claude <noreply@anthropic.com>
113 lines
4.3 KiB
Go
113 lines
4.3 KiB
Go
package admin
|
|
|
|
import (
|
|
"log/slog"
|
|
"net/http"
|
|
"sync"
|
|
"sync/atomic"
|
|
)
|
|
|
|
// The admin panel has three actions that end in a process restart: applying a
|
|
// server update, restoring a database backup, and finishing the setup wizard
|
|
// with listener-affecting changes. None of them stop or spawn processes
|
|
// directly — they request a restart through the hook below, and the main
|
|
// package performs the actual work: trigger the same graceful drain a SIGTERM
|
|
// would (which also works on Windows, where a process cannot signal itself),
|
|
// and only after run() has fully torn down either spawn the replacement
|
|
// binary or exit for the process supervisor to relaunch, per
|
|
// server.restart_mode (see Server/restart.go).
|
|
|
|
// restartSelf is the process-restart request hook. Swapped by
|
|
// SetRestartHandoff at startup and by tests (StubRestart); the default logs
|
|
// loudly instead of exiting, so a mis-wired binary degrades to "restart
|
|
// manually" rather than to a silent no-op or a test-killing os.Exit.
|
|
var (
|
|
restartMu sync.Mutex
|
|
restartSelf = restartUnwired
|
|
)
|
|
|
|
func restartUnwired(reason string) {
|
|
slog.Error("restart requested but no restart coordinator is wired — restart the server manually",
|
|
"reason", reason)
|
|
}
|
|
|
|
// SetRestartHandoff wires restart requests to the main package's restart
|
|
// coordinator. Call once at startup, before the router serves (main.go, next
|
|
// to SetDatabasePath).
|
|
func SetRestartHandoff(fn func(reason string)) {
|
|
restartMu.Lock()
|
|
restartSelf = fn
|
|
restartMu.Unlock()
|
|
}
|
|
|
|
// requestRestart invokes the current restart hook.
|
|
func requestRestart(reason string) {
|
|
restartMu.Lock()
|
|
fn := restartSelf
|
|
restartMu.Unlock()
|
|
fn(reason)
|
|
}
|
|
|
|
// restartState serializes the restart-ending admin actions against each
|
|
// other. Two problems it closes: concurrent update applies raced the same
|
|
// staged .new file (each download deletes the other's staged file, and the
|
|
// loser broadcast a spurious update_aborted to every client while a restart
|
|
// was actually happening), and a restore could tear the database down under
|
|
// an apply that had already responded 200.
|
|
//
|
|
// idle ──beginRestartSensitiveOp──▶ busy ──commitRestartPending──▶ pending
|
|
// ▲ │
|
|
// └────abortRestartSensitiveOp─────┘
|
|
//
|
|
// pending is terminal: it means a restart request has been (or is about to
|
|
// be) issued and only the process replacement clears it. Ownership of the
|
|
// busy state transfers to the background goroutine that finishes the work —
|
|
// the HTTP handler responds while the state is still busy, so it must NOT
|
|
// defer a reset.
|
|
const (
|
|
restartStateIdle int32 = 0
|
|
restartStateBusy int32 = 1 // an update apply or restore is in flight
|
|
restartStatePending int32 = 2 // a restart has been requested
|
|
)
|
|
|
|
var restartState atomic.Int32
|
|
|
|
// beginRestartSensitiveOp claims the exclusive restart-sensitive slot.
|
|
// Callers that fail any later step must release it with
|
|
// abortRestartSensitiveOp; callers that reach the point of no return promote
|
|
// it with commitRestartPending.
|
|
func beginRestartSensitiveOp() bool {
|
|
return restartState.CompareAndSwap(restartStateIdle, restartStateBusy)
|
|
}
|
|
|
|
// abortRestartSensitiveOp releases the slot after a failed operation. CAS,
|
|
// not a blind store: an abort must never demote an already-pending restart.
|
|
func abortRestartSensitiveOp() {
|
|
restartState.CompareAndSwap(restartStateBusy, restartStateIdle)
|
|
}
|
|
|
|
// commitRestartPending marks the process as committed to restarting.
|
|
func commitRestartPending() {
|
|
restartState.Store(restartStatePending)
|
|
}
|
|
|
|
// tryDirectRestartPending is commitRestartPending for paths with no failable
|
|
// work between claiming the slot and requesting the restart (setup wizard):
|
|
// idle → pending in one step. Reports whether the claim won.
|
|
func tryDirectRestartPending() bool {
|
|
return restartState.CompareAndSwap(restartStateIdle, restartStatePending)
|
|
}
|
|
|
|
// writeRestartConflict answers a request that lost to an in-flight
|
|
// restart-sensitive operation, distinguishing "busy, try again shortly" from
|
|
// "the process is about to be replaced".
|
|
func writeRestartConflict(w http.ResponseWriter) {
|
|
if restartState.Load() == restartStatePending {
|
|
writeErr(w, http.StatusConflict, "RESTART_PENDING",
|
|
"the server is restarting — retry after it comes back")
|
|
return
|
|
}
|
|
writeErr(w, http.StatusConflict, "UPDATE_IN_PROGRESS",
|
|
"another update or restore is already in progress")
|
|
}
|