Files
OwnCord/Server/admin/export_test.go
T
J3vbandClaude Fable 5 6a26f2a839 fix(server): drain fully before the self-update/restore restart handoff (#1380)
* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 08:25:40 +02:00

165 lines
5.6 KiB
Go

package admin
import (
"errors"
"sync"
"sync/atomic"
"time"
"github.com/owncord/server/auth"
"github.com/owncord/server/db"
)
// CaptureSetupLimiter installs h so the next NewAdminAPI call reports the
// *auth.RateLimiter it creates for the /setup endpoint. NewAdminAPI returns
// only an http.Handler, so this is the only way tests can reach that limiter
// to check whether its stale entries get reaped.
func CaptureSetupLimiter(h func(*auth.RateLimiter)) (restore func()) {
prev := setupLimiterHook
setupLimiterHook = h
return func() { setupLimiterHook = prev }
}
// SetSetupLimiterReapTiming overrides the interval and max-window the setup
// endpoint's rate-limiter reaper uses, so tests don't wait on the real
// 5-minute interval.
func SetSetupLimiterReapTiming(interval, maxWindow time.Duration) (restore func()) {
prevI, prevW := setupLimiterReapInterval, setupLimiterReapMaxWindow
setupLimiterReapInterval = interval
setupLimiterReapMaxWindow = maxWindow
return func() {
setupLimiterReapInterval = prevI
setupLimiterReapMaxWindow = prevW
}
}
// SetBackupBaseDir overrides backupBaseDir so tests can point backup handlers
// at a temp dir. Lives here so it stays out of the production binary.
func SetBackupBaseDir(dir string) { backupBaseDir = dir }
// StubCopyBackup swaps the restore path's file-copy hook so tests can inject
// mid-copy failures that pass the pre-copy integrity gate. CopyBackupForTest
// is the real implementation, for stubs that only want to fail once.
func StubCopyBackup(fn func(src, dst string) error) (restore func()) {
prev := copyBackupFile
copyBackupFile = fn
return func() { copyBackupFile = prev }
}
// CopyBackupForTest exposes the real copyFile for StubCopyBackup delegates.
var CopyBackupForTest = copyFile
// StubCloseError makes the next handleRestoreBackup call's database.Close()
// return err instead of actually closing the pools, so tests can exercise the
// Close-failure branch without a genuine driver-level close error (see
// dbCloser's doc comment for why that's not otherwise reachable in a test).
func StubCloseError(msg string) (restore func()) {
closeMu.Lock()
prev := dbCloser
dbCloser = func(*db.DB) error { return errors.New(msg) }
closeMu.Unlock()
return func() {
closeMu.Lock()
dbCloser = prev
closeMu.Unlock()
}
}
// ApplyStagedUpdate exposes applyStagedUpdate (the on-disk swap behind
// POST /updates/apply's background goroutine) so tests can drive it directly
// with fake filesystem paths, instead of exercising the full HTTP handler —
// which resolves exePath via os.Executable() and would rename/replace the
// running test binary itself. The swap has no process side effects anymore
// (no spawn, no signal, no exit), so the success path is testable too.
var ApplyStagedUpdate = applyStagedUpdate
// ApplyAndRestart exposes the whole background tail (countdown broadcast →
// swap → restart request / guard release) for tests, combined with
// StubRestart and SetApplyRestartDelay.
var ApplyAndRestart = applyAndRestart
// SetApplyRestartDelay shrinks the client-facing restart countdown so tests
// of applyAndRestart don't sleep the real 5 seconds.
func SetApplyRestartDelay(d time.Duration) (restore func()) {
prev := applyRestartDelay
applyRestartDelay = d
return func() { applyRestartDelay = prev }
}
// ResetRestartState returns the restart-serialization guard to idle. Tests
// that drive a restart-committing path (restore, apply success, setup
// restart) must call it afterwards — the state is process-global and would
// otherwise 409 every later test's request.
func ResetRestartState() { restartState.Store(restartStateIdle) }
// ForceRestartState pins the guard for conflict-path tests: busy=false sets
// restart-pending, busy=true sets an in-flight exclusive operation.
func ForceRestartState(busy bool) {
if busy {
restartState.Store(restartStateBusy)
} else {
restartState.Store(restartStatePending)
}
}
// CurrentRestartState reports the guard state by name for assertions.
func CurrentRestartState() string {
switch restartState.Load() {
case restartStateBusy:
return "busy"
case restartStatePending:
return "pending"
default:
return "idle"
}
}
// RequestRestartForTest exposes requestRestart so the unwired default hook's
// inertness (log, no exit, no spawn) is directly testable.
var RequestRestartForTest = requestRestart
// StubRestart replaces the process-restart hook for the duration of a test and
// returns a func reporting whether a restart was requested. Without this the
// restore handler's request would hit the unwired-hook error log; with it the
// test can assert the request happened. The restore func also returns the
// restart-serialization guard to idle, since any test that triggered the hook
// has necessarily left it in restart-pending.
func StubRestart() (restarted func() bool, restore func()) {
restartMu.Lock()
prev := restartSelf
called := &atomic.Bool{}
restartSelf = func(string) { called.Store(true) }
restartMu.Unlock()
return called.Load, func() {
restartMu.Lock()
restartSelf = prev
restartMu.Unlock()
ResetRestartState()
}
}
// StubRestartCapture is StubRestart recording the reasons passed to the hook,
// for tests that assert which path requested the restart.
func StubRestartCapture() (reasons func() []string, restore func()) {
restartMu.Lock()
prev := restartSelf
var mu sync.Mutex
var got []string
restartSelf = func(reason string) {
mu.Lock()
got = append(got, reason)
mu.Unlock()
}
restartMu.Unlock()
return func() []string {
mu.Lock()
defer mu.Unlock()
return append([]string(nil), got...)
}, func() {
restartMu.Lock()
restartSelf = prev
restartMu.Unlock()
ResetRestartState()
}
}