Files
OwnCord/Server/admin/update_handlers.go
T
J3vbandClaude Fable 5 6a26f2a839 fix(server): drain fully before the self-update/restore restart handoff (#1380)
* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 08:25:40 +02:00

218 lines
8.3 KiB
Go

package admin
import (
"context"
"log/slog"
"net/http"
"os"
"path/filepath"
"time"
"github.com/owncord/server/updater"
"golang.org/x/mod/semver"
)
// handleCheckUpdate returns the current update status. can_apply tells the
// admin SPA whether POST /updates/apply is usable in this deployment (false
// in containers, where upgrades are image pulls).
func handleCheckUpdate(u *updater.Updater) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if u == nil {
writeErr(w, http.StatusServiceUnavailable, "UPDATE_UNAVAILABLE", "update checking is not configured")
return
}
info, err := u.CheckForUpdate(r.Context())
if err != nil {
slog.Error("update check failed", "err", err)
writeErr(w, http.StatusBadGateway, "UPDATE_CHECK_FAILED", "failed to check for updates — see server logs")
return
}
writeJSON(w, http.StatusOK, struct {
updater.UpdateInfo
CanApply bool `json:"can_apply"`
}{info, !updater.RunningInContainer()})
}
}
// applyRestartDelay is how long the "restarting in 5s" countdown broadcast to
// clients actually gets before the swap + restart request. A var so tests can
// shrink it; the broadcast countdown below stays 5 to match this value.
var applyRestartDelay = 5 * time.Second
// handleApplyUpdate downloads and applies a server update.
func handleApplyUpdate(u *updater.Updater, hub HubBroadcaster, _ string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// In a container the running binary is image content: the staged
// replacement dies with the container and the restart comes back as
// the old image. Refuse before any nil/availability logic so the
// answer does not depend on updater configuration.
if updater.RunningInContainer() {
writeErr(w, http.StatusServiceUnavailable, "CONTAINER_DEPLOYMENT",
"in-place self-update is disabled in container deployments — upgrade by pulling the new image")
return
}
if u == nil {
writeErr(w, http.StatusServiceUnavailable, "UPDATE_UNAVAILABLE", "update checking is not configured")
return
}
// Serialize against concurrent applies, restores, and an
// already-requested restart — claimed before any updater work so a
// pending restart answers 409 without an outbound GitHub call, and
// so two concurrent applies can never both stage into the same .new
// path (each download removes the other's staged file). The deferred
// release covers every early return; ownership transfers to the
// applyAndRestart goroutine at the bottom.
if !beginRestartSensitiveOp() {
writeRestartConflict(w)
return
}
claimed := true
defer func() {
if claimed {
abortRestartSensitiveOp()
}
}()
// Check for available update.
info, err := u.CheckForUpdate(r.Context())
if err != nil {
slog.Error("update check failed during apply", "err", err)
writeErr(w, http.StatusBadGateway, "UPDATE_CHECK_FAILED", "failed to check for updates — see server logs")
return
}
if !info.UpdateAvailable {
if semver.Compare(info.Current, info.Latest) < 0 && !info.RequiredAssetsPresent {
writeErr(w, http.StatusBadGateway, "MISSING_ASSETS", "release is missing required assets")
return
}
writeErr(w, http.StatusConflict, "NO_UPDATE", "server is already up to date")
return
}
if !info.RequiredAssetsPresent {
writeErr(w, http.StatusBadGateway, "MISSING_ASSETS", "release is missing required assets")
return
}
// Get current executable path.
exePath, err := os.Executable()
if err != nil {
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "cannot determine executable path")
return
}
exePath, err = filepath.EvalSymlinks(exePath)
if err != nil {
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "cannot resolve executable path")
return
}
newPath := exePath + ".new"
oldPath := exePath + ".old"
// Download and verify.
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
defer cancel()
// DownloadAndVerify stages the binary and returns its trusted hash
// (bound to the signed release manifest). The apply goroutine below
// re-verifies the staged file against this hash through an open
// handle — never by path — before the rename.
stagedHash, err := u.DownloadAndVerify(ctx, info.Latest, info.DownloadURL, info.ChecksumURL, info.SignatureURL, info.ManifestURL, info.ManifestSignatureURL, newPath)
if err != nil {
slog.Error("update download/verify failed", "err", err)
writeErr(w, http.StatusBadGateway, "DOWNLOAD_FAILED", "download or verification failed — see server logs")
return
}
// Respond to the client before shutting down.
writeJSON(w, http.StatusOK, map[string]string{
"status": "applying",
"version": info.Latest,
})
// Broadcast the restart countdown and finish in the background — the
// goroutine takes over the busy state claimed above, so the deferred
// release must stand down.
claimed = false
go applyAndRestart(hub, exePath, oldPath, newPath, stagedHash)
})
}
// applyAndRestart is POST /updates/apply's background tail: give clients the
// promised countdown, swap the binary on disk, and hand the process over to
// the main package's restart coordinator. On a failed swap it releases the
// exclusive slot claimed by the handler so a corrected release can be applied
// without a manual restart (the corrective update_aborted broadcast is sent
// by applyStagedUpdate's deferred guard).
func applyAndRestart(hub HubBroadcaster, exePath, oldPath, newPath, stagedHash string) {
if hub != nil {
hub.BroadcastServerRestart("update", 5)
}
time.Sleep(applyRestartDelay)
if applyStagedUpdate(hub, exePath, oldPath, newPath, stagedHash) {
commitRestartPending()
requestRestart("update")
return
}
abortRestartSensitiveOp()
}
// applyStagedUpdate performs the on-disk swap: verified staged binary ->
// exePath, previous binary -> .old. It reports whether the swap committed —
// on true the caller must request a restart, because the file at exePath is
// no longer the binary this process is running.
//
// The caller has already broadcast "restarting in 5s" to every connected
// client before invoking this, so every failure path must correct that
// promise — otherwise the client's restart banner counts down to a permanent
// "Reconnecting..." over a connection that never actually dropped (OC-0226).
// The deferred broadcast below covers all such paths (verification failure,
// rename failure, commit failure) with one guard; it is cancelled by setting
// committed=true once the verified binary is in place.
//
// It does NOT spawn, signal, or exit: the restart itself is the main
// package's job, after run() has fully drained (Server/restart.go). Keeping
// the swap free of process side effects is also what makes the success path
// unit-testable.
func applyStagedUpdate(hub HubBroadcaster, exePath, oldPath, newPath, stagedHash string) bool {
committed := false
defer func() {
if !committed && hub != nil {
hub.BroadcastServerRestart("update_aborted", 0)
}
}()
// TOCTOU guard: open the staged binary once, verify its hash
// through that handle, and commit (rename) that exact file.
// Commit fails if the path was swapped after verification, so
// the bytes verified are the bytes the restart will execute.
staged, err := updater.OpenVerifiedBinary(newPath, stagedHash)
if err != nil {
slog.Error("update: staged binary re-verification failed, aborting update", "error", err)
return false
}
defer staged.Close() //nolint:errcheck
// Rename: current -> .old, verified staged binary -> current
_ = os.Remove(oldPath) // remove any stale .old
if err := os.Rename(exePath, oldPath); err != nil {
slog.Error("update: rename current to old failed", "error", err)
return false
}
if err := staged.Commit(exePath); err != nil {
slog.Error("update: committing staged binary failed, restoring original binary", "error", err)
// Whatever is at exePath now (if anything) is not the verified
// binary; restoring .old replaces it.
if restoreErr := os.Rename(oldPath, exePath); restoreErr != nil {
slog.Error("update: CRITICAL — recovery rename also failed, server binary may be missing",
"restore_error", restoreErr, "original_error", err,
"old_path", oldPath, "exe_path", exePath)
}
return false
}
committed = true
slog.Info("update: staged binary committed — requesting restart", "path", exePath)
return true
}