Files
OwnCord/Server/admin/handlers_backup.go
T
J3vbandClaude Fable 5 6a26f2a839 fix(server): drain fully before the self-update/restore restart handoff (#1380)
* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 08:25:40 +02:00

380 lines
15 KiB
Go

package admin
import (
"context"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"github.com/go-chi/chi/v5"
"github.com/owncord/server/db"
)
// backupBaseDir is the directory for backup files, resolved to an absolute
// path at package init time so handlers don't depend on the process CWD (L14).
// Overridden at startup via SetBackupDir with cfg.Backup.Dir.
var backupBaseDir string
func init() {
backupBaseDir = absOrRaw(filepath.Join("data", "backups"))
}
// SetBackupDir points every backup handler and the scheduled-backup
// maintenance at the operator-configured directory. Call once at startup with
// cfg.Backup.Dir (main.go, next to SetDatabasePath); tests use it to isolate
// a temp dir. Mirrors SetDatabasePath: without it, a configured backup.dir
// would be ignored while backups keep landing in the default location.
func SetBackupDir(dir string) {
if dir == "" {
return
}
backupBaseDir = absOrRaw(dir)
}
func absOrRaw(p string) string {
if abs, err := filepath.Abs(p); err == nil {
return abs
}
return p
}
// dbFilePath is the live SQLite database file that "Restore backup"
// overwrites. It defaults to the historical "data/chatserver.db" but must be
// pointed at cfg.Database.Path via SetDatabasePath before the server starts
// serving requests (main.go, right after db.Open): without that call, a
// server configured with a non-default database.path would open its real
// database at cfg.Database.Path while restore keeps copying backups over an
// unrelated (possibly newly created) file at the default path, reporting
// success while the live database is never touched.
var dbFilePath = filepath.Join("data", "chatserver.db")
// SetDatabasePath points the restore handler at the SQLite file the server
// actually opened. Call once at startup with cfg.Database.Path; tests use it
// to point restore at an isolated temp file.
func SetDatabasePath(path string) {
dbFilePath = path
}
// ─── Backup Handlers ─────────────────────────────────────────────────────────
func handleBackup(database *db.DB) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
backupDir := backupBaseDir
if err := os.MkdirAll(backupDir, 0o750); err != nil {
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to create backup directory")
return
}
timestamp := time.Now().UTC().Format("20060102_150405")
backupPath := filepath.Join(backupDir, "chatserver_"+timestamp+".db")
// Detached like the restore path's safety backup: an interrupted
// VACUUM INTO leaves a truncated .db that handleListBackups would
// present as restorable. BackupToSafe is rooted at the configured
// backup dir (SetBackupDir), not the historical hardcoded default.
if err := database.BackupToSafe(context.WithoutCancel(r.Context()), backupPath, backupDir); err != nil {
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "backup failed")
return
}
// Verify before reporting success: a backup that fails integrity_check
// is worse than no backup, because the operator believes they have one.
if err := db.CheckBackupIntegrity(context.WithoutCancel(r.Context()), backupPath); err != nil {
slog.Error("backup failed integrity check — removing", "path", backupPath, "err", err)
_ = os.Remove(backupPath)
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "backup failed verification")
return
}
actor := actorFromContext(r)
backupName := filepath.Base(backupPath)
slog.Info("database backup created", "actor_id", actor, "name", backupName)
db.WriteAudit(context.WithoutCancel(r.Context()), database, actor, "backup_create", "server", 0,
fmt.Sprintf("backup saved: %s", backupName))
writeJSON(w, http.StatusOK, map[string]string{
"path": filepath.Base(backupPath),
"created": timestamp,
})
})
}
// backupEntry is the JSON shape returned by GET /admin/api/backups.
type backupEntry struct {
Name string `json:"name"`
Size int64 `json:"size"`
Date string `json:"date"`
}
func handleListBackups() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
backupDir := backupBaseDir
entries, err := os.ReadDir(backupDir)
if err != nil {
if os.IsNotExist(err) {
writeJSON(w, http.StatusOK, []backupEntry{})
return
}
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to list backups")
return
}
var backups []backupEntry
for _, e := range entries {
if e.IsDir() || filepath.Ext(e.Name()) != ".db" {
continue
}
info, err := e.Info()
if err != nil {
continue
}
backups = append(backups, backupEntry{
Name: e.Name(),
Size: info.Size(),
Date: info.ModTime().UTC().Format(time.RFC3339),
})
}
if backups == nil {
backups = []backupEntry{}
}
// Sort newest first.
sort.Slice(backups, func(i, j int) bool {
return backups[i].Date > backups[j].Date
})
writeJSON(w, http.StatusOK, backups)
}
}
func handleDeleteBackup(database *db.DB) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "name")
if name == "" || strings.Contains(name, "..") || strings.ContainsAny(name, `/\`) {
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
return
}
target := filepath.Join(backupBaseDir, name)
if !strings.HasPrefix(target, backupBaseDir+string(filepath.Separator)) {
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
return
}
if _, err := os.Stat(target); os.IsNotExist(err) { //nolint:gosec // G703: path sanitized by HasPrefix check above
writeErr(w, http.StatusNotFound, "NOT_FOUND", "backup not found")
return
}
if err := os.Remove(target); err != nil { //nolint:gosec // G703: path sanitized by HasPrefix check above
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to delete backup")
return
}
actor := actorFromContext(r)
slog.Info("backup deleted", "actor_id", actor, "name", name)
db.WriteAudit(context.WithoutCancel(r.Context()), database, actor, "backup_delete", "server", 0, "deleted backup "+name)
w.WriteHeader(http.StatusNoContent)
})
}
func handleRestoreBackup(database *db.DB, hub HubBroadcaster) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "name")
if name == "" || strings.Contains(name, "..") || strings.ContainsAny(name, `/\`) {
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
return
}
target := filepath.Join(backupBaseDir, name)
if !strings.HasPrefix(target, backupBaseDir+string(filepath.Separator)) {
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
return
}
// Serialize against a concurrent update apply (which stages a new
// binary and then restarts) and an already-requested restart — a
// restore must not close the database out from under either. The
// deferred release is a busy→idle CAS, so it covers every failure
// return below and harmlessly no-ops once a branch has promoted the
// state to restart-pending via commitRestartPending.
if !beginRestartSensitiveOp() {
writeRestartConflict(w)
return
}
defer abortRestartSensitiveOp()
if _, err := os.Stat(target); os.IsNotExist(err) { //nolint:gosec // G703: path sanitized by HasPrefix check above
writeErr(w, http.StatusNotFound, "NOT_FOUND", "backup not found")
return
}
// Refuse to overwrite the live database with a file SQLite itself
// rejects — a truncated pre-crash backup, a stray non-database .db.
// The pre-restore safety copy would make this survivable, but "restore
// succeeded" followed by a broken server is still the worst UX here.
if err := db.CheckBackupIntegrity(context.WithoutCancel(r.Context()), target); err != nil {
slog.Error("restore refused: backup failed integrity check", "backup", name, "err", err)
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "backup file failed integrity verification")
return
}
dbPath := dbFilePath
actor := actorFromContext(r)
// Audit the restore BEFORE the pre-restore safety copy is taken, and
// synchronously (LogAudit, not WriteAudit): the restore overwrites the
// live database file, so the only durable home for this row is the
// pre_restore_* backup captured below — an entry enqueued on the async
// WriteAudit path could still be sitting in the writer's buffer when
// BackupTo snapshots the DB. Best-effort per policy D8: a failed write
// is logged, never a reason to refuse the restore.
if err := database.LogAudit(context.WithoutCancel(r.Context()), actor, "backup_restore", "server", 0,
fmt.Sprintf("restoring backup %s", name)); err != nil {
slog.Error("audit log write failed", "action", "backup_restore", "actor_id", actor, "error", err)
}
// Safety: create a pre-restore backup before overwriting. WithoutCancel:
// the restore proceeds regardless of client disconnect (Close/copyFile
// below are not ctx-aware), so the safety backup must not be skippable
// by a canceled request ctx.
// backupBaseDir, not a cwd-relative path: the safety copy has to land in
// the same directory the rest of the backup handlers read and write, or
// a server started from another working directory writes it somewhere
// the operator will never find it.
preRestore := filepath.Join(backupBaseDir, "pre_restore_"+time.Now().UTC().Format("20060102_150405")+".db")
if err := database.BackupToSafe(context.WithoutCancel(r.Context()), preRestore, backupBaseDir); err != nil {
// Fail closed. The admin panel promises "a pre-restore backup will
// be created" before an irreversible overwrite; proceeding without
// one takes away the safety net the operator was shown, exactly
// when they need it (restoring the wrong or a corrupt backup).
slog.Error("pre-restore backup failed — aborting restore", "err", err)
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR",
"could not create the pre-restore safety backup — restore aborted, database untouched")
return
}
// Notify clients that the server is restarting.
hub.BroadcastServerRestart("backup_restore", 5)
// Checkpoint the WAL and close the database connection before overwriting
// to prevent corruption from concurrent writes (BUG-096).
if _, checkpointErr := database.SQLDb().ExecContext(context.WithoutCancel(r.Context()), "PRAGMA wal_checkpoint(TRUNCATE)"); checkpointErr != nil {
slog.Warn("pre-restore WAL checkpoint failed", "err", checkpointErr)
}
slog.Warn("database restored from backup — closing DB", "actor_id", actor, "backup", name)
if err := closeDatabase(database); err != nil {
// database.Close() closes the writer and reader pools regardless of
// the error it returns (Server/db/db.go), so this process cannot
// serve anything more either way — every other failure path below
// (copyFile failing, and the success path itself) restarts for
// exactly that reason. The live database file is still intact here
// (copyFile hasn't run yet), so the restarted process comes back on
// the pre-restore data rather than leaving clients pinned on
// "Reconnecting..." against a process that never actually restarts.
slog.Error("failed to close database before restore — restarting anyway, DB pools are closed either way", "err", err)
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to close database")
commitRestartPending()
go requestRestart("backup_restore_close_failed")
return
}
// Stream the backup file over the (now closed) database to avoid loading
// the entire DB into memory (could be hundreds of MiB).
if err := copyBackupFile(target, dbPath); err != nil {
// copyFile truncates the destination with os.Create before it can know
// whether the read will succeed, so the live database file is already
// destroyed by the time we get here — and the DB is closed, so nothing
// is holding the old contents. Put the safety copy back rather than
// leaving the operator with a zero-byte database.
slog.Error("restore copy failed — rolling back to the pre-restore safety copy", "backup", name, "err", err)
msg := "failed to restore database file — the pre-restore safety copy was put back, server restarting"
if rbErr := copyBackupFile(preRestore, dbPath); rbErr != nil {
slog.Error("rollback from the pre-restore safety copy failed — recover manually",
"safety_copy", preRestore, "err", rbErr)
msg = "failed to restore database file AND failed to roll back — recover manually from " + filepath.Base(preRestore)
}
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", msg)
// The database was closed before the copy: this process cannot serve
// anything more either way, so it must restart exactly as it does on
// the success path.
commitRestartPending()
go requestRestart("backup_restore_failed")
return
}
slog.Warn("database file replaced — restarting to load the restored data", "backup", name)
writeJSON(w, http.StatusOK, map[string]string{
"message": "database restored — server restarting",
"backup": name,
})
// The database is closed and the file underneath it has been swapped:
// this process can serve nothing more. It used to stop here, leaving a
// live server answering every request against a closed DB while the
// response and the restart broadcast both claimed a restart was
// happening. Restart for real, the same way applying an update does.
commitRestartPending()
go requestRestart("backup_restore")
})
}
// dbCloser is swappable in tests to simulate database.Close() returning an
// error. modernc.org/sqlite's sqlite3_close_v2 essentially never fails on a
// normally-open connection, so there is no portable way to provoke a genuine
// Close() error from a real driver in a unit test; this seam lets tests
// exercise that branch directly. Guarded like restartSelf, for the same
// reason (swap happens on the test goroutine, read on the handler's).
var (
closeMu sync.Mutex
dbCloser = func(database *db.DB) error { return database.Close() }
)
// closeDatabase invokes the current close hook.
func closeDatabase(database *db.DB) error {
closeMu.Lock()
fn := dbCloser
closeMu.Unlock()
return fn(database)
}
// copyBackupFile is the restore path's file-copy hook. It exists as a var so
// tests can inject the hard-to-simulate mid-copy failure (truncate-then-fail)
// the rollback branch exists for; production never swaps it.
var copyBackupFile = copyFile
// copyFile streams src to dst without loading the entire file into memory.
func copyFile(src, dst string) error {
in, err := os.Open(src) //nolint:gosec // G703: src is from sanitized backup path
if err != nil {
return fmt.Errorf("open source: %w", err)
}
defer in.Close() //nolint:errcheck
out, err := os.Create(dst)
if err != nil {
return fmt.Errorf("create destination: %w", err)
}
if _, err := io.Copy(out, in); err != nil {
_ = out.Close()
return fmt.Errorf("copy: %w", err)
}
if err := out.Sync(); err != nil {
_ = out.Close()
return fmt.Errorf("sync: %w", err)
}
return out.Close()
}