mirror of
https://github.com/J3vb/OwnCord.git
synced 2026-09-03 03:50:00 +03:00
* feat(server): supervisor detection and server.restart_mode config key RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort, NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set the mode explicitly). server.restart_mode (auto|spawn|supervised, default auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands off after the server drains: exit for the supervisor to relaunch, or spawn the replacement directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp * fix(server): make the self-restart handoff drain fully before starting the successor The update/restore/wizard restart previously spawned the replacement while the old server was still serving, then SIGTERMed itself and hard-exited after 10s. That design failed in every documented deployment mode: under the shipped systemd unit the spawned child (same cgroup) was killed when the old main process exited and Restart=on-failure never relaunched a clean exit; on Windows the self-SIGTERM is unsupported and silently dropped, so graceful shutdown never ran — hub.GracefulStop (the only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and NSSM's relaunch raced the self-spawned replacement for the database lock. Admin handlers now perform only the on-disk swap and request a restart through an injected hook (admin.SetRestartHandoff). The main package's restart coordinator cancels the parent of run()'s signal.NotifyContext — the exact drain a SIGTERM triggers, on every platform — and after run() has fully torn down (listeners closed, hub and LiveKit stopped, queues flushed, DB closed and its lock released) main() performs the handoff: spawn the replacement in spawn mode, or exit 0 for the supervisor in supervised mode. A 90s backstop force-exits a wedged teardown; the DB-lock and bind retries demote to safety nets. A three-state guard (idle/busy/restart-pending) serializes update apply, backup restore, and setup-wizard restarts against each other: concurrent applies no longer race the same staged .new file or broadcast a spurious update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS / RESTART_PENDING. The swap being free of process side effects also makes the apply success path unit-testable for the first time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp * fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE 10048 on Windows) with the English strings kept only as fallback — the string-only match never fired on localized Windows, silently disabling the bind retry. The retry loop is extracted into serveWithBindRetry and now also covers the ACME :80 challenge server, which previously gave up on first conflict and stayed dead (breaking HTTP-01 renewals) until the next restart. The .old-binary boot cleanup retries briefly for the window where a spawn-mode predecessor has not fully exited. The companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent killed without teardown (kill -9, OOM, backstop exit) cannot orphan it with the voice ports held. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp * docs(deploy): Restart=always unit and per-supervisor restart-mode guidance Restart=always is what lets the deliberate clean exit after a self-update/restore relaunch under systemd (systemctl stop is never auto-restarted; failure exits behave as before). Deployment docs gain the required NSSM AppEnvironmentExtra line, the Task Scheduler and Docker restart-policy notes, and the new drain-then-handoff update flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp --------- Co-authored-by: Claude <noreply@anthropic.com>
380 lines
15 KiB
Go
380 lines
15 KiB
Go
package admin
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"io"
|
|
"log/slog"
|
|
"net/http"
|
|
"os"
|
|
"path/filepath"
|
|
"sort"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/go-chi/chi/v5"
|
|
"github.com/owncord/server/db"
|
|
)
|
|
|
|
// backupBaseDir is the directory for backup files, resolved to an absolute
|
|
// path at package init time so handlers don't depend on the process CWD (L14).
|
|
// Overridden at startup via SetBackupDir with cfg.Backup.Dir.
|
|
var backupBaseDir string
|
|
|
|
func init() {
|
|
backupBaseDir = absOrRaw(filepath.Join("data", "backups"))
|
|
}
|
|
|
|
// SetBackupDir points every backup handler and the scheduled-backup
|
|
// maintenance at the operator-configured directory. Call once at startup with
|
|
// cfg.Backup.Dir (main.go, next to SetDatabasePath); tests use it to isolate
|
|
// a temp dir. Mirrors SetDatabasePath: without it, a configured backup.dir
|
|
// would be ignored while backups keep landing in the default location.
|
|
func SetBackupDir(dir string) {
|
|
if dir == "" {
|
|
return
|
|
}
|
|
backupBaseDir = absOrRaw(dir)
|
|
}
|
|
|
|
func absOrRaw(p string) string {
|
|
if abs, err := filepath.Abs(p); err == nil {
|
|
return abs
|
|
}
|
|
return p
|
|
}
|
|
|
|
// dbFilePath is the live SQLite database file that "Restore backup"
|
|
// overwrites. It defaults to the historical "data/chatserver.db" but must be
|
|
// pointed at cfg.Database.Path via SetDatabasePath before the server starts
|
|
// serving requests (main.go, right after db.Open): without that call, a
|
|
// server configured with a non-default database.path would open its real
|
|
// database at cfg.Database.Path while restore keeps copying backups over an
|
|
// unrelated (possibly newly created) file at the default path, reporting
|
|
// success while the live database is never touched.
|
|
var dbFilePath = filepath.Join("data", "chatserver.db")
|
|
|
|
// SetDatabasePath points the restore handler at the SQLite file the server
|
|
// actually opened. Call once at startup with cfg.Database.Path; tests use it
|
|
// to point restore at an isolated temp file.
|
|
func SetDatabasePath(path string) {
|
|
dbFilePath = path
|
|
}
|
|
|
|
// ─── Backup Handlers ─────────────────────────────────────────────────────────
|
|
|
|
func handleBackup(database *db.DB) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
backupDir := backupBaseDir
|
|
if err := os.MkdirAll(backupDir, 0o750); err != nil {
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to create backup directory")
|
|
return
|
|
}
|
|
|
|
timestamp := time.Now().UTC().Format("20060102_150405")
|
|
backupPath := filepath.Join(backupDir, "chatserver_"+timestamp+".db")
|
|
|
|
// Detached like the restore path's safety backup: an interrupted
|
|
// VACUUM INTO leaves a truncated .db that handleListBackups would
|
|
// present as restorable. BackupToSafe is rooted at the configured
|
|
// backup dir (SetBackupDir), not the historical hardcoded default.
|
|
if err := database.BackupToSafe(context.WithoutCancel(r.Context()), backupPath, backupDir); err != nil {
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "backup failed")
|
|
return
|
|
}
|
|
|
|
// Verify before reporting success: a backup that fails integrity_check
|
|
// is worse than no backup, because the operator believes they have one.
|
|
if err := db.CheckBackupIntegrity(context.WithoutCancel(r.Context()), backupPath); err != nil {
|
|
slog.Error("backup failed integrity check — removing", "path", backupPath, "err", err)
|
|
_ = os.Remove(backupPath)
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "backup failed verification")
|
|
return
|
|
}
|
|
|
|
actor := actorFromContext(r)
|
|
backupName := filepath.Base(backupPath)
|
|
slog.Info("database backup created", "actor_id", actor, "name", backupName)
|
|
db.WriteAudit(context.WithoutCancel(r.Context()), database, actor, "backup_create", "server", 0,
|
|
fmt.Sprintf("backup saved: %s", backupName))
|
|
|
|
writeJSON(w, http.StatusOK, map[string]string{
|
|
"path": filepath.Base(backupPath),
|
|
"created": timestamp,
|
|
})
|
|
})
|
|
}
|
|
|
|
// backupEntry is the JSON shape returned by GET /admin/api/backups.
|
|
type backupEntry struct {
|
|
Name string `json:"name"`
|
|
Size int64 `json:"size"`
|
|
Date string `json:"date"`
|
|
}
|
|
|
|
func handleListBackups() http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
backupDir := backupBaseDir
|
|
entries, err := os.ReadDir(backupDir)
|
|
if err != nil {
|
|
if os.IsNotExist(err) {
|
|
writeJSON(w, http.StatusOK, []backupEntry{})
|
|
return
|
|
}
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to list backups")
|
|
return
|
|
}
|
|
|
|
var backups []backupEntry
|
|
for _, e := range entries {
|
|
if e.IsDir() || filepath.Ext(e.Name()) != ".db" {
|
|
continue
|
|
}
|
|
info, err := e.Info()
|
|
if err != nil {
|
|
continue
|
|
}
|
|
backups = append(backups, backupEntry{
|
|
Name: e.Name(),
|
|
Size: info.Size(),
|
|
Date: info.ModTime().UTC().Format(time.RFC3339),
|
|
})
|
|
}
|
|
if backups == nil {
|
|
backups = []backupEntry{}
|
|
}
|
|
|
|
// Sort newest first.
|
|
sort.Slice(backups, func(i, j int) bool {
|
|
return backups[i].Date > backups[j].Date
|
|
})
|
|
|
|
writeJSON(w, http.StatusOK, backups)
|
|
}
|
|
}
|
|
|
|
func handleDeleteBackup(database *db.DB) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
name := chi.URLParam(r, "name")
|
|
if name == "" || strings.Contains(name, "..") || strings.ContainsAny(name, `/\`) {
|
|
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
|
|
return
|
|
}
|
|
|
|
target := filepath.Join(backupBaseDir, name)
|
|
if !strings.HasPrefix(target, backupBaseDir+string(filepath.Separator)) {
|
|
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
|
|
return
|
|
}
|
|
|
|
if _, err := os.Stat(target); os.IsNotExist(err) { //nolint:gosec // G703: path sanitized by HasPrefix check above
|
|
writeErr(w, http.StatusNotFound, "NOT_FOUND", "backup not found")
|
|
return
|
|
}
|
|
|
|
if err := os.Remove(target); err != nil { //nolint:gosec // G703: path sanitized by HasPrefix check above
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to delete backup")
|
|
return
|
|
}
|
|
|
|
actor := actorFromContext(r)
|
|
slog.Info("backup deleted", "actor_id", actor, "name", name)
|
|
db.WriteAudit(context.WithoutCancel(r.Context()), database, actor, "backup_delete", "server", 0, "deleted backup "+name)
|
|
|
|
w.WriteHeader(http.StatusNoContent)
|
|
})
|
|
}
|
|
|
|
func handleRestoreBackup(database *db.DB, hub HubBroadcaster) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
name := chi.URLParam(r, "name")
|
|
if name == "" || strings.Contains(name, "..") || strings.ContainsAny(name, `/\`) {
|
|
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
|
|
return
|
|
}
|
|
|
|
target := filepath.Join(backupBaseDir, name)
|
|
if !strings.HasPrefix(target, backupBaseDir+string(filepath.Separator)) {
|
|
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "invalid backup name")
|
|
return
|
|
}
|
|
|
|
// Serialize against a concurrent update apply (which stages a new
|
|
// binary and then restarts) and an already-requested restart — a
|
|
// restore must not close the database out from under either. The
|
|
// deferred release is a busy→idle CAS, so it covers every failure
|
|
// return below and harmlessly no-ops once a branch has promoted the
|
|
// state to restart-pending via commitRestartPending.
|
|
if !beginRestartSensitiveOp() {
|
|
writeRestartConflict(w)
|
|
return
|
|
}
|
|
defer abortRestartSensitiveOp()
|
|
|
|
if _, err := os.Stat(target); os.IsNotExist(err) { //nolint:gosec // G703: path sanitized by HasPrefix check above
|
|
writeErr(w, http.StatusNotFound, "NOT_FOUND", "backup not found")
|
|
return
|
|
}
|
|
|
|
// Refuse to overwrite the live database with a file SQLite itself
|
|
// rejects — a truncated pre-crash backup, a stray non-database .db.
|
|
// The pre-restore safety copy would make this survivable, but "restore
|
|
// succeeded" followed by a broken server is still the worst UX here.
|
|
if err := db.CheckBackupIntegrity(context.WithoutCancel(r.Context()), target); err != nil {
|
|
slog.Error("restore refused: backup failed integrity check", "backup", name, "err", err)
|
|
writeErr(w, http.StatusBadRequest, "BAD_REQUEST", "backup file failed integrity verification")
|
|
return
|
|
}
|
|
|
|
dbPath := dbFilePath
|
|
|
|
actor := actorFromContext(r)
|
|
// Audit the restore BEFORE the pre-restore safety copy is taken, and
|
|
// synchronously (LogAudit, not WriteAudit): the restore overwrites the
|
|
// live database file, so the only durable home for this row is the
|
|
// pre_restore_* backup captured below — an entry enqueued on the async
|
|
// WriteAudit path could still be sitting in the writer's buffer when
|
|
// BackupTo snapshots the DB. Best-effort per policy D8: a failed write
|
|
// is logged, never a reason to refuse the restore.
|
|
if err := database.LogAudit(context.WithoutCancel(r.Context()), actor, "backup_restore", "server", 0,
|
|
fmt.Sprintf("restoring backup %s", name)); err != nil {
|
|
slog.Error("audit log write failed", "action", "backup_restore", "actor_id", actor, "error", err)
|
|
}
|
|
|
|
// Safety: create a pre-restore backup before overwriting. WithoutCancel:
|
|
// the restore proceeds regardless of client disconnect (Close/copyFile
|
|
// below are not ctx-aware), so the safety backup must not be skippable
|
|
// by a canceled request ctx.
|
|
// backupBaseDir, not a cwd-relative path: the safety copy has to land in
|
|
// the same directory the rest of the backup handlers read and write, or
|
|
// a server started from another working directory writes it somewhere
|
|
// the operator will never find it.
|
|
preRestore := filepath.Join(backupBaseDir, "pre_restore_"+time.Now().UTC().Format("20060102_150405")+".db")
|
|
if err := database.BackupToSafe(context.WithoutCancel(r.Context()), preRestore, backupBaseDir); err != nil {
|
|
// Fail closed. The admin panel promises "a pre-restore backup will
|
|
// be created" before an irreversible overwrite; proceeding without
|
|
// one takes away the safety net the operator was shown, exactly
|
|
// when they need it (restoring the wrong or a corrupt backup).
|
|
slog.Error("pre-restore backup failed — aborting restore", "err", err)
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR",
|
|
"could not create the pre-restore safety backup — restore aborted, database untouched")
|
|
return
|
|
}
|
|
|
|
// Notify clients that the server is restarting.
|
|
hub.BroadcastServerRestart("backup_restore", 5)
|
|
|
|
// Checkpoint the WAL and close the database connection before overwriting
|
|
// to prevent corruption from concurrent writes (BUG-096).
|
|
if _, checkpointErr := database.SQLDb().ExecContext(context.WithoutCancel(r.Context()), "PRAGMA wal_checkpoint(TRUNCATE)"); checkpointErr != nil {
|
|
slog.Warn("pre-restore WAL checkpoint failed", "err", checkpointErr)
|
|
}
|
|
|
|
slog.Warn("database restored from backup — closing DB", "actor_id", actor, "backup", name)
|
|
|
|
if err := closeDatabase(database); err != nil {
|
|
// database.Close() closes the writer and reader pools regardless of
|
|
// the error it returns (Server/db/db.go), so this process cannot
|
|
// serve anything more either way — every other failure path below
|
|
// (copyFile failing, and the success path itself) restarts for
|
|
// exactly that reason. The live database file is still intact here
|
|
// (copyFile hasn't run yet), so the restarted process comes back on
|
|
// the pre-restore data rather than leaving clients pinned on
|
|
// "Reconnecting..." against a process that never actually restarts.
|
|
slog.Error("failed to close database before restore — restarting anyway, DB pools are closed either way", "err", err)
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", "failed to close database")
|
|
commitRestartPending()
|
|
go requestRestart("backup_restore_close_failed")
|
|
return
|
|
}
|
|
|
|
// Stream the backup file over the (now closed) database to avoid loading
|
|
// the entire DB into memory (could be hundreds of MiB).
|
|
if err := copyBackupFile(target, dbPath); err != nil {
|
|
// copyFile truncates the destination with os.Create before it can know
|
|
// whether the read will succeed, so the live database file is already
|
|
// destroyed by the time we get here — and the DB is closed, so nothing
|
|
// is holding the old contents. Put the safety copy back rather than
|
|
// leaving the operator with a zero-byte database.
|
|
slog.Error("restore copy failed — rolling back to the pre-restore safety copy", "backup", name, "err", err)
|
|
msg := "failed to restore database file — the pre-restore safety copy was put back, server restarting"
|
|
if rbErr := copyBackupFile(preRestore, dbPath); rbErr != nil {
|
|
slog.Error("rollback from the pre-restore safety copy failed — recover manually",
|
|
"safety_copy", preRestore, "err", rbErr)
|
|
msg = "failed to restore database file AND failed to roll back — recover manually from " + filepath.Base(preRestore)
|
|
}
|
|
writeErr(w, http.StatusInternalServerError, "INTERNAL_ERROR", msg)
|
|
// The database was closed before the copy: this process cannot serve
|
|
// anything more either way, so it must restart exactly as it does on
|
|
// the success path.
|
|
commitRestartPending()
|
|
go requestRestart("backup_restore_failed")
|
|
return
|
|
}
|
|
|
|
slog.Warn("database file replaced — restarting to load the restored data", "backup", name)
|
|
|
|
writeJSON(w, http.StatusOK, map[string]string{
|
|
"message": "database restored — server restarting",
|
|
"backup": name,
|
|
})
|
|
|
|
// The database is closed and the file underneath it has been swapped:
|
|
// this process can serve nothing more. It used to stop here, leaving a
|
|
// live server answering every request against a closed DB while the
|
|
// response and the restart broadcast both claimed a restart was
|
|
// happening. Restart for real, the same way applying an update does.
|
|
commitRestartPending()
|
|
go requestRestart("backup_restore")
|
|
})
|
|
}
|
|
|
|
// dbCloser is swappable in tests to simulate database.Close() returning an
|
|
// error. modernc.org/sqlite's sqlite3_close_v2 essentially never fails on a
|
|
// normally-open connection, so there is no portable way to provoke a genuine
|
|
// Close() error from a real driver in a unit test; this seam lets tests
|
|
// exercise that branch directly. Guarded like restartSelf, for the same
|
|
// reason (swap happens on the test goroutine, read on the handler's).
|
|
var (
|
|
closeMu sync.Mutex
|
|
dbCloser = func(database *db.DB) error { return database.Close() }
|
|
)
|
|
|
|
// closeDatabase invokes the current close hook.
|
|
func closeDatabase(database *db.DB) error {
|
|
closeMu.Lock()
|
|
fn := dbCloser
|
|
closeMu.Unlock()
|
|
return fn(database)
|
|
}
|
|
|
|
// copyBackupFile is the restore path's file-copy hook. It exists as a var so
|
|
// tests can inject the hard-to-simulate mid-copy failure (truncate-then-fail)
|
|
// the rollback branch exists for; production never swaps it.
|
|
var copyBackupFile = copyFile
|
|
|
|
// copyFile streams src to dst without loading the entire file into memory.
|
|
func copyFile(src, dst string) error {
|
|
in, err := os.Open(src) //nolint:gosec // G703: src is from sanitized backup path
|
|
if err != nil {
|
|
return fmt.Errorf("open source: %w", err)
|
|
}
|
|
defer in.Close() //nolint:errcheck
|
|
|
|
out, err := os.Create(dst)
|
|
if err != nil {
|
|
return fmt.Errorf("create destination: %w", err)
|
|
}
|
|
|
|
if _, err := io.Copy(out, in); err != nil {
|
|
_ = out.Close()
|
|
return fmt.Errorf("copy: %w", err)
|
|
}
|
|
if err := out.Sync(); err != nil {
|
|
_ = out.Close()
|
|
return fmt.Errorf("sync: %w", err)
|
|
}
|
|
return out.Close()
|
|
}
|