Files
OwnCord/Server/admin/handlers_backup_test.go
T
J3vbandClaude Fable 5 f5faf82a60 infra: observability, backups, guardrails, and deployment hardening (#1376)
* docs: add infrastructure roadmap plan

Records the verified recommendations from an infrastructure review in three
tracks: raising the single-instance ceiling, cheap seams for a possible
multi-instance future, and ops hygiene. Includes explicit anti-recommendations
and sequencing. Security-sensitive detail is intentionally excluded per
docs/security.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* feat(server): real health checks and saturation metrics

/api/v1/metrics now exposes signals that were already computed in memory but
never surfaced: reconnect replay tier hits, event-persister counters, SQLite
writer-pool wait stats, aggregate per-client backpressure counters (including
previously invisible low-priority drops), and permission-cache hit/miss.

/health now returns a real verdict: hub dispatch-loop liveness, a bounded
database ping, and a free-disk check, returning 503 with a subsystem reason
when degraded. Checks are cached so the unauthenticated endpoint cannot
amplify load. The hub's panic breaker now exits the process so a supervisor
can restart it, instead of leaving broadcast delivery silently dead while
clients still appear online.

OTel instruments that were declared but never recorded are now wired
(ws_active_connections, ws_broadcast_latency_seconds, ws_messages_total,
ws_events_dropped_total, voice gauges) or removed (db_query_duration_seconds).
Also corrects the docs/api.md description of broadcast_drops, which counts
hub-queue overflow, not client send-queue overflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* feat(server): implement scheduled backups, retention, and backup verification

The backup_schedule and backup_retention settings have existed in the admin
panel and API since the initial schema but were never read by any code. The
15-minute maintenance loop now enforces them: a scheduled backup is taken
when the newest backup on disk is older than the schedule interval (manual
backups reset the clock), and retention prunes backups older than the
configured days while always keeping the newest one.

Backups are now verified with PRAGMA integrity_check immediately after
VACUUM INTO (a failed backup is removed rather than listed as restorable)
and again before a restore may overwrite the live database. A failed VACUUM
INTO also cleans up its partial output file — but never a pre-existing one.

The backup directory is configurable via a new backup.dir key (default
data/backups) so operators can point backups at another disk or an off-host
mount, mirroring the SetDatabasePath plumb.

Restore-handler tests now use real SQLite fixtures (the integrity gate
correctly refuses text files) with the mid-copy failure injected through a
test-only copy hook. Also adds audited gosec suppressions to the Windows
disk-free syscall added in the previous commit, which the Windows lint leg
flagged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* feat(server): capacity and failure-mode guardrails

- server.max_ws_connections: optional cap on concurrent WebSocket clients,
  checked before the upgrade with a 503 + Retry-After; rejections are counted
  and exposed as ws_conn_rejects in /api/v1/metrics.
- Single-process database lock: an OS-level advisory lock (flock / exclusive
  handle) beside the SQLite file makes a second server process fail fast with
  a clear message instead of silently fighting the first over process-local
  state. A bounded retry covers the self-update/restore restart handoff, and
  the lock mechanism failing (e.g. network filesystems) only warns.
- Disk-space awareness: boot-time warnings for the data and backup volumes,
  plus a disk_free_mb metrics field, via a small cross-platform diskutil
  package (already used by /health).
- Upload storage failures: storage.Save now marks server-side filesystem
  failures with a sentinel (storage.ErrIO); handlers return 507 for those
  instead of blaming the client with a 400, and the emoji route stops echoing
  raw storage errors (which embed absolute paths) into responses.
- Unknown config keys now warn at startup — a typo like admin_alowed_cidrs
  previously kept the default silently while the operator believed the
  setting changed. Never fatal: newer servers tolerate older configs.
- Admin settings honesty: the three stored-but-inert settings (server_icon,
  max_upload_bytes, voice_quality) are shown read-only with a note pointing
  at the real config.yaml keys, instead of pretending to apply.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* perf(db): write-path efficiency and capacity knobs

- channel_focus/mark_read now skip the read-state UPSERT when the stored row
  already matches (same last_message_id, no mentions) — refocus events fire
  at up to 10/s/user and every no-op write still occupied the single SQLite
  writer connection. The extra existence check runs on the reader pool, which
  doesn't serialize. Same shape as the session-touch throttle.
- DeleteExpiredSessions is now sargable: migration 031 normalizes legacy
  expiry formats to the RFC3339-Z layout the server writes and indexes
  expires_at, replacing the strftime full-table scan that ran on the writer
  every 15 minutes.
- Boot-time ANALYZE runs only when a migration actually applied; unchanged
  schemas get the cheap PRAGMA optimize instead (which also covers
  crash-restarts that never reached the shutdown optimize).
- The read/write SQL router gets a table-driven test with explicit expected
  values (INSERT ... RETURNING must hit the writer despite being :one).
- New knobs, all defaulting to current behavior: database.max_readers,
  security.auth_rate_limit_multiplier (for shared-NAT communities),
  event_persistence.replay_ring_size and replay_cold_limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* fix(server): shutdown lifecycle ordering

- The event pruner and maintenance loop are now joined (bounded) before the
  database closes: bgCtx cancellation used to run AFTER database.Close via
  LIFO defers, contradicting its own comment, and neither goroutine was ever
  waited on — a mid-tick scheduled backup or prune could still hold the
  writer while the pool tore down. StartEventPruner returns a done channel
  with the same join contract EventPersister.Stop already had.
- srv.Shutdown now runs before hub.GracefulStop, so in-flight HTTP handlers'
  broadcasts still reach a live hub and the event persister instead of
  vanishing from the replay/event store across a restart. Shutdown does not
  wait on hijacked WebSocket connections, so the swap adds no delay.
- GracefulStopContext threads the 30s shutdown budget into the hub: the 5s
  client-notice window (matching the countdown clients are shown) ends early
  when the budget expires, and is skipped entirely when nobody is connected —
  early-return startup paths and idle servers no longer sleep 5s for an
  audience of zero.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* build(deploy): systemd unit, compose hardening, boot-smoked releases, CI polish

- deploy/owncord.service: hardened systemd unit template with the two
  verified caveats encoded (install dir stays writable for self-update under
  ProtectSystem=strict; CAP_NET_BIND_SERVICE for ACME's :80), plus a
  'Linux (systemd)' deployment docs section — the Linux service story was
  previously 'Docker or nothing'.
- New 'Reverse Proxy Topology' docs section with a working nginx snippet and
  the correct signaling-vs-media distinction: /livekit/* is already proxied
  by the server, only WebRTC media ports must be directly reachable.
- docker-compose: log rotation, commented resource limits, and a healthcheck
  backed by a new 'chatserver healthcheck' subcommand (the distroless image
  has no shell) that probes /health without config side effects.
- release.yml: a concurrency group (queue, never cancel), and boot-smoke
  gates — the freshly built server binaries and the Docker image are cold
  booted and probed healthy BEFORE anything is signed or pushed. The release
  feed drives signed self-updates, so a binary that compiles but dies on
  boot previously would have shipped itself to every auto-updating instance.
- ci.yml: client-check/client-tests move to ubuntu with the reasoning
  recorded (no win32 code paths, LF enforced repo-wide); admin-e2e gets a
  written graduation criterion instead of an open-ended non-blocking status.
- docs: Tailscale guide notes the CGNAT range vs the default admin CIDRs;
  architecture overview records presence/voice state as the fifth
  single-instance blocker and the macOS client scope decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* perf(server): measured load tooling, narrowed invalidation, presence coalescing, storage and CIDR seams

- Fix scripts/k6/ws-load.js against the real wire protocol: envelope-wrapped
  frames, correct message types (typing_start, presence_update), the correct
  /api/v1/ws path, and thresholds that fail a run where nobody authenticated
  or went ready — the script had drifted to pre-envelope framing and reported
  100% green while every auth failed on the first frame. A new
  workflow_dispatch-only load-baseline workflow boots a real server, seeds
  users through the setup/invite APIs, runs the script, and uploads the k6
  summary plus a metrics snapshot for before/after comparison.
- Role-scoped channel-override changes now evict only the affected role's
  members from the permission cache (fail-safe: unreadable member list still
  flushes everything). InvalidateAll here repopulated every connected user —
  two reads each — synchronously inside the admin request via
  RefreshChannelVisibility, a stampede that scaled with total population
  rather than the role's size. Same pattern the per-user override endpoints
  already used.
- Connect/disconnect presence broadcasts now pass through a 300ms latest-wins
  coalescer (QueuePresence): each un-coalesced presence change is a sequenced
  global broadcast (an O(clients) fan-out under seqMu), so a reconnect storm
  fired O(users) of them from the connect critical path. A flap inside the
  window collapses to its final state; the wire format, seq ordering, and
  replay behaviour are unchanged, and the delivery path (BroadcastPresence)
  is untouched.
- Storage seam: api handlers now consume a FileStore interface (consumer-side,
  same pattern as service.Store) with Open returning a seekable storage.File —
  writing down the contract (range-request seeks included) an alternative
  backend would have to meet, without building one.
- The metrics surfaces and the LiveKit webhook/health endpoints get their own
  allowlist keys (metrics_allowed_cidrs, livekit_webhook_allowed_cidrs, both
  defaulting to admin_allowed_cidrs), so a central Prometheus scraper or an
  externally-hosted LiveKit no longer requires widening the admin panel's
  perimeter. Startup now also warns when admin_allowed_cidrs is customized
  while trusted_proxies is empty — behind a proxy or container network the
  check would otherwise compare the proxy's private address, not the client's.
- The container healthcheck probe now PINS the server's own certificate from
  disk (VerifyConnection, exact-match) instead of skipping TLS verification,
  addressing the CodeQL finding on the previous commit; WebPKI verification
  is used when no local cert exists (ACME).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* fix(server): address self-review findings on the hardening branch

Seven fixes from a high-effort review of the full branch diff:

- healthcheck CLI now works under tls.mode acme: it overrides ServerName
  with the configured domain for WebPKI verification instead of pinning a
  cert that doesn't exist (or is stale) in that mode. Previously an ACME
  deployment's container healthcheck failed forever.
- /health pings the READER pool (new db.PingRead): the writer ping queued
  behind a scheduled backup's VACUUM INTO and reported the server degraded
  for the whole backup — which an autoheal watchdog would turn into a
  nightly mid-backup restart.
- /health runs its cached checks under context.WithoutCancel so a probe
  that disconnects mid-request cannot poison the shared cache with a false
  degraded verdict for the next 5 seconds.
- The token CLI uses a new db.OpenShared that skips the single-process
  lock: minting a token against a running server is safe under WAL and was
  a documented workflow the lock had broken.
- The per-user TOTP failure cap is no longer scaled by
  security.auth_rate_limit_multiplier — that knob exists for per-IP limits;
  scaling the only cross-IP brute-force defence multiplied an attacker's
  distributed guess budget. Mirrors the unscaled per-user login threshold.
- A direct presence_update now drops the user's queued entry in the
  connect/disconnect coalescer, so a stale connect-time presence can no
  longer flush 300ms later over the user's fresher chosen status.
- The scheduled-backup filename collision loop breaks on any stat error
  and bounds its suffix probing, instead of spinning the maintenance
  goroutine forever on a persistent EACCES.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

* test(admin): real SQLite fixture for the merged Close-failure restore test

TestHandleRestoreBackup_RestartsWhenCloseFails arrived from main (#1375)
with a plain-text backup fixture; this branch's restore handler verifies
backups with integrity_check before touching the live database, so the text
fixture was (correctly) refused with 400 before the Close-failure branch
under test was reached. Use a real backup via BackupToSafe, matching the
other restore tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-15 20:50:47 +02:00

703 lines
26 KiB
Go

package admin_test
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/owncord/server/admin"
"github.com/owncord/server/auth"
"github.com/owncord/server/db"
)
// chdirTemp changes the working directory to a fresh temp directory for the
// duration of t and restores the original on cleanup. Backup handlers use
// relative paths ("data/backups") that are resolved against cwd.
func chdirTemp(t *testing.T) string {
t.Helper()
tmpDir := t.TempDir()
origDir, err := os.Getwd()
if err != nil {
t.Fatalf("os.Getwd: %v", err)
}
if err := os.Chdir(tmpDir); err != nil {
t.Fatalf("os.Chdir(%q): %v", tmpDir, err)
}
// Update the package-level backup dir to match the new CWD (L14).
admin.SetBackupBaseDir(filepath.Join(tmpDir, "data", "backups"))
t.Cleanup(func() {
_ = os.Chdir(origDir)
admin.SetBackupBaseDir(filepath.Join(origDir, "data", "backups"))
})
return tmpDir
}
// ─── POST /backup ─────────────────────────────────────────────────────────────
// TestHandleBackup_Success verifies that the backup endpoint creates a backup
// file and returns 200 with path and created fields.
func TestHandleBackup_Success(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
w := doRequest(t, handler, http.MethodPost, "/backup", token, nil)
if w.Code != http.StatusOK {
t.Fatalf("POST /backup status = %d, want 200; body: %s", w.Code, w.Body.String())
}
var resp map[string]string
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("unmarshal response: %v", err)
}
if resp["path"] == "" {
t.Error("response missing 'path' field")
}
if resp["created"] == "" {
t.Error("response missing 'created' field")
}
// Verify the backup file actually exists on disk.
backupDir := filepath.Join(tmpDir, "data", "backups")
entries, err := os.ReadDir(backupDir)
if err != nil {
t.Fatalf("ReadDir(%q): %v", backupDir, err)
}
if len(entries) == 0 {
t.Error("no backup files found after successful backup")
}
}
// TestHandleBackup_RequiresOwner verifies that admin-role (not owner) receives 403.
func TestHandleBackup_RequiresOwner(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
adminUID, _ := database.CreateUser(context.Background(), "backupadmin", "hash", 2)
token := "backup-admin-token"
_, _ = database.CreateSession(context.Background(), adminUID, auth.HashToken(token), "test", "127.0.0.1")
w := doRequest(t, handler, http.MethodPost, "/backup", token, nil)
if w.Code != http.StatusForbidden {
t.Errorf("admin user on /backup status = %d, want 403", w.Code)
}
}
// ─── GET /backups ─────────────────────────────────────────────────────────────
// TestHandleListBackups_EmptyWhenNoDirExists verifies that the endpoint returns
// an empty JSON array when the backups directory does not exist.
func TestHandleListBackups_EmptyWhenNoDirExists(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
w := doRequest(t, handler, http.MethodGet, "/backups", token, nil)
if w.Code != http.StatusOK {
t.Fatalf("GET /backups status = %d, want 200; body: %s", w.Code, w.Body.String())
}
var backups []any
if err := json.Unmarshal(w.Body.Bytes(), &backups); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(backups) != 0 {
t.Errorf("expected 0 backups when dir missing, got %d", len(backups))
}
}
// TestHandleListBackups_ReturnsCreatedBackup verifies that a backup created via
// POST /backup appears in GET /backups.
func TestHandleListBackups_ReturnsCreatedBackup(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
// Create a backup first.
wBackup := doRequest(t, handler, http.MethodPost, "/backup", token, nil)
if wBackup.Code != http.StatusOK {
t.Fatalf("POST /backup failed: %d %s", wBackup.Code, wBackup.Body.String())
}
// Now list them.
w := doRequest(t, handler, http.MethodGet, "/backups", token, nil)
if w.Code != http.StatusOK {
t.Fatalf("GET /backups status = %d, want 200; body: %s", w.Code, w.Body.String())
}
var backups []map[string]any
if err := json.Unmarshal(w.Body.Bytes(), &backups); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(backups) == 0 {
t.Fatal("expected at least 1 backup in list after POST /backup")
}
b := backups[0]
if b["name"] == "" {
t.Error("backup entry missing 'name'")
}
if b["size"] == nil {
t.Error("backup entry missing 'size'")
}
if b["date"] == "" {
t.Error("backup entry missing 'date'")
}
}
// ─── DELETE /backups/{name} ───────────────────────────────────────────────────
// TestHandleDeleteBackup_Success verifies that an existing backup file is
// deleted and 204 is returned.
func TestHandleDeleteBackup_Success(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
// Create a real backup file to delete.
backupDir := filepath.Join(tmpDir, "data", "backups")
if err := os.MkdirAll(backupDir, 0o750); err != nil {
t.Fatalf("MkdirAll: %v", err)
}
backupName := "chatserver_20240101_120000.db"
backupPath := filepath.Join(backupDir, backupName)
if err := os.WriteFile(backupPath, []byte("fake backup"), 0o644); err != nil {
t.Fatalf("WriteFile: %v", err)
}
w := doRequest(t, handler, http.MethodDelete, "/backups/"+backupName, token, nil)
if w.Code != http.StatusNoContent {
t.Errorf("DELETE /backups/%s status = %d, want 204; body: %s", backupName, w.Code, w.Body.String())
}
// Verify the file is gone.
if _, err := os.Stat(backupPath); !os.IsNotExist(err) {
t.Error("backup file still exists after delete")
}
}
// TestHandleDeleteBackup_NotFound verifies that deleting a nonexistent backup
// returns 404.
func TestHandleDeleteBackup_NotFound(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
w := doRequest(t, handler, http.MethodDelete, "/backups/nonexistent.db", token, nil)
if w.Code != http.StatusNotFound {
t.Errorf("status = %d, want 404", w.Code)
}
}
// TestHandleDeleteBackup_InvalidNameTraversal verifies that path traversal
// names are rejected with 400.
func TestHandleDeleteBackup_InvalidNameTraversal(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
// The chi router URL-decodes the path parameter, so ".." arrives decoded.
// The handler checks for ".." and returns 400.
w := doRequest(t, handler, http.MethodDelete, "/backups/..evil.db", token, nil)
// Either 400 (blocked) or 404 (file not found) is acceptable.
// What must NOT happen is 204 (successful delete).
if w.Code == http.StatusNoContent {
t.Error("path traversal name resulted in 204 — traversal not blocked")
}
}
// TestHandleDeleteBackup_RequiresOwner verifies that admin-role is denied.
func TestHandleDeleteBackup_RequiresOwner(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
adminUID, _ := database.CreateUser(context.Background(), "deladmin", "hash", 2)
token := "del-admin-token"
_, _ = database.CreateSession(context.Background(), adminUID, auth.HashToken(token), "test", "127.0.0.1")
// Create the file so path validation doesn't return 404 before the 403.
backupDir := filepath.Join(tmpDir, "data", "backups")
_ = os.MkdirAll(backupDir, 0o750)
_ = os.WriteFile(filepath.Join(backupDir, "test.db"), []byte("x"), 0o644)
w := doRequest(t, handler, http.MethodDelete, "/backups/test.db", token, nil)
if w.Code != http.StatusForbidden {
t.Errorf("admin user on delete-backup status = %d, want 403", w.Code)
}
}
// ─── POST /backups/{name}/restore ─────────────────────────────────────────────
// TestHandleRestoreBackup_Success verifies that a restore operation returns 200
// with the expected message and backup name.
func TestHandleRestoreBackup_Success(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
// Set up backup and data directories.
backupDir := filepath.Join(tmpDir, "data", "backups")
dataDir := filepath.Join(tmpDir, "data")
if err := os.MkdirAll(backupDir, 0o750); err != nil {
t.Fatalf("MkdirAll backups: %v", err)
}
if err := os.MkdirAll(dataDir, 0o750); err != nil {
t.Fatalf("MkdirAll data: %v", err)
}
// A real SQLite backup to restore from — the handler now verifies backups
// with integrity_check before touching the live database, so a text
// fixture would be (correctly) refused.
backupName := "chatserver_20240101_120000.db"
backupPath := filepath.Join(backupDir, backupName)
if err := database.BackupToSafe(context.Background(), backupPath, backupDir); err != nil {
t.Fatalf("BackupToSafe fixture: %v", err)
}
restarted, restoreHook := admin.StubRestart()
defer restoreHook()
w := doRequest(t, handler, http.MethodPost, "/backups/"+backupName+"/restore", token, nil)
if w.Code != http.StatusOK {
t.Fatalf("POST /backups/%s/restore status = %d, want 200; body: %s", backupName, w.Code, w.Body.String())
}
var resp map[string]string
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if resp["message"] == "" {
t.Error("response missing 'message' field")
}
if resp["backup"] != backupName {
t.Errorf("backup = %q, want %q", resp["backup"], backupName)
}
// The response and the server_restart broadcast both promise a restart.
// Without one the process keeps serving requests against a closed DB.
deadline := time.Now().Add(2 * time.Second)
for !restarted() && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if !restarted() {
t.Error("restore did not request a process restart")
}
// The safety copy the panel promises must exist on disk.
entries, err := os.ReadDir(backupDir)
if err != nil {
t.Fatalf("ReadDir backups: %v", err)
}
preRestore := ""
for _, e := range entries {
if strings.HasPrefix(e.Name(), "pre_restore_") {
preRestore = filepath.Join(backupDir, e.Name())
}
}
if preRestore == "" {
t.Fatal("no pre_restore_*.db safety backup was created")
}
// The backup_restore audit row must be INSIDE the safety copy — the live
// DB file is replaced by the restore, so the pre_restore backup is that
// row's only durable home. Asserting against the reopened backup file (not
// the handler's DB, which is closed by now) proves both the write and its
// ordering before BackupTo.
restoredDB, err := db.Open(preRestore)
if err != nil {
t.Fatalf("db.Open(pre-restore backup): %v", err)
}
defer restoredDB.Close() //nolint:errcheck
audits, err := restoredDB.GetAuditLog(context.Background(), 10, 0)
if err != nil {
t.Fatalf("GetAuditLog on pre-restore backup: %v", err)
}
foundAudit := false
for _, e := range audits {
if e.Action == "backup_restore" {
foundAudit = true
}
}
if !foundAudit {
t.Error("expected a backup_restore audit entry inside the pre-restore safety backup")
}
}
// TestHandleRestoreBackup_RollsBackWhenCopyFails verifies the live database file
// is not left destroyed when the copy fails partway. copyFile truncates the live
// DB with os.Create before it can know whether the read will succeed, so a
// failure there leaves a closed DB and a zero-byte file underneath it; the
// pre-restore safety copy must be put back, and the process must still respawn
// because the DB is closed either way.
//
// The failure is injected by making the "backup" a directory: it passes the
// handler's existence check and opens, but reading it fails after the truncate.
func TestHandleRestoreBackup_RollsBackWhenCopyFails(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
backupDir := filepath.Join(tmpDir, "data", "backups")
if err := os.MkdirAll(backupDir, 0o750); err != nil {
t.Fatalf("MkdirAll backups: %v", err)
}
dbPath := filepath.Join(tmpDir, "data", "chatserver.db")
if err := os.WriteFile(dbPath, []byte("live database contents"), 0o600); err != nil {
t.Fatalf("WriteFile live db: %v", err)
}
// A valid backup (it must pass the pre-copy integrity gate); the mid-copy
// failure is injected through the copy hook below, reproducing the exact
// failure mode the rollback exists for: os.Create truncates the live DB,
// then the copy dies.
backupName := "chatserver_20240102_120000.db"
if err := database.BackupToSafe(context.Background(), filepath.Join(backupDir, backupName), backupDir); err != nil {
t.Fatalf("BackupToSafe fixture: %v", err)
}
failedOnce := false
restoreCopy := admin.StubCopyBackup(func(src, dst string) error {
if !failedOnce {
failedOnce = true
// Truncate the destination the way the real copy's os.Create
// does, then fail — the state the rollback must repair.
f, createErr := os.Create(dst)
if createErr == nil {
_ = f.Close()
}
return fmt.Errorf("injected copy failure")
}
return admin.CopyBackupForTest(src, dst)
})
defer restoreCopy()
restarted, restoreHook := admin.StubRestart()
defer restoreHook()
w := doRequest(t, handler, http.MethodPost, "/backups/"+backupName+"/restore", token, nil)
if w.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want 500; body: %s", w.Code, w.Body.String())
}
entries, err := os.ReadDir(backupDir)
if err != nil {
t.Fatalf("ReadDir backups: %v", err)
}
preRestore := ""
for _, e := range entries {
if strings.HasPrefix(e.Name(), "pre_restore_") {
preRestore = filepath.Join(backupDir, e.Name())
}
}
if preRestore == "" {
t.Fatal("no pre_restore_*.db safety backup was created")
}
want, err := os.Stat(preRestore)
if err != nil {
t.Fatalf("Stat pre-restore backup: %v", err)
}
got, err := os.Stat(dbPath)
if err != nil {
t.Fatalf("Stat live db after failed restore: %v", err)
}
if got.Size() == 0 {
t.Error("live database file was left truncated after the failed restore")
}
if got.Size() != want.Size() {
t.Errorf("live db size = %d, want %d (the safety copy should have been put back)", got.Size(), want.Size())
}
deadline := time.Now().Add(2 * time.Second)
for !restarted() && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if !restarted() {
t.Error("failed restore did not request a process restart, but the database is closed")
}
}
// TestHandleRestoreBackup_RestartsWhenCloseFails verifies OC-0209: a failed
// database.Close() must still schedule a process restart. database.Close()
// closes the writer and reader pools regardless of the error it returns
// (Server/db/db.go), and the server_restart broadcast already went out to
// every client before Close() is even called — so a process that answers 500
// here without respawning leaves clients pinned on "Reconnecting..." forever
// while the process quietly keeps failing every request with a closed DB.
func TestHandleRestoreBackup_RestartsWhenCloseFails(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
backupDir := filepath.Join(tmpDir, "data", "backups")
if err := os.MkdirAll(backupDir, 0o750); err != nil {
t.Fatalf("MkdirAll backups: %v", err)
}
dbPath := filepath.Join(tmpDir, "data", "chatserver.db")
if err := os.WriteFile(dbPath, []byte("original live contents"), 0o600); err != nil {
t.Fatalf("WriteFile live db: %v", err)
}
// A real SQLite backup — the restore handler verifies backups with
// integrity_check before touching the live database, so a text fixture
// would be (correctly) refused with 400 before the Close-failure branch
// under test is ever reached.
backupName := "chatserver_20240103_120000.db"
if err := database.BackupToSafe(context.Background(), filepath.Join(backupDir, backupName), backupDir); err != nil {
t.Fatalf("BackupToSafe fixture: %v", err)
}
restarted, restoreRestartHook := admin.StubRestart()
defer restoreRestartHook()
restoreCloseHook := admin.StubCloseError("simulated close failure")
defer restoreCloseHook()
w := doRequest(t, handler, http.MethodPost, "/backups/"+backupName+"/restore", token, nil)
if w.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want 500; body: %s", w.Code, w.Body.String())
}
deadline := time.Now().Add(2 * time.Second)
for !restarted() && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if !restarted() {
t.Error("a failed database.Close() did not request a process restart, " +
"leaving a live server answering requests against closed DB pools")
}
}
// TestHandleRestoreBackup_AbortsWithoutSafetyBackup verifies the restore fails
// closed when the pre-restore backup can't be written: the panel promises that
// safety copy, and overwriting the live database without one is unrecoverable.
func TestHandleRestoreBackup_AbortsWithoutSafetyBackup(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
backupDir := filepath.Join(tmpDir, "data", "backups")
if err := os.MkdirAll(backupDir, 0o750); err != nil {
t.Fatalf("MkdirAll backups: %v", err)
}
backupName := "chatserver_20240101_120000.db"
dbFile := filepath.Join(tmpDir, "data", "chatserver.db")
if err := database.BackupToSafe(context.Background(), filepath.Join(backupDir, backupName), backupDir); err != nil {
t.Fatalf("BackupToSafe fixture: %v", err)
}
if err := os.WriteFile(dbFile, []byte("original"), 0o644); err != nil {
t.Fatalf("WriteFile db: %v", err)
}
restarted, restoreHook := admin.StubRestart()
defer restoreHook()
// Make the safety copy impossible: VACUUM INTO refuses a destination that
// already exists. The name is pre_restore_<UTC seconds>.db, so occupy the
// next two minutes' worth of candidates — a 4-second window flaked on slow
// Windows CI runners where the request itself outlived it.
admin.SetBackupBaseDir(backupDir)
for i := range 120 {
name := "pre_restore_" + time.Now().UTC().Add(time.Duration(i)*time.Second).Format("20060102_150405") + ".db"
if err := os.WriteFile(filepath.Join(backupDir, name), []byte("occupied"), 0o644); err != nil {
t.Fatalf("WriteFile blocker: %v", err)
}
}
w := doRequest(t, handler, http.MethodPost, "/backups/"+backupName+"/restore", token, nil)
if w.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want 500 (restore must abort); body: %s", w.Code, w.Body.String())
}
if restarted() {
t.Error("aborted restore must not restart the process")
}
data, err := os.ReadFile(dbFile)
if err != nil {
t.Fatalf("ReadFile db: %v", err)
}
if string(data) != "original" {
t.Errorf("database was overwritten despite the abort: %q", string(data))
}
}
// TestHandleRestoreBackup_UsesConfiguredDatabasePath verifies that the
// restore handler writes to the SQLite file the server was actually
// configured to use (SetDatabasePath), not a hardcoded "data/chatserver.db".
// A server with database.path set to anything else must not have its real
// database silently left untouched by a "successful" restore (OC-0097).
func TestHandleRestoreBackup_UsesConfiguredDatabasePath(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
backupDir := filepath.Join(tmpDir, "data", "backups")
if err := os.MkdirAll(backupDir, 0o750); err != nil {
t.Fatalf("MkdirAll backups: %v", err)
}
// Configure a non-default database path, as an operator would via
// database.path in config.yaml.
customDBPath := filepath.Join(tmpDir, "custom", "oc.db")
if err := os.MkdirAll(filepath.Dir(customDBPath), 0o750); err != nil {
t.Fatalf("MkdirAll custom db dir: %v", err)
}
if err := os.WriteFile(customDBPath, []byte("original live contents"), 0o644); err != nil {
t.Fatalf("WriteFile custom db: %v", err)
}
admin.SetDatabasePath(customDBPath)
t.Cleanup(func() { admin.SetDatabasePath(filepath.Join("data", "chatserver.db")) })
backupName := "chatserver_20240101_120000.db"
backupPath := filepath.Join(backupDir, backupName)
if err := database.BackupToSafe(context.Background(), backupPath, backupDir); err != nil {
t.Fatalf("BackupToSafe fixture: %v", err)
}
backupContent, err := os.ReadFile(backupPath)
if err != nil {
t.Fatalf("ReadFile fixture: %v", err)
}
restarted, restoreHook := admin.StubRestart()
defer restoreHook()
w := doRequest(t, handler, http.MethodPost, "/backups/"+backupName+"/restore", token, nil)
if w.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body: %s", w.Code, w.Body.String())
}
deadline := time.Now().Add(2 * time.Second)
for !restarted() && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if !restarted() {
t.Error("restore did not request a process restart")
}
got, err := os.ReadFile(customDBPath)
if err != nil {
t.Fatalf("ReadFile(%q): %v", customDBPath, err)
}
if !bytes.Equal(got, backupContent) {
t.Errorf("configured database file content = %q, want %q — restore wrote to the wrong path", got, backupContent)
}
// The hardcoded default path must NOT have been created/touched.
defaultPath := filepath.Join(tmpDir, "data", "chatserver.db")
if _, err := os.Stat(defaultPath); err == nil {
t.Error("restore wrote to the hardcoded default database path instead of the configured one")
}
}
// TestHandleRestoreBackup_NotFound verifies that restoring a missing backup
// returns 404.
func TestHandleRestoreBackup_NotFound(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
w := doRequest(t, handler, http.MethodPost, "/backups/missing.db/restore", token, nil)
if w.Code != http.StatusNotFound {
t.Errorf("status = %d, want 404", w.Code)
}
}
// TestHandleRestoreBackup_InvalidName verifies that a name containing ".." is
// rejected with 400.
func TestHandleRestoreBackup_InvalidName(t *testing.T) {
_ = chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
w := doRequest(t, handler, http.MethodPost, "/backups/..evil.db/restore", token, nil)
// Must not return 200 OK.
if w.Code == http.StatusOK {
t.Error("path-traversal restore name returned 200 — traversal not blocked")
}
}
// TestHandleListBackups_ErrorReadingDir verifies that if the backups path
// exists but is a file (not a directory), the endpoint returns 500.
func TestHandleListBackups_ErrorReadingDir(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
token := createAdminUser(t, database)
// Create data/ directory but make "backups" a file instead of a directory.
dataDir := filepath.Join(tmpDir, "data")
if err := os.MkdirAll(dataDir, 0o750); err != nil {
t.Fatalf("MkdirAll data: %v", err)
}
backupsFile := filepath.Join(dataDir, "backups")
if err := os.WriteFile(backupsFile, []byte("not a directory"), 0o644); err != nil {
t.Fatalf("WriteFile: %v", err)
}
w := doRequest(t, handler, http.MethodGet, "/backups", token, nil)
// os.ReadDir on a file (not a directory) fails with a non-IsNotExist error
// on most platforms, but the exact behavior is platform-dependent.
// On Windows, ReadDir on a file returns an error that is NOT os.IsNotExist.
// So we expect either 500 or (in edge cases) 200 with empty list.
if w.Code != http.StatusInternalServerError && w.Code != http.StatusOK {
t.Errorf("status = %d, want 500 or 200 (platform dependent)", w.Code)
}
}
// TestHandleRestoreBackup_RequiresOwner verifies that admin-role is denied.
func TestHandleRestoreBackup_RequiresOwner(t *testing.T) {
tmpDir := chdirTemp(t)
database := openAdminTestDB(t)
handler := admin.NewAdminAPI(database, "1.0.0", &mockHub{}, nil, nil, nil, nil, newTestModService(database), newTestRoleService(database))
adminUID, _ := database.CreateUser(context.Background(), "restoreadmin", "hash", 2)
token := "restore-admin-token"
_, _ = database.CreateSession(context.Background(), adminUID, auth.HashToken(token), "test", "127.0.0.1")
// Create files so path checks pass before auth check.
backupDir := filepath.Join(tmpDir, "data", "backups")
dataDir := filepath.Join(tmpDir, "data")
_ = os.MkdirAll(backupDir, 0o750)
_ = os.MkdirAll(dataDir, 0o750)
_ = os.WriteFile(filepath.Join(backupDir, "test.db"), []byte("x"), 0o644)
w := doRequest(t, handler, http.MethodPost, "/backups/test.db/restore", token, nil)
if w.Code != http.StatusForbidden {
t.Errorf("admin user on restore status = %d, want 403", w.Code)
}
}