mirror of
https://github.com/J3vb/OwnCord.git
synced 2026-09-03 03:50:00 +03:00
* docs: add infrastructure roadmap plan Records the verified recommendations from an infrastructure review in three tracks: raising the single-instance ceiling, cheap seams for a possible multi-instance future, and ops hygiene. Includes explicit anti-recommendations and sequencing. Security-sensitive detail is intentionally excluded per docs/security.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * feat(server): real health checks and saturation metrics /api/v1/metrics now exposes signals that were already computed in memory but never surfaced: reconnect replay tier hits, event-persister counters, SQLite writer-pool wait stats, aggregate per-client backpressure counters (including previously invisible low-priority drops), and permission-cache hit/miss. /health now returns a real verdict: hub dispatch-loop liveness, a bounded database ping, and a free-disk check, returning 503 with a subsystem reason when degraded. Checks are cached so the unauthenticated endpoint cannot amplify load. The hub's panic breaker now exits the process so a supervisor can restart it, instead of leaving broadcast delivery silently dead while clients still appear online. OTel instruments that were declared but never recorded are now wired (ws_active_connections, ws_broadcast_latency_seconds, ws_messages_total, ws_events_dropped_total, voice gauges) or removed (db_query_duration_seconds). Also corrects the docs/api.md description of broadcast_drops, which counts hub-queue overflow, not client send-queue overflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * feat(server): implement scheduled backups, retention, and backup verification The backup_schedule and backup_retention settings have existed in the admin panel and API since the initial schema but were never read by any code. The 15-minute maintenance loop now enforces them: a scheduled backup is taken when the newest backup on disk is older than the schedule interval (manual backups reset the clock), and retention prunes backups older than the configured days while always keeping the newest one. Backups are now verified with PRAGMA integrity_check immediately after VACUUM INTO (a failed backup is removed rather than listed as restorable) and again before a restore may overwrite the live database. A failed VACUUM INTO also cleans up its partial output file — but never a pre-existing one. The backup directory is configurable via a new backup.dir key (default data/backups) so operators can point backups at another disk or an off-host mount, mirroring the SetDatabasePath plumb. Restore-handler tests now use real SQLite fixtures (the integrity gate correctly refuses text files) with the mid-copy failure injected through a test-only copy hook. Also adds audited gosec suppressions to the Windows disk-free syscall added in the previous commit, which the Windows lint leg flagged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * feat(server): capacity and failure-mode guardrails - server.max_ws_connections: optional cap on concurrent WebSocket clients, checked before the upgrade with a 503 + Retry-After; rejections are counted and exposed as ws_conn_rejects in /api/v1/metrics. - Single-process database lock: an OS-level advisory lock (flock / exclusive handle) beside the SQLite file makes a second server process fail fast with a clear message instead of silently fighting the first over process-local state. A bounded retry covers the self-update/restore restart handoff, and the lock mechanism failing (e.g. network filesystems) only warns. - Disk-space awareness: boot-time warnings for the data and backup volumes, plus a disk_free_mb metrics field, via a small cross-platform diskutil package (already used by /health). - Upload storage failures: storage.Save now marks server-side filesystem failures with a sentinel (storage.ErrIO); handlers return 507 for those instead of blaming the client with a 400, and the emoji route stops echoing raw storage errors (which embed absolute paths) into responses. - Unknown config keys now warn at startup — a typo like admin_alowed_cidrs previously kept the default silently while the operator believed the setting changed. Never fatal: newer servers tolerate older configs. - Admin settings honesty: the three stored-but-inert settings (server_icon, max_upload_bytes, voice_quality) are shown read-only with a note pointing at the real config.yaml keys, instead of pretending to apply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * perf(db): write-path efficiency and capacity knobs - channel_focus/mark_read now skip the read-state UPSERT when the stored row already matches (same last_message_id, no mentions) — refocus events fire at up to 10/s/user and every no-op write still occupied the single SQLite writer connection. The extra existence check runs on the reader pool, which doesn't serialize. Same shape as the session-touch throttle. - DeleteExpiredSessions is now sargable: migration 031 normalizes legacy expiry formats to the RFC3339-Z layout the server writes and indexes expires_at, replacing the strftime full-table scan that ran on the writer every 15 minutes. - Boot-time ANALYZE runs only when a migration actually applied; unchanged schemas get the cheap PRAGMA optimize instead (which also covers crash-restarts that never reached the shutdown optimize). - The read/write SQL router gets a table-driven test with explicit expected values (INSERT ... RETURNING must hit the writer despite being :one). - New knobs, all defaulting to current behavior: database.max_readers, security.auth_rate_limit_multiplier (for shared-NAT communities), event_persistence.replay_ring_size and replay_cold_limit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * fix(server): shutdown lifecycle ordering - The event pruner and maintenance loop are now joined (bounded) before the database closes: bgCtx cancellation used to run AFTER database.Close via LIFO defers, contradicting its own comment, and neither goroutine was ever waited on — a mid-tick scheduled backup or prune could still hold the writer while the pool tore down. StartEventPruner returns a done channel with the same join contract EventPersister.Stop already had. - srv.Shutdown now runs before hub.GracefulStop, so in-flight HTTP handlers' broadcasts still reach a live hub and the event persister instead of vanishing from the replay/event store across a restart. Shutdown does not wait on hijacked WebSocket connections, so the swap adds no delay. - GracefulStopContext threads the 30s shutdown budget into the hub: the 5s client-notice window (matching the countdown clients are shown) ends early when the budget expires, and is skipped entirely when nobody is connected — early-return startup paths and idle servers no longer sleep 5s for an audience of zero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * build(deploy): systemd unit, compose hardening, boot-smoked releases, CI polish - deploy/owncord.service: hardened systemd unit template with the two verified caveats encoded (install dir stays writable for self-update under ProtectSystem=strict; CAP_NET_BIND_SERVICE for ACME's :80), plus a 'Linux (systemd)' deployment docs section — the Linux service story was previously 'Docker or nothing'. - New 'Reverse Proxy Topology' docs section with a working nginx snippet and the correct signaling-vs-media distinction: /livekit/* is already proxied by the server, only WebRTC media ports must be directly reachable. - docker-compose: log rotation, commented resource limits, and a healthcheck backed by a new 'chatserver healthcheck' subcommand (the distroless image has no shell) that probes /health without config side effects. - release.yml: a concurrency group (queue, never cancel), and boot-smoke gates — the freshly built server binaries and the Docker image are cold booted and probed healthy BEFORE anything is signed or pushed. The release feed drives signed self-updates, so a binary that compiles but dies on boot previously would have shipped itself to every auto-updating instance. - ci.yml: client-check/client-tests move to ubuntu with the reasoning recorded (no win32 code paths, LF enforced repo-wide); admin-e2e gets a written graduation criterion instead of an open-ended non-blocking status. - docs: Tailscale guide notes the CGNAT range vs the default admin CIDRs; architecture overview records presence/voice state as the fifth single-instance blocker and the macOS client scope decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * perf(server): measured load tooling, narrowed invalidation, presence coalescing, storage and CIDR seams - Fix scripts/k6/ws-load.js against the real wire protocol: envelope-wrapped frames, correct message types (typing_start, presence_update), the correct /api/v1/ws path, and thresholds that fail a run where nobody authenticated or went ready — the script had drifted to pre-envelope framing and reported 100% green while every auth failed on the first frame. A new workflow_dispatch-only load-baseline workflow boots a real server, seeds users through the setup/invite APIs, runs the script, and uploads the k6 summary plus a metrics snapshot for before/after comparison. - Role-scoped channel-override changes now evict only the affected role's members from the permission cache (fail-safe: unreadable member list still flushes everything). InvalidateAll here repopulated every connected user — two reads each — synchronously inside the admin request via RefreshChannelVisibility, a stampede that scaled with total population rather than the role's size. Same pattern the per-user override endpoints already used. - Connect/disconnect presence broadcasts now pass through a 300ms latest-wins coalescer (QueuePresence): each un-coalesced presence change is a sequenced global broadcast (an O(clients) fan-out under seqMu), so a reconnect storm fired O(users) of them from the connect critical path. A flap inside the window collapses to its final state; the wire format, seq ordering, and replay behaviour are unchanged, and the delivery path (BroadcastPresence) is untouched. - Storage seam: api handlers now consume a FileStore interface (consumer-side, same pattern as service.Store) with Open returning a seekable storage.File — writing down the contract (range-request seeks included) an alternative backend would have to meet, without building one. - The metrics surfaces and the LiveKit webhook/health endpoints get their own allowlist keys (metrics_allowed_cidrs, livekit_webhook_allowed_cidrs, both defaulting to admin_allowed_cidrs), so a central Prometheus scraper or an externally-hosted LiveKit no longer requires widening the admin panel's perimeter. Startup now also warns when admin_allowed_cidrs is customized while trusted_proxies is empty — behind a proxy or container network the check would otherwise compare the proxy's private address, not the client's. - The container healthcheck probe now PINS the server's own certificate from disk (VerifyConnection, exact-match) instead of skipping TLS verification, addressing the CodeQL finding on the previous commit; WebPKI verification is used when no local cert exists (ACME). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * fix(server): address self-review findings on the hardening branch Seven fixes from a high-effort review of the full branch diff: - healthcheck CLI now works under tls.mode acme: it overrides ServerName with the configured domain for WebPKI verification instead of pinning a cert that doesn't exist (or is stale) in that mode. Previously an ACME deployment's container healthcheck failed forever. - /health pings the READER pool (new db.PingRead): the writer ping queued behind a scheduled backup's VACUUM INTO and reported the server degraded for the whole backup — which an autoheal watchdog would turn into a nightly mid-backup restart. - /health runs its cached checks under context.WithoutCancel so a probe that disconnects mid-request cannot poison the shared cache with a false degraded verdict for the next 5 seconds. - The token CLI uses a new db.OpenShared that skips the single-process lock: minting a token against a running server is safe under WAL and was a documented workflow the lock had broken. - The per-user TOTP failure cap is no longer scaled by security.auth_rate_limit_multiplier — that knob exists for per-IP limits; scaling the only cross-IP brute-force defence multiplied an attacker's distributed guess budget. Mirrors the unscaled per-user login threshold. - A direct presence_update now drops the user's queued entry in the connect/disconnect coalescer, so a stale connect-time presence can no longer flush 300ms later over the user's fresher chosen status. - The scheduled-backup filename collision loop breaks on any stat error and bounds its suffix probing, instead of spinning the maintenance goroutine forever on a persistent EACCES. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj * test(admin): real SQLite fixture for the merged Close-failure restore test TestHandleRestoreBackup_RestartsWhenCloseFails arrived from main (#1375) with a plain-text backup fixture; this branch's restore handler verifies backups with integrity_check before touching the live database, so the text fixture was (correctly) refused with 400 before the Close-failure branch under test was reached. Use a real backup via BackupToSafe, matching the other restore tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RtDNHSYWwPKArL8MsRdbj --------- Co-authored-by: Claude <noreply@anthropic.com>
746 lines
28 KiB
Go
746 lines
28 KiB
Go
// OwnCord chat server — self-hosted, Windows-native.
|
|
// Build: go build -o chatserver.exe -ldflags "-s -w -X main.version=1.0.0" .
|
|
package main
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"crypto/tls"
|
|
"encoding/pem"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
stdlog "log"
|
|
"log/slog"
|
|
"net"
|
|
"net/http"
|
|
"os"
|
|
"os/signal"
|
|
"path/filepath"
|
|
"runtime"
|
|
"strconv"
|
|
"strings"
|
|
"syscall"
|
|
"time"
|
|
|
|
"gopkg.in/yaml.v3"
|
|
|
|
"github.com/owncord/server/admin"
|
|
"github.com/owncord/server/api"
|
|
"github.com/owncord/server/auth"
|
|
"github.com/owncord/server/config"
|
|
"github.com/owncord/server/db"
|
|
"github.com/owncord/server/diskutil"
|
|
"github.com/owncord/server/logctx"
|
|
"github.com/owncord/server/plugin"
|
|
"github.com/owncord/server/storage"
|
|
"github.com/owncord/server/telemetry"
|
|
"github.com/owncord/server/ws"
|
|
)
|
|
|
|
// version is overridden at build time via -ldflags "-X main.version=1.0.0".
|
|
var version = "dev"
|
|
|
|
func main() {
|
|
// `server healthcheck` probes the running instance's /health and exits
|
|
// 0/1. It exists for container healthchecks: the distroless image has no
|
|
// shell or curl, so the binary is its own probe.
|
|
if len(os.Args) > 1 && os.Args[1] == "healthcheck" {
|
|
os.Exit(runHealthcheckCLI())
|
|
}
|
|
// `server token ...` is a direct-to-DB CLI (mint/list/revoke API tokens) —
|
|
// handled before any server/logging setup so it stays quiet and standalone.
|
|
if len(os.Args) > 1 && os.Args[1] == "token" {
|
|
os.Exit(runTokenCLI(os.Args[2:]))
|
|
}
|
|
|
|
// Create ring buffer for admin log viewer, then build a multi-handler
|
|
// that tees log records to both stdout and the ring buffer.
|
|
logBuf := admin.NewRingBuffer(2000)
|
|
// levelVar controls both handlers' thresholds. It starts at INFO (the
|
|
// zero value) so early-startup logs are captured, then run() raises/lowers
|
|
// it once config.yaml / OWNCORD_LOGGING_LEVEL is loaded. The ring buffer
|
|
// shares it rather than hard-wiring DEBUG: with both sinks gated, Enabled
|
|
// returns false for suppressed levels and every gated Debug call across
|
|
// the server becomes a no-op instead of formatting a ring entry.
|
|
levelVar := new(slog.LevelVar)
|
|
stdoutHandler := slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: levelVar})
|
|
multiHandler := admin.NewMultiHandler(stdoutHandler, logBuf, levelVar)
|
|
// logctx enriches records logged with a request/trace context (the
|
|
// ...Context slog variants) with req_id and, under -tags otel, trace_id.
|
|
log := slog.New(logctx.New(multiHandler))
|
|
slog.SetDefault(log)
|
|
|
|
if err := run(log, logBuf, levelVar); err != nil {
|
|
_, _ = fmt.Fprintf(os.Stderr, "\n [ERROR] %v\n\n", err)
|
|
log.Error("server exited with error", "error", err)
|
|
os.Exit(1)
|
|
}
|
|
}
|
|
|
|
// run is the real entrypoint — separated for testability.
|
|
func run(log *slog.Logger, logBuf *admin.RingBuffer, levelVar *slog.LevelVar) error {
|
|
// bgCtx is a cancellable context shared by all background goroutines
|
|
// (event persister, event pruner, plugin loader, maintenance loop).
|
|
//
|
|
// This first deferred bgCancel is only the LIFO backstop — because it is
|
|
// registered before `defer database.Close()`, it would otherwise run
|
|
// AFTER the database is closed, leaving background goroutines running
|
|
// through teardown. The persistence and maintenance blocks below register
|
|
// their own later (= earlier-running) defers that cancel bgCtx and JOIN
|
|
// their goroutines before the database closes.
|
|
bgCtx, bgCancel := context.WithCancel(context.Background())
|
|
defer bgCancel()
|
|
|
|
// Clean up old binary from a previous update.
|
|
exePath, exeErr := os.Executable()
|
|
if exeErr != nil {
|
|
log.Warn("failed to determine executable path", "error", exeErr)
|
|
} else {
|
|
oldPath := exePath + ".old"
|
|
if _, statErr := os.Stat(oldPath); statErr == nil {
|
|
if rmErr := os.Remove(oldPath); rmErr != nil {
|
|
log.Warn("failed to remove old binary", "path", oldPath, "error", rmErr)
|
|
} else {
|
|
log.Info("removed old binary from previous update", "path", oldPath)
|
|
}
|
|
}
|
|
}
|
|
|
|
// ── 1. Load configuration ──────────────────────────────────────────────
|
|
cfg, err := config.Load(config.DefaultPath)
|
|
if err != nil {
|
|
return fmt.Errorf("loading config: %w", err)
|
|
}
|
|
|
|
// Apply the configured log level. The admin panel's live log view (ring
|
|
// buffer) follows the same threshold — set logging.level to "debug" to
|
|
// capture debug records there.
|
|
if lvl, ok := config.ParseLevel(cfg.Logging.Level); ok {
|
|
levelVar.Set(lvl)
|
|
} else {
|
|
log.Warn("unknown logging.level, keeping info", "value", cfg.Logging.Level)
|
|
}
|
|
|
|
// ── 2. Ensure data directory exists ────────────────────────────────────
|
|
if mkdirErr := os.MkdirAll(cfg.Server.DataDir, 0o750); mkdirErr != nil {
|
|
return fmt.Errorf("creating data dir %s: %w", cfg.Server.DataDir, mkdirErr)
|
|
}
|
|
|
|
// Disk-space awareness: the database (WAL growth included), uploads,
|
|
// certs, and by default backups all live on this volume, and running it
|
|
// dry breaks several of them at once. Probe errors are ignored — unknown
|
|
// is not "full". /health repeats this check continuously at 256 MiB.
|
|
warnLowDisk(log, "data dir", cfg.Server.DataDir)
|
|
if cfg.Backup.Dir != "" && cfg.Backup.Dir != filepath.Join(cfg.Server.DataDir, "backups") {
|
|
warnLowDisk(log, "backup dir", cfg.Backup.Dir)
|
|
}
|
|
|
|
// ── 3. TLS ────────────────────────────────────────────────────────────
|
|
tlsResult, err := auth.LoadOrGenerate(cfg.TLS)
|
|
if err != nil {
|
|
return fmt.Errorf("configuring TLS: %w", err)
|
|
}
|
|
tlsCfg := tlsResult.TLSConfig
|
|
|
|
// Print startup banner first so it appears above all init logs.
|
|
printBanner(cfg, version, tlsCfg != nil)
|
|
|
|
// ── 4. Open database + run migrations ─────────────────────────────────
|
|
// SQLite is the only supported backend; the unfinished Postgres
|
|
// scaffolding (stubbed query layer, never wired into the runtime) was
|
|
// removed rather than completed.
|
|
if t := cfg.Database.Type; t != "" && t != "sqlite" {
|
|
return fmt.Errorf("database.type=%q is not supported; set \"sqlite\" or omit it", t)
|
|
}
|
|
|
|
database, err := db.OpenWithMaxReaders(cfg.Database.Path, cfg.Database.MaxReaders)
|
|
if err != nil {
|
|
return fmt.Errorf("opening database: %w", err)
|
|
}
|
|
defer database.Close() //nolint:errcheck
|
|
|
|
// The admin "Restore backup" handler needs the real database file path:
|
|
// without this, it falls back to a hardcoded "data/chatserver.db" and
|
|
// silently no-ops on any server with a configured database.path.
|
|
admin.SetDatabasePath(cfg.Database.Path)
|
|
// Backup handlers and the scheduled-backup maintenance write to the
|
|
// configured backup directory (defaults to data/backups).
|
|
admin.SetBackupDir(cfg.Backup.Dir)
|
|
|
|
if err := db.Migrate(database); err != nil {
|
|
return fmt.Errorf("running migrations: %w", err)
|
|
}
|
|
|
|
// Clear stale state from a previous run or crash. Startup work — nothing
|
|
// to inherit a context from yet.
|
|
if err := database.ResetAllUserStatuses(context.Background()); err != nil {
|
|
log.Warn("failed to reset stale user statuses", "error", err)
|
|
} else {
|
|
log.Info("reset all user statuses to offline")
|
|
}
|
|
if err := database.ClearAllVoiceStates(context.Background()); err != nil {
|
|
log.Warn("failed to clear stale voice states", "error", err)
|
|
} else {
|
|
log.Info("cleared stale voice states")
|
|
}
|
|
|
|
// ── 4b. Telemetry (Phase B Step 8) ─────────────────────────────────────
|
|
// Init can return (nil, err) when the otel build-tag skeleton hasn't been
|
|
// finished wiring to the upstream SDK. Normalise to a no-op shutdown so
|
|
// the deferred closure never calls a nil function.
|
|
telemetryShutdown, telErr := telemetry.Init(context.Background(), cfg.Telemetry)
|
|
if telErr != nil {
|
|
log.Warn("telemetry init failed; continuing without OpenTelemetry", "error", telErr)
|
|
}
|
|
if telemetryShutdown == nil {
|
|
telemetryShutdown = func(context.Context) error { return nil }
|
|
}
|
|
defer func() {
|
|
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
if err := telemetryShutdown(shutdownCtx); err != nil {
|
|
log.Warn("telemetry shutdown returned error", "error", err)
|
|
}
|
|
}()
|
|
|
|
// ── 5a. Construct plugin runtime BEFORE the router so the router can
|
|
// wire the live registry into the plugin admin handler. ────────────────
|
|
var pluginRegistry *plugin.Registry
|
|
if cfg.Plugins.Enabled {
|
|
registry, plugErr := plugin.NewRegistry(plugin.Config{
|
|
Directory: cfg.Plugins.Directory,
|
|
MaxMemoryMB: cfg.Plugins.MaxMemoryMB,
|
|
CPUBudgetMs: cfg.Plugins.CPUBudgetMs,
|
|
HTTPAllowlist: cfg.Plugins.HTTPAllowlist,
|
|
Store: database,
|
|
})
|
|
if plugErr != nil {
|
|
log.Warn("plugin runtime init failed; continuing without plugins", "error", plugErr)
|
|
} else {
|
|
pluginRegistry = registry
|
|
if err := registry.LoadAll(bgCtx); err != nil {
|
|
log.Warn("plugin loader: failed to scan directory", "error", err)
|
|
}
|
|
defer func() {
|
|
closeCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_ = registry.Close(closeCtx)
|
|
}()
|
|
}
|
|
}
|
|
|
|
// ── 5b. Build HTTP router ──────────────────────────────────────────────
|
|
router, hub, routerCleanup := api.NewRouter(cfg, database, version, logBuf, pluginRegistry)
|
|
defer routerCleanup()
|
|
// Backstop for every early return below (serve error, ACME shutdown
|
|
// failure, etc.): hub.GracefulStop is the only caller of
|
|
// LiveKitProcess.Stop(), so skipping it orphans the companion
|
|
// livekit-server process and leaves the hub's dispatch goroutine
|
|
// running. gracefulOnce makes it idempotent alongside the explicit call
|
|
// on the normal shutdown path below.
|
|
defer hub.GracefulStop()
|
|
|
|
// ── 5c. Wire event persistence (Phase B Step 7) ────────────────────────
|
|
if cfg.EventPersistence.Enabled && hub != nil {
|
|
seedHubReplayState(bgCtx, hub, database, log)
|
|
|
|
persister := ws.NewEventPersister(
|
|
database,
|
|
4096,
|
|
cfg.EventPersistence.BatchSize,
|
|
time.Duration(cfg.EventPersistence.BatchFlushMs)*time.Millisecond,
|
|
)
|
|
persister.Start(bgCtx)
|
|
hub.SetEventPersister(persister)
|
|
hub.SetEventStore(database)
|
|
|
|
retention := time.Duration(cfg.EventPersistence.RetentionHours) * time.Hour
|
|
prunerInterval := time.Duration(cfg.EventPersistence.PrunerIntervalMinutes) * time.Minute
|
|
prunerDone := ws.StartEventPruner(bgCtx, database, retention, prunerInterval)
|
|
defer func() {
|
|
stopCtx, stopCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer stopCancel()
|
|
persister.Stop(stopCtx)
|
|
// Cancel the shared background context and JOIN the pruner before
|
|
// the (LIFO-later) database.Close defer runs, so no prune is still
|
|
// mid-query against a closing pool. Bounded: a stuck prune delays
|
|
// shutdown by at most the timeout, then Close proceeds anyway.
|
|
bgCancel()
|
|
select {
|
|
case <-prunerDone:
|
|
case <-stopCtx.Done():
|
|
log.Warn("event pruner did not exit before shutdown timeout")
|
|
}
|
|
}()
|
|
}
|
|
|
|
// ── 5d. Async audit writer ─────────────────────────────────────────────
|
|
// Moves audit-log INSERTs off the request path: once the writer is
|
|
// installed, WriteAudit enqueues here and a background goroutine batches
|
|
// the writes (same shape as the event persister above). Paths that never
|
|
// install a writer — the token CLI, tests — keep the synchronous
|
|
// behavior. This defer is registered after `defer database.Close()` so
|
|
// LIFO ordering drains the queue before the database is torn down.
|
|
auditWriter := db.NewAuditWriter(database, 1024, 50, 100*time.Millisecond)
|
|
auditWriter.Start(bgCtx)
|
|
database.SetAuditWriter(auditWriter)
|
|
defer func() {
|
|
stopCtx, stopCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer stopCancel()
|
|
auditWriter.Stop(stopCtx)
|
|
}()
|
|
|
|
// ── 6. Start server ────────────────────────────────────────────────────
|
|
addr := fmt.Sprintf(":%d", cfg.Server.Port)
|
|
srv := &http.Server{
|
|
Addr: addr,
|
|
Handler: router,
|
|
TLSConfig: tlsCfg,
|
|
ReadTimeout: 30 * time.Second,
|
|
WriteTimeout: 30 * time.Second,
|
|
IdleTimeout: 120 * time.Second,
|
|
ErrorLog: stdlog.New(io.Discard, "", 0), // suppress TLS handshake noise
|
|
}
|
|
|
|
// ── 6b. ACME HTTP challenge server on :80 ─────────────────────────────
|
|
// When using Let's Encrypt (tls.mode: acme), an HTTP server on port 80
|
|
// is needed for HTTP-01 challenge validation and HTTP→HTTPS redirect.
|
|
var acmeSrv *http.Server
|
|
if tlsResult.HTTPHandler != nil {
|
|
acmeSrv = &http.Server{
|
|
Addr: ":80",
|
|
Handler: tlsResult.HTTPHandler,
|
|
ReadTimeout: 10 * time.Second,
|
|
WriteTimeout: 10 * time.Second,
|
|
}
|
|
go func() {
|
|
log.Info("ACME HTTP challenge server starting on :80")
|
|
if err := acmeSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
|
log.Error("ACME HTTP server error", "error", err)
|
|
}
|
|
}()
|
|
}
|
|
|
|
// ── 7. Background maintenance ────────────────────────────────────────
|
|
// Periodically purge expired sessions and orphaned attachments.
|
|
fileStorage, fileStorageErr := storage.New(cfg.Upload.StorageDir, cfg.Upload.MaxSizeMB)
|
|
if fileStorageErr != nil {
|
|
log.Warn("failed to create file storage for maintenance; orphan file cleanup disabled", "error", fileStorageErr)
|
|
}
|
|
|
|
stopMaintenance := make(chan struct{})
|
|
maintenanceDone := make(chan struct{})
|
|
defer func() {
|
|
// Backstop for early returns below (see hub.GracefulStop defer above),
|
|
// and a bounded join so an in-flight tick (which can hold the writer —
|
|
// scheduled backups run VACUUM INTO) isn't still using the database
|
|
// while the LIFO-later Close defer tears it down.
|
|
close(stopMaintenance)
|
|
select {
|
|
case <-maintenanceDone:
|
|
case <-time.After(5 * time.Second):
|
|
log.Warn("maintenance loop did not exit before shutdown timeout")
|
|
}
|
|
}()
|
|
go func() {
|
|
defer close(maintenanceDone)
|
|
ticker := time.NewTicker(15 * time.Minute)
|
|
defer ticker.Stop()
|
|
consecutiveFailures := 0
|
|
const maxConsecutiveFailures = 5
|
|
for {
|
|
select {
|
|
case <-ticker.C:
|
|
if consecutiveFailures >= maxConsecutiveFailures {
|
|
log.Error("maintenance loop: circuit breaker open, skipping tick",
|
|
"consecutive_failures", consecutiveFailures)
|
|
// Reset after one skip to allow retry next tick.
|
|
consecutiveFailures = maxConsecutiveFailures - 1
|
|
continue
|
|
}
|
|
|
|
tickFailed := false
|
|
if err := database.DeleteExpiredSessions(bgCtx); err != nil {
|
|
log.Warn("failed to delete expired sessions", "error", err)
|
|
tickFailed = true
|
|
}
|
|
|
|
// Scheduled backups + retention pruning, driven by the
|
|
// backup_schedule / backup_retention admin settings.
|
|
if err := admin.MaintainBackups(bgCtx, database); err != nil {
|
|
log.Warn("backup maintenance failed", "error", err)
|
|
tickFailed = true
|
|
}
|
|
|
|
// Clean up orphaned attachments (uploaded but never linked to a message).
|
|
//
|
|
// Skipped entirely with no file storage configured: the delete is
|
|
// atomic (row goes the instant it's selected, by design — see
|
|
// db/attachment_queries.go), so with fileStorage nil the returned
|
|
// stored_as names — the only remaining handle on those blobs —
|
|
// would just be discarded and the files stranded on disk with no
|
|
// query left able to name them. Leaving the rows in place keeps
|
|
// them reclaimable once storage is available again.
|
|
if fileStorage != nil {
|
|
cutoff := time.Now().Add(-1 * time.Hour)
|
|
orphanFiles, orphanErr := database.DeleteOrphanedAttachments(bgCtx, cutoff)
|
|
if orphanErr != nil {
|
|
log.Warn("failed to delete orphaned attachments", "error", orphanErr)
|
|
tickFailed = true
|
|
} else if len(orphanFiles) > 0 {
|
|
// Best-effort file cleanup.
|
|
for _, filename := range orphanFiles {
|
|
if delErr := fileStorage.Delete(filename); delErr != nil {
|
|
log.Warn("failed to delete orphan file", "file", filename, "error", delErr)
|
|
}
|
|
}
|
|
log.Info("cleaned up orphaned attachments", "count", len(orphanFiles))
|
|
}
|
|
}
|
|
|
|
if tickFailed {
|
|
consecutiveFailures++
|
|
} else {
|
|
consecutiveFailures = 0
|
|
}
|
|
case <-stopMaintenance:
|
|
return
|
|
}
|
|
}
|
|
}()
|
|
|
|
// Listen for OS signals for graceful shutdown.
|
|
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
|
defer stop()
|
|
|
|
// Start serving in a goroutine.
|
|
serveErr := make(chan error, 1)
|
|
go func() {
|
|
log.Info("server starting", "addr", addr, "tls", tlsCfg != nil, "version", version)
|
|
|
|
for attempt := range 20 {
|
|
var listenErr error
|
|
if tlsCfg != nil {
|
|
listenErr = srv.ListenAndServeTLS("", "")
|
|
} else {
|
|
listenErr = srv.ListenAndServe()
|
|
}
|
|
if listenErr != nil && !errors.Is(listenErr, http.ErrServerClosed) {
|
|
// Check if it's an "address already in use" error (port not released yet from old process)
|
|
if attempt < 19 && isAddrInUse(listenErr) {
|
|
log.Warn("port in use, retrying...", "attempt", attempt+1, "error", listenErr)
|
|
time.Sleep(500 * time.Millisecond)
|
|
continue
|
|
}
|
|
serveErr <- listenErr
|
|
}
|
|
break
|
|
}
|
|
close(serveErr)
|
|
}()
|
|
|
|
// Wait for shutdown signal or server error.
|
|
select {
|
|
case err := <-serveErr:
|
|
if err != nil {
|
|
return fmt.Errorf("server error: %w", err)
|
|
}
|
|
case <-ctx.Done():
|
|
log.Info("shutdown signal received, draining connections (30s timeout)")
|
|
}
|
|
|
|
// Graceful shutdown.
|
|
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
defer cancel()
|
|
|
|
if acmeSrv != nil {
|
|
if err := acmeSrv.Shutdown(shutdownCtx); err != nil {
|
|
log.Warn("ACME HTTP server shutdown error", "error", err)
|
|
}
|
|
}
|
|
|
|
// Drain in-flight HTTP handlers FIRST: their broadcasts must still reach
|
|
// a live hub (and the event persister) or the frames vanish from the
|
|
// replay/event store across the restart. Shutdown does not wait on
|
|
// hijacked WebSocket connections, so the hub's own stop below is not
|
|
// delayed by connected clients — they get the restart notice right after
|
|
// the drain instead of right before it.
|
|
shutdownErr := srv.Shutdown(shutdownCtx)
|
|
|
|
// Stop the WebSocket hub: notify clients, stop LiveKit, close all client
|
|
// connections. Threaded with the same 30s budget the operator was told
|
|
// about — the notice sleep and LiveKit stop count against it rather than
|
|
// extending it.
|
|
hub.GracefulStopContext(shutdownCtx)
|
|
|
|
if shutdownErr != nil {
|
|
return fmt.Errorf("graceful shutdown: %w", shutdownErr)
|
|
}
|
|
|
|
log.Info("server stopped cleanly")
|
|
return nil
|
|
}
|
|
|
|
// runHealthcheckCLI probes the local server's /health endpoint and returns a
|
|
// process exit code: 0 healthy, 1 degraded or unreachable. /health answers
|
|
// 503 with a subsystem reason when the hub, database, or disk is unhealthy,
|
|
// so a container orchestrator's healthcheck surfaces those too.
|
|
func runHealthcheckCLI() int {
|
|
// Deliberately NOT config.Load: that writes a default config.yaml when
|
|
// none exists, and a probe must have no side effects. Peek at the file
|
|
// (and the env overrides) for just the values that shape the URL and the
|
|
// certificate pin.
|
|
port := 8443
|
|
scheme := "https"
|
|
certFile := "data/cert.pem"
|
|
tlsMode := ""
|
|
acmeDomain := ""
|
|
if raw, err := os.ReadFile(config.DefaultPath); err == nil {
|
|
var partial struct {
|
|
Server struct {
|
|
Port int `yaml:"port"`
|
|
} `yaml:"server"`
|
|
TLS struct {
|
|
Mode string `yaml:"mode"`
|
|
CertFile string `yaml:"cert_file"`
|
|
Domain string `yaml:"domain"`
|
|
} `yaml:"tls"`
|
|
}
|
|
if yaml.Unmarshal(raw, &partial) == nil {
|
|
if partial.Server.Port > 0 {
|
|
port = partial.Server.Port
|
|
}
|
|
tlsMode = partial.TLS.Mode
|
|
if partial.TLS.Mode == "off" {
|
|
scheme = "http"
|
|
}
|
|
if partial.TLS.CertFile != "" {
|
|
certFile = partial.TLS.CertFile
|
|
}
|
|
acmeDomain = partial.TLS.Domain
|
|
}
|
|
}
|
|
if env := os.Getenv("OWNCORD_SERVER_PORT"); env != "" {
|
|
if p, err := strconv.Atoi(env); err == nil && p > 0 {
|
|
port = p
|
|
}
|
|
}
|
|
if env := os.Getenv("OWNCORD_TLS_MODE"); env != "" {
|
|
tlsMode = env
|
|
if env == "off" {
|
|
scheme = "http"
|
|
}
|
|
}
|
|
if env := os.Getenv("OWNCORD_TLS_DOMAIN"); env != "" {
|
|
acmeDomain = env
|
|
}
|
|
client := &http.Client{
|
|
Timeout: 5 * time.Second,
|
|
Transport: &http.Transport{
|
|
TLSClientConfig: healthcheckTLSConfig(tlsMode, certFile, acmeDomain),
|
|
},
|
|
}
|
|
if port < 1 || port > 65535 {
|
|
port = 8443
|
|
}
|
|
resp, err := client.Get(fmt.Sprintf("%s://127.0.0.1:%d/health", scheme, port)) //nolint:gosec // G704: host is hardcoded loopback; only the port comes from the operator's own config
|
|
if err != nil {
|
|
fmt.Fprintln(os.Stderr, "healthcheck: unreachable:", err)
|
|
return 1
|
|
}
|
|
defer resp.Body.Close() //nolint:errcheck
|
|
if resp.StatusCode != http.StatusOK {
|
|
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
|
fmt.Fprintf(os.Stderr, "healthcheck: status %d: %s\n", resp.StatusCode, body)
|
|
return 1
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// healthcheckTLSConfig builds the probe's TLS config, per TLS mode:
|
|
//
|
|
// - acme: the served cert is CA-issued for the configured domain, so
|
|
// standard WebPKI verification works — but the probe dials 127.0.0.1, so
|
|
// ServerName must be overridden to the domain or hostname verification
|
|
// fails unconditionally and the probe reports a healthy server as down.
|
|
// A stale pre-ACME data/cert.pem must NOT be pinned in this mode either;
|
|
// the pin would mismatch the served ACME leaf forever.
|
|
// - self_signed / manual: the cert can never pass WebPKI (the generated one
|
|
// has no SANs and IsCA=false), so hostname/chain checks are replaced (not
|
|
// skipped) by pinning: the presented leaf must be byte-identical to the
|
|
// local cert file.
|
|
// - anything else with no readable local cert: plain WebPKI.
|
|
func healthcheckTLSConfig(tlsMode, certFile, acmeDomain string) *tls.Config {
|
|
if tlsMode == "acme" && acmeDomain != "" {
|
|
return &tls.Config{MinVersion: tls.VersionTLS12, ServerName: acmeDomain}
|
|
}
|
|
pinned := loadPinnedCert(certFile)
|
|
if pinned == nil {
|
|
return &tls.Config{MinVersion: tls.VersionTLS12}
|
|
}
|
|
return &tls.Config{
|
|
MinVersion: tls.VersionTLS12,
|
|
// Chain/hostname verification is replaced by the exact-match pin
|
|
// below, which is strictly stronger for a cert we hold on disk.
|
|
// VerifyConnection (not VerifyPeerCertificate) so the pin also runs
|
|
// on resumed sessions (gosec G123).
|
|
InsecureSkipVerify: true, //nolint:gosec // G402: VerifyConnection below pins the exact local certificate
|
|
VerifyConnection: func(cs tls.ConnectionState) error {
|
|
if len(cs.PeerCertificates) == 0 {
|
|
return errors.New("healthcheck: server presented no certificate")
|
|
}
|
|
if !bytes.Equal(cs.PeerCertificates[0].Raw, pinned) {
|
|
return errors.New("healthcheck: server certificate does not match " + certFile)
|
|
}
|
|
return nil
|
|
},
|
|
}
|
|
}
|
|
|
|
// loadPinnedCert reads the first PEM certificate block from path, returning
|
|
// its DER bytes, or nil when unavailable.
|
|
func loadPinnedCert(path string) []byte {
|
|
raw, err := os.ReadFile(path) //nolint:gosec // G304: path is the operator's own configured cert file
|
|
if err != nil {
|
|
return nil
|
|
}
|
|
block, _ := pem.Decode(raw)
|
|
if block == nil || block.Type != "CERTIFICATE" {
|
|
return nil
|
|
}
|
|
return block.Bytes
|
|
}
|
|
|
|
// seedHubReplayState restores the hub's monotonic seq counter from the
|
|
// persisted MAX(events.seq) so wrapped-payload seqs stay monotonic across
|
|
// restarts. Without this, the events table accumulates rows whose payload
|
|
// seqs reset to 1 after every restart, breaking the reconnect "events since
|
|
// last_seq" contract.
|
|
//
|
|
// It also forces every client resuming from at or before that restored seq
|
|
// onto the full-ready path for this boot. h.seq is persisted and restored
|
|
// here, but the paired watermark that tells a resuming client whether a
|
|
// channel-visibility change happened since its last_seq
|
|
// (visibilityChangeSeq) is in-memory only and always starts at 0 on a fresh
|
|
// process — see ws/hub_events.go's mustFullResync. Channel-visibility
|
|
// changes made to an offline client (RefreshChannelVisibility,
|
|
// revokeUnreadableChannels) are sent as targeted, unsequenced messages that
|
|
// are never written to the events table, so replay can never recover them.
|
|
// Without the MarkVisibilityChanged call below, a client resuming with
|
|
// last_seq at or before the pre-restart max sails straight through
|
|
// mustFullResync's zeroed watermark and can silently miss a visibility
|
|
// change it should have converged on.
|
|
func seedHubReplayState(ctx context.Context, hub *ws.Hub, database *db.DB, log *slog.Logger) {
|
|
maxSeq, seedErr := database.GetMaxEventSeq(ctx)
|
|
if seedErr != nil {
|
|
log.Warn("event persistence: failed to read MAX(events.seq); starting hub seq from 0", "error", seedErr)
|
|
return
|
|
}
|
|
if maxSeq <= 0 {
|
|
return
|
|
}
|
|
hub.SeedSeq(uint64(maxSeq))
|
|
log.Info("event persistence: seeded hub seq from persisted events", "seq", maxSeq)
|
|
hub.MarkVisibilityChanged()
|
|
}
|
|
|
|
// isAddrInUse checks if an error is an "address already in use" error.
|
|
func isAddrInUse(err error) bool {
|
|
return err != nil && (strings.Contains(err.Error(), "address already in use") || strings.Contains(err.Error(), "Only one usage of each socket address"))
|
|
}
|
|
|
|
// printBanner writes the startup banner to stderr (so it doesn't mix with
|
|
// the structured log output on stdout).
|
|
func printBanner(cfg *config.Config, ver string, tls bool) {
|
|
scheme := "http"
|
|
if tls {
|
|
scheme = "https"
|
|
}
|
|
|
|
localIP := getOutboundIP()
|
|
port := cfg.Server.Port
|
|
baseURL := fmt.Sprintf("%s://%s:%d", scheme, localIP, port)
|
|
adminURL := baseURL + "/admin"
|
|
|
|
tlsStatus := "disabled"
|
|
if tls {
|
|
tlsStatus = "enabled"
|
|
}
|
|
|
|
banner := fmt.Sprintf(`
|
|
|
|
___ ____ _
|
|
/ _ \__ ___ __ / ___|___ _ __ __| |
|
|
| | | \ \ /\ / / '_ \| | / _ \| '__/ _`+"`"+` |
|
|
| |_| |\ V V /| | | | |__| (_) | | | (_| |
|
|
\___/ \_/\_/ |_| |_|\____\___/|_| \__,_|
|
|
|
|
─────────────────────────────────────────────
|
|
Server %s
|
|
Version %s
|
|
TLS %s
|
|
Platform %s/%s
|
|
─────────────────────────────────────────────
|
|
API %s/api/v1/info
|
|
WebSocket %s/api/v1/ws
|
|
Admin %s
|
|
Health %s/health
|
|
─────────────────────────────────────────────
|
|
Press Ctrl+C to stop the server.
|
|
|
|
`, cfg.Server.Name, ver, tlsStatus, runtime.GOOS, runtime.GOARCH,
|
|
baseURL, wsURL(scheme, localIP, port), adminURL, baseURL)
|
|
|
|
_, _ = fmt.Fprint(os.Stderr, banner)
|
|
}
|
|
|
|
// wsURL builds the WebSocket URL with the correct scheme.
|
|
func wsURL(httpScheme, ip string, port int) string {
|
|
ws := "ws"
|
|
if httpScheme == "https" {
|
|
ws = "wss"
|
|
}
|
|
return fmt.Sprintf("%s://%s:%d", ws, ip, port)
|
|
}
|
|
|
|
// Free-space thresholds for the boot-time disk warning. /health uses its own
|
|
// (lower) continuous threshold; these only shape startup log noise.
|
|
const (
|
|
diskWarnBytes = 1 << 30 // 1 GiB — warn
|
|
diskCriticalBytes = 256 << 20 // 256 MiB — error
|
|
)
|
|
|
|
// warnLowDisk logs when the volume holding path is low on space. Probe
|
|
// failures (unsupported platform, missing dir) are silent — unknown ≠ full.
|
|
func warnLowDisk(log *slog.Logger, label, path string) {
|
|
free, err := diskutil.FreeBytes(path)
|
|
if err != nil {
|
|
return
|
|
}
|
|
switch {
|
|
case free < diskCriticalBytes:
|
|
log.Error("disk space critically low — writes will start failing soon",
|
|
"volume", label, "path", path, "free_mb", free>>20)
|
|
case free < diskWarnBytes:
|
|
log.Warn("disk space low", "volume", label, "path", path, "free_mb", free>>20)
|
|
}
|
|
}
|
|
|
|
// getOutboundIP returns the preferred outbound IP of this machine by dialing
|
|
// a known external address (no actual connection is made with UDP).
|
|
func getOutboundIP() string {
|
|
conn, err := net.Dial("udp", "8.8.8.8:80")
|
|
if err != nil {
|
|
return "localhost"
|
|
}
|
|
defer conn.Close() //nolint:errcheck
|
|
addr, ok := conn.LocalAddr().(*net.UDPAddr)
|
|
if !ok {
|
|
slog.Warn("getOutboundIP: unexpected LocalAddr type, falling back to localhost",
|
|
"type", fmt.Sprintf("%T", conn.LocalAddr()))
|
|
return "localhost"
|
|
}
|
|
return addr.IP.String()
|
|
}
|