Files
OwnCord/Server/addrinuse_test.go
J3vbandClaude Fable 5 6a26f2a839 fix(server): drain fully before the self-update/restore restart handoff (#1380)
* feat(server): supervisor detection and server.restart_mode config key

RunningUnderSupervisor detects systemd (INVOCATION_ID) and, best-effort,
NSSM (NSSM_SERVICE_NAME — 2.24 does not set it, so NSSM deployments set
the mode explicitly). server.restart_mode (auto|spawn|supervised, default
auto, env OWNCORD_SERVER_RESTART_MODE) selects how a self-restart hands
off after the server drains: exit for the supervisor to relaunch, or
spawn the replacement directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): make the self-restart handoff drain fully before starting the successor

The update/restore/wizard restart previously spawned the replacement
while the old server was still serving, then SIGTERMed itself and
hard-exited after 10s. That design failed in every documented deployment
mode: under the shipped systemd unit the spawned child (same cgroup) was
killed when the old main process exited and Restart=on-failure never
relaunched a clean exit; on Windows the self-SIGTERM is unsupported and
silently dropped, so graceful shutdown never ran — hub.GracefulStop (the
only caller of LiveKitProcess.Stop) was skipped, orphaning livekit-server
on TCP 7880/UDP 50000-60000 and dropping queued event/audit batches; and
NSSM's relaunch raced the self-spawned replacement for the database lock.

Admin handlers now perform only the on-disk swap and request a restart
through an injected hook (admin.SetRestartHandoff). The main package's
restart coordinator cancels the parent of run()'s signal.NotifyContext —
the exact drain a SIGTERM triggers, on every platform — and after run()
has fully torn down (listeners closed, hub and LiveKit stopped, queues
flushed, DB closed and its lock released) main() performs the handoff:
spawn the replacement in spawn mode, or exit 0 for the supervisor in
supervised mode. A 90s backstop force-exits a wedged teardown; the
DB-lock and bind retries demote to safety nets.

A three-state guard (idle/busy/restart-pending) serializes update apply,
backup restore, and setup-wizard restarts against each other: concurrent
applies no longer race the same staged .new file or broadcast a spurious
update_aborted, and conflicting requests get 409 UPDATE_IN_PROGRESS /
RESTART_PENDING. The swap being free of process side effects also makes
the apply success path unit-testable for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* fix(server): errno-based bind-conflict detection, ACME bind retry, LiveKit Pdeathsig

isAddrInUse now unwraps to the platform errno (EADDRINUSE; WSAEADDRINUSE
10048 on Windows) with the English strings kept only as fallback — the
string-only match never fired on localized Windows, silently disabling
the bind retry. The retry loop is extracted into serveWithBindRetry and
now also covers the ACME :80 challenge server, which previously gave up
on first conflict and stayed dead (breaking HTTP-01 renewals) until the
next restart. The .old-binary boot cleanup retries briefly for the
window where a spawn-mode predecessor has not fully exited. The
companion livekit-server gets Pdeathsig SIGKILL on Linux so a parent
killed without teardown (kill -9, OOM, backstop exit) cannot orphan it
with the voice ports held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

* docs(deploy): Restart=always unit and per-supervisor restart-mode guidance

Restart=always is what lets the deliberate clean exit after a
self-update/restore relaunch under systemd (systemctl stop is never
auto-restarted; failure exits behave as before). Deployment docs gain
the required NSSM AppEnvironmentExtra line, the Task Scheduler and
Docker restart-policy notes, and the new drain-then-handoff update flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ngzj2Rx9UGC35uLHAfErMp

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 08:25:40 +02:00

115 lines
3.3 KiB
Go

package main
import (
"errors"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"testing"
"time"
)
// A real double-bind must be recognized through the errno chain
// (net.OpError → os.SyscallError → Errno) — this is what keeps the bind
// retry alive on non-English Windows, where the old string-only match never
// fired against localized error text.
func TestIsAddrInUse_RealBindConflict(t *testing.T) {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
defer l.Close() //nolint:errcheck
_, err = net.Listen("tcp", l.Addr().String())
if err == nil {
t.Fatal("second listen on the same address unexpectedly succeeded")
}
if !isAddrInUse(err) {
t.Errorf("isAddrInUse(%v) = false, want true for a real bind conflict", err)
}
}
func TestIsAddrInUse_Table(t *testing.T) {
cases := []struct {
name string
err error
want bool
}{
{"nil", nil, false},
{"unrelated error", errors.New("connection refused"), false},
{"unix string fallback", errors.New("listen tcp :8443: bind: address already in use"), true},
{"windows string fallback", errors.New("listen tcp :8443: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted."), true},
{"out of range port", fmt.Errorf("listen tcp: address 99999: invalid port"), false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := isAddrInUse(tc.err); got != tc.want {
t.Errorf("isAddrInUse(%v) = %v, want %v", tc.err, got, tc.want)
}
})
}
}
func TestServeWithBindRetry(t *testing.T) {
log := slog.New(slog.NewTextHandler(io.Discard, nil))
addrInUse := errors.New("listen tcp :1: bind: address already in use")
prevEvery := bindRetryEvery
bindRetryEvery = time.Millisecond
t.Cleanup(func() { bindRetryEvery = prevEvery })
t.Run("retries through a transient conflict", func(t *testing.T) {
calls := 0
err := serveWithBindRetry(log, "test", func() error {
calls++
if calls < 3 {
return addrInUse
}
return http.ErrServerClosed
})
if !errors.Is(err, http.ErrServerClosed) {
t.Errorf("err = %v, want ErrServerClosed after the conflict clears", err)
}
if calls != 3 {
t.Errorf("serve calls = %d, want 3", calls)
}
})
t.Run("gives up on a non-conflict error immediately", func(t *testing.T) {
calls := 0
otherErr := errors.New("listen tcp: address 99999: invalid port")
err := serveWithBindRetry(log, "test", func() error {
calls++
return otherErr
})
if !errors.Is(err, otherErr) {
t.Errorf("err = %v, want the serve error passed through", err)
}
if calls != 1 {
t.Errorf("serve calls = %d, want 1 (no retry for non-conflict errors)", calls)
}
})
t.Run("bounded attempts on a persistent conflict", func(t *testing.T) {
calls := 0
err := serveWithBindRetry(log, "test", func() error {
calls++
return addrInUse
})
if !errors.Is(err, addrInUse) {
t.Errorf("err = %v, want the final conflict error", err)
}
if calls != 20 {
t.Errorf("serve calls = %d, want exactly 20", calls)
}
})
t.Run("clean shutdown passes through untouched", func(t *testing.T) {
if err := serveWithBindRetry(log, "test", func() error { return http.ErrServerClosed }); !errors.Is(err, http.ErrServerClosed) {
t.Errorf("err = %v, want ErrServerClosed", err)
}
})
}