mirror of
https://github.com/J3vb/OwnCord.git
synced 2026-09-03 03:50:00 +03:00
* test(b3-6): seeded hub simulation and fault-injected transport (items 2 and 3) Server/ws/hub_sim_test.go drives a PCG-seeded interleaving of subscribe, broadcast (global, channel, recipients-scoped, sequenced DM), ack, disconnect and reconnect-transfer over a real Hub with eight headless clients, and a model client checks the per-client FIFO/seq oracle from Server/CLAUDE.md after every step: strictly increasing seq per connection, exact audience delivery (nothing lost, extra or twice), a resume replayed exactly from the watermark to the seq at which registerNow ran, h.seq advancing only for a frame that reached the ring, an evicted watermark refused a replay, and a replaced socket's late teardown reporting replaced=true. The resume step runs reconnectRegister as-is (snapshot and registerNow under one seqMu section) on a goroutine while up to three broadcasts race it; the model recovers the snapshot point from the replay burst, so any interleaving is checkable. OWNCORD_SIM_SEED replays one seed, OWNCORD_SIM_SEEDS (default 20) and OWNCORD_SIM_STEPS (default 200) size a run, and a failure prints the seed, the step, a ready-to-paste replay line and the last steps. The default runs in about 2.3 s under -race; `make sim` runs 10,000 steps per seed. Server/ws/faultconn_test.go is the seeded, deterministic frame transport the simulation reads through: drop, tail cut, duplicate, bounded reorder and an order-preserving lag from its own PCG stream, exported to ws_test through export_test.go as NewFaultConnForTest. The simulation's default wire is a lag plus tail cuts, the one fault a TCP-backed WebSocket really has; the silent drop is the negative control that proves the oracle notices a lost replay. BenchmarkReconnectStorm resumes 50 live clients per op through the same path. newTestHub and its three seed helpers take testing.TB so the benchmark can share them. No production code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmiqjgTuov1stBTB6uGkvo * docs(b3-6): evidence block for items 2 and 3 (hub simulation, fault transport) Oracle, the RED/GREEN excerpts (inverted assertion, seed replay, drop-all wire, unsynchronized registerNow), wall-clock and benchmark figures, gate results and the epoch-harness decision, under B3-6 in the plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmiqjgTuov1stBTB6uGkvo * fix(b3-6): hub sim — deterministic topic limiter for exact replay, a floor on the step mix, auth-frame-wins under transfer, wire-seed mixing Review fixes for items 2 and 3. Exact replay. TopicRateLimiter keys its window on time.Now(), so at 10,000 steps the shed boundary was a timing-dependent step and every later seq differed between runs; the printed OWNCORD_SIM_SEED line could not reproduce a failure. FreezeTopicLimiterForTest (export_test.go) swaps the hub's limiter for one whose window never rolls over inside a run, so the shed is a per-channel count. Three more leaks of the scheduler's interleaving into the trajectory surfaced once that was fixed, and are closed the same way — by taking the decision away from the race or making both outcomes read the same: racing frames are pulled into the wire at attach time (queue fill no longer depends on which side of the snapshot they fell), the racing burst is aimed at the resuming client's own audience (a replay-superset frame was read iff it landed before the snapshot), and a resume within the burst's reach of the ring's eviction boundary is not raced (the allocations could evict the watermark before or after the snapshot and pick replay or fallback). Three runs of one seed now print byte-identical stats; what still varies — how many racing seqs land in the replay burst — is printed on its own line and stated in the doc comment. Floor. TestHubSimulation aggregates the per-seed stats and requires every load-bearing transition (the four broadcast kinds, resume, fallback, fresh, cut, kicked, racing-in-replay) at least once across the default run, so a constant change cannot turn the simulation into no-ops with CI green. Its first run found that the overflow kick had become unreachable at 200 steps; the sim's queue is 12 now (production stays 256). Also: the resume step draws active_channel_id as none / the open channel / another channel whether or not the old socket is registered, so registerNow's auth-frame-wins branch runs under the transfer; the wire's PCG takes the seed and (idx<<32|conns) as its two words instead of an arithmetic mix that collided past 131 connections; seedTestUser takes testing.TB like its siblings; the evidence block lists what the simulation does not cover and the -timeout 60m the ten-pass deadlock gate needs, with the same line under Traps carried forward. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmiqjgTuov1stBTB6uGkvo * test(b3-6): hub sim — floor on raced resumes instead of the scheduler-decided key; log when the floor is skipped racing-in-replay was the one floor key the scheduler decides, so a correct hub could in principle fail the floor on a run where no racing seq landed inside a burst. The floor now keys on raced resumes — a resume that got a replay while a burst ran (burst > 0 && ok), which the seed determines — and racing-in-replay stays a printed count. The floor also says so when it is skipped for OWNCORD_SIM_SEED or a shorter seed list instead of returning silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmiqjgTuov1stBTB6uGkvo * test(b3-6): hub sim — raced counts a broadcast allocated while registration was in progress, not the requested burst Codex P2 on #1458: raced incremented on burst > 0, which counted a resume whose goroutine had returned before the first broadcast ran and one whose every channel broadcast the limiter shed, so the floor could pass with no broadcast overlapping a registration. raced now counts a resume where a racing broadcast allocated a seq while the reconnect goroutine had not yet been observed to return (the driver's done handshake, checked after each allocation). That is the scheduler's call, so raced moves off the deterministic stats line and is floored only in aggregate across the 20 default seeds — 179 bursts per run, 178–179 observed overlapping in three measured runs, odds named in the comment and the evidence block. The requested burst stays a printed, seed-determined count (bursts) and is floored as before; the floor logs its totals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmiqjgTuov1stBTB6uGkvo --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
625 lines
23 KiB
Go
625 lines
23 KiB
Go
package ws_test
|
||
|
||
// hub_sim_test.go — B3-6 item 2: a seeded simulation of the hub's ordering
|
||
// space (Tier 3b of docs/plans/bug-detection-improvements.md). Every recurring
|
||
// bug in this package's history is a bad ordering — registerNow's
|
||
// reconnect-transfer, replay against a moving ring, a socket dying with
|
||
// frames in flight — and nothing else in the repo generates orderings. This
|
||
// does: a PCG stream picks each step from {subscribe, broadcast, ack,
|
||
// disconnect, reconnect-transfer} over eight headless clients on a real Hub,
|
||
// and a model client checks the oracle below after every step.
|
||
//
|
||
// ORACLE — written before the driver, from Server/CLAUDE.md ("Sequenced
|
||
// frames share one per-client FIFO because clients ack only max(seq) — a
|
||
// frame that skips the queue, or a seq allocated for a frame that is then
|
||
// dropped, is silently unrecoverable") and the replay semantics pinned by
|
||
// ringbuffer_test.go, hub_register_race_test.go and serve.go's
|
||
// reconnectRegister. The only ack this protocol has is last_seq on the next
|
||
// auth frame, so "ack" here is the client reading frames and advancing its
|
||
// watermark W = max(seq read).
|
||
//
|
||
// I1 Within one connection, sequenced frames arrive with strictly
|
||
// increasing seq — the replay burst first, then the live queue — and a
|
||
// seq-stamped frame never rides the high- or low-priority queue.
|
||
// I2 A live connection yields, in order, exactly the seqs the hub allocated
|
||
// for events whose audience contained it since it registered: nothing
|
||
// missing (a seq allocated for a frame that was then dropped is a gap),
|
||
// nothing extra, nothing twice. After every step the frames still
|
||
// unread (queue + transport) equal the seqs still owed.
|
||
// I3 On a resume from W that the ring covers, the replay burst is exactly
|
||
// {s in (W, S] : channel(s) is 0 or READ-allowed}, in seq order, where
|
||
// S is the hub seq at the instant registerNow ran (atomic with the
|
||
// snapshot under h.seqMu), and every audience seq above S arrives live —
|
||
// so across a drop and a resume no audience seq above W is lost or
|
||
// skipped. Frames the dying socket still held above W are delivered
|
||
// again by the replay; that cross-connection duplicate is by design
|
||
// (the client acks max(seq)) and I1 still holds per connection.
|
||
// I4 h.seq advances only for a frame that reached the ring: a
|
||
// topic-limiter shed allocates nothing, and after every allocation
|
||
// ring.NewestSeq() == h.seq.
|
||
// I5 When the ring cannot cover W the resume is refused a replay
|
||
// (ok=false), the connection is registered the way handleFreshConnect
|
||
// registers a replay-failure fallback, and the obligations restart
|
||
// there — a full ready carries the state.
|
||
// I6 A dead socket's late teardown (unregisterNow after its replacement
|
||
// registered) reports replaced=true and leaves the replacement in
|
||
// place; tearing down the live connection reports false.
|
||
//
|
||
// Audience: a global broadcast reaches every registered client; a channel
|
||
// broadcast the clients focused on it (after the topic limiter); a
|
||
// recipients-scoped broadcast (voice_state's path) exactly its listed users;
|
||
// a sequenced DM its participants. The replay filter is coarser than the
|
||
// audience — it replays every READ-allowed channel, focused or not — so a
|
||
// resume legitimately carries frames the client would not have received live.
|
||
//
|
||
// Knobs: OWNCORD_SIM_SEED replays one seed; OWNCORD_SIM_SEEDS (default 20)
|
||
// and OWNCORD_SIM_STEPS (default 200) size a run; `make sim` runs 10,000
|
||
// steps. A failure prints the seed, the step, a ready-to-paste replay line
|
||
// and the last steps.
|
||
//
|
||
// Determinism: the step sequence, every seq allocation (the topic limiter is
|
||
// frozen to a per-channel count, FreezeTopicLimiterForTest), every wire fault
|
||
// and every figure on the stats line are a pure function of the seed — three
|
||
// runs of one seed print byte-identical stats — so a failure in any of them
|
||
// replays exactly from the printed line. What still varies between runs is
|
||
// the scheduler's interleaving inside a racing reconnect step: which of the
|
||
// racing seqs land in the replay burst and which arrive live. The oracle
|
||
// holds for every interleaving, and the burst is shaped so the client reads
|
||
// the same frames either way (see broadcast), but a defect that depends on
|
||
// that split — the I3 class — replays only probabilistically; run the seed
|
||
// with -count. The default run also carries a floor (TestHubSimulation):
|
||
// every load-bearing transition must occur at least once across the seeds,
|
||
// so a constant change cannot quietly turn the simulation into no-ops — and
|
||
// one of them, a broadcast observed running against a registration in
|
||
// flight, is the scheduler's and is floored only in aggregate.
|
||
|
||
import (
|
||
"fmt"
|
||
"math/rand/v2"
|
||
"os"
|
||
"slices"
|
||
"strconv"
|
||
"strings"
|
||
"testing"
|
||
|
||
"github.com/J3vb/OwnCord/Server/db"
|
||
"github.com/J3vb/OwnCord/Server/ws"
|
||
)
|
||
|
||
const (
|
||
simClients = 8
|
||
simChannels = 3
|
||
simRing = 48 // replay ring (production 1000): small enough that I5's eviction is reachable in 200 steps
|
||
simSendBuf = 12 // normal queue (production 256, client.go sendBufSize): small enough that the BUG-124 overflow kick happens a few times per seed at 200 steps
|
||
simDefaultSeeds = 20
|
||
simDefaultSteps = 200
|
||
)
|
||
|
||
type simOp int
|
||
|
||
const (
|
||
opSubscribe simOp = iota
|
||
opBroadcast
|
||
opAck
|
||
opDisconnect
|
||
opReconnect
|
||
)
|
||
|
||
var simOpWeights = [...]int{opSubscribe: 15, opBroadcast: 40, opAck: 20, opDisconnect: 10, opReconnect: 15}
|
||
|
||
type simKind int
|
||
|
||
const (
|
||
kindGlobal simKind = iota
|
||
kindChannel
|
||
kindRecipients
|
||
kindDM
|
||
)
|
||
|
||
var (
|
||
simKindNames = [...]string{kindGlobal: "global", kindChannel: "channel", kindRecipients: "recipients", kindDM: "dm"}
|
||
simKindWeights = [...]int{kindGlobal: 30, kindChannel: 35, kindRecipients: 15, kindDM: 20}
|
||
)
|
||
|
||
// simAlloc is one allocated seq and how the hub addressed it.
|
||
type simAlloc struct {
|
||
seq uint64
|
||
ch int64
|
||
kind simKind
|
||
users []int64 // recipients or DM participants
|
||
}
|
||
|
||
// reaches is the live audience rule for a registered client.
|
||
func (a simAlloc) reaches(c *simClient) bool {
|
||
switch a.kind {
|
||
case kindGlobal:
|
||
return true
|
||
case kindChannel:
|
||
return c.focus == a.ch
|
||
default:
|
||
return slices.Contains(a.users, c.user.ID)
|
||
}
|
||
}
|
||
|
||
type simClient struct {
|
||
idx int
|
||
user *db.User
|
||
allowed map[int64]bool // READ-allowed channels: every text channel plus this user's DM channels
|
||
conn *ws.Client // nil while disconnected
|
||
send chan []byte
|
||
high chan []byte
|
||
low chan []byte
|
||
wire *ws.FaultConn
|
||
conns int
|
||
cut bool // the transport cut the socket; the server has not noticed yet
|
||
focus int64 // focused channel, 0 = none
|
||
w uint64 // watermark: max seq read
|
||
owed []uint64
|
||
}
|
||
|
||
type sim struct {
|
||
t *testing.T
|
||
hub *ws.Hub
|
||
rng *rand.Rand
|
||
seed uint64
|
||
steps int
|
||
step int
|
||
chIDs []int64
|
||
clients []*simClient
|
||
chanOf []int64 // chanOf[seq] = channel of every allocated seq; index 0 unused
|
||
sched ws.FaultSchedule
|
||
trace []string
|
||
stats map[string]int // every figure here is a pure function of the seed
|
||
racing int // racing seqs that landed inside a replay burst: the scheduler's call, not the seed's
|
||
raced int // resumes where a broadcast allocated a seq while registration was still in flight: also the scheduler's
|
||
}
|
||
|
||
func TestHubSimulation(t *testing.T) {
|
||
seeds, steps := simConfig(t)
|
||
total := map[string]int{}
|
||
for _, seed := range seeds {
|
||
t.Run(fmt.Sprintf("seed=%d", seed), func(t *testing.T) {
|
||
stats, raced := runHubSim(t, seed, steps)
|
||
for k, v := range stats {
|
||
total[k] += v
|
||
}
|
||
total["raced"] += raced
|
||
})
|
||
}
|
||
// The floor: every load-bearing transition must have happened at least
|
||
// once across the run, or a constant change (weights, ring, queue, wire
|
||
// schedule) could turn the simulation into no-ops with CI still green.
|
||
// Every key but one is a function of the seed. "raced" — a broadcast
|
||
// that allocated a seq while a resume's registration was still in flight
|
||
// — is the scheduler's, so it is floored only in aggregate: the default
|
||
// run has 179 resumes with a burst ("bursts", seed-determined), and each
|
||
// one overlaps with odds far above a coin flip — registerNow makes two
|
||
// slog syscalls before the goroutine can return, and 178–179 of the 179
|
||
// were observed overlapping in three measured runs — so an all-miss run
|
||
// is not luck but a scheduler or lock change that no longer lets a
|
||
// broadcast run against a registration, the very thing this simulation
|
||
// exists to exercise. Calibrated for the default seed count; a
|
||
// single-seed replay or a shorter seed list skips it.
|
||
if t.Failed() {
|
||
return
|
||
}
|
||
if os.Getenv("OWNCORD_SIM_SEED") != "" || len(seeds) < simDefaultSeeds {
|
||
t.Logf("hub simulation: floor skipped — OWNCORD_SIM_SEED=%q, %d seed(s) (calibrated for %d)", os.Getenv("OWNCORD_SIM_SEED"), len(seeds), simDefaultSeeds)
|
||
return
|
||
}
|
||
t.Logf("hub simulation: floor totals %v", total)
|
||
for _, k := range []string{"global", "channel", "recipients", "dm", "resume", "bursts", "raced", "fallback", "fresh", "cut", "kicked"} {
|
||
if total[k] == 0 {
|
||
t.Errorf("hub simulation: %q never happened across %d seeds x %d steps — the step mix no longer reaches it", k, len(seeds), steps)
|
||
}
|
||
}
|
||
}
|
||
|
||
func simConfig(t *testing.T) (seeds []uint64, steps int) {
|
||
t.Helper()
|
||
steps = simEnvInt(t, "OWNCORD_SIM_STEPS", simDefaultSteps)
|
||
if v := os.Getenv("OWNCORD_SIM_SEED"); v != "" {
|
||
seed, err := strconv.ParseUint(v, 10, 64)
|
||
if err != nil {
|
||
t.Fatalf("OWNCORD_SIM_SEED=%q: %v", v, err)
|
||
}
|
||
return []uint64{seed}, steps
|
||
}
|
||
for i := range simEnvInt(t, "OWNCORD_SIM_SEEDS", simDefaultSeeds) {
|
||
seeds = append(seeds, uint64(i+1))
|
||
}
|
||
return seeds, steps
|
||
}
|
||
|
||
func simEnvInt(t *testing.T, name string, def int) int {
|
||
t.Helper()
|
||
v := os.Getenv(name)
|
||
if v == "" {
|
||
return def
|
||
}
|
||
n, err := strconv.Atoi(v)
|
||
if err != nil || n <= 0 {
|
||
t.Fatalf("%s=%q: want a positive integer", name, v)
|
||
}
|
||
return n
|
||
}
|
||
|
||
func simDMChannel(i, j int) int64 {
|
||
return 1000 + int64(min(i, j))*simClients + int64(max(i, j))
|
||
}
|
||
|
||
func runHubSim(t *testing.T, seed uint64, steps int) (stats map[string]int, raced int) {
|
||
hub, database := newTestHub(t)
|
||
hub.ConfigureReplay(simRing, 0)
|
||
hub.FreezeTopicLimiterForTest()
|
||
s := &sim{
|
||
t: t, hub: hub, seed: seed, steps: steps,
|
||
rng: rand.New(rand.NewPCG(seed, seed^0xD1B54A32D192ED03)),
|
||
chanOf: []int64{0},
|
||
stats: map[string]int{},
|
||
}
|
||
for i := range simChannels {
|
||
s.chIDs = append(s.chIDs, seedTestChannel(t, database, fmt.Sprintf("sim-%d", i)))
|
||
}
|
||
for i := range simClients {
|
||
c := &simClient{idx: i, user: seedOwnerUser(t, database, fmt.Sprintf("sim-user-%d", i)), allowed: map[int64]bool{}}
|
||
for _, ch := range s.chIDs {
|
||
c.allowed[ch] = true
|
||
}
|
||
for j := range simClients {
|
||
if j != i {
|
||
c.allowed[simDMChannel(i, j)] = true
|
||
}
|
||
}
|
||
s.clients = append(s.clients, c)
|
||
}
|
||
// The wire: an order-preserving lag and the one fault TCP really has, a cut.
|
||
s.sched = ws.FaultSchedule{Delay: s.rng.IntN(3), Drop: 0.02, DropTail: true}
|
||
for _, c := range s.clients {
|
||
s.connect(c)
|
||
}
|
||
for s.step = 1; s.step <= steps; s.step++ {
|
||
c := s.clients[s.rng.IntN(simClients)]
|
||
switch simOp(s.pick(simOpWeights[:])) {
|
||
case opSubscribe:
|
||
s.subscribe(c)
|
||
case opBroadcast:
|
||
s.broadcast(nil)
|
||
case opAck:
|
||
s.ack(c)
|
||
case opDisconnect:
|
||
s.disconnect(c)
|
||
case opReconnect:
|
||
s.reconnect(c)
|
||
}
|
||
s.checkCounts()
|
||
}
|
||
for _, c := range s.clients {
|
||
if c.conn != nil && !c.cut {
|
||
s.read(c, -1)
|
||
}
|
||
}
|
||
t.Logf("seed %d: %d steps, seq %d, %v", seed, steps, hub.SeqForTest(), s.stats)
|
||
t.Logf("seed %d: %d resume(s) overlapped a broadcast (raced), %d racing seq(s) landed inside a replay burst — both scheduler-decided, kept off the stats line", seed, s.raced, s.racing)
|
||
return s.stats, s.raced
|
||
}
|
||
|
||
func (s *sim) pick(weights []int) int {
|
||
total := 0
|
||
for _, w := range weights {
|
||
total += w
|
||
}
|
||
r := s.rng.IntN(total)
|
||
for i, w := range weights {
|
||
if r < w {
|
||
return i
|
||
}
|
||
r -= w
|
||
}
|
||
return len(weights) - 1
|
||
}
|
||
|
||
func (s *sim) note(format string, args ...any) {
|
||
s.trace = append(s.trace, fmt.Sprintf("#%d ", s.step)+fmt.Sprintf(format, args...))
|
||
if len(s.trace) > 24 {
|
||
s.trace = s.trace[1:]
|
||
}
|
||
}
|
||
|
||
func (s *sim) failf(format string, args ...any) {
|
||
s.t.Helper()
|
||
s.t.Fatalf("hub simulation: seed %d step %d: %s\nreplay: OWNCORD_SIM_SEED=%d OWNCORD_SIM_STEPS=%d go test -race -count=1 -run '^TestHubSimulation$' ./ws/\nlast steps:\n %s",
|
||
s.seed, s.step, fmt.Sprintf(format, args...), s.seed, s.steps, strings.Join(s.trace, "\n "))
|
||
}
|
||
|
||
// ─── connections ─────────────────────────────────────────────────────────────
|
||
|
||
func (s *sim) newConn(c *simClient, channelID int64, lastSeq uint64) *ws.Client {
|
||
c.send = make(chan []byte, simSendBuf)
|
||
c.high = make(chan []byte, 64)
|
||
c.low = make(chan []byte, 64)
|
||
return ws.NewSimClientForTest(s.hub, c.user, channelID, lastSeq, c.send, c.high, c.low)
|
||
}
|
||
|
||
func (s *sim) attach(c *simClient, conn *ws.Client, preface [][]byte, owed []uint64) {
|
||
c.conns++
|
||
c.conn, c.cut, c.owed = conn, false, owed
|
||
c.wire = ws.NewFaultConnForTest(s.seed, uint64(c.idx)<<32|uint64(c.conns), s.sched, preface, c.send)
|
||
// Take what the queue already holds — the racing broadcasts that landed
|
||
// after registerNow — into the wire now: the queue's fill, and so the
|
||
// step at which it could overflow, must not depend on which side of the
|
||
// snapshot those frames fell, the one thing the scheduler rather than
|
||
// the seed decides.
|
||
c.wire.Pull()
|
||
if got := s.hub.GetClient(c.user.ID); got != conn {
|
||
s.failf("c%d: hub holds %p for the user after registration, want %p", c.idx, got, conn)
|
||
}
|
||
}
|
||
|
||
// connect is a fresh connect (last_seq 0): handleFreshConnect registers with
|
||
// a nil readable set, so nothing is inherited and there is no replay.
|
||
func (s *sim) connect(c *simClient) {
|
||
conn := s.newConn(c, 0, 0)
|
||
s.hub.RegisterNowForTest(conn)
|
||
c.focus = 0
|
||
s.attach(c, conn, nil, nil)
|
||
}
|
||
|
||
// teardownOld is the replaced socket's readPump defer finishing late (I6).
|
||
func (s *sim) teardownOld(c *simClient, old *ws.Client) {
|
||
if old != nil && !s.hub.UnregisterNowForTest(old) {
|
||
s.failf("I6: c%d: the replaced connection's teardown evicted the replacement", c.idx)
|
||
}
|
||
}
|
||
|
||
// ─── steps ───────────────────────────────────────────────────────────────────
|
||
|
||
func (s *sim) subscribe(c *simClient) {
|
||
// A queue the server already closed (overflow kick, read as FaultClosed
|
||
// at the next ack) is a dying socket: Subscribe refuses it by design, and
|
||
// the client is not sending channel_focus on it anyway.
|
||
if c.conn == nil || c.cut || ws.IsSendClosedForTest(c.conn) {
|
||
s.note("subscribe c%d: no live connection", c.idx)
|
||
return
|
||
}
|
||
var ch int64
|
||
if s.rng.IntN(4) != 0 {
|
||
ch = s.chIDs[s.rng.IntN(simChannels)]
|
||
}
|
||
s.hub.ApplySetChannelIDForTest(c.conn, ch)
|
||
c.focus = ch
|
||
if got := ws.ClientChannelIDForTest(c.conn); got != ch || (ch != 0 && !s.hub.SubscribedToChannelTopicForTest(c.conn, ch)) {
|
||
s.failf("channel_focus %d left c%d focused on %d", ch, c.idx, got)
|
||
}
|
||
s.note("subscribe c%d -> ch%d", c.idx, ch)
|
||
}
|
||
|
||
// broadcast allocates one seq through the real delivery path and records who
|
||
// is owed it. racing, when set, is the client mid-reconnect: the caller
|
||
// resolves its audience itself, and the frame is aimed so that it reaches
|
||
// that client whether it lands before the snapshot (replayed) or after it
|
||
// (live) — global, the channel it is focused on, a recipient list it is on,
|
||
// a DM it is in. A frame the replay filter would carry but the live audience
|
||
// would not (another channel, someone else's recipients) would make what the
|
||
// client reads depend on the interleaving, and with it every later wire
|
||
// draw; the replay-superset case is covered by every non-racing broadcast.
|
||
func (s *sim) broadcast(racing *simClient) simAlloc {
|
||
kind := simKind(s.pick(simKindWeights[:]))
|
||
if racing != nil && kind == kindChannel && racing.focus == 0 {
|
||
kind = kindGlobal
|
||
}
|
||
a := simAlloc{kind: kind}
|
||
payload := fmt.Appendf(nil, `{"type":"sim","step":%d}`, s.step)
|
||
before := s.hub.SeqForTest()
|
||
switch a.kind {
|
||
case kindGlobal:
|
||
a.seq = s.hub.DeliverBroadcastForTest(0, nil, payload)
|
||
case kindChannel:
|
||
a.ch = s.chIDs[s.rng.IntN(simChannels)]
|
||
if racing != nil {
|
||
a.ch = racing.focus
|
||
}
|
||
a.seq = s.hub.DeliverBroadcastForTest(a.ch, nil, payload)
|
||
case kindRecipients:
|
||
a.ch = s.chIDs[s.rng.IntN(simChannels)]
|
||
a.users = make([]int64, 0, simClients)
|
||
for _, c := range s.clients {
|
||
if c == racing || s.rng.IntN(2) == 0 {
|
||
a.users = append(a.users, c.user.ID)
|
||
}
|
||
}
|
||
a.seq = s.hub.DeliverBroadcastForTest(a.ch, a.users, payload)
|
||
case kindDM:
|
||
i := s.rng.IntN(simClients)
|
||
if racing != nil {
|
||
i = racing.idx
|
||
}
|
||
j := (i + 1 + s.rng.IntN(simClients-1)) % simClients
|
||
a.ch = simDMChannel(i, j)
|
||
a.users = []int64{s.clients[i].user.ID, s.clients[j].user.ID}
|
||
a.seq = s.hub.SendSequencedToUsersForTest(a.ch, a.users, payload)
|
||
}
|
||
after := s.hub.SeqForTest()
|
||
switch {
|
||
case a.seq == 0 && after != before:
|
||
s.failf("I4: a shed %s frame advanced seq %d -> %d", simKindNames[a.kind], before, after)
|
||
case a.seq == 0:
|
||
s.stats["shed"]++
|
||
s.note("broadcast %s ch%d: shed by the topic limiter", simKindNames[a.kind], a.ch)
|
||
return a
|
||
case after != a.seq || s.hub.ReplayBuffer().NewestSeq() != a.seq:
|
||
s.failf("I4: seq %d allocated but ring newest is %d (hub seq %d)", a.seq, s.hub.ReplayBuffer().NewestSeq(), after)
|
||
case uint64(len(s.chanOf)) != a.seq:
|
||
s.failf("I4: seq %d is not the successor of %d", a.seq, len(s.chanOf)-1)
|
||
}
|
||
s.chanOf = append(s.chanOf, a.ch)
|
||
for _, c := range s.clients {
|
||
if c != racing && c.conn != nil && !c.cut && a.reaches(c) {
|
||
c.owed = append(c.owed, a.seq)
|
||
}
|
||
}
|
||
s.stats[simKindNames[a.kind]]++
|
||
s.note("broadcast %s ch%d users%v -> seq %d", simKindNames[a.kind], a.ch, a.users, a.seq)
|
||
return a
|
||
}
|
||
|
||
func (s *sim) ack(c *simClient) {
|
||
if c.conn == nil || c.cut {
|
||
s.note("ack c%d: no live connection", c.idx)
|
||
return
|
||
}
|
||
n := 1 + s.rng.IntN(8)
|
||
if s.rng.IntN(2) == 0 {
|
||
n = -1 // drain
|
||
}
|
||
s.read(c, n)
|
||
}
|
||
|
||
// read pulls up to n frames (n < 0: until the wire is empty) through the
|
||
// transport and checks each against the model.
|
||
func (s *sim) read(c *simClient, n int) {
|
||
defer s.checkPriorityQueues(c)
|
||
read := 0
|
||
for n < 0 || read < n {
|
||
frame, st := c.wire.Recv()
|
||
if st == ws.FaultOK {
|
||
s.observe(c, frame)
|
||
read++
|
||
continue
|
||
}
|
||
switch st {
|
||
case ws.FaultEmpty:
|
||
if len(c.owed) != c.wire.Buffered() {
|
||
s.failf("I2: c%d conn %d: owed %v but only %d unread frame(s) remain (W=%d)", c.idx, c.conns, c.owed, c.wire.Buffered(), c.w)
|
||
}
|
||
s.note("ack c%d: read %d, W=%d, %d in flight", c.idx, read, c.w, c.wire.Buffered())
|
||
case ws.FaultClosed:
|
||
// The server closed the queue: the normal-queue overflow kick
|
||
// (client.go sendMsg, BUG-124). Its readPump defer follows.
|
||
if !ws.IsSendClosedForTest(c.conn) || s.hub.UnregisterNowForTest(c.conn) {
|
||
s.failf("I6: c%d: queue reported closed on a live connection, or its teardown reported replaced", c.idx)
|
||
}
|
||
c.conn, c.owed = nil, nil
|
||
s.stats["kicked"]++
|
||
s.note("ack c%d: read %d then the server had closed the queue (overflow kick), W=%d", c.idx, read, c.w)
|
||
case ws.FaultCut:
|
||
c.cut, c.owed = true, nil
|
||
s.stats["cut"]++
|
||
s.note("ack c%d: read %d then the wire cut, W=%d", c.idx, read, c.w)
|
||
}
|
||
return
|
||
}
|
||
s.note("ack c%d: read %d, W=%d", c.idx, read, c.w)
|
||
}
|
||
|
||
func (s *sim) disconnect(c *simClient) {
|
||
if c.conn == nil {
|
||
s.note("disconnect c%d: already gone", c.idx)
|
||
return
|
||
}
|
||
if s.hub.UnregisterNowForTest(c.conn) || s.hub.GetClient(c.user.ID) != nil {
|
||
s.failf("I6: c%d: tearing down the live connection reported replaced or left a client registered", c.idx)
|
||
}
|
||
ws.CloseSendForTest(c.conn)
|
||
c.conn, c.owed, c.cut = nil, nil, false
|
||
s.note("disconnect c%d (W=%d)", c.idx, c.w)
|
||
}
|
||
|
||
// reconnect is the resume: a new socket for the same user carrying W as
|
||
// last_seq, registered through reconnectRegister — the replay snapshot and
|
||
// registerNow under one h.seqMu section — while up to three broadcasts race
|
||
// it from this goroutine, exactly the interleaving the seqMu discipline
|
||
// exists for. The old socket may still be registered (a network blip: the
|
||
// transfer path) or already gone (active_channel_id restores the focus).
|
||
func (s *sim) reconnect(c *simClient) {
|
||
old := c.conn
|
||
if c.w == 0 {
|
||
s.connect(c)
|
||
s.teardownOld(c, old)
|
||
s.stats["fresh"]++
|
||
s.note("reconnect c%d: fresh (W=0)", c.idx)
|
||
return
|
||
}
|
||
// The auth frame's active_channel_id: none, the channel the client had
|
||
// open, or one it switched to while offline. handleReconnect honours it
|
||
// when READ-allowed (every text channel here), and registerNow lets it
|
||
// win over the transfer, which lands only on a client that declared none.
|
||
var authCh int64
|
||
switch s.rng.IntN(3) {
|
||
case 1:
|
||
authCh = c.focus
|
||
case 2:
|
||
authCh = s.chIDs[s.rng.IntN(simChannels)]
|
||
}
|
||
if authCh != 0 || old == nil {
|
||
c.focus = authCh
|
||
}
|
||
conn := s.newConn(c, authCh, c.w)
|
||
s0 := s.hub.SeqForTest()
|
||
var (
|
||
events [][]byte
|
||
ok bool
|
||
done = make(chan struct{})
|
||
)
|
||
go func() {
|
||
defer close(done)
|
||
events, ok = s.hub.ReconnectRegisterForTest(conn, c.w, c.allowed)
|
||
}()
|
||
burst := s.rng.IntN(4)
|
||
// Racing allocations can evict W from the ring before or after the
|
||
// snapshot. Where that would decide replay-or-fallback, the scheduler
|
||
// would be choosing the path and everything the client reads after it,
|
||
// so a resume that close to the eviction boundary is not raced; the
|
||
// paths on either side of the boundary are the seed's alone. What stays
|
||
// the scheduler's is only the split of the racing seqs between the
|
||
// replay burst and the live queue.
|
||
if oldest := s.hub.ReplayBuffer().OldestSeq(); oldest < c.w && c.w <= oldest+uint64(burst) {
|
||
burst = 0
|
||
}
|
||
var racing []simAlloc
|
||
overlapped := false
|
||
for range burst {
|
||
a := s.broadcast(c)
|
||
if a.seq == 0 {
|
||
continue
|
||
}
|
||
racing = append(racing, a)
|
||
// An overlap on record: this seq was allocated while the goroutine
|
||
// had not yet been seen to return, so the broadcast ran against a
|
||
// registration in flight. A burst the goroutine beat to the lock, or
|
||
// one the limiter shed entirely, is not one.
|
||
select {
|
||
case <-done:
|
||
default:
|
||
overlapped = true
|
||
}
|
||
}
|
||
<-done
|
||
var owed []uint64
|
||
if ok {
|
||
owed = s.checkReplay(c, events, s0, racing)
|
||
s.stats["resume"]++
|
||
if burst > 0 {
|
||
s.stats["bursts"]++
|
||
}
|
||
if overlapped {
|
||
s.raced++
|
||
}
|
||
} else {
|
||
if s.hub.ReplayBuffer().EventsSinceFiltered(c.w, c.allowed) != nil {
|
||
s.failf("I5: c%d: replay refused although the ring (oldest %d) covers W=%d", c.idx, s.hub.ReplayBuffer().OldestSeq(), c.w)
|
||
}
|
||
// handleFreshConnect's replay-failure fallback: registerNow with the
|
||
// readable set, so the focus still transfers; a full ready follows.
|
||
s.hub.RegisterNowWithReadableForTest(conn, c.allowed)
|
||
events = nil
|
||
s.stats["fallback"]++
|
||
}
|
||
s.teardownOld(c, old)
|
||
s.attach(c, conn, events, owed)
|
||
if got := ws.ClientChannelIDForTest(conn); got != c.focus {
|
||
s.failf("c%d: resumed connection focused on %d, want %d", c.idx, got, c.focus)
|
||
}
|
||
s.note("reconnect c%d: W=%d old=%v auth_ch=%d replay=%v racing=%d owed=%v", c.idx, c.w, old != nil, authCh, ok, len(racing), owed)
|
||
}
|