J3vb/OwnCord
1
0
Fork 0
mirror of https://github.com/J3vb/OwnCord.git synced 2026-09-03 03:50:00 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
dev
OwnCord/docs/architecture/ux/voice-and-e2ee.md
T
J3vbandClaude Fable 5 1fe3df7962 test(b2-1): capture the epoch-1 protocol fixtures and retire S-15 (#1435) 
* refactor(protocol): retire reserved voice_speakers and member_leave (S-15)

Neither type was ever emitted by the server; B2-1 clears them from the
schema before the epoch-1 wire fixtures are captured, so the frozen epoch
does not carry two dead message types.

Client: dropped the dead `ws.on(MEMBER_LEAVE)` / `ws.on(VOICE_SPEAKERS)`
dispatcher handlers, the `MemberLeavePayload` type and both `ServerMessage`
union members, and the tests that only exercised those WS paths.
`removeMember` (member_ban) and `setSpeakers` (LiveKit ActiveSpeakers) stay
live and keep their direct unit tests.

* test(ws): capture the epoch-1 wire fixtures

alpha.4 is the last client on the pre-epoch wire and B2-2 adds a protocol
epoch to the auth handshake next, so record what epoch 1 actually looks
like while it is still observable.

TestEpoch1Fixtures drives eleven journeys through the ws package's
in-process hub harness (full migrations, real hub, httptest WebSocket
server) and compares each journey's per-connection frame sequence with a
transcript under protocol/fixtures/epoch-1/:

- fresh-connect, auth-failure, ping
- chat-send-fanout, chat-edit-delete, reaction-add-remove
- typing, mark-read, dm-send
- resume-replay (last_seq + buffer-tier replay burst)
- voice-join-e2ee-leave (join, both voice_state forms, announce and
  offer relay, leave)

Volatile values are replaced by typed placeholders before both writing
and comparison -- any key that is id/seq/last_seq, ends in _id (except
channel_id and role_id) or _at, is timestamp/ts/last_seen, or contains
token, becomes "<class:json-type>" so a field that changes type is still
a diff, while everything else is compared verbatim.

Regenerate with: go test ./ws -run TestEpoch1Fixtures -update

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(client): pin the epoch-1 auth frame contract

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(updater): pin the epoch-1 manifest and client-update shapes

* test(ws): harden the epoch-1 fixtures against trailing frames and absent optionals

Every journey now ends with the ping/pong barrier on each recorded
connection, so a frame the server emits after the last read fails as
`expected "pong", got "X"` instead of going unrecorded; auth-failure
asserts the StatusPolicyViolation close instead, its socket being gone.

alice carries a display name, avatar, about text, custom status, identity
public key and an announce signature (bob carries none), so every optional
field is frozen in its present form as well as its absent one — a rename or
a retype of display_name or identity_public_key now moves a fixture.

The typing journey focuses the channel on "a" before typing: without the
subscription registerNow only makes for a focused client, its ping/pong
proved nothing about excludeUserID.

Comment fixes: the escaped placeholder form MarshalIndent would write, the
real (headroom) reason for the raised read limit, a note that bare id and
active_channel_id are normalised by design, and a .prettierignore line
saying these fixtures are verified by the Go comparison, not by git diff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(protocol): document the epoch-1 fixtures

* docs(protocol): match the epoch-1 wire where the fixtures contradicted the doc

- voice_state: note the unsequenced relay-to-joiner form (table row + section)
- auth_error: correct the example message and list the real rejection messages
- member_join/presence connect example: add the seq both frames actually carry
- chat_message.user: document display_name
- voice_join reply order: state that the joiner's own voice_state broadcast is not ordered against the other three frames

* test(ws): freeze the null forms of auth_ok and member_join user fields

buildAuthOK emits display_name/about/custom_status/avatar as
always-present nulls, but only alice — who has all four set — ever
authenticated on a recorded connection, so the fixtures froze those
fields in their populated form alone. A rename, a retype or a dropped
null would have moved nothing, on the very frame B2-2 edits.

fresh-connect now records bob's handshake too, on a second connection:
his auth_ok carries the four nulls, his member_join carries avatar null
with display_name and identity_public_key omitted, and alice — idle by
then, so her reads stay in hub order — records the same pair as an
already-connected observer sees it. In voice-join-e2ee-leave bob answers
alice's signed announce with a legacy unsigned one, which freezes the
absent form of signature next to her present one. The auth frames the
test writes now carry the correlation id the real client stamps on every
frame (ws.ts send()); normalisation renders it <id:string>.

expectClosed also asserts the close reason ("authentication failed"),
not just code 1008 — HP-2 asks for both.

Comment precision, no behaviour change: the barrier guarantee now states
that pong may overtake a pending LOW-priority frame (writePump) and that
no journey is affected because every barrier is sent on an idle
connection; the ping-budget ceiling is six connections, not four; the
typing journey cites handleChannelFocusV2 rather than registerNow as the
subscribe site; resume-replay's b barrier explains why moving it past
the resume would be a flake, not a fix; and the header notes that
normalisation hides that chat_send_ok.id echoes the request id.

The client contract test's cited range for ws.ts's send() call is
441-453, not 441-454.

Regenerate with: go test ./ws -run TestEpoch1Fixtures -update

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(protocol): shape-not-value fixture rule, voice_max_video default, auth close code

The fixture rule read "a fixture may only change with an epoch bump",
which is false for the seeded values the transcripts record verbatim:
role permission masks and colours, motd, server_name, voice_max_video,
the voice_config preset. A migration that changes a default mask diffs
fresh-connect.json, and the README told the author to revert a change
that never touched the wire.

Split the rule along shape versus value. A key set, a JSON type, a key
appearing or disappearing, or per-connection frame order is a protocol
change and earns fixtures/epoch-<n+1>/. A seeded default value is a seed
change: regenerate in the same PR and read the diff frame by frame.
Normalising those values is explicitly not the answer — a placeholder
over a mask or over an enum such as voice_config.threshold_mode would
hide the drift the fixtures exist to catch.

Two wire facts corrected against the fixtures:

- voice_max_video on an unconfigured channel is 25, not 0 (migration
  004 is DEFAULT 25); the doc listed it among the zero values.
- auth_error is followed by a close with code 1008 (policy violation)
  and reason "authentication failed" (serve.go:128), which the doc left
  as "closes the connection".

Also: voice_speakers moves from discord-parity's "still dead" list to
"came off the list" — it was retired earlier on this branch. And Kick
says sessions are revoked and sockets drop on the next sweep, which is
what ForceLogout does (moderation.go:236); it does not cut sockets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(protocol): fixture rule covers enum vocabularies; header comment precision

Re-check minors from the whole-branch review: a fixed-vocabulary value the client switches on is shape, not a seeded value; drop the false 'new default channel' clause (the seed asserts channel ids); the low-priority frame is presence_update, not connect presence.

* docs(plans): record B2-1 evidence for HP-2 (PR #1435, fixture commit SHAs)

* docs(protocol): additive changes stay within an epoch; bump only for what old clients cannot process

Codex review on #1435: B2-2 keeps protocol_epoch = 1 while adding auth/ready/auth_error fields, which the previous wording would have called a break to revert. An epoch is a compatibility boundary, not a snapshot: additive keys regenerate in the same PR and are documented; removals, renames, retypes, dropped frames and reordering bump the epoch. The plan's B2-1 evidence records the refinement and hands B2-2/B2-4 the open questions (additive-tolerant replay of the epoch-1 transcript; epoch 0 vs 1 naming).

* test(client): compare auth-frame key sets order-independently

Codex review on #1435: Object.keys preserves insertion order, so a harmless property reorder in ws.ts would fail the pin. Key order has no wire meaning; the Go fixtures already compare with sorted keys. Sort both sides.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-28 15:43:30 +02:00

221 lines
16 KiB
Markdown
Raw Permalink Blame History

# Voice, Video & E2EE — target UX
**Verified against:** commit `5630aa1`, 2026-08-04
Part of the [Client UX Specification](README.md). The signaling/crypto mechanics
are mapped structurally in [../voice-e2ee.md](../voice-e2ee.md); this document
specifies the **user-facing** states and reactions.
Covers: joining/leaving voice, mute/deafen/camera/screenshare, push-to-talk, the
active-speaker display, and — the main gap — the E2EE "securing / secured"
indicators.
---
## 1. Two state machines, one status
Internally there are **two** FSMs:
- The **WS connection** FSM (`ws.ts`: `disconnected…connected`) — the socket.
- The **voice session** FSM (`livekitSession.ts`: `idle | connecting |
connected | reconnecting`) — the LiveKit room.
Plus the user-facing booleans in `voice.store` (`localMuted`, `localDeafened`,
`localCamera`, `localScreenshare`, `listenOnly`, `joinedAt`) and the per-user
roster (`voiceUsers` with per-user `speaking/muted/deafened/camera/screenshare`).
**Target:** expose the voice session as one observable `voiceStatus` the widgets
read — `idle | joining | securing | connected | reconnecting` — rather than
inferring it from `isVoiceConnected()` alone.
> **✓ Implemented (2026-07).** `voice.store.voiceStatus`
> (`idle | joining | securing | connected | reconnecting`) is now the observable
> voice-session status. `livekitSession.ts` is the single writer: `joining` at the
> start of `connectAndSetup`, `securing` when the ECDH key exchange begins,
> `connected` on the atomic `connected` transition (both the initial join and a
> successful auto-reconnect), `reconnecting` when the room drops and the reconnect
> loop forms its state, and `idle` on `leaveVoice`. `joinVoiceChannel` seeds
> `joining` optimistically on click so the widget reacts before the `voice_token`
> round-trip. The VoiceWidget reads it to distinguish "connecting to the room"
> from "securing the encryption" from "reconnecting". `failed` is not a persisted
> status: an E2EE-timeout / connection error auto-leaves to `idle` and surfaces a
> toast via `onErrorCallback` (§2).
---
## 2. Join / leave
```mermaid
stateDiagram-v2
idle --> joining: click voice channel → voice_join → voice_token
joining --> securing: room.connect ok, E2EE key exchange begins
securing --> connected: room key ready (holder generates / member receives)
securing --> failed: e2ee_timeout (no key within ~15s)
joining --> reconnecting: transient connect failure (retry ≤3)
connected --> reconnecting: socket/room drop
reconnecting --> connected: re-announce key + rejoin (≤2 attempts)
reconnecting --> failed: attempts exhausted
connected --> idle: leave
failed --> idle: auto-leave + error
```
| Status | Presentation | Notes |
| -------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `joining` | Voice widget shows "Connecting…"; channel roster shows self pending | `handleVoiceToken` → `connectAndSetup` |
| `securing` | "Securing connection…" indicator (lock, in-progress) | Non-key-holders block here until a room key arrives (10 s + 5 s retry, the "securing" key-exchange block in `connectAndSetup` (`lib/livekitSession.ts`) / `E2EEManager.setupKeyExchange` (`lib/livekitE2EE.ts`)) |
| `connected` | "Voice connected · secured 🔒" + elapsed timer (from `joinedAt`) | E2EE active; per-user tiles live |
| `reconnecting` | "Reconnecting voice…"; controls frozen, not torn down | Keypair regenerated for forward secrecy (`attemptAutoReconnect()` → `reannounceForReconnect()`, `lib/livekitSession.ts`) |
| `failed` | Toast "Voice connection lost" / "Couldn't secure the call"; auto-leave | `onErrorCallback` fires |
**Target rules:**
- The "connecting" vs "securing" distinction is user-visible: while a non-key-holder
waits for the room key, show **securing**, not a generic spinner — an E2EE call
that's still exchanging keys is not yet private.
- Leaving is immediate and local (`leaveVoice`): tear down tracks, clear E2EE
state, reset camera/screenshare, `idle`.
> **✓ Implemented (2026-07).** The VoiceWidget header now renders the E2EE phase
> from `voiceStatus`: a "Securing…" label (amber) while the key exchange runs and
> a persistent "🔒 Secured" badge once the room key is ready and the room is
> connected — replacing the log-line-only feedback. `joining` shows "Connecting…"
> and `reconnecting` shows "Reconnecting voice…", neither showing the secured
> badge. An E2EE-timeout still surfaces its `"e2ee_timeout"` toast and auto-leaves
> (`livekitSession.ts` `connectAndSetup`). **Code vs. diagram note:** the client
> actually runs the ECDH key exchange _before_ `room.connect()`, so `securing`
> spans the key wait and the media connect; the state diagram below draws them in
> the reverse order for readability. The distinction users see is unchanged:
> non-key-holders sit in `securing` until a room key arrives.
---
## 3. Local controls
All four are optimistic with rollback; each also emits a WS control message.
| Control | Local state | WS message | Rollback |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------- |
| **Mute** | `localMuted` (`setLocalMuted`) — fully unpublishes the mic track | `voice_mute{muted}` | n/a (local-authoritative) |
| **Deafen** | `localDeafened` + forces mute — unsubscribes remote _voice_ audio only; screen-share/stream audio keeps playing (it has its own per-tile mute/volume) | `voice_deafen` + `voice_mute` | implies mute |
| **Camera** | `localCamera` set optimistically, rolled back on device failure (`enableCamera()` in `lib/screenShare.ts`) | `voice_camera{enabled}` | revert on failure + toast |
| **Screenshare** | `localScreenshare` optimistic, rollback on failure (`enableScreenshare()` in `lib/screenShare.ts`); rate-limited | `voice_screenshare{enabled}` | revert + toast |
| Control state | Presentation |
| -------------- | ------------------------------------------------------------------------------------------ |
| mic muted | Mic-slash icon on self tile + control bar |
| deafened | Headphone-slash; implies muted styling |
| listen-only | Badge "Listen only — no microphone" with a **Retry mic** affordance (`retryMicPermission`) |
| camera on | Self video tile in the grid |
| screenshare on | Screen tile; a stop-share affordance always visible |
| speaking | Green ring on the speaking user's tile/avatar (from LiveKit's ActiveSpeakers) |
**Mic-permission failure** (`restoreLocalVoiceState`): on denied/absent mic, set
`listenOnly` and surface the specific reason ("Microphone permission denied" /
"No microphone found") as a toast with a retry — already wired to
`onErrorCallback` (the mic-unavailable branches of `restoreLocalVoiceState()`, `lib/livekitSession.ts`); the spec makes the **Retry mic**
control a permanent part of the listen-only badge.
---
## 4. Push-to-talk
PTT is a Rust key-poller (`ptt.rs`, 20 ms) emitting `ptt-state{pressed}` →
`setMuted(!pressed)` only while in a channel (the `ptt-state` listener inside `initPtt()`, `lib/ptt.ts`). **Target UX:**
| State | Presentation |
| ------------------- | --------------------------------------------------------------------------------------------------------------------- |
| PTT bound, released | Muted; hint "Hold {key} to talk" |
| PTT pressed | Unmuted + speaking ring |
| binding a key | Keybinds tab: "Press a key…" (10 s capture window, `ptt_listen_for_key`); reject text keys with "Pick a non-text key" |
| PTT thread error | Toast "Push-to-talk stopped unexpectedly" on `ptt-error`, offer re-enable |
---
## 5. Voice roster (per channel)
The channel's voice roster renders from `voiceUsers`. Each participant tile
reflects their `speaking/muted/deafened/camera/screenshare`. **Target:**
| Signal | Tile reaction |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `voice_state` | Add/update the participant with their flags |
| `voice_leave` | Remove the tile; if it's us (kick/disconnect), clear local voice state (already the `voice_leave` handler in `wireDispatcher()`, `lib/dispatcher.ts`) |
| key-holder change | Invisible to users (re-election is automatic on leave); no UI churn |
Per-user volume is adjustable and persisted (`userVolume_{id}` in the Rust store).
---
## 6. Token refresh & reconnect (invisible)
Token refresh (23 h timer) and voice reconnect (≤2 attempts, 3 s apart) should be
**invisible on success**. Only exhaustion surfaces: "Voice connection lost —
failed to reconnect" + auto-leave. The 60 s token-refresh response guard and the
forward-secrecy keypair rotation on reconnect are mechanics the user never sees.
---
## 7. E2EE identity verification surface
Peer identity state lives in `voice.store` (per-participant
`status: verified | unverified | mismatch` + `safetyNumber`), written by
`lib/livekitE2EE.ts` as announces are verified against the pinned identity
keys (`lib/identity.ts`).
| State | Roster badge (`verifyPresentation()`, `components/ChannelSidebar.ts`) | Interaction |
| ------------ | --------------------------------------------------------------------------- | ---------------------------------------- |
| `verified` | Green shield; title "Identity verified · Safety number: {n}" | none needed |
| `unverified` | Neutral shield; no pinned key yet | none — pins on first verified announce |
| `mismatch` | Red shield-alert; title "Identity key changed — click to review and re-pin" | Click → blocking identity-mismatch modal |
The mismatch modal (`createIdentityMismatchModal()`, `components/CertMismatchModal.ts`;
opened from `openIdentityMismatchModal()` in `components/ChannelSidebar.ts`) shows the **new key's fingerprint** so
the user can verify it out-of-band before trusting. "Trust New Key" re-pins
via `rePinPeerIdentity` — deliberately pinning the exact key whose fingerprint
was displayed, not a fresh store read, so a malicious server cannot swap the
key during the human verification window (TOCTOU). Reject leaves the peer
blocked for E2EE media. A stripped or malformed published key disables the
trust action entirely (a blind accept is refused).
## 8. Media processing & devices
- **Noise suppression:** RNNoise WASM worklet (`lib/noise-suppression.ts`,
assets `public/rnnoise.wasm` + `public/rnnoise-worklet.js`), toggled in
Settings → Voice & Audio; falls back to a ScriptProcessorNode pipeline when
AudioWorklet is unavailable (`createScriptProcessorPipeline()` in `lib/noise-suppression.ts`).
- **Input volume & VAD:** `lib/audioPipeline.ts` applies input gain and
voice-activity gating ahead of publish.
- **Device hot-swap:** `lib/deviceManager.ts` follows OS device
plug/unplug and re-routes the active input/output without rejoining.
- **Stream preview:** `lib/streamPreview.ts` renders the pre-share preview in
the screen-share picker.
## 9. DM calls (ring)
DM voice is the same voice machinery on the DM's voice channel, plus a ring
layer (no server-side call state — presence in the DM voice channel _is_ the
call):
| Event | Reaction |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Outgoing: user clicks Call | `call_ring` sent (rate-limited 1/3 s server-side); caller joins the DM voice channel |
| Incoming: `call_incoming` | `components/IncomingCallBanner.ts` banner + ring chime (`lib/notifications.ts`), driven by the `lib/call-ring.ts` state machine (30 s auto-timeout) |
| Accept | Join the DM voice channel; banner clears |
| Decline | `call_decline` sent → other participants' ringing stops via `call_declined` |
| Timeout / caller leaves | Banner clears silently |
`call_incoming` / `call_declined` are page-scoped listeners in `MainPage.ts`,
not dispatcher handlers (see [README §4](README.md)).
---
## Source of truth
`src/lib/livekitSession.ts`, `src/lib/livekitE2EE.ts`,
`src/stores/voice.store.ts`, `src/lib/screenShare.ts`,
`src/lib/ptt.ts`, `src/lib/roomEventHandlers.ts`, `src/components/VoiceWidget.ts`,
`src/components/ChannelSidebar.ts` (voice rows, join freeze on WS reconnect,
and the E2EE verification badge), `src/components/VideoGrid.ts`,
`src-tauri/src/livekit_proxy.rs`, `src-tauri/src/ptt.rs`,
`src/lib/e2eeCrypto.ts`, `src/lib/identity.ts`; and the structural map in
[../voice-e2ee.md](../voice-e2ee.md).
Reference in New Issue View Git Blame Copy Permalink