|
|
|
@@ -1,113 +0,0 @@
|
|
|
|
|
# Phase A Implementation Status
|
|
|
|
|
|
|
|
|
|
**Branch:** `claude/phase-a-foundation-plan-eUys1`
|
|
|
|
|
|
|
|
|
|
This document tracks what shipped during Phase A (Foundation) and what is still pending. The original Phase A design brief (`phase-a-foundation.md`) was removed from the repo root on `dev`; this status doc preserves the actionable follow-up work.
|
|
|
|
|
|
|
|
|
|
## Done
|
|
|
|
|
|
|
|
|
|
- **Step 1 — Service layer + permission cache.** `Server/service/` contains 12 service files covering message, channel, permission, moderation, user, dm, block, invite, voice. Services depend on `store.Store`, not `*db.DB`. PermissionService maintains a per-user cache. REST and WS handlers (auth, channel, dm, profile, upload, chat, reaction, presence, voice) are migrated. Service tests live in `service/message_test.go` and `service/permission_test.go`.
|
|
|
|
|
- **Step 2 — sqlc adoption.** `Server/sqlc.yaml` configures both the SQLite engine (queries in `Server/db/queries/sqlite/`, generated output in `Server/db/dbgen/`) and the PostgreSQL engine (queries in `Server/db/queries/postgres/`, output in `Server/db/pgdbgen/`). sqlc v1.30.0 is pinned in `Server/sqlc.version`. `Server/Makefile` exposes `sqlc-install`, `sqlc-generate`, `sqlc-verify` targets covering both engines. 14 SQL query files per engine cover all DB domains; the SQLite `dbgen` package is committed, the PostgreSQL `pgdbgen` package will be generated on the next `make sqlc-generate` run. FTS search queries remain hand-written; transactional multi-step operations are unchanged.
|
|
|
|
|
- **Step 3 (partial) — Store interface + SQLiteStore + MemStore.** `Server/store/store.go` defines the full Store interface (12 sub-interfaces). `Server/store/sqlite.go` wraps `*db.DB`. `Server/store/memstore.go` provides an in-memory implementation used by service tests.
|
|
|
|
|
- **Step 4 — Logging consolidation.** OwnCord code uses `log/slog` exclusively. `go.uber.org/zap` and `rs/zerolog` appear only as transitive dependencies of livekit and are not imported by any OwnCord `.go` file. The plan's "pick one and find-and-replace" item is satisfied.
|
|
|
|
|
- **Step 5 — Pub/sub broadcast model.** `Server/ws/pubsub.go`, `topic_rate_limiter.go`, `ringbuffer.go`, and the three-tier priority queue (commit `7ccc93f`) implement the topic-based broadcast model with global rate limits, backpressure, and priority tiers.
|
|
|
|
|
|
|
|
|
|
## Pending
|
|
|
|
|
|
|
|
|
|
### Step 3 — PostgreSQL backend (final wiring)
|
|
|
|
|
|
|
|
|
|
Scaffolding has landed on this branch:
|
|
|
|
|
|
|
|
|
|
- `Server/migrations/postgres/001_initial_schema.sql` — consolidated postgres schema with `tsvector` + GIN full-text search, `CITEXT` usernames, native CHECK constraints replacing SQLite triggers, and seeded role/setting rows.
|
|
|
|
|
- `Server/migrations/postgres/migrations.go` — embed FS for the postgres migration set.
|
|
|
|
|
- `Server/db/queries/postgres/*.sql` — 14 query files (users, sessions, roles, invites, channels, messages, reactions, voice, attachments, admin, dm, blocks, lockouts, profile) translated to postgres dialect: `$N` placeholders, `NOW()`/native `TIMESTAMPTZ`, `TRUE`/`FALSE` for BOOLEAN columns, `ON CONFLICT ... DO UPDATE` for upserts, `RETURNING id` for creates (postgres has no `LastInsertId`), `:execrows` for mutations that need rows-affected.
|
|
|
|
|
- `Server/sqlc.yaml` — second engine entry (`engine: "postgresql"`, `sql_package: "pgx/v5"`) generating into the `pgdbgen` package under `Server/db/pgdbgen/`.
|
|
|
|
|
- `Server/Makefile` — `sqlc-generate` and `sqlc-verify` cover both engines.
|
|
|
|
|
- `Server/store/postgres.go` — `PostgresStore` type behind the `//go:build postgres` tag, implementing the full `store.Store` interface. Connection lifecycle (`OpenPostgres`, `Close`, `SQLDb`, `WithTx`) is fully implemented using `database/sql` + the pgx stdlib driver. Query methods are stubs that return `ErrPostgresNotImplemented`, waiting for the `pgdbgen` querier to land so they can be replaced with wrappers around generated code. The build tag keeps pgx out of the default build; operators who want to enable postgres run `go get github.com/jackc/pgx/v5 && go build -tags postgres ./...`.
|
|
|
|
|
- `Server/config/config.go` — `DatabaseConfig` extended with `Type`, `Host`, `Port`, `User`, `Password`, `Name`, `SSLMode`, `MaxConns`. Defaults set to `type: "sqlite"` so existing operators are unaffected.
|
|
|
|
|
- `Server/main.go` — explicit dispatch on `database.type`. Selecting `postgres` produces a clear startup error pointing at the remaining work, instead of silently falling through to sqlite.
|
|
|
|
|
|
|
|
|
|
What still needs to land for postgres to be runnable:
|
|
|
|
|
|
|
|
|
|
1. Add `github.com/jackc/pgx/v5` to `go.mod` and run `go mod tidy`. This happens naturally the first time an operator runs `go get github.com/jackc/pgx/v5` — the package only needs to be in the module graph when building with `-tags postgres`.
|
|
|
|
|
2. Run `make sqlc-generate` to produce `Server/db/pgdbgen/` from the committed query files. This requires either network access to download sqlc or a pre-installed `sqlc` binary at v1.30.0.
|
|
|
|
|
3. Replace the stub query methods in `Server/store/postgres.go` with real implementations that wrap the generated `pgdbgen` querier. The connection lifecycle and interface assertion are already in place; each stub carries the same method signature as the sqlite version, so the migration is mechanical. Where the schema produces different Go types than sqlite (e.g. `bool` vs `int64` for boolean columns, `time.Time` vs `string` for timestamps), the store wrapper performs the conversion so services see a uniform API.
|
|
|
|
|
4. Refactor `main.go` and `Server/api/router.go` to thread `store.Store` through the boundary instead of `*db.DB`. Today the router constructs `dbstore.NewSQLiteStore(database)` inline, so services are store-aware but everything else still consumes `*db.DB` directly. The store-everywhere migration is the gating step before either backend can be swapped at runtime.
|
|
|
|
|
5. FTS dispatch in `MessageStore.SearchMessages` / `SearchMessagesInChannels` — sqlite uses `MATCH` against the FTS5 virtual table, postgres uses `@@ to_tsquery(...)` against the `messages.fts` tsvector column. Both remain hand-written (outside the sqlc-generated set) for their respective backends.
|
|
|
|
|
6. CI matrix to run the test suite against both backends.
|
|
|
|
|
|
|
|
|
|
### One-way sqlite → postgres data migration
|
|
|
|
|
|
|
|
|
|
Operators who start a community on the default sqlite backend and later outgrow it must be able to carry their history over. The migration must be forward-only (once postgres is selected, the server stays on postgres unless the operator deliberately wipes the postgres database and re-initialises), both to keep the contract simple and to avoid the support burden of "I reverted to sqlite yesterday and now my data is gone".
|
|
|
|
|
|
|
|
|
|
Proposed design:
|
|
|
|
|
|
|
|
|
|
- A one-shot CLI flag, e.g. `chatserver --migrate-to-postgres`, that exits after completion. No continuous sync.
|
|
|
|
|
- Pre-flight checks: `database.type` in config is already `postgres`; postgres connection succeeds; the target postgres database contains no rows in `users` (or equivalent sentinel) — if it does, refuse to run, printing the exact row count so the operator can confirm they meant to target this database.
|
|
|
|
|
- Open sqlite read-only alongside postgres. Begin one postgres transaction for the entire migration so partial failures roll back cleanly.
|
|
|
|
|
- Copy rows in foreign-key order (`roles` → `users` → `channels` → `channel_overrides` → `messages` → `reactions` → `attachments` → `invites` → `sessions` → `voice_states` → `dm_participants` → `dm_open_state` → `read_states` → `audit_log` → `user_blocks` → `rate_lockouts` → `settings` → `emoji` → `sounds`). Disable the `trg_messages_fts_update` trigger for the bulk copy and re-populate `messages.fts` in a single `UPDATE messages SET fts = to_tsvector('simple', content)` at the end, so FTS doesn't fire per row.
|
|
|
|
|
- Convert types at the boundary: sqlite RFC3339 strings → postgres `TIMESTAMPTZ` via `time.Parse`, sqlite `INTEGER` boolean (0/1) → postgres `BOOLEAN`. Drop the preserved `id` values straight through since both schemas are `BIGINT`-compatible.
|
|
|
|
|
- After copying, reset every postgres sequence with `SELECT setval('<table>_id_seq', COALESCE(MAX(id), 1)) FROM <table>` so subsequent inserts don't collide with migrated IDs.
|
|
|
|
|
- Write a marker row into `settings`: `migrated_from_sqlite = <RFC3339 timestamp>`. On subsequent startups with `type: postgres`, the marker's presence (or simply the presence of a non-empty `users` table) is the signal that the migration has already run and must not be repeated.
|
|
|
|
|
- Uploaded files on disk (`data/uploads/`) are not touched — the migration only moves database rows. Attachments reference filenames, not blobs, so the filesystem copy is a separate `cp -a data/uploads old-host:/new-path` step the operator does out-of-band.
|
|
|
|
|
- **No reverse migration.** The code does not include a postgres → sqlite path. If an operator wants to return to sqlite, they stop the server, restore a sqlite backup, edit `config.yaml` back to `type: sqlite`, and start fresh. This is deliberately inconvenient.
|
|
|
|
|
|
|
|
|
|
This feature is gated on PostgresStore's query methods being real (pending item 3). The migration implementation itself lives in a new `Server/migrate/sqlite_to_postgres.go` file and is invoked from `main.go` before the normal startup path.
|
|
|
|
|
|
|
|
|
|
## Verification status
|
|
|
|
|
|
|
|
|
|
`go build ./...` and `go test ./...` were not run in the session that produced this branch — the development environment lacked network access to fetch the Go 1.25.0 toolchain required by `go.mod`. Manual audit of the touched Go files (the `go-build-check` skill) found no compile errors. The branch should be verified locally by the next operator before merge.
|
|
|
|
|
|
|
|
|
|
## Actionable TODOs
|
|
|
|
|
|
|
|
|
|
Scannable checklist extracted from the prose above. Work top-to-bottom; most items unblock the ones below them.
|
|
|
|
|
|
|
|
|
|
### Verification (do first, cheap)
|
|
|
|
|
|
|
|
|
|
- [ ] Run `go build ./...` in `Server/` to confirm the branch compiles
|
|
|
|
|
- [ ] Run `go test ./...` in `Server/` and fix any regressions
|
|
|
|
|
- [ ] Run `go build -tags postgres ./...` after adding pgx (below) to verify `store/postgres.go` compiles under the postgres tag
|
|
|
|
|
|
|
|
|
|
### Postgres enablement (Step 3 final wiring)
|
|
|
|
|
|
|
|
|
|
- [ ] `cd Server && go get github.com/jackc/pgx/v5 && go mod tidy` — add pgx to the module graph
|
|
|
|
|
- [ ] `make sqlc-generate` — produce `Server/db/pgdbgen/` from the committed `db/queries/postgres/*.sql`
|
|
|
|
|
- [ ] Commit the generated `Server/db/pgdbgen/` output
|
|
|
|
|
- [ ] Replace each stub in `Server/store/postgres.go` with a real implementation wrapping `pgdbgen`; keep the `//go:build postgres` tag. Expect per-method type conversion (sqlite `string` timestamps vs postgres `time.Time`, sqlite `int64` bools vs postgres `bool`)
|
|
|
|
|
- [ ] Implement FTS dispatch: `SearchMessages` and `SearchMessagesInChannels` currently use sqlite `MATCH`; add a postgres branch using `@@ to_tsquery(...)` against `messages.fts`
|
|
|
|
|
- [ ] Remove `ErrPostgresNotImplemented` once every method is real
|
|
|
|
|
|
|
|
|
|
### Store-everywhere boundary refactor (unblocks runtime backend selection)
|
|
|
|
|
|
|
|
|
|
- [ ] Audit every `*db.DB` parameter in `Server/api/` and `Server/ws/`; replace with `store.Store` where possible, using `store.Store.SQLDb()` at the leaves that truly need `*sql.DB` (backups, migrations)
|
|
|
|
|
- [ ] Update `Server/api/router.go`'s `NewRouter` signature: take `store.Store` instead of `*db.DB`
|
|
|
|
|
- [ ] Update every `Mount*Routes(r, database, …)` call site to pass the store
|
|
|
|
|
- [ ] Update `ws.NewHub` and `auth.NewPersistentRateLimiter` to accept `store.Store`
|
|
|
|
|
- [ ] Update `Server/admin/handler.go` (`admin.NewHandler`) the same way
|
|
|
|
|
- [ ] Update `Server/main.go` to construct the store via a factory, `store.Open(&cfg.Database)`, that dispatches on `cfg.Database.Type`
|
|
|
|
|
- [ ] Delete the explicit postgres error in `main.go`'s switch — selecting postgres should Just Work once PostgresStore is real
|
|
|
|
|
- [ ] Fix all handler tests that construct handlers from `*db.DB` — they now take `store.Store` (use `store.MemStore` for unit tests)
|
|
|
|
|
|
|
|
|
|
### Data migration (one-way sqlite → postgres)
|
|
|
|
|
|
|
|
|
|
- [ ] Create `Server/migrate/sqlite_to_postgres.go`
|
|
|
|
|
- [ ] Add `--migrate-to-postgres` CLI flag to `main.go`, parsed before normal startup
|
|
|
|
|
- [ ] Pre-flight: verify `database.type == postgres`, verify target postgres `users` table is empty, refuse with exact row count if not
|
|
|
|
|
- [ ] Implement bulk copy in FK order: `roles → users → channels → channel_overrides → messages → reactions → attachments → invites → sessions → voice_states → dm_participants → dm_open_state → read_states → audit_log → user_blocks → rate_lockouts → settings → emoji → sounds`
|
|
|
|
|
- [ ] Disable `trg_messages_fts_update` during bulk message copy; repopulate `messages.fts` in one statement at the end
|
|
|
|
|
- [ ] Convert types at the boundary: `time.Parse(time.RFC3339, …)` for timestamps, `int != 0` for booleans
|
|
|
|
|
- [ ] Reset every `<table>_id_seq` with `SELECT setval(…, COALESCE(MAX(id), 1))` after copy
|
|
|
|
|
- [ ] Write marker row: `INSERT INTO settings (key, value) VALUES ('migrated_from_sqlite', NOW()::text)`
|
|
|
|
|
- [ ] Wrap everything in one postgres transaction so partial failures roll back
|
|
|
|
|
- [ ] **Do NOT implement a reverse migration** — documented as deliberately unavailable
|
|
|
|
|
|
|
|
|
|
### CI
|
|
|
|
|
|
|
|
|
|
- [ ] Add a postgres job to `.github/workflows/*.yml` that spins up a postgres service container and runs `go test -tags postgres ./...`
|
|
|
|
|
- [ ] Keep the existing sqlite job unchanged
|
|
|
|
|
- [ ] Add a `sqlc-verify` job that runs `make sqlc-verify` to catch stale generated code
|
|
|
|
|
|
|
|
|
|
### Hygiene (optional, do whenever)
|
|
|
|
|
|
|
|
|
|
- [ ] Expand `Server/service/*_test.go` coverage — currently only `message_test.go` and `permission_test.go` exist; add channel, dm, voice, invite, moderation tests using `store.MemStore`
|
|
|
|
|
- [ ] Split the postgres migration file if it grows beyond ~300 lines; for now it's consolidated intentionally
|
J3vb