2 Commits

Include renames

Author	SHA1	Message	Date
ConnorYohandJames Brunton	f60a75c253	chore(saas): remove unused Flyway migration system (#7100) ... ## What this PR does Removes the **Flyway** migration system from the SaaS build: - drops `flyway-core` + `flyway-database-postgresql` from `app/saas/build.gradle` - deletes all `Vxx__*.sql` files under `app/saas/src/main/resources/db/migration/` - removes the `spring.flyway.*` config from `application-saas.properties` - clears the now-inert `SPRING_FLYWAY_ENABLED` override and stale Flyway comments in `testing/compose/docker-compose-saas.yml` `ddl-auto` is **left on `update`** (unchanged) — that's a separate decision (see "Not in this PR"). ## Why — Flyway never actually ran anywhere From a full schema-management review (`notes/FLYWAY_MIGRATION_REVIEW.md`), verified against the live databases: - **Prod & dev (v3):** no `flyway_schema_history` table exists in any schema → Flyway has never executed. Schema is authored by the Supabase migrations in `Stirling-PDF-SaaS` and applied by that repo's **GitHub integration** (merge to `main` → prod). - **Tests:** the saas module has zero `@SpringBootTest`; the only real-DB integration tests (in `proprietary`) use `ddl-auto=create-drop` and don't have Flyway on the classpath. - **The mock-DB harness** (`testing/compose/docker-compose-saas.yml`, PAYG cucumber) *explicitly disabled* Flyway, because the `Vxx` migrations can't run against a clean Postgres — they assume Supabase has already provisioned `users`/`teams` (`V2` ALTERs `users`, `V5` references `teams`). So Flyway was dead weight, and its 4 duplicate versions (`V25/V26/V30/V31`) were a latent trap: re-enabling it would crash boot on the collision. Everything it contained (schema + seeds like the default pricing policy) is already mirrored by the Supabase migrations, and by `saas-seed.sql` for the cucumber stack. ## ⚠️ Required follow-up (item #1) — capture the Flyway-only tables into Supabase migrations **This is documentation of the next step, not done in this PR.** Seven tables were defined in Flyway with **no matching Supabase migration**. They exist in prod today only because `ddl-auto=update` created them from their entities. Before `ddl-auto` is ever tightened to `validate` (see #3 below), and so any fresh Supabase branch is complete, they must be added as Supabase migrations in `Stirling-PDF-SaaS/supabase/migrations/`. **Capture (CREATE) — 6 live tables** (definitions are visible in the deleted files in this PR's diff): | Table | Source (deleted here) | Backing entity | |---|---|---| | `resource_grants` | `V25__resource_grants.sql` | `ResourceGrant` | | `integration_configs` | `V26__integration_configs.sql` | `IntegrationConfig` | | `policy_sources` | `V22__policy_engine_tables.sql` | `SourceEntity` | | `policy_source_doc_counts` | `V23__policy_source_doc_counts.sql` | `SourceDocCountEntity` | | `policy_source_doc_totals` | `V23__policy_source_doc_counts.sql` | `SourceDocTotalEntity` | | `saas_user_extensions` | `V9__saas_user_team_extensions.sql` | `SaasUserExtensions` | Write them as `CREATE TABLE IF NOT EXISTS stirling_pdf.<name> (...)` (idempotent — no-op against the existing prod/v3 tables). Preserve column types/defaults/constraints from the deleted `Vxx` files. **Drop (do NOT recreate) — 1 orphaned table:** - `classification_labels` — created by `V30` and dropped by `V39` within Flyway; its `ClassificationLabel` is now a plain `record`, not a JPA entity. It lingers in prod only because Flyway's `V39` drop never ran. The follow-up should emit a `DROP TABLE IF EXISTS stirling_pdf.classification_labels` (mirroring `V39`'s intent), after confirming nothing reads it. ## Not in this PR (deliberately) - **#3 — flip saas `ddl-auto` `update` → `validate`.** Held pending team confirmation; it has boot-risk and should be gated by a CI "validate-boot against a fresh Supabase branch" first. Self-hosted stays on `update` regardless. - **#4 — `billing_subscriptions` split-brain** (prod `public` has 23,164 rows, `stirling_pdf` has 0, Java reads the empty one). Tracked separately. ## Verification - `:saas:compileJava` succeeds with Flyway removed (no code imports `org.flywaydb.*`). - No test or ArchUnit rule references Flyway or the migration files. - No runtime/data impact: Flyway never ran against any live database. --------- Co-authored-by: James Brunton <jbrunton96@gmail.com>	2026-07-21 09:32:56 +00:00
ConnorYoh	ff96a80947	PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow (#6522) ... ## What this PR is End-to-end cucumber coverage for the PAYG shadow charging engine (the filter + interceptor stack from #6519), wired into CI via a new `docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching PRs. Stacked on #6519. ## Automated scenarios (run by `docker-compose-tests-saas.yml`) See [`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature): | Scenario | Validates | |---|---| | First tool call writes a CHARGED row | Filter + interceptor fire end-to-end | | Lineage join — second call on output | `JobService.joinOrOpen` matching; no new shadow row | | 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics | | ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor` unpacks + records signatures | | Multi-file input writes a single shadow row | Multi-input group sizing | | `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource` detection | All 6 run locally via `./testing/test-payg.sh` and will run on CI for any PR that touches `app/saas/**`, the PAYG cucumber features, the saas compose stack, or the workflow itself. ## Manual-only scenarios — documented in design doc, not in this suite Two parts of the shadow engine are deliberately not automated; the engine paths are unit-tested in `PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual procedures (which require a temporary throw endpoint or a container restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2 "PAYG cucumber: manual-only scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md). - **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing endpoint exists; manual procedure adds a throw endpoint, runs, asserts, removes. - **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container restart mid-suite; manual procedure tears down, flips env, brings up, asserts zero shadow rows. If either gets a hot-reload path (test-only throw endpoint shipped behind a profile gate, or admin endpoint for the kill switch), automate it in a follow-up and drop the manual procedure. ## CI workflow `.github/workflows/docker-compose-tests-saas.yml` (new) — self-contained, not wired into `build.yml`'s `files-changed` matrix so the saas-cucumber job fails and succeeds independently. Triggers only on PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have the coverage override; can add later). ## Test infrastructure (recap) - **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf` schema. Supabase JWT auto-config disabled; API-key auth via `SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests exercise. - **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** — schema bootstrap + idempotent seed (team / user / wallet_policy). - **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6 scenarios above. - **`testing/cucumber/features/steps/payg_step_definitions.py`** — step defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer. - **`testing/test-payg.sh`** — companion runner to `testing/test.sh`. Brings up the saas compose, waits for health, seeds, runs behave, tears down. - **`behave.ini`** excludes `features/payg` from the default behave run (the saas-cucumber CI job invokes it explicitly). ## Why a separate harness from `testing/test.sh` The existing `test.sh` covers the proprietary-flavour stack (no PAYG tables, no saas profile). Coupling two CI matrices that fail and succeed independently into one script is asking for trouble. Keep the saas-cucumber job focused on its own concerns; once the harness is mature, the wider team can decide whether to merge them. ## Tracked in `notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios).	2026-06-09 14:47:40 +00:00