Reworks the Processor (PAYG) meter to **size-scaled units** while
keeping a true **PDF count** visible and distinct from units, and
replaces the fragile content+time lineage grouping with **explicit
per-run grouping**. Built as one PR across three slices.
> Status: **all three slices committed + verified.** `:saas` payg suite
green (418 tests, 0 failures); FE green (typecheck 0, 1260 tests, lint
0, format 0). Remaining before it takes effect in prod: run the
size-scaled default-policy SQL (below) in the Supabase SQL editor +
attach the $0.01/unit Stripe price.
## Model (what we're implementing)
- **Size scaling**: 1 unit per 50 MiB (bytes only, no page charge, no
cap). *(policy-row config, applied separately via SQL.)*
- **Charge = number of input files**: split (1→N outputs) = 1 charge;
merge (N→1) = N charges. `doc_count` = input files, fixed at open;
joined steps add 0.
- **Grouping by run id, not time**: a pipeline/policy/AI run = one
`run_id`; its tool sub-steps group into one charge (content-lineage
still maps split/merge journeys *within* the run). Two separate runs on
identical bytes = two charges. The 5-min window survives only as a
stale-job janitor.
- **10-tool split kept**: within a run's single-file lineage, an 11th
tool run opens a 2nd charge (step limit 10).
- **Count vs units surfaced**: usage page shows unique PDFs,
per-category (automation/AI/API) counts + units, and how many PDFs hit a
size multiplier with avg units/PDF.
## Slice 1 — run-id grouping (behavioural core)
- `AutomationRunContext` (common) — thread-scoped run id.
- `InternalApiClient` — stamps `X-Stirling-Run-Id`.
- Orchestrators open a run scope **on the worker thread that
dispatches** (async-safe): `PipelineProcessor.runPipelineAgainstFiles`,
`PolicyEngine.runToCompletion` (uses `run.getRunId()`),
`AiWorkflowService.orchestrate`.
- `ChargeContext` + `JobContext`: add `runId`; the charge interceptor
reads the header.
- `JobService.joinOrOpen`: `runId == null` → always open fresh
(standalone never joins); non-null → match scoped to the same `run_id`.
`JpaJobLineageStore`/`JobArtifactHashRepository`: add `run_id` filter to
the match query. Step-limit 10 unchanged.
## Slice 2 — doc_count + document_fingerprint
- V33 migration + entity fields.
- `JobService.openFresh`: set `docCount = inputs.size()`, compute
`document_fingerprint` from input signatures, and denormalise both onto
the DEBIT row in `JobChargeService.recordLedgerDebit`.
## Slice 3 — usage analytics API + FE
- `WalletLedgerRepository`: per-category `SUM(units)` +
`SUM(doc_count)`, `COUNT(DISTINCT document_fingerprint)`, and count of
rows whose units exceed their doc_count (a size multiplier fired), over
the period.
- `WalletSnapshotResponse` + `PaygWalletController`: add `categoryDocs`,
`docsProcessedThisPeriod`, `uniquePdfsThisPeriod`,
`sizeMultiplierPdfsThisPeriod`.
- FE `types.ts` + `PdfsProcessedCard` + `useWallet` + `walletFixtures` +
i18n: headline is the **PDF count**; a summary line shows "{unique}
unique · {units} meter units · {avg} avg units/PDF"; the split bar is
per-category PDF counts; a size-multiplier line shows how many PDFs
scaled. Count is separated from meter units so a 5-unit large PDF reads
as "1 PDF, 5 units".
## Config (out of PR — run in the Supabase SQL editor)
Wrap in one transaction. The partial-unique `is_default` index only
allows one default, so the old default is flipped off **before** the new
one is inserted. The new policy carries the prior default's
`free_tier_units` forward (change the literal if the launch grant should
differ).
```sql
BEGIN;
-- 1) flip default off the current policy + close its effective window
UPDATE stirling_pdf.pricing_policy
SET is_default = FALSE, effective_to = now()
WHERE is_default = TRUE;
-- 2) new default: 1 unit / 5 MiB, no page charge, no scaling cap.
-- free_tier_units carried from whatever the last policy granted (COALESCE→0).
INSERT INTO stirling_pdf.pricing_policy
(version, effective_from, doc_pages_per_unit, doc_bytes_per_unit,
min_charge_units, file_unit_cap, free_tier_units, is_default, notes, created_by)
VALUES
('v2-size-scaled-2026-07', now(),
2147483647, -- doc_pages_per_unit = INT_MAX → pages never drive units
52428800, -- doc_bytes_per_unit = 50 MiB → +1 unit per 50 MiB
1, -- min_charge_units
2147483647, -- file_unit_cap = INT_MAX → no cap on size scaling
COALESCE((SELECT free_tier_units FROM stirling_pdf.pricing_policy
ORDER BY effective_from DESC LIMIT 1), 0),
TRUE, 'Size-scaled: 1 unit/5MiB, bytes only, no cap', 'connor');
-- 3) per-source step limits: standalone ops = own charge; pipelines split at 10
INSERT INTO stirling_pdf.pricing_policy_step_limit (policy_id, job_source, step_limit)
SELECT p.policy_id, s.src, s.lim
FROM stirling_pdf.pricing_policy p
CROSS JOIN (VALUES
('WEB',1),('API',1),('DESKTOP_APP',1),('LINKED_INSTANCE',1),('PIPELINE',10)
) AS s(src, lim)
WHERE p.version = 'v2-size-scaled-2026-07';
-- 4) attach the $0.01/unit Stripe price (you handle the real price id)
INSERT INTO stirling_pdf.pricing_policy_stripe_price (policy_id, stripe_price_id)
SELECT policy_id, 'price_XXXXXXXX'
FROM stirling_pdf.pricing_policy WHERE version = 'v2-size-scaled-2026-07';
COMMIT;
```
Note: the `free_tier_units` subquery reads the most-recent policy
*before* the insert — run it as written (the new row doesn't exist yet
at step 2's SELECT).
## Self-hosted parity — tracked follow-up (not in this PR)
Combined-billing (`stirling.billing.account-link.enabled`) is a
**separate metering engine** (`app/proprietary/accountlink` —
`UsageMeterService`/`LocalUsageService`/`UsageSyncService`). The unit
*math* is shared (`DocumentUnitCalculator`), so size scaling matches
once the policy is pushed. But run-id grouping, `doc_count`, and
fingerprints must be mirrored there, and the usage-sync protocol
extended to report counts/fingerprints, before the self-hosted usage
page shows the same breakdown. Frozen/deferred, so this PR does SaaS;
self-hosted mirrors when it ships.
## What this changes
Moves the admin portal's browser route from **`/portal`** to
**`/processor`**, to match the "Processor" product name (the in-app app
switcher already says "processor").
- `PORTAL_BASENAME` `"/portal"` → `"/processor"` — the single source of
truth for all portal paths on the frontend, so every `toPortalPath(...)`
link and redirect follows automatically.
- `adminRouteExtensions` now mounts at `` `${PORTAL_BASENAME}/*` ``
instead of a hardcoded `"/portal/*"`, so the route can't drift from the
constant.
- **Backend:** `RequestUriUtils.isStaticResource` now treats
`/processor` (not `/portal`) as the SPA shell, so a direct navigation or
hard refresh to `/processor` serves the app instead of 404ing. (This is
the only backend URL reference — there's no Spring Security matcher for
it.)
- Updated the two portal route tests and the doc comments that named the
old path.
**Not changed:** the `@portal/*` import alias — that's the code layer /
flavor-layering path, not the URL. Renaming it would be a much larger,
unrelated churn.
The portal isn't publicly launched yet, so there are no existing links
to preserve — no redirect from the old path is included.
## Testing
- `task frontend:typecheck:all`, full test suite (1,211), lint, format —
green
- `./gradlew :common:test --tests "*RequestUriUtilsTest"` — green
## Overview
Adds **AI document classification** and a **classification-aware Files
sidebar**: uploaded documents are automatically tagged with
document-type labels (Invoice, Contract, Lab report, …), and the sidebar
groups files under editable parent categories so a large library stays
navigable.
> [!IMPORTANT]
> **This feature only runs in the SaaS build.** Classification depends
on the AI engine and team-scoped label storage, so it's gated to SaaS
end-to-end:
> - The sidebar grouping is a `saas/`-layer override of the
`fileSidebarGrouping` seam; every other build (OSS core, self-hosted
proprietary, desktop) gets the null stub and renders the **unchanged
flat, recency-sorted list** — no categories, no "Other", no picker.
> - The classify/labels backend endpoints are gated on
`policies.enabled` (on in SaaS) and live in `app/proprietary`, so
they're absent from pure OSS and dormant in self-hosted unless
explicitly enabled.
> - The Python classifier is reached only via that gated path.
>
> Shared-layer changes that do compile everywhere are inert without the
engine (dormant schema/field additions) or intentional (`GetInfoOnPDF`
surfacing custom metadata).
## What it does
- **Classifier (engine):** reads the first/last two pages of a PDF and
assigns document-type labels from an allowed vocabulary. Labels are
deliberately document-*type* descriptors — no deep-content/PII
detection, since only a page window is read.
- **Team label vocabulary:** ~270 built-in defaults across ~15 families,
seeded per team. Editable by team leaders/admins in the Classification
policy settings (import/export/reset). Team-scoped and shared;
**per-user personal labels are intentionally out of scope** — the
vocabulary is team-level only.
- **Sidebar categories:** files group under parent categories
(Financial, Legal, Medical, …), busiest-first, collapsible, with a
"Recent" group on top and an "Other" group for anything uncategorised.
The category structure (names, icons, membership, custom categories) is
**device-local and user-editable** via a "Customize" picker — the only
per-user personalization; it never changes the team's label vocabulary.
- Classification results are written to PDF metadata
(`StirlingPDFClassification`), read back to keep files in their groups
without re-parsing.
## Architecture
Spans all three layers, mirroring the existing policy/source subsystem
conventions:
- **`frontend/editor`** — sidebar grouping seam + SaaS override,
category manager, labels editor, icon palette, file grouping, tests,
`en-US` i18n.
- **`app/proprietary` + `app/common` + `app/core`** —
`ClassifyLabelController`, team-scoped `ClassificationLabelStore` (Jpa +
in-process impls, same shape as `PolicyStore`/`SourceStore`), metadata
read/write.
- **`engine`** — the document-classifier agent, contracts, routes,
tests.
## Screenshots
**Files sidebar — grouped by category (SaaS)**
### Loading view
<img width="2056" height="1046" alt="Screenshot 2026-07-07 at 5 12
56 PM"
src="https://github.com/user-attachments/assets/1d712da5-50ae-4349-b0cd-e62665c3ec0c"
/>
### Organized in the sidebar
<img width="2056" height="1045" alt="Screenshot 2026-07-07 at 5 14
05 PM"
src="https://github.com/user-attachments/assets/3ea4fe21-da51-4cea-bc3a-18ce040d3d05"
/>
**Customize categories picker**
### Personal settings to change how labels are grouped in an individual
users editor
<img width="2056" height="1044" alt="Screenshot 2026-07-07 at 5 52
42 PM"
src="https://github.com/user-attachments/assets/40be03ce-0f63-4d1e-b58b-cec045d01cb2"
/>
**Classification labels editor (team settings)**
<img width="2056" height="1042" alt="Screenshot 2026-07-07 at 5 53
00 PM"
src="https://github.com/user-attachments/assets/337b0739-15c9-4749-9c6b-22e3b20825b8"
/>
## Testing
- Frontend `task frontend:check` — green (editor + portal tests,
typecheck across all flavors, lint, label-drift guard).
- Backend `task backend:check` (proprietary) and `:saas:test` — green.
- Engine `task engine:check` — green.
## What
Lets the admin portal ("Stirling Processor") ship **inside the JAR**,
gated by a build flag. On `main` the portal already exists as a lazy
`/portal/*` route in the editor but isn't included in production builds
and isn't reachable in a login-enabled server. This PR makes it a
**flag-gated, directly-navigable** part of the editor bundle, and wires
it into the PR preview deployment so it can be tried live.
It keeps the exact architecture `main` uses (portal = a lazy chunk of
the editor, not a separate app), so it inherits all the editor's global
providers/styles and there's no second build to maintain.
## How
**Frontend - gate the existing lazy route**
([`adminRouteExtensions.tsx`](frontend/editor/src/proprietary/routes/adminRouteExtensions.tsx))
```ts
const includePortal = import.meta.env.VITE_INCLUDE_PORTAL === "true" || import.meta.env.DEV;
const PortalApp = includePortal ? lazy(() => import("@portal/PortalApp")) : null;
```
Vite bakes the env to a literal, so when off the dynamic import is
**tree-shaken out entirely** (no `PortalApp` chunk emitted). Always on
in dev. `VITE_INCLUDE_PORTAL` is typed in `vite-env.d.ts` and declared
(default `false`) in `editor/.env`.
**Gradle** ([`build.gradle`](app/core/build.gradle)) -
`-PbuildWithPortal=true` forces `buildWithFrontend=true` and sets
`VITE_INCLUDE_PORTAL=true` on the editor build. Process-env takes
priority over `.env`, so the flag wins for JAR builds while plain `vite
build` / Cloudflare Pages default to off.
**Backend - make the shell reachable**
([`RequestUriUtils`](app/common/src/main/java/stirling/software/common/util/RequestUriUtils.java))
- permits `/portal` + `/portal/*` as public SPA routes. The editor keeps
its JWT in localStorage (not a cookie), so a direct nav/refresh to
`/portal` isn't authenticated at the server and would otherwise redirect
to `/login` and never load. Serving the shell pre-auth (like the editor
root already is) lets it load; **access control is unchanged** - the
portal has its own auth gate + `RequirePortalAccess`, and its data APIs
stay protected.
**Docker** - the embedded Dockerfiles take `ARG BUILD_PORTAL=false` →
`-PbuildWithPortal=${BUILD_PORTAL}`. Default off, so official
`push-docker` images do **not** bundle the portal.
**CI - scoped to the PR preview deploy only**
([`PR-Auto-Deploy-V2.yml`](.github/workflows/PR-Auto-Deploy-V2.yml)) -
the one job that builds the JAR and comments owns all portal wiring:
passes `BUILD_PORTAL=true`, enables the portal's backend features
(`POLICIES_ENABLED`, `STIRLING_BILLING_ACCOUNT_LINK_ENABLED`), and adds
an "Admin portal included" line (linking `/portal` via the direct IP) to
the deployment comment. `push-docker`, `build.yml`, `test-build-docker`,
and the shared paths-filter are untouched.
## Validation (real, in the JAR)
Built and booted the JAR with `-PbuildWithPortal=true` and login
enabled:
- `/portal` and `/portal/users` load via direct nav and render **fully
themed** (dark surfaces, gradients, filled buttons).
- Editor-only build (`-PbuildWithFrontend=true`, no portal flag) →
editor ships, **0 portal chunks** (tree-shaken).
- `-PbuildWithPortal=true` → `PortalApp` chunk present.
Green: `frontend:check:all` (typecheck all variants, lint, format,
build, tests incl. the `VITE_*` env guard), backend compile,
`RequestUriUtilsTest`, spotless.
## Notes
- **Official images never bundle the portal** (Dockerfile default off);
only the PR preview does. Flip `BUILD_PORTAL` / `-PbuildWithPortal` to
include it elsewhere.
- The `/portal` shell being public is the one deviation from `main`, and
it's required for the route to be reachable at all in a login-enabled
server; data access is still fully gated.
# Description of Changes
Redesign policies backend to treat sources a lot closer to how the
frontend imagined them working (they're persistent now and have an API).
Then connect the portal to the sources when mocks are off to allow for
source creation in the UI. It's not particularly useful to do that right
now because there's no policies UI, but I've tested manually that
sources set up in the UI are usable by policies created via the API.
I had to change the portal so that when mocks are off, it doesn't just
hard crash when attempting to connect to all the backend APIs that don't
exist yet. It'll still log the errors, but just continues on rendering
the UI now.
I also changed all the policies backend APIs to be gated behind a flag
instead of behind the SaaS profile. This is because we haven't yet got
the payment model sorted, but we're going to need this stuff running
self-hosted to be able to test it locally.
# Description of Changes
This change centralizes several dependency version declarations into
shared Gradle version properties and updates module build files to
reference those properties instead of hardcoded version strings.
### What was changed
- Added centralized version properties in the root `build.gradle` for:
- commons-io
- commons-lang3
- rhino
- okhttp BOM
- gson
- guava
- bucket4j
- archunit
- batik
- jpdfium
- JWT
- AWS SDK
- Testcontainers
- Replaced hardcoded dependency versions across multiple modules with
shared version variables.
- Updated `resolutionStrategy.force` declarations to use centralized
version properties.
- Updated dependency constraints and BOM references to use shared
version variables.
- Removed module-specific duplicate version declarations from
`app/proprietary/build.gradle`.
- Standardized dependency declarations across `common`, `core`,
`proprietary`, and `saas` modules.
## Why the change was made
- Reduce duplication of dependency version definitions.
- Simplify future dependency upgrades and maintenance.
- Ensure consistent dependency versions across all modules.
- Improve readability and reduce the risk of version drift between
subprojects.
- Make security-related dependency overrides easier to maintain from a
single location.
---
## Checklist
### General
- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings
### Documentation
- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)
### Translations (if applicable)
- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)
### UI Changes (if applicable)
- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)
### Testing (if applicable)
- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
# Description of Changes
> [!warning]
> **Do not** squash this on merge. It should be merged via a merge
commit
Fixes conflicts in `pgvector_store.py`.
Also since codespell is failing, add comments to ignore the errors in
`sync_en_us_spelling.py`
---------
Co-authored-by: Ludy <Ludy87@users.noreply.github.com>
Co-authored-by: Anthony Stirling <77850077+Frooodle@users.noreply.github.com>
## Summary
Pay-as-you-go (PAYG) billing for Stirling-PDF SaaS. Manual PDF editing
stays free forever; only **automation, AI, and API** usage is metered.
Every team gets a **one-time lifetime free grant** (default 500 PDFs)
before any billing; past that, a team adds a card and pays per metered
document, with a self-set monthly spending cap.
This branch combines and supersedes the in-flight BE (#6574) and FE
(#6579) work plus the SaaS edge functions (Stirling-PDF-SaaS PR, now on
`v3`), hardened into a single reviewable feature after a pre-merge
dead-code/security review.
## Billing model
- **Always free:** manual / JWT web-tool usage is `BYPASSED` — never
metered, no matter where it's triggered.
- **Billable categories:** `AUTOMATION`, `AI`, `API`.
- **One-time lifetime free grant** (`pricing_policy.free_tier_units`,
default 500): never resets, survives subscribing. It gates unsubscribed
teams (billable API calls hard-stop with a 402 once exhausted) and
decides the free-vs-paid split of every job.
- **Subscribed:** paid documents (beyond the grant) are metered to a
Stripe Billing Meter; an optional monthly spending cap degrades billable
categories when reached.
- **Dedup:** the same file pushed through several steps within a
workflow window counts **once** (lineage join), so API/AI chaining on
one file isn't double-charged.
## What's included
**Database** — Flyway migrations `V11`→`V21` with matching Supabase
twins: pricing policy + per-team sidecar (`payg_team_extensions`:
subscription id, Stripe customer, free-grant counter), append-only
`wallet_ledger`, shadow charges, subscription-state RPCs (`V14`), audit
logs (`V15`), billing category (`V16`), one-time lifetime free grant
(`V19`), launch-grant seed (`V20`), drop of the unused
`wallet_category_summary` view (`V21`).
**Charge pipeline** — `PaygChargeInterceptor` (open/join a process,
split the free grant, write the ledger DEBIT), `JobChargeService`
(consume the grant under a row lock, restore it on a first-step refund,
meter only the paid portion on completion), `StaleJobCloser` fallback
(idempotent close → meter).
**Entitlement** — `EntitlementService` (per-team cached snapshot:
grant-gated for free teams, monthly-cap-gated for subscribed) +
`EntitlementGuard` (401 `SIGNUP_REQUIRED` / 402 `FEATURE_DEGRADED` /
`PAYG_LIMIT_REACHED`).
**Metering** — `PaygMeterReportingService` writes a durable
`payg_meter_event_log` row around every POST to the `meter-payg-units`
edge fn (pending → posted/failed); `PaygMeterReconcileScheduler` retries
unposted events under the same idempotency key inside Stripe's 24h dedup
window.
**Billing facts** — `TeamBillingService` reads the synced `stripe.*`
mirror (subscription window, per-document rate; the unsubscribed-team
estimate resolves the rate by Price `lookup_key = plan:processor`).
**Wallet API** — `PaygWalletController`: `GET /api/v1/payg/wallet`,
`PATCH /api/v1/payg/cap`.
**Frontend** — PAYG Plan page (two-card free layout + subscribed views),
`useWallet`, upgrade modal with lazy-loaded Stripe Embedded Checkout and
a shared `SpendCapControl`, customer-portal link, 402/401 interceptor
toast, en-GB i18n. (Per-member usage shows each teammate's spend; the
activity feed is behind a flag until polished.)
**SaaS edge functions** (`Stirling-PDF-SaaS` `v3`) —
`create-checkout-session`, `create-payg-team-subscription`,
`create-customer-portal-session`, `meter-payg-units`,
`payg-subscription-webhook`, `stripe-sync`, plus the stripe-sync
`migrate` + scoped-`backfill` scripts. All price lookup is DB-driven (no
`STRIPE_PAYG_PRICE_ID_*` env vars).
## Release prerequisites (prod)
1. Apply Flyway migrations (`V11`→`V21`) and the Supabase migration
twins.
2. Stripe Sync Engine: run `stripe-sync:migrate`, then a **scoped**
backfill — `product`, `price`, `customer`, `subscription` only (not
`all`, which rate-limits).
3. Register 2 PAYG webhook endpoints (each its own signing secret):
`stripe-sync` (product/price/customer/subscription `.*`) and
`payg-subscription-webhook` (`customer.subscription.created`/`.deleted`
drive state; `.updated` + `invoice.*` observed). Keep the legacy
`stripe-webhook` only if credits/self-hosted flows still run.
4. Stripe Billing Meter: `event_name = payg_doc_units`, value key
`processed_documents`.
5. Env: `PAYG_METER_ENDPOINT` + `SUPABASE_EDGE_FUNCTION_SECRET`
(backend); the webhook signing secrets (edge fns). The default pricing
policy must point at the PAYG Stripe Price(s); `V20` seeds
`free_tier_units = 500`.
## Testing
- `:saas:test` green, `:saas:spotlessCheck` clean, edge-fn Deno tests
green, FE saas typecheck clean (the remaining errors are pre-existing
`proprietary/*` + `prototypes/*`, untouched here). Cucumber shadow-mode
suite + CI workflow included.
## Pre-merge review
An independent dead-code/security pass came back **clean on security**
(team-derived authz / no IDOR, leader-only cap mutation, no
billing-category downgrade, dev/mock hooks gated to
`import.meta.env.DEV` + `/dev/`, no secrets/injection, fail-open
metering by design). The dead/unwired code it flagged has been removed
in this branch (unenforced sub-cap control, an unused JDBC DAO + its
view, dead methods).
## Follow-ups (tracked, not blocking)
- **Enforce per-member sub-caps** — the control was removed because it
read for display but never gated a request; the per-member usage display
and `cap_units` column are retained for when enforcement is wired.
- **API/AI chaining billing model + `ProcessType` enum** — confirm
same-file dedup covers API chaining; define per-tool AI charging; decide
whether the unused enum values stay.
- **Activity feed** — hidden behind a flag until the meter-event surface
is polished.
---------
Co-authored-by: Reece <reece@stirlingpdf.com>
Added the create agent. Use [these
prompts](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/blob/main/docgen/backend/default_templates/sample_prompts.md)
to test or try your own :)
Here’s the one I use
```
Hey, I need to generate an employee expense report for reimbursement.
Company: Summit Consulting Partners Company address: 88 Riverside Plaza, Suite 1400, New York, NY 10069 Accounting department email: expenses@example.com
Employee details:
* Employee Name: Michael Tran
* Employee ID: EMP-1047
* Department: Client Services
* Report Date: January 20th, 2026
* Reporting Period: January 5th, 2026 – January 16th, 2026
* Manager Approver: Laura Simmons
Trip purpose: Client onsite meetings with Atlantic Energy Solutions in Boston, MA.
Expense items:
* Flight (NYC to Boston roundtrip) — $325.40 — January 5th, 2026 — Airline ticket
* Hotel (3 nights at Harborview Hotel) — $822.75 — January 5th-8th, 2026
* Taxi from airport to hotel — $48.00 — January 5th, 2026
* Client dinner (3 attendees) — $186.20 — January 6th, 2026
* Parking at JFK Airport — $72.00 — January 5th-8th, 2026
* Breakfast (per diem not used) — $18.50 — January 7th, 2026
* Uber to client office — $22.10 — January 7th, 2026
* Printing + presentation materials — $46.90 — January 8th, 2026
* Lunch with client — $39.75 — January 8th, 2026
* Office supplies (notebooks, pens) — $27.60 — January 10th, 2026
* Mileage reimbursement (client visit in NJ, 42 miles @ $0.67/mile) — $28.14 — January 14th, 2026
* Team lunch meeting (internal) — $64.30 — January 15th, 2026
Reimbursement method should be direct deposit.
Add a notes section stating: "All receipts attached. Expenses are business-related and comply with company travel policy."
```
---------
Co-authored-by: Anthony Stirling <77850077+frooodle@users.noreply.github.com>
# Description of Changes
Add new triggers:
- Schedule (fires every X amount of time)
- Folder watch (fires whenever the OS tells us a folder has a new file
in it; on Mac this is technically a 2s schedule but that's just how Java
implements it)
Add new sources:
- Folder (reads from this directory)
Add new sinks:
- Inline (stores in FileStorage)
- Folder (stores in specified directory)
Still want to do S3 buckets and web hooks and stuff, but they can come
in a future PR. I'm hoping this should make it sufficient to be able to
integrate with processing folders frontend etc. I've also changed it so
that policies can have multiple sources and triggers at once, which
seems like it might be useful.
Adds an optional MCP server (proprietary module) that exposes Stirling's
PDF operations and AI capabilities to MCP clients. Off by default, zero
footprint when disabled.
### What
- New `/mcp` endpoint: streamable-HTTP + JSON-RPC 2.0; 8 tools
(describe_operation, pages/convert/misc/security category tools, AI,
upload, download).
- Runs real operations over an internal loopback; results returned
inline as base64 (small) or by fileId (large).
### Auth (two modes)
- OAuth2 resource server: RFC 9728 protected-resource metadata, RFC 8707
audience binding, JWKS, `mcp.tools.read/write` scopes; binds each token
to a provisioned Stirling account.
- API-key mode: reuses Stirling per-user `X-API-KEY` (no IdP needed).
### Security
- Per-user file ownership in FileStorage: async/queued writes scoped to
the submitting user; legacy/owner-less files stay readable.
- Admin allow/block list controls which operations are exposed.
- Python engine gated behind a shared secret (`X-Engine-Auth`).
- MCP filter chain is isolated and cannot weaken the main app's
security.
- Hardened: no upstream error-body leakage, log injection sanitized,
fileId path/sidecar enumeration blocked.
### Config / footprint
- Disabled by default (`mcp.enabled=false`); all beans
`@ConditionalOnProperty`.
---
## Checklist
### General
- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings
### Documentation
- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)
### Translations (if applicable)
- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)
### UI Changes (if applicable)
- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)
### Testing (if applicable)
- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
# Description of Changes
Add a backend for running any multi-step PDF operations. This is
designed to be used for the upcoming Policies feature, along with
anything else that will require automated running of PDF operations,
like the Automate tool or Processing Folders.
The implementation is not complete. I've tried to get all the
infrastructure in there so that we can add in whichever triggers we need
in the future (like cron triggers or watching folders on disk) but
currently it just supports manual triggering of the policy.
The basis of this work was the operation running from the Stirling
Engine, which this PR removes in favour of this new system. The only
currently accessible frontend way to test this work is to ask the AI
chat to execute multiple operations on a PDF, but I've also extensively
tested with direct API calls to make sure that the policies work and
persist properly.
# Description of Changes
Currently, it's not possible to develop the backend on Mac without
manually signing the JPDFium binaries yourself since macOS will reject
running the unsigned binaries. [We've now updated JPDFium to sign the
Mac binaries in
v1.0.2](https://github.com/Stirling-Tools/JPDFium/releases/tag/v1.0.2),
so update to use that version.
# Description of Changes
Two narrowly-scoped hardening changes to the credits engine.
## 1. CreditService — move Stripe meter call to `afterCommit`
The Stripe metered-usage call sits inside the surrounding
`@Transactional`, holding the `user_credits` row lock for the duration
of an HTTP round-trip to Supabase. Under load this starves concurrent
debits; a transient Stripe blip rolls back a (correct) free-credit
consumption and forces the caller to retry.
The Stripe call now runs in a `TransactionSynchronization.afterCommit`
hook — DB commits first, Stripe fires immediately after. If Stripe fails
after commit, we log + increment a new `credits.stripe_report.failures`
counter; the idempotency key is stable, so a manual replay recovers
without double-charging.
Applied to both `consumeCreditBySupabaseId` and
`consumeCreditWithWaterfall`.
**Dead-code removed:**
- Unreachable UUID fallback for MDC `requestId` — `CorrelationIdFilter`
already guarantees the key on every request.
- The `"Unable to report usage to Stripe"` `RuntimeException` and its
catch block — the afterCommit refactor eliminates the throw path.
- `StripeRollbackOnFailureTest` — pinned the rollback-on-Stripe-fail
behaviour this refactor replaces.
## 2. `@AutoJobPostMapping` — build-time lint for `resourceWeight`
`UnifiedCreditInterceptor` multiplies `resourceWeight` into the per-call
charge. An endpoint that falls through to the annotation default
produces a charge derived from a value nobody chose.
- Annotation default flipped from `1` to `Integer.MIN_VALUE` (sentinel).
Both runtime readers (`UnifiedCreditInterceptor`, `AutoJobAspect`)
already clamp into `[1, 100]` so behaviour is unchanged.
- New `AutoJobPostMappingWeightTest` scans the classpath and fails the
build if any method leaves the sentinel.
- Initial run caught 11 endpoints relying on the default. Explicit
weights now declared, chosen by comparing to peer endpoints:
- `EditTextController` — LARGE
- `EmailController#sendEmailWithAttachment` — SMALL
- `ConvertPDFToMarkdown` — MEDIUM
- `AttachmentController` (extract/list/rename/delete) — SMALL × 4
- `ConvertImgPDFController` (cbr/cbz ↔ pdf) — MEDIUM × 2, LARGE × 2
## Tests
- `StripeUsageIdempotencyKeyTest` — pins the `(supabaseId, overage,
requestId)` idempotency key shape so Stripe always dedupes a retry.
- `StripeAfterCommitOrderingTest` — pins that `afterCommit` fires after
commit and NOT on rollback.
- `AutoJobPostMappingWeightTest` — the lint itself, plus a self-check
that the classpath scan finds at least 10 `@AutoJobPostMapping` methods
(guards against the lint passing vacuously).
Build verified: `ENABLE_SAAS=true ./gradlew :stirling-pdf:test
:saas:test`.
---
## Checklist
### General
- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — no translation changes
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
### Documentation
- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
— internal-billing change, no public docs impact
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
— N/A
### Translations (if applicable)
- [ ] Not applicable
### UI Changes (if applicable)
- [ ] Not applicable
### Testing (if applicable)
- [x] I have run `task check` (via `./gradlew :stirling-pdf:test
:saas:test` with `ENABLE_SAAS=true`) — passes
- [x] I have tested my changes locally
# Description of Changes
## What & why
Customers using ADFS (or any generic OIDC provider that doesn't emit
`email`) hit `Attribute value for 'email' cannot be null` during OAuth2
login with no visibility into what claims the provider actually sent.
The only available remedy was guessing at
`security.oauth2.useAsUsername` until something worked.
This PR adds a new opt-in `security.oauth2.debugLogging` flag (default
`false`). When enabled, `CustomOAuth2UserService` logs:
- All ID token claims (sorted, with values)
- All UserInfo endpoint claims (if any)
- The merged attribute key set Spring exposes to `getAttribute()`
- The value the configured `useAsUsername` actually resolved to
- A **`Hint:`** line listing the claim keys present in the token that
map to a valid `UsernameAttribute` enum value — i.e. exactly what the
operator could put in `useAsUsername` to make login work
Logged at `INFO` on the success path and `ERROR` on failure (inside the
existing `catch (IllegalArgumentException)` block that throws
`OAuth2AuthenticationException`). The block is wrapped with a `[OAUTH2
DEBUG] ... [/OAUTH2 DEBUG]` banner and ends with a PII warning so
operators don't leave it on in production.
Default off → zero observable change for anyone not actively
troubleshooting.
## Files changed
| File | Why |
|---|---|
| `app/common/.../ApplicationProperties.java` | New `debugLogging` field
on the `OAUTH2` config class with javadoc warning about PII |
| `app/core/src/main/resources/settings.yml.template` | Documents
`oauth2.debugLogging` so it appears on next startup |
| `app/proprietary/.../security/service/CustomOAuth2UserService.java` |
Emits the claim dump + suggestion hint when the flag is on |
|
`app/proprietary/.../security/service/CustomOAuth2UserServiceDebugLoggingTest.java`
(new) | Unit test: mocks the OIDC delegate, asserts off-path is silent
and on-path emits the dump with the right Hint contents |
## End-to-end verification
Ran the bundled `testing/compose/docker-compose-keycloak-oauth.yml`
Keycloak realm, configured `security.oauth2.useAsUsername: mail`
(Keycloak emits `email`, not `mail`) and `provider: demarest` (matches
the original customer bug report). Triggered the OAuth flow at
`http://localhost:8080/oauth2/authorization/demarest` and confirmed:
- The ERROR-level dump fires with the full 19-claim ID token decoded
- `-- Value at 'mail' : <NULL — this is why login fails>` correctly
identifies the missing claim
- `-- Hint:` correctly suggests `[email, family_name, given_name,
preferred_username]` (the four keys present that map to valid
`UsernameAttribute` values)
- Auth still fails with the original `OAuth2AuthenticationException` —
no change to control flow, just added diagnostic logging
Unit test (`CustomOAuth2UserServiceDebugLoggingTest`) covers both
branches.
## Reviewer notes
- **No new public APIs.** The flag is config-only; no servlet endpoints
exposed.
- **PII is logged when the flag is on.** This is the whole point —
operators need to see the claims to fix their config — but it's gated,
defaults off, and the dump self-documents with a `WARNING: ... Set
security.oauth2.debugLogging=false once troubleshooting is complete.`
footer.
- **Why log everything, not just sub/email?** Because the operator
doesn't know in advance which claim they actually want. ADFS uses `upn`
in some configs and `preferred_username` in others; Azure AD uses `oid`;
the customer here had neither. Dumping the full set is the only way to
make the diagnostic self-service.
- **Out of scope for this PR (follow-ups):**
- The `UsernameAttribute` enum doesn't include `upn` / `unique_name`
(common ADFS claims). If the customer's token only has `upn`, the Hint
will be empty even though the operator can see `upn` in the dump. Worth
a separate PR to extend the enum.
- The known-provider validator in `Provider.java` (rejects e.g.
`useAsUsername: mail` for `provider: keycloak` at startup) bypasses our
diagnostic for those provider names. ADFS customers using `provider:
<name>` fall into the `default` branch so are not affected — but it's a
sharp edge worth documenting.
---
## Checklist
### General
- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — N/A, backend-only change
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
### Documentation
- [ ] Doc-repo update (if functionality has heavily changed) —
diagnostic flag is self-documenting via the `settings.yml.template`
comment and the in-log warning; happy to add a doc-repo entry if
reviewers want one
- [ ] Translation tags — N/A
### UI Changes (if applicable)
- [ ] N/A — backend-only
### Testing (if applicable)
- [x] Unit test added (`CustomOAuth2UserServiceDebugLoggingTest`)
covering on/off paths and Hint correctness
- [x] End-to-end verified locally against bundled Keycloak compose with
intentionally misconfigured `useAsUsername`
- [x] Full `:proprietary:test` suite passes
`app/common/src/main/java/stirling/software/common/util/RequestUriUtils.java:175`
section comment said "Health Endoints" → "Health Endpoints".
Comment-only.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
# Description of Changes
Give Edit Agent access to descriptions of the request from the Java API.
This opens the door to us better documenting our Java APIs to give the
stirling engine better knowledge of what the various tools are and how
to use them.
Also improves the tool selection sub-agent to get the tool parameters
and descriptions so it can more intelligently decide which operations
should be used to fulfil the user's request. Also provides it more
encouragement to string together multiple operations if necessary.
## Description
Consolidates Playwright running under cohesive Task namespaces, isolates
Playwright state from the developer's local working tree, and swaps CI's
frontend webserver from `vite` dev to `vite preview` against a pre-built
`dist/`.
### `e2e:*` namespace
Renames `.taskfiles/testing.yml` to `.taskfiles/e2e.yml` and
consolidates everything Playwright-related under one `e2e:` namespace:
- `e2e:stubbed` / `e2e:live` / `e2e:enterprise` / `e2e:cross-browser`:
project-specific runners
- `e2e:check` (no-Docker subset) and `e2e:check:all` (full)
- `e2e:oauth:up` / `:down`, `e2e:saml:up` / `:down`: symmetric lifecycle
for the keycloak compose stacks
- `e2e:install`: Playwright browser install
- `docker:test`: full Docker integration suite
The redundant `frontend:test:e2e:*` project shortcuts are removed. CI
workflows (`e2e-stubbed.yml`, `e2e-live.yml`, `build-enterprise.yml`,
`nightly.yml`) are updated to call the new task names.
### Isolated Playwright state
New `STIRLING_BASE_PATH` (and `-Dstirling.base-path=`) override in
`InstallationPathConfig` redirects the entire state tree (configs,
backups, customFiles, pipeline, logs) at startup. `task e2e:live` points
it at `.test-state/playwright/` (purged on every invocation) so the
suite never touches the developer's local DB, settings.yml or backups.
`task e2e:live` auto-spawns gradle, waits for `/api/v1/info/status` to
come up, runs Playwright, then tears down the whole backend process
tree.
### CI runs Playwright against `vite preview`
Builds the frontend up-front with `VITE_BUILD_FOR_PREVIEW=1` (forces
absolute base so deep SPA routes resolve `/assets/...`) and the
playwright `webServer` now uses `vite preview --port 5173 --strictPort`
in CI. Avoids the per-page on-demand transform cost that was blowing the
30s navigation timeout under `--workers=3` on
`all-tool-pages-load.spec.ts`. Local dev keeps `vite` dev for HMR.
### OAuth/SAML compose helpers
`start-oauth-test.sh` and `start-saml-test.sh` gain a `--license-key
<KEY>` (`-k`) flag so CI and scripted runs can skip the interactive
license prompt. `start-oauth-test.sh` also moves from `for arg in "$@"`
to a `while`-with-`shift` arg loop to support multi-arg flags
consistently with the SAML script.
### Backend gradlew unification
Drops the per-platform `cmd /c gradlew.bat` branches from `backend.yml`
and routes every gradle invocation through `bash gradlew`. Works
uniformly on Linux/macOS and Windows-with-Git-Bash.
### Compare.tsx flake fix (re-land of
[#6316](https://github.com/Stirling-Tools/Stirling-PDF/pull/6316))
Piggybacks Anthony's never-merged fix from #6316. Without it,
`e2e:stubbed` continues to flake under `--workers=3` on
`compare.spec.ts`'s second-upload case via a React "Maximum update depth
exceeded" infinite loop in the Compare auto-fill effect. CI traces from
recent failed runs match exactly; 10 local runs of `compare.spec.ts`
with `CI=1 --workers=3` pass cleanly with the fix applied.
---------
Co-authored-by: James Brunton <jbrunton96@gmail.com>
# Description of Changes
Hooks up the (alpha) PDF Editor backend to the AI engine Edit Agent via
an intermediary API which is easier for the agent to call. It suffers
from all the same issues that the PDF Editor does in actually editing
the text, but should also benefit from any fixes to that.
It also adds protection against the underlying tools misbehaving by
hanging, and fixes a hanging bug in the PDF Editor.
---------
Co-authored-by: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com>
# Description of Changes
Have the Java send a list of enabled endpoints to the AI engine so it
can intelligently respond to the user that the tool does exist but is
disabled on the server so it can't acutally run the operation, instead
of the current behaviour where it sends the API call back and then 503
errors because the execution fails when the URL is disabled.
<img width="380" height="208" alt="image"
src="https://github.com/user-attachments/assets/5842fb2e-2e55-45a5-8205-25515636daae"
/>
---------
Co-authored-by: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com>
# Description of Changes
Flesh out the RAG system and connect it to the PDF Question Agent so it
can respond to questions about PDFs of an extremely large size.
I'd expect lots more work will need to be done to finish off the RAG
system to really be what we need, but this should be a reasonable start
which will let us connect it to tools and have the ingestion mostly
handled automatically. I'm leaving file deletion and proper file ID
management to be done in a future PR. We also need to consider whether
all tools should retrieve content exclusively via RAG, or whether it's
beneficial to have tools sometimes fetch the direct content and other
times fetch it from RAG.
A diagram of the expected interaction is as follows:
```mermaid
sequenceDiagram
autonumber
actor U as User
participant FE as Frontend<br/>(ChatPanel)
participant J as Java<br/>(AiWorkflowService)
participant O as Engine:<br/>OrchestratorAgent
participant QA as Engine:<br/>PdfQuestionAgent
participant RAG as Engine:<br/>RagService + SqliteVecStore
participant V as VoyageAI<br/>(embeddings)
participant L as LLM<br/>(Claude / etc.)
U->>FE: types "Summarise this PDF"<br/>(PDF already uploaded)
FE->>J: POST /api/v1/ai/orchestrate/stream<br/>multipart: fileInputs[], userMessage
Note over J: ByteHashFileIdStrategy<br/>id = sha256(bytes)[:16]
J->>O: POST /api/v1/orchestrator<br/>{ files:[{id,name}], userMessage }
O->>L: route via fast model
L-->>O: delegate_pdf_question
O->>QA: PdfQuestionRequest
loop for each file
QA->>RAG: has_collection(file.id)
RAG-->>QA: false
end
QA-->>O: NeedIngestResponse(files_to_ingest)
O-->>J: { outcome:"need_ingest", filesToIngest:[...] }
Note over J: onNeedIngest
loop per file
J->>J: PDFBox: extract page text
J->>O: POST /api/v1/rag/documents<br/>(long-running timeout)
O->>RAG: chunk + stage documents
O->>V: embed_documents (batches of 256)
V-->>O: embeddings
O->>RAG: add_documents
O-->>J: { chunks_indexed: N }
end
Note over J: retry with resumeWith=pdf_question
J->>O: POST /api/v1/orchestrator
Note over O: fast-path to PdfQuestionAgent
O->>QA: PdfQuestionRequest
Note over QA: build RagCapability<br/>pinned to file IDs
QA->>L: run(prompt) with search_knowledge tool
loop up to max_searches
L->>QA: search_knowledge(query)
QA->>V: embed_query
V-->>QA: query vector
QA->>RAG: search(vector, collections=[file.id])
RAG-->>QA: top-k chunks
QA-->>L: formatted chunks
end
Note over QA: once budget spent,<br/>prepare() hides the tool
L-->>QA: PdfQuestionAnswerResponse
QA-->>O: answer
O-->>J: { outcome:"answer", answer, evidence }
J-->>FE: SSE "result"
FE->>U: assistant bubble
```
ConnorYoh