Compare commits

...
Author SHA1 Message Date
EthanHealy01 fc9960b6e6 fast path added 2026-06-02 01:20:35 +01:00
EthanHealy01 f8ce30ff37 Add AGPL-3.0 license text and fix formatting
- Add licenses/AGPL-3.0.txt (full license text for scripts/pymupdf_convert.py)
- Update root LICENSE to reference it
- Fix Spotless formatting in PyMuPdfConverter and ExternalAppDepConfig
2026-06-02 00:58:39 +01:00
EthanHealy01 f0a7f9af78 Collapse pymupdf-worker package into a single script
The pymupdf-worker/ directory existed only to register a pip entry point
for pymupdf-convert. Replace it with scripts/pymupdf_convert.py — a
standalone script installed directly onto PATH in the Dockerfile.

- Add scripts/pymupdf_convert.py (AGPL-3.0, same logic, shebang added)
- Update docker/base/Dockerfile to copy + install the script directly
  instead of pip-installing the package; remove now-redundant symlink
- Delete pymupdf-worker/ and its taskfile
- Update root LICENSE to reference the script file instead of directory
2026-06-02 00:48:55 +01:00
EthanHealy01 6309ca7234 Clean up pymupdf-convert integration per review feedback
- Remove AGPL/license comments from main codebase (Dockerfile, Java
  classes, javadocs) — legal isolation is structural, comments were noise
- PyMuPdfConverter: check CLI availability once at @PostConstruct instead
  of caching with a TTL; simpler and consistent with how LibreOffice/
  Tesseract availability is handled
2026-06-02 00:44:41 +01:00
EthanHealy01 f3f17d6381 fix: non-root user in Dockerfile, update stale HTTP references in worker 2026-06-01 19:45:35 +01:00
EthanHealy01 4487f23ff7 feat(pdf-to-markdown): add pymupdf-convert CLI for faster PDF→Markdown conversion
Introduces a new  directory containing a small standalone
CLI tool () that converts PDFs to Markdown using PyMuPDF
and pymupdf4llm. When installed, it replaces the slower pdftohtml-based
converter with a significantly faster path.
The tool is invoked as an OS subprocess, following the same pattern already
used for LibreOffice and Tesseract. A new  service handles
discovery (PATH probe, cached 30 s), temp file I/O, and process execution via
the existing  infrastructure. Auto-detected at startup via
 — no configuration flag required. Falls back
transparently to the bundled pdftohtml converter if not found.
 is baked into the Docker base image via the existing
Python venv build stage and symlinked onto PATH, so all Docker users get
the faster path automatically with no manual steps.
AGPL notice:  is licensed under AGPL-3.0-or-later because
it depends on PyMuPDF (AGPL-3.0 / Artifex commercial). It is a SEPARATE
program from the rest of Stirling PDF and shares no source code with the
MIT-licensed core or the proprietary engine. The subprocess boundary
ensures the AGPL copyleft attaches to this tool alone and does not extend
to Stirling PDF. The LICENSE file and pymupdf-worker/LICENSE document this
separation explicitly. Do not replace the subprocess invocation with an
in-process import of PyMuPDF.
2026-06-01 19:42:49 +01:00
brios 2c0ebc28a7 perf(api): optimize static asset caching, enable ETag support, and expand response compression mime types. (#6273) 2026-06-01 16:37:56 +01:00
Ludy d1486c7762 ci(github-actions): replace deprecated app-id input with client-id (#6485)
# Description of Changes

This change updates the GitHub App token generation step in the custom
`setup-bot` GitHub Action to use the new `client-id` input instead of
the deprecated `app-id` input when invoking
`actions/create-github-app-token`.

### What was changed

- Replaced the deprecated `app-id` parameter with `client-id` in
`.github/actions/setup-bot/action.yml`.
- Continued sourcing the value from the existing `inputs.app-id` action
input to avoid broader interface changes.

### Why the change was made

- `actions/create-github-app-token` has deprecated the `app-id` input
and now expects `client-id`.
- Updating the workflow removes the deprecation warning and ensures
compatibility with current and future versions of the action.


---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
2026-05-31 08:07:30 +01:00
Ludy 2ccff6f73f fix(update-service): correct GitHub branch reference for version retrieval (#6333) 2026-05-30 22:44:42 +01:00
Ludy 78da227eba fix: Use frontend/editor for locales paths (#6483) 2026-05-30 22:00:17 +01:00
Anthony Stirling 30e782e29c Disable Save-to-server when storage off, fix QR port 0 (#6473) 2026-05-29 19:37:42 +01:00
Anthony Stirling 2b0905887b Add desktop multi-window support (#6463) 2026-05-29 19:35:54 +01:00
ConnorYoh 28b81828b5 PAYG: PricingPolicyService + admin REST + 30s read cache (#6469)
## What this is

PR-I1 service half from `notes/PAYG_DESIGN.md`. Built on top of the data
model from #6460 — answers "what pricing policy applies to this team
right now?" with a fast cache and an admin write surface.

## Scope

| Piece | Where |
|---|---|
| `PricingPolicyService` — `getEffectivePolicy(teamId)` with 30s
Caffeine cache + admin write paths |
`app/saas/.../payg/policy/PricingPolicyService.java` |
| `PolicyChangedEvent` — published after admin writes for in-process
cache invalidation | `app/saas/.../payg/policy/PolicyChangedEvent.java`
|
| Admin REST — list / get / create / set-default / set team override /
get effective |
`app/saas/.../payg/policy/admin/PricingPolicyAdminController.java` +
DTOs |
| `PricingPolicyRepository.clearDefaultFlag()` — atomic clear for
set-default | repository update |
| `SaasJpaConfigScanTest` — drift guard against the JPA scan paths going
stale (carried over from the #6460 review concern) | new test |
| V12 default-policy seed (`v1-initial`, 25 pages/unit, 5 MiB/unit,
per-`JobSource` step limits) | `V12__seed_default_payg_policy.sql` |

## Lookup precedence

1. `PaygTeamExtensions.pricingPolicyId` set → return that policy
2. Else return the `pricing_policy` row with `is_default = TRUE`
3. Override row points at a deleted policy → log warn, fall back to
default (safety net for racing deletes)
4. No default → `IllegalStateException` (V12 seed guarantees one exists)

## Cache behaviour

- 30s `expireAfterWrite` Caffeine, max 10k entries, keyed by `teamId`.
- **Single correctness model: the TTL.** Cross-instance propagation is
at-most-30-seconds. The writer instance sees its own change immediately
via the `PolicyChangedEvent` after-commit publish. Other instances pick
it up on the next TTL expiry.
- Admin reads use `getEffectivePolicyUncached` so admins always see
their own write straight back.

**Why no LISTEN/NOTIFY runner.** An earlier cut of this PR included a
Postgres `LISTEN policy_changed` runner so cross-instance propagation
was instant. Dropped — admin policy changes are events-per-week and the
30s TTL is already the correctness floor; the listener was ~250 lines of
nontrivial code (raw JDBC outside HikariCP, daemon thread, reconnect
loop, lock-protected connection lifecycle) for a use case that isn't on
the hot path. Trade-off is documented in `notes/PAYG_DESIGN.md` §9 with
three concrete triggers that would justify reintroducing it (aggressive
cap enforcement, Redis landing for other reasons, real-time admin UI).

## Writes — transactional, fire `PolicyChangedEvent` after commit

- `create(draft)` — rejects pre-set `policy_id` or `is_default=true`
(promotion must go through `setDefault` so the partial unique idx is
freed first).
- `setDefault(id)` — atomically clears the existing default via
`clearDefaultFlag()` then flips the new row. Idempotent: silent no-op if
the row is already default.
- `setTeamOverride(teamId, policyId | null)` — validates the policy
exists before save; `null` clears the override.

`publishOnCommit` uses `TransactionSynchronizationManager.afterCommit`
so listeners never see pre-commit state. Outside a transaction (test
paths) falls through to immediate publish.

## Admin REST surface — `/api/v1/admin/payg/...`

All endpoints `@PreAuthorize("hasRole('ADMIN')")`:

- `GET  /policies` — list all
- `GET  /policies/{id}` — read one
- `POST /policies` — create new (non-default)
- `POST /policies/{id}/set-default` — atomic promote
- `PUT /teams/{teamId}/policy-override` — set or clear per-team override
- `GET  /teams/{teamId}/effective-policy` — cache-bypassing live read

Validation errors → 400, unknown rows → 404.

## Counterpart Supabase PR


[`Stirling-PDF-SaaS#298`](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/298)
— seeds the same V1 default policy on the Supabase side via
`20260528000002_payg_seed_default_policy.sql`.

## Tests

- 17 × `PricingPolicyServiceTest` — lookup precedence, cache hit/miss,
invalidation on event, mutation paths publishing event, error cases.
- 14 × `PricingPolicyAdminControllerTest` — every endpoint's happy path
+ error mapping, DTO defensive-copy invariant.
- 2 × `SaasJpaConfigScanTest` — reflection-based guard that
`payg.repository` is in `@EnableJpaRepositories` and `payg` is in
`@EntityScan`. Without this, new sub-packages can silently fail to wire
at runtime — same class of bug that the #6460 review caught.

Full `:saas:test` BUILD SUCCESSFUL.

## Design doc

`notes/PAYG_DESIGN.md` §7.4 PR-I1 — completes the service half (the
schema half landed in #6460). §9 carries the 30s-TTL trade-off note.
2026-05-29 16:27:01 +00:00
James Brunton 2c01f41142 Update indexeddb to v9 to unify SaaS and OSS users (#6474)
# Description of Changes
The production SaaS is currently on v8 of IndexedDB due to various
schema changes for Smart Folders, which haven't made their way into OSS.
OSS is currently on v4 of IndexedDB, so if we release an OSS build to
the SaaS deployment, existing users will not be able to use it because
the DB version is 'too old'.

This PR updates the IDB version number to v9 so both OSS and SaaS users
will be able to upgrade to it. Theoretically both types of user should
be able to keep their IDB files without issue. SaaS previously actively
wiped the user's files in an old version (v6/v7) and users who haven't
used it since then will have their DBs wiped, but that'd happen anyway
if they use current SaaS so I don't think that matters.
2026-05-29 15:23:40 +00:00
James Brunton 4d5eeb103f Fix username display issues (#6471)
# Description of Changes
Main fixes:
- Fix the display of the username in the bottom left
- Now displays as "User" when not logged in on self-hosted (desktop) and
"Guest" on SaaS when logged in anonymously
- Now updates properly when the user logs in/out in SaaS, desktop and
self-hosted
- Fix incremental build issues in the desktop app that have been here
since the start (I hope at least - I think the issue is that the JLink
is built read-only and then on subsequent builds you get OS errors when
trying to override the JLink with the new version. There's no real need
for it to be read-only that I know of, so we might as well just make it
R/W and ship like that)
2026-05-29 14:35:47 +00:00
ConnorYoh 83ea07ed6a saas: DocumentClassifier + PAYG data model (#6460)
# Description of Changes

Two layers — the `DocumentClassifier` utility plus the full data model
for the new billing engine. Nothing wires the entities into application
behaviour yet; services and controllers land in follow-up PRs.

**Companion PR:**
[Stirling-PDF-SaaS#296](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/296)
— Supabase migration for the v3 dev branch, schema-equivalent to the
Flyway migration in this PR.

## 1. DocumentClassifier (under `payg.docs`)

`DocumentClassifier` computes the doc-unit cost of an uploaded file (or
multi-file input) under a `PricingPolicy`. PDFs read page count via
`stirling.software.jpdfium.PdfDocument`; non-PDFs are bytes-only.
Formula: `max(ceil(pages / docPagesPerUnit), ceil(bytes /
docBytesPerUnit))` clamped to `[1, fileUnitCap]`. Multi-file is the sum
of raw per-file units capped at `fileUnitCap × file_count`.

Two floors, by design: the classifier returns `docUnits` with an
absolute `1` floor for non-empty input; the policy-level
`minChargeUnits` is intentionally applied later, at process-open time in
`JobChargeService`, per design § 3.4 (`unitsForProcess =
max(policy.min_charge_units, docUnits)`). Documented in the interface +
impl javadoc.

Upload bytes are materialised through
`TempFileManager.createManagedTempFile` so jpdfium gets a `Path`; the
temp file auto-deletes on close.

Twelve tests, all in-memory fixtures generated with PDFBox at test time
— no committed binary blobs.

## 2. PAYG data model (under `payg.*`)

JPA entities, repositories, and a Flyway migration covering the full
schema in §6 of the design.

**Enums** (`payg.model`):

`JobSource`, `ProcessType`, `JobStatus`, `JobStepStatus`,
`ArtifactKind`, `LedgerEntryType`, `LedgerBucket`, `ReferenceType`,
`EntitlementState`, `FeatureSet`, `FeatureGate`, `WalletEngine`,
`CapPeriod`, `AutoGroupStrategy`.

**Entities + repositories:**

| Entity | Table | Notes |
|---|---|---|
| `PricingPolicy` | `pricing_policy` | Promoted from a record.
`stepLimits` is `Map<JobSource, Integer>` persisted via normalised child
table `pricing_policy_step_limit`. `stripePriceIds` is `Set<String>`
persisted via `pricing_policy_stripe_price` — currency comes from
`stripe.prices` via Sync Engine, not stored locally. |
| `ProcessingJob` | `processing_job` | UUID PK. Tracks lineage window
via `step_count` and `last_step_at`. |
| `ProcessingJobStep` | `processing_job_step` | Per-tool-call audit. |
| `JobArtifactHash` | `job_artifact_hash` | Composite key `(job_id,
content_hash, kind)`. `content_hash VARCHAR(128)` so multiple signature
schemes coexist as `"type:value"` storage keys. Lineage detector queries
this. |
| `WalletLedgerEntry` | `wallet_ledger` | Append-only, signed
`amount_units`. Two unique indexes kill double-posting. |
| `WalletPolicy` | `wallet_policy` | Per-team engine + cap + degradation
rules + lineage strategy. No `@Version` — admin-only writes (documented
in javadoc). |
| `WalletEntitlementSnapshot` | `wallet_entitlement_snapshot` |
Composite key `(team_id, user_id)`; `user_id = 0` is the team-wide
sentinel. No `@Version` — full-row recompute via
`EntitlementService.recompute` (documented in javadoc). |
| `PaygShadowCharge` | `payg_shadow_charge` | Per-job diff while in
`PAYG_SHADOW` engine mode. |
| `PaygTeamExtensions` | `payg_team_extensions` | Sidecar 1:1 with
`teams` carrying `pricing_policy_id` (per-team override) +
`stripe_customer_id`. Sidecar pattern (mirrors `saas_team_extensions`)
so OSS Hibernate ddl-auto never sees PAYG columns on `teams`. |

**Column adds:**

- `team_memberships.cap_units` (optional per-member sub-cap)

**Width split (intentional, documented in V11):** per-row deltas
(`wallet_ledger.amount_units`, `processing_job.charged_units`) are
`INTEGER` because no single charge realistically approaches 2B units.
Cap and period-rollup columns (`team_memberships.cap_units`,
`wallet_policy.cap_units`,
`wallet_entitlement_snapshot.period_spend_units / period_cap_units`) are
`BIGINT` because they accumulate across a billing period and admins may
legitimately set headroom-cap values into the millions.

**JPA wiring:** `SaasJpaConfig` was updated to include
`stirling.software.saas.payg.repository` in
`@EnableJpaRepositories.basePackages` and `stirling.software.saas.payg`
in `@EntityScan` (covers `payg.policy` / `payg.job` / `payg.wallet` /
`payg.entitlement` / `payg.shadow` recursively). New
`SaasJpaConfigScanTest` reads the annotations reflectively and asserts
every expected package is wired — catches the next time someone adds a
new sub-package without updating the scan paths.

**Migration:** `V11__saas_payg_model.sql` (purely additive).
Schema-equivalent to the Supabase migration in the companion PR —
including the `VARCHAR(128) content_hash` width that's needed for the
multi-signature-scheme storage encoding the lineage layer uses.

## 3. Smoke tests

`PaygEntitiesSmokeTest` exercises each entity via the no-arg ctor JPA
requires, plus getter/setter round-trips and composite-key equality —
catches Lombok/annotation regressions without needing a database.
Real-DB integration coverage lands alongside the services that consume
each entity.

## Why this is safe to land now

- All schema changes are additive — no existing rows modified, no
columns dropped.
- The entities are not yet referenced from any production code path;
they exist for the next PRs to build on.
- The v3 Supabase dev branch picks up the schema via the companion PR;
the main repo's Flyway migration applies the same shape when an instance
boots against a freshly-migrated v3 database.

## Open decisions made

- **Step-limits keyed by `JobSource`** rather than by `ProcessType`.
Captures the "self-hosted gets a different knob" framing in earlier
feedback. Trivially overridable per pricing policy version.
- **Step limits + Stripe price IDs normalised into child tables** rather
than JSONB on `pricing_policy` (per Connor's review on #296). Typed
columns, queryable directly, no JSON parsing.
- **Currency dropped from `pricing_policy_stripe_price`** — it lives on
`stripe.prices.currency` and is resolved via Sync Engine. App is
currency-blind.

## Rollback

Straight `git revert` on this PR. The Supabase migration in #296 is
additive and can be left in place safely — the running app ignores
tables it doesn't reference.

---

## Checklist

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
- [x] I have run `task check` (via `./gradlew :saas:test` with
`ENABLE_SAAS=true`) — passes
2026-05-29 12:03:01 +00:00
James Brunton 61ebe977d3 Auto-delete CI linting comments on success (#6465)
# Description of Changes
Set CI backend & engine comments to auto-delete once the CI has passed. 

Also redesign the engine CI to call `task engine:check` like it should
have been, and make it post a comment when the tool models need to be
updated.

Also makes the comment wording more consistent between the three
languages.
2026-05-29 10:13:12 +00:00
EthanHealy01 763595a5a3 feat: add Agents UI to proprietary right sidebar (#6454)
Update UI to include agents

Run `task dev:all` to test
2026-05-28 17:26:23 +00:00
Anthony Stirling 398617391b Fix SSO auto-login and custom metadata settings not persisting on restart (#6468) 2026-05-28 17:39:05 +01:00
ConnorYoh a0e0e88f07 saas: harden CreditService Stripe ordering + lint @AutoJobPostMapping weights (#6458)
# Description of Changes

Two narrowly-scoped hardening changes to the credits engine.

## 1. CreditService — move Stripe meter call to `afterCommit`

The Stripe metered-usage call sits inside the surrounding
`@Transactional`, holding the `user_credits` row lock for the duration
of an HTTP round-trip to Supabase. Under load this starves concurrent
debits; a transient Stripe blip rolls back a (correct) free-credit
consumption and forces the caller to retry.

The Stripe call now runs in a `TransactionSynchronization.afterCommit`
hook — DB commits first, Stripe fires immediately after. If Stripe fails
after commit, we log + increment a new `credits.stripe_report.failures`
counter; the idempotency key is stable, so a manual replay recovers
without double-charging.

Applied to both `consumeCreditBySupabaseId` and
`consumeCreditWithWaterfall`.

**Dead-code removed:**
- Unreachable UUID fallback for MDC `requestId` — `CorrelationIdFilter`
already guarantees the key on every request.
- The `"Unable to report usage to Stripe"` `RuntimeException` and its
catch block — the afterCommit refactor eliminates the throw path.
- `StripeRollbackOnFailureTest` — pinned the rollback-on-Stripe-fail
behaviour this refactor replaces.

## 2. `@AutoJobPostMapping` — build-time lint for `resourceWeight`

`UnifiedCreditInterceptor` multiplies `resourceWeight` into the per-call
charge. An endpoint that falls through to the annotation default
produces a charge derived from a value nobody chose.

- Annotation default flipped from `1` to `Integer.MIN_VALUE` (sentinel).
Both runtime readers (`UnifiedCreditInterceptor`, `AutoJobAspect`)
already clamp into `[1, 100]` so behaviour is unchanged.
- New `AutoJobPostMappingWeightTest` scans the classpath and fails the
build if any method leaves the sentinel.
- Initial run caught 11 endpoints relying on the default. Explicit
weights now declared, chosen by comparing to peer endpoints:
  - `EditTextController` — LARGE
  - `EmailController#sendEmailWithAttachment` — SMALL
  - `ConvertPDFToMarkdown` — MEDIUM
  - `AttachmentController` (extract/list/rename/delete) — SMALL × 4
  - `ConvertImgPDFController` (cbr/cbz ↔ pdf) — MEDIUM × 2, LARGE × 2

## Tests

- `StripeUsageIdempotencyKeyTest` — pins the `(supabaseId, overage,
requestId)` idempotency key shape so Stripe always dedupes a retry.
- `StripeAfterCommitOrderingTest` — pins that `afterCommit` fires after
commit and NOT on rollback.
- `AutoJobPostMappingWeightTest` — the lint itself, plus a self-check
that the classpath scan finds at least 10 `@AutoJobPostMapping` methods
(guards against the lint passing vacuously).

Build verified: `ENABLE_SAAS=true ./gradlew :stirling-pdf:test
:saas:test`.

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — no translation changes
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
— internal-billing change, no public docs impact
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
— N/A

### Translations (if applicable)

- [ ] Not applicable

### UI Changes (if applicable)

- [ ] Not applicable

### Testing (if applicable)

- [x] I have run `task check` (via `./gradlew :stirling-pdf:test
:saas:test` with `ENABLE_SAAS=true`) — passes
- [x] I have tested my changes locally
2026-05-28 14:57:59 +00:00
Anthony Stirling c80a5db5f5 folder and file fixes (#6461) 2026-05-28 15:57:35 +01:00
Anthony Stirling 4fa67afc3d Fix Tauri artifact copy path so installers upload (smoke + release) (#6466)
## Summary
Regression from #6404 (Restructure/frontend editor). Two CI workflows
copy the built installers to the wrong directory, so installer artifacts
(MSI / DMG / DEB / RPM / AppImage) silently vanish:

- **`tauri-build.yml`** (PR/desktop smoke builds) - uploads zero
installer artifacts.
- **`multiOSReleases.yml`** (production releases) - the empty artifacts
are downloaded by `create-release` and fed to `action-gh-release`, so a
release would publish **only the JARs, no desktop installers**.

## Root cause
#6404 moved the Tauri project from `frontend/` to `frontend/editor/` and
updated every **absolute** path (`projectPath`, `cd`, `Get-ChildItem`)
to add the `editor/` segment - but left the **relative** copy targets
`../../../dist`. Those resolve against the (now one level deeper)
working dir after `cd ./frontend/editor/src-tauri/target`:

| | resolves to |
|---|---|
| before #6404 (`frontend/src-tauri/target`) | repo-root `dist/`  |
| after #6404 (`frontend/editor/src-tauri/target`) | `frontend/dist/` 
(missing) |

The `cp` fails, repo-root `dist/` (from `mkdir -p ./dist`) stays empty,
and the upload finds nothing. `find -exec cp` failing is non-fatal, so
jobs still report success - that's why it went unnoticed. No release has
shipped broken yet: the last release (v2.11.0, 2026-05-19) predates
#6404 (2026-05-22).

## Fix
Copy to an absolute `$GITHUB_WORKSPACE/dist` in both workflows so the
`cd` can't drift the destination again. This matches where the upload /
signature-verify steps already read from.

## Evidence (run 26574078559, all 3 OS legs)
```
cp: cannot create regular file '../../../dist/Stirling-PDF-windows-x86_64.msi': No such file or directory
##[warning]No files were found with the provided path: ./dist/*. No artifacts will be uploaded.
```
The Tauri builds themselves succeeded - only the copy/upload was broken.

## Test plan
- [ ] `tauri-build` on this PR uploads non-empty `Stirling-PDF-<name>`
artifacts on Windows/macOS/Linux.
- [ ] Next release (or a `workflow_dispatch` of multiOSReleases)
attaches MSI/DMG/DEB/RPM/AppImage to the release.
2026-05-28 15:57:01 +01:00
Anthony StirlingandConnorYoh 8bd78d2624 Add landscape page size options (#6248)
# Description of Changes

Adds orientation (portrait/landscape) to the Adjust Page Scale tool.

- Orientation as a separate parameter (per review), sent through to the
backend
- ScalePagesController simplified; PDFWithPageSize gains the orientation
field
- Regenerated tool_models.py; frontend + backend tests added

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.

---------

Co-authored-by: ConnorYoh <40631091+ConnorYoh@users.noreply.github.com>
2026-05-28 14:16:15 +00:00
Anthony Stirling b3c4b8b463 Add S3 storage and cluster artifact backend (#6457) 2026-05-28 13:06:27 +01:00
James Brunton 57af5b9dc2 Fix Tauri testing (#6462)
# Description of Changes
#6402 introduced a Rust test `refresh_token_fallback.rs`, but it wasn't
moved properly after the restructure of the `frontend/` folder in #6404.
This PR moves the file to the right place, and also hooks up Task and CI
rules for `cargo test` since nothing was actually running the test in
the first place.
2026-05-28 11:05:56 +00:00
James Brunton 44fbf8c587 Various bug fixes found while testing SaaS build (#6459)
# Description of Changes
Various fixes and improvements I made while testing the SaaS code:
- Changes the new `.env.saas` file to live in `app/` and match the
semantics of the other `.env` files
- Adds top-level `task dev:saas` command to spawn SaaS frontend &
backend
- Deletes dead SaaS code and improves some overriding logic
- Fixes refreshing issue when coming back to the tab
- Fix the Compare tool's selection logic
- Make Compare handle error cases properly
- Fixes the location of the "Dismiss All Errors" button (was rendering
on top of the top-bar with a transparent background previously so it
looked rubbish)
- Fixes file selection in PDF Editor
2026-05-28 11:05:30 +00:00
Anthony Stirling 76840d8a57 Add CI DB migration smoke test against v2.0/v2.5/v2.10 updates (#6453) 2026-05-28 11:36:07 +01:00
James Brunton d459ded168 Add cancel button to kill long-running AI tasks (#6351)
# Description of Changes
Adds a cancel button to the AI chat to allow the user to abort
long-running AI tasks. Just disconnects the SSE stream (all the backend
code already interrupts when it notices the stream is dead).
2026-05-28 09:25:23 +00:00
ConnorYoh 43b67d213d feat(oauth2): opt-in claim-dump diagnostics for OIDC login failures (#6456)
# Description of Changes

## What & why

Customers using ADFS (or any generic OIDC provider that doesn't emit
`email`) hit `Attribute value for 'email' cannot be null` during OAuth2
login with no visibility into what claims the provider actually sent.
The only available remedy was guessing at
`security.oauth2.useAsUsername` until something worked.

This PR adds a new opt-in `security.oauth2.debugLogging` flag (default
`false`). When enabled, `CustomOAuth2UserService` logs:

- All ID token claims (sorted, with values)
- All UserInfo endpoint claims (if any)
- The merged attribute key set Spring exposes to `getAttribute()`
- The value the configured `useAsUsername` actually resolved to
- A **`Hint:`** line listing the claim keys present in the token that
map to a valid `UsernameAttribute` enum value — i.e. exactly what the
operator could put in `useAsUsername` to make login work

Logged at `INFO` on the success path and `ERROR` on failure (inside the
existing `catch (IllegalArgumentException)` block that throws
`OAuth2AuthenticationException`). The block is wrapped with a `[OAUTH2
DEBUG] ... [/OAUTH2 DEBUG]` banner and ends with a PII warning so
operators don't leave it on in production.

Default off → zero observable change for anyone not actively
troubleshooting.

## Files changed

| File | Why |
|---|---|
| `app/common/.../ApplicationProperties.java` | New `debugLogging` field
on the `OAUTH2` config class with javadoc warning about PII |
| `app/core/src/main/resources/settings.yml.template` | Documents
`oauth2.debugLogging` so it appears on next startup |
| `app/proprietary/.../security/service/CustomOAuth2UserService.java` |
Emits the claim dump + suggestion hint when the flag is on |
|
`app/proprietary/.../security/service/CustomOAuth2UserServiceDebugLoggingTest.java`
(new) | Unit test: mocks the OIDC delegate, asserts off-path is silent
and on-path emits the dump with the right Hint contents |

## End-to-end verification

Ran the bundled `testing/compose/docker-compose-keycloak-oauth.yml`
Keycloak realm, configured `security.oauth2.useAsUsername: mail`
(Keycloak emits `email`, not `mail`) and `provider: demarest` (matches
the original customer bug report). Triggered the OAuth flow at
`http://localhost:8080/oauth2/authorization/demarest` and confirmed:

- The ERROR-level dump fires with the full 19-claim ID token decoded
- `-- Value at 'mail' : <NULL — this is why login fails>` correctly
identifies the missing claim
- `-- Hint:` correctly suggests `[email, family_name, given_name,
preferred_username]` (the four keys present that map to valid
`UsernameAttribute` values)
- Auth still fails with the original `OAuth2AuthenticationException` —
no change to control flow, just added diagnostic logging

Unit test (`CustomOAuth2UserServiceDebugLoggingTest`) covers both
branches.

## Reviewer notes

- **No new public APIs.** The flag is config-only; no servlet endpoints
exposed.
- **PII is logged when the flag is on.** This is the whole point —
operators need to see the claims to fix their config — but it's gated,
defaults off, and the dump self-documents with a `WARNING: ... Set
security.oauth2.debugLogging=false once troubleshooting is complete.`
footer.
- **Why log everything, not just sub/email?** Because the operator
doesn't know in advance which claim they actually want. ADFS uses `upn`
in some configs and `preferred_username` in others; Azure AD uses `oid`;
the customer here had neither. Dumping the full set is the only way to
make the diagnostic self-service.
- **Out of scope for this PR (follow-ups):**
- The `UsernameAttribute` enum doesn't include `upn` / `unique_name`
(common ADFS claims). If the customer's token only has `upn`, the Hint
will be empty even though the operator can see `upn` in the dump. Worth
a separate PR to extend the enum.
- The known-provider validator in `Provider.java` (rejects e.g.
`useAsUsername: mail` for `provider: keycloak` at startup) bypasses our
diagnostic for those provider names. ADFS customers using `provider:
<name>` fall into the `default` branch so are not affected — but it's a
sharp edge worth documenting.

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — N/A, backend-only change
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] Doc-repo update (if functionality has heavily changed) —
diagnostic flag is self-documenting via the `settings.yml.template`
comment and the in-log warning; happy to add a doc-repo entry if
reviewers want one
- [ ] Translation tags — N/A

### UI Changes (if applicable)

- [ ] N/A — backend-only

### Testing (if applicable)

- [x] Unit test added (`CustomOAuth2UserServiceDebugLoggingTest`)
covering on/off paths and Hint correctness
- [x] End-to-end verified locally against bundled Keycloak compose with
intentionally misconfigured `useAsUsername`
- [x] Full `:proprietary:test` suite passes
2026-05-27 13:01:51 +00:00
Anthony Stirling d42b779644 Add server-side folders and files page UI (#6383) 2026-05-27 12:52:46 +01:00
317 changed files with 29633 additions and 2790 deletions
+1 -1
View File
@@ -24,7 +24,7 @@ runs:
id: generate-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ inputs.app-id }}
client-id: ${{ inputs.app-id }}
private-key: ${{ inputs.private-key }}
- name: Configure Git
run: |
+2
View File
@@ -38,6 +38,8 @@ project: &project
- frontend/**
- docker/**
- scripts/RestartHelper.java
- scripts/db-migration/**
- .github/workflows/db-migration-test.yml
frontend: &frontend
- frontend/**
+4 -3
View File
@@ -13,7 +13,7 @@ Usage:
"""
# Sample for Windows:
# python .github/scripts/check_language_toml.py --reference-file frontend/public/locales/en-GB/translation.toml --branch "" --files frontend/public/locales/de-DE/translation.toml frontend/public/locales/fr-FR/translation.toml
# python .github/scripts/check_language_toml.py --reference-file frontend/editor/public/locales/en-GB/translation.toml --branch "" --files frontend/editor/public/locales/de-DE/translation.toml frontend/editor/public/locales/fr-FR/translation.toml
import argparse
import glob
@@ -184,7 +184,7 @@ def check_for_differences(reference_file, file_list, branch, actor):
if len(file_list) == 1:
file_arr = file_list[0].split()
base_dir = Path.cwd() / "frontend" / "public" / "locales"
base_dir = Path.cwd() / "frontend" / "editor" / "public" / "locales"
for file_path in file_arr:
file_path = Path(file_path)
@@ -308,7 +308,7 @@ def check_for_differences(reference_file, file_list, branch, actor):
report.append("## ❌ Overall Check Status: **_Failed_**")
report.append("")
report.append(
f"@{actor} please check your translation if it conforms to the standard. Follow the format of [en-GB/translation.toml](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/frontend/public/locales/en-GB/translation.toml)"
f"@{actor} please check your translation if it conforms to the standard. Follow the format of [en-GB/translation.toml](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/frontend/editor/public/locales/en-GB/translation.toml)"
)
else:
report.append("## ✅ Overall Check Status: **_Success_**")
@@ -372,6 +372,7 @@ if __name__ == "__main__":
os.path.join(
os.getcwd(),
"frontend",
"editor",
"public",
"locales",
"*",
+2 -1
View File
@@ -287,6 +287,7 @@ jobs:
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/data:/usr/share/tessdata:rw
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/config:/configs:rw
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/logs:/logs:rw
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/storage:/storage:rw
environment:
DISABLE_ADDITIONAL_FEATURES: "false"
SECURITY_ENABLELOGIN: "true"
@@ -309,7 +310,7 @@ jobs:
ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -T ${{ secrets.NEW_VPS_USERNAME }}@${{ secrets.NEW_VPS_HOST }} << ENDSSH
# Create V2 PR-specific directories
mkdir -p /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/{data,config,logs}
mkdir -p /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/{data,config,logs,storage}
# Move docker-compose file to correct location
mv /tmp/docker-compose-v2.yml /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/docker-compose.yml
+124 -27
View File
@@ -1,8 +1,9 @@
name: AI Engine CI
# Validates the Python AI engine: regenerates tool models, runs fixers,
# lint, type-check, and tests. Called from build.yml on PRs and merge_group;
# also runs directly on push to main as a post-merge safety net.
# Validates the Python AI engine: regenerates tool models and runs the
# engine quality gate (lint, type-check, format-check, tests). Called from
# build.yml on PRs and merge_group; also runs directly on push to main as
# a post-merge safety net.
on:
workflow_call:
push:
@@ -51,27 +52,95 @@ jobs:
run: task engine:tool-models
- name: Verify tool models are up to date
id: tool-models-check
continue-on-error: true
run: git diff --exit-code engine/src/stirling/models/tool_models.py
- name: Comment on tool models check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.tool-models-check.outcome == 'failure' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- tool-models-check -->';
const body = [
marker,
'### Tool Models Check Failed',
'',
'The generated `engine/src/stirling/models/tool_models.py` is out of date with the Java OpenAPI spec and will need to be regenerated before it can be merged in.',
'',
'Run `task engine:tool-models` to regenerate, then commit the updated file.',
].join('\n');
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});
}
- name: Fail if tool models check failed
if: steps.tool-models-check.outcome == 'failure'
run: |
if ! git diff --exit-code engine/src/stirling/models/tool_models.py; then
echo "tool_models.py is out of date."
echo "Run 'task engine:tool-models' locally and commit the updated file."
exit 1
fi
echo "============================================"
echo " Tool Models Check Failed"
echo "============================================"
echo ""
echo "The generated engine/src/stirling/models/tool_models.py"
echo "is out of date with the Java OpenAPI spec and will"
echo "need to be regenerated before it can be merged in."
echo ""
echo "Run 'task engine:tool-models' to regenerate, then"
echo "commit the updated file."
echo "============================================"
exit 1
- name: Run fixers
run: task engine:fix
- name: Remove tool models check comment on success
if: steps.tool-models-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- tool-models-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
- name: Verify fixes are committed
id: fixer_changes
run: |
if ! git diff --quiet; then
git --no-pager diff --stat
echo "::error::There are issues with your Python code that will need to be fixed before they can be merged in. Run 'task engine:fix' to auto-fix what can be fixed automatically, then run 'task engine:check' to see what still needs fixing manually."
exit 1
fi
- name: Quality-check engine
id: engine-check
run: task engine:check
continue-on-error: true
- name: Comment on fixer failures
if: steps.fixer_changes.outcome == 'failure' && github.event_name == 'pull_request'
- name: Comment on engine check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.engine-check.outcome == 'failure' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
@@ -107,11 +176,39 @@ jobs:
});
}
- name: Run linting
run: task engine:lint
- name: Fail if engine check failed
if: steps.engine-check.outcome == 'failure'
run: |
echo "============================================"
echo " Engine Check Failed"
echo "============================================"
echo ""
echo "There are issues with your Python code that"
echo "will need to be fixed before they can be merged in."
echo ""
echo "Run 'task engine:fix' to auto-fix what can be"
echo "fixed automatically, then run 'task engine:check'"
echo "to see what still needs fixing manually."
echo "============================================"
exit 1
- name: Run type checking
run: task engine:typecheck
- name: Run tests
run: task engine:test
- name: Remove engine check comment on success
if: steps.engine-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- engine-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
+32 -15
View File
@@ -67,7 +67,7 @@ jobs:
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
- name: Comment on Java formatting failure
- name: Comment on backend format check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.spotless-check.outcome == 'failure' && github.event_name == 'pull_request'
@@ -78,15 +78,11 @@ jobs:
const marker = '<!-- java-formatting-check -->';
const body = [
marker,
'### Java Formatting Check Failed',
'### Backend Format Check Failed',
'',
'Your code has formatting issues. Run the following command to fix them:',
'There are formatting issues in your Java code that will need to be fixed before they can be merged in.',
'',
'```bash',
'task backend:format',
'```',
'',
'Then commit and push the changes.',
'Run `task backend:format` to auto-fix, then commit and push the changes.',
].join('\n');
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
@@ -110,22 +106,43 @@ jobs:
});
}
- name: Fail if Java formatting issues found
- name: Fail if backend format check failed
if: steps.spotless-check.outcome == 'failure'
run: |
echo "============================================"
echo " Java Formatting Check Failed"
echo " Backend Format Check Failed"
echo "============================================"
echo ""
echo "Your code has formatting issues."
echo "Run the following command to fix them:"
echo "There are formatting issues in your Java code"
echo "that will need to be fixed before they can be"
echo "merged in."
echo ""
echo " task backend:format"
echo ""
echo "Then commit and push the changes."
echo "Run 'task backend:format' to auto-fix, then"
echo "commit and push the changes."
echo "============================================"
exit 1
- name: Remove backend format check comment on success
if: steps.spotless-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- java-formatting-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
- name: Build with Gradle and spring security ${{ matrix.spring-security }}
run: task backend:build:ci
env:
+14
View File
@@ -68,6 +68,18 @@ jobs:
uses: ./.github/workflows/backend-build.yml
secrets: inherit
db-migration-test:
# Boots the current bootJar against H2 fixtures captured from past
# releases (v2.0.0 / v2.5.0 / v2.10.0) and verifies admin login still
# works after Hibernate's ddl-auto=update migrates the schema. Gated on
# the `project` filter so doc-only PRs skip this ~5-minute job.
if: needs.files-changed.outputs.project == 'true'
needs: [files-changed]
permissions:
contents: read
uses: ./.github/workflows/db-migration-test.yml
secrets: inherit
check-generateOpenApiDocs:
if: needs.files-changed.outputs.openapi == 'true'
needs: [files-changed]
@@ -184,6 +196,7 @@ jobs:
needs:
- files-changed
- build
- db-migration-test
- check-generateOpenApiDocs
- frontend-validation
- playwright-e2e
@@ -208,6 +221,7 @@ jobs:
RESULTS: |
files-changed=${{ needs.files-changed.result }}
build=${{ needs.build.result }}
db-migration-test=${{ needs.db-migration-test.result }}
check-generateOpenApiDocs=${{ needs.check-generateOpenApiDocs.result }}
frontend-validation=${{ needs.frontend-validation.result }}
playwright-e2e=${{ needs.playwright-e2e.result }}
+93
View File
@@ -0,0 +1,93 @@
name: DB migration smoke test
# Boots the current Stirling-PDF JAR against H2 fixtures captured from past
# releases (v2.0.0 / v2.5.0 / v2.10.0) and verifies admin login still works.
# Catches schema changes that would break existing user databases under
# Hibernate's `ddl-auto=update` upgrade path.
on:
workflow_call:
permissions:
contents: read
jobs:
pick:
uses: ./.github/workflows/_runner-pick.yml
migration-test:
needs: pick
runs-on: ${{ needs.pick.outputs.is_fork == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-8' }}
timeout-minutes: 30
env:
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
steps:
- name: Harden Runner
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up JDK 25
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
java-version: 25
distribution: temurin
- name: Cache Gradle dependency artifacts
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: |
~/.gradle/wrapper
~/.gradle/caches/modules-2/files-2.1
~/.gradle/caches/modules-2/metadata-2.*
key: gradle-deps-${{ runner.os }}-jdk-25-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties', '**/*.gradle', '**/*.gradle.kts', 'settings.gradle', 'settings.gradle.kts', 'gradle/libs.versions.toml') }}
- name: Setup Gradle
uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6.1.0
with:
gradle-version: 9.3.1
cache-disabled: true
# No `-PnoSpotless` here yet because the upstream cache layer matches the
# backend build's; reuse keeps cold-cache cost identical.
- name: Build Stirling-PDF JAR
env:
MAVEN_USER: ${{ secrets.MAVEN_USER }}
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
run: ./gradlew :stirling-pdf:bootJar -PnoSpotless --no-daemon
- name: Locate built JAR
id: jar
run: |
jar=$(find app/core/build/libs -maxdepth 1 -name 'Stirling-PDF*.jar' -o -name 'stirling-pdf*.jar' 2>/dev/null \
| grep -vE '(-plain|-sources)\.jar$' | head -n 1)
if [[ -z "$jar" ]]; then
echo "::error::No JAR under app/core/build/libs"
ls -lah app/core/build/libs || true
exit 1
fi
# Absolute path - the migration script pushd's into a temp workdir
# before invoking java, which would dangle a relative path.
jar=$(realpath "$jar")
echo "path=$jar" >> "$GITHUB_OUTPUT"
echo "Built JAR: $jar"
- name: Run migration smoke test
env:
STIRLING_JAR: ${{ steps.jar.outputs.path }}
run: bash scripts/db-migration/run-migration-test.sh
- name: Upload app logs on failure
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: db-migration-app-logs
# Path matches the preserved workdir in run-migration-test.sh -
# only failing fixtures leave a directory behind.
path: /tmp/stirling-migration-failed-*/app.log
retention-days: 7
if-no-files-found: warn
+9 -7
View File
@@ -586,21 +586,23 @@ jobs:
if: always() && steps.digicert-setup.conclusion != 'failure'
shell: bash
run: |
mkdir -p ./dist
# Absolute dist path so the cd below can't break the copy targets.
DIST="$GITHUB_WORKSPACE/dist"
mkdir -p "$DIST"
cd ./frontend/editor/src-tauri/target
# Find and rename artifacts based on platform
if [ "${{ matrix.platform }}" = "windows-latest" ]; then
# Only ship the MSI installer on Windows. The loose exe and WiX toolset exes
# are not the user-facing installer - the MSI contains the signed inner exe.
find . -name "*.msi" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.msi" \;
find . -name "*.msi" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.msi" \;
elif [ "${{ matrix.platform }}" = "macos-15" ]; then
find . -name "*.dmg" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.dmg" \;
find . -name "*.app" -exec cp -r {} "../../../dist/Stirling-PDF-${{ matrix.name }}.app" \;
find . -name "*.dmg" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.dmg" \;
find . -name "*.app" -exec cp -r {} "$DIST/Stirling-PDF-${{ matrix.name }}.app" \;
else
find . -name "*.deb" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.AppImage" \;
find . -name "*.deb" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.AppImage" \;
fi
- name: Upload build artifacts
+11 -6
View File
@@ -157,6 +157,9 @@ jobs:
JPDFIUM_PLATFORMS: ${{ matrix.jpdfium_platforms }}
run: task desktop:prepare
- name: Run Tauri/Cargo tests
run: task desktop:test
# DigiCert KeyLocker Setup (Cloud HSM)
- name: Setup DigiCert KeyLocker
id: digicert-setup
@@ -417,20 +420,22 @@ jobs:
- name: Rename artifacts
shell: bash
run: |
mkdir -p ./dist
# Absolute dist path so the cd below can't break the copy targets.
DIST="$GITHUB_WORKSPACE/dist"
mkdir -p "$DIST"
cd ./frontend/editor/src-tauri/target
# Find and rename artifacts based on platform
if [ "${{ matrix.platform }}" = "windows-latest" ]; then
# Only ship the MSI installer. The loose exe and WiX toolset exes
# are not the user-facing installer - the MSI contains the signed inner exe.
find . -name "*.msi" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.msi" \;
find . -name "*.msi" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.msi" \;
elif [ "${{ matrix.platform }}" = "macos-15" ]; then
find . -name "*.dmg" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.dmg" \;
find . -name "*.dmg" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.dmg" \;
else
find . -name "*.deb" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.AppImage" \;
find . -name "*.deb" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.AppImage" \;
fi
# Verify the MSI AND the inner exe extracted from it are signed.
+7 -1
View File
@@ -23,6 +23,10 @@ customFiles/
configs/
watchedFolders/
clientWebUI/
# Scratch dir used by local fixture-regeneration runs (see
# app/proprietary/src/test/resources/db-migration-fixtures/README.md).
# Holds downloaded JARs and disposable workdirs. Never committed.
.alpha-local/
!cucumber/
!cucumber/exampleFiles/
!cucumber/exampleFiles/example_html.zip
@@ -174,7 +178,6 @@ venv.bak/
# Env files (secrets / local overrides). Subproject .gitignore files whitelist any committed defaults.
.env*
!.env.saas.example
# VS Code
/.vscode/**/*
@@ -274,3 +277,6 @@ docs/type3/signatures/
# Playwright MCP screenshots / traces
.playwright-mcp/
*.playwright-mcp.png
# Local screenshot artifacts from *-screenshots.spec.ts
frontend/screenshots/
+6 -5
View File
@@ -22,12 +22,13 @@ tasks:
vars:
PORT: '{{.PORT | default "8080"}}'
AIENGINE_URL: '{{.AIENGINE_URL | default ""}}'
AIENGINE_TIMEOUTSECONDS: '{{.AIENGINE_TIMEOUTSECONDS | default "120"}}'
env:
SERVER_PORT: '{{.PORT}}'
cmds:
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true {{end}}cmd /c ".\gradlew.bat :stirling-pdf:bootRun"'
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true AIENGINE_TIMEOUTSECONDS={{.AIENGINE_TIMEOUTSECONDS}} {{end}}cmd /c ".\gradlew.bat :stirling-pdf:bootRun"'
platforms: [windows]
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true {{end}}./gradlew :stirling-pdf:bootRun'
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true AIENGINE_TIMEOUTSECONDS={{.AIENGINE_TIMEOUTSECONDS}} {{end}}./gradlew :stirling-pdf:bootRun'
platforms: [linux, darwin]
dev:bundled:
@@ -40,10 +41,10 @@ tasks:
platforms: [linux, darwin]
dev:saas:
desc: "Start backend in SaaS flavor against Supabase (loads .env.saas.local)"
desc: "Start backend in SaaS flavor against Supabase"
# `dotenv:` reads from the root Taskfile's directory (".") because this
# subtaskfile is included with `dir: .`. Drop the file at the repo root.
dotenv: ['.env.saas.local']
# subtaskfile is included with `dir: .`.
dotenv: ['app/.env.saas.local', 'app/.env.saas']
ignore_error: true
vars:
PORT: '{{.PORT | default "8080"}}'
+29 -1
View File
@@ -78,6 +78,13 @@ tasks:
cmds:
- npx tauri build --bundles appimage
test:
desc: "Run Tauri/Cargo tests"
deps: [prepare]
dir: editor/src-tauri
cmds:
- cargo test
clean:
desc: "Clean Tauri/Cargo build artifacts"
dir: editor
@@ -126,8 +133,29 @@ tasks:
--no-header-files
--no-man-pages
--output runtime/jre
# jlink emits its files mode 444 (read-only). Tauri's build-script
# resource copier preserves source permissions when staging
# `runtime/jre/**/*` into `target/<profile>/runtime/jre/...`, so the
# staged copies are read-only too. On any subsequent incremental
# build the copier tries to overwrite them and fails with a bare
# `Permission denied (os error 13)` (Rust's io::Error Display drops
# the path, so the failure is opaque). Make the source writable here
# so the staged destinations are writable and can be overwritten.
#
# Trade-off: this task runs for both `task desktop:dev` and
# `task desktop:build`, so production bundles also ship mode-644
# JRE files instead of 444. Functionally harmless on POSIX (the
# `other` bit is `r--` either way, and on macOS code signing is the
# real integrity check) and on Windows the DOS read-only attribute
# isn't load-bearing for the bundled JDK. If we ever need strict
# 444 in production, split the chmod into a dev-only step and have
# `desktop:build` run `jlink:clean` first to force a fresh build.
- cmd: chmod -R u+w runtime/jre
platforms: [linux, darwin]
- cmd: powershell -NoProfile -Command "Get-ChildItem -Recurse runtime/jre | ForEach-Object { $_.IsReadOnly = $false }"
platforms: [windows]
status:
- test -d editor/src-tauri/runtime/jre
- test -f runtime/jre/release
jlink:clean:
desc: "Remove JLink runtime and bundled JARs"
+4
View File
@@ -10,6 +10,10 @@ if that directory exists, is licensed under the license defined in "app/propriet
if that directory exists, is licensed under the license defined in "app/saas/LICENSE".
* All content that resides under the "engine/" directory of this repository,
if that directory exists, is licensed under the license defined in "engine/LICENSE".
* "scripts/pymupdf_convert.py", if that file exists, is licensed under the GNU Affero
General Public License v3.0 (or later) as declared in its file header. It is a separate
program invoked as an OS subprocess; its license does not extend to other content in
this repository.
* All content that resides under the "frontend/src/proprietary/" directory of this repository,
if that directory exists, is licensed under the license defined in "frontend/src/proprietary/LICENSE".
* All content that resides under the "frontend/src/desktop/" directory of this repository,
+18 -1
View File
@@ -58,6 +58,23 @@ tasks:
BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}'
OPEN: "true"
dev:saas:
desc: "Start SaaS backend + frontend concurrently on free ports"
vars:
PORTS:
sh: '{{if eq OS "windows"}}{{.FIND_FREE_PORT_PS}} 8080 5173{{else}}{{.FIND_FREE_PORT_SH}} 8080 5173{{end}}'
BACKEND_PORT: '{{index (splitList "\n" .PORTS) 0}}'
FRONTEND_PORT: '{{index (splitList "\n" .PORTS) 1}}'
deps:
- task: backend:dev:saas
vars:
PORT: '{{.BACKEND_PORT}}'
- task: frontend:dev:saas
vars:
PORT: '{{.FRONTEND_PORT}}'
BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}'
OPEN: "true"
dev:all:
desc: "Start backend + frontend + engine concurrently on free ports"
vars:
@@ -74,7 +91,7 @@ tasks:
vars:
PORT: '{{.BACKEND_PORT}}'
AIENGINE_URL: 'http://localhost:{{.ENGINE_PORT}}'
- task: frontend:dev:prototypes
- task: frontend:dev
vars:
PORT: '{{.FRONTEND_PORT}}'
BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}'
+9 -9
View File
@@ -1,20 +1,20 @@
###############################################################################
# Stirling-PDF SaaS local environment template.
# Stirling-PDF SaaS environment defaults.
#
# Copy this file to `.env.saas.local` (gitignored) and fill in real values.
# Loaded by `task backend:dev:saas` via Taskfile's `dotenv:` directive, then
# read by Spring Boot's `${...}` placeholders in application-saas.properties
# and application-dev.properties.
# This file is committed and provides non-secret defaults loaded by
# `task backend:dev:saas`. Put real values for secrets (passwords, project
# refs, edge function secrets) in `.env.saas.local` - any variable set there
# takes precedence over what's defined here.
#
# DO NOT commit `.env.saas.local`. Only `.env.saas.example` is checked in.
# DO NOT commit `.env.saas.local`. Only `.env.saas` is checked in.
###############################################################################
# ---------- Supabase project ----------
# Project reference (the subdomain part of <ref>.supabase.co). Required.
# Example dev project:
# Set in .env.saas.local.
SAAS_DB_PROJECT_REF=
# Edge function secret used by billing/license rollup calls.
# Edge function secret used by billing/license rollup calls. Set in .env.saas.local.
SUPABASE_EDGE_FUNCTION_SECRET=
# ---------- Database (saas profile) ----------
@@ -28,7 +28,7 @@ SAAS_DB_PASSWORD=
# ---------- Database (dev profile overrides) ----------
# Used when `--spring.profiles.include=dev` is active. The dev profile
# defaults the URL/username to the shared dev Supabase project, but the
# password must still be provided here.
# password must still be provided in .env.saas.local.
SAAS_DEV_DB_URL=
SAAS_DEV_DB_USERNAME=postgres
SAAS_DEV_DB_PASSWORD=
+3
View File
@@ -0,0 +1,3 @@
# Whitelist committed env defaults. `.env.saas.local` (and any other .env*)
# stays ignored via the root .gitignore.
!.env.saas
+4
View File
@@ -44,6 +44,10 @@
"moduleName": ".*",
"moduleLicense": "The MIT License"
},
{
"moduleName": ".*",
"moduleLicense": "MIT-0"
},
{
"moduleName": "com.github.jai-imageio:jai-imageio-core",
"moduleLicense": "LICENSE.txt"
@@ -77,6 +77,10 @@ public @interface AutoJobPostMapping {
/**
* Relative resource weight (1-100). See {@link
* stirling.software.common.enumeration.ResourceWeight} for the standard tiers.
*
* <p>The default is a sentinel ({@link Integer#MIN_VALUE}); {@code
* AutoJobPostMappingWeightTest} fails the build if any endpoint leaves it unset. Runtime
* readers clamp the value into {@code [1, 100]}.
*/
int resourceWeight() default 1;
int resourceWeight() default Integer.MIN_VALUE;
}
@@ -80,6 +80,7 @@ public class ConfigInitializer {
YamlHelper settingsFile = new YamlHelper(settingTempPath);
migrateEnterpriseEditionToPremium(settingsFile, settingsTemplateFile);
migrateProFeaturesKeyCasing(settingsFile, settingsTemplateFile);
boolean changesMade =
settingsTemplateFile.updateValuesFromYaml(settingsFile, settingsTemplateFile);
@@ -116,31 +117,52 @@ public class ConfigInitializer {
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "SSOAutoLogin") != null) {
template.updateValue(
List.of("premium", "proFeatures", "SSOAutoLogin"),
List.of("premium", "proFeatures", "ssoAutoLogin"),
yaml.getValueByExactKeyPath("enterpriseEdition", "SSOAutoLogin"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "autoUpdateMetadata")
!= null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "autoUpdateMetadata"),
List.of("premium", "proFeatures", "customMetadata", "autoUpdateMetadata"),
yaml.getValueByExactKeyPath(
"enterpriseEdition", "CustomMetadata", "autoUpdateMetadata"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "author") != null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "author"),
List.of("premium", "proFeatures", "customMetadata", "author"),
yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "author"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "creator") != null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "creator"),
List.of("premium", "proFeatures", "customMetadata", "creator"),
yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "creator"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "producer")
!= null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "producer"),
List.of("premium", "proFeatures", "customMetadata", "producer"),
yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "producer"));
}
}
// TODO: Remove post migration
// settings.yml.template renamed the two non-camelCase proFeatures keys
// ("SSOAutoLogin" -> "ssoAutoLogin", "CustomMetadata" -> "customMetadata") so the whole
// settings pipeline is consistent camelCase. The save path (YamlHelper.updateValue) matches
// keys case-sensitively, so without this carry-forward an existing install's values written
// under the old PascalCase keys would be dropped on upgrade and reset to template defaults.
void migrateProFeaturesKeyCasing(YamlHelper yaml, YamlHelper template) {
Object ssoAutoLogin = yaml.getValueByExactKeyPath("premium", "proFeatures", "SSOAutoLogin");
if (ssoAutoLogin != null) {
template.updateValue(List.of("premium", "proFeatures", "ssoAutoLogin"), ssoAutoLogin);
}
for (String field : List.of("autoUpdateMetadata", "author", "creator", "producer")) {
Object value =
yaml.getValueByExactKeyPath("premium", "proFeatures", "CustomMetadata", field);
if (value != null) {
template.updateValue(
List.of("premium", "proFeatures", "customMetadata", field), value);
}
}
}
}
@@ -528,6 +528,16 @@ public class ApplicationProperties {
private String provider;
private Client client = new Client();
/**
* When true, the OAuth2/OIDC login flow logs the full set of ID token and UserInfo
* claims at INFO level (and again at ERROR level if the username attribute cannot be
* resolved). Used to diagnose provider misconfiguration (for example ADFS not returning
* an {@code email} claim). WARNING: writes PII (sub, email, name) to application logs.
* Leave disabled in production; enable only while actively troubleshooting and disable
* again afterwards.
*/
private Boolean debugLogging = false;
public void setScopes(String scopes) {
List<String> scopesList =
Arrays.stream(scopes.split(",")).map(String::trim).toList();
@@ -778,6 +788,7 @@ public class ApplicationProperties {
private boolean enabled = false;
private String provider = "local";
private Local local = new Local();
private S3 s3 = new S3();
private Quotas quotas = new Quotas();
private Sharing sharing = new Sharing();
private Signing signing = new Signing();
@@ -787,6 +798,57 @@ public class ApplicationProperties {
private String basePath = InstallationPathConfig.getPath() + "storage";
}
@Data
public static class S3 {
/**
* Optional custom endpoint (e.g. {@code https://<account>.r2.cloudflarestorage.com},
* {@code https://<project>.supabase.co/storage/v1/s3}, or {@code http://localhost:9000}
* for MinIO). Blank = use AWS regional default.
*/
private String endpoint = "";
private String bucket = "";
private String region = "us-east-1";
private String accessKey = "";
private String secretKey = "";
/**
* When {@code true} use path-style URLs ({@code <endpoint>/<bucket>/<key>}) instead of
* virtual-hosted ({@code <bucket>.<endpoint>/<key>}). MinIO and most S3-compatible
* gateways require path-style; AWS S3 prefers virtual-hosted.
*/
private boolean pathStyleAccess = false;
/**
* When {@code false} (default), {@code endpoint} hostnames that resolve to private,
* loopback, or link-local addresses are rejected at startup to block SSRF attacks via
* the cloud metadata service (e.g. {@code http://169.254.169.254/}). Set to {@code
* true} to opt in for MinIO / in-cluster S3 endpoints on private networks.
*/
private boolean allowPrivateEndpoints = false;
/**
* Controls when the SDK adds an {@code x-amz-checksum-*} header on PUT/UploadPart.
* Default {@code WHEN_SUPPORTED} (the SDK default since 2.30) makes the SDK send a
* CRC32 checksum on every upload - this works on AWS S3, MinIO, current Supabase,
* Backblaze B2 (post-July-2025), and modern R2. Set to {@code WHEN_REQUIRED} to
* suppress the auto-checksum on vendors that reject unknown {@code x-amz-checksum-*}
* headers (older Backblaze B2, some R2 corner cases, GCS S3 endpoint). Invalid values
* fall back to {@code WHEN_SUPPORTED}.
*/
private String requestChecksumCalculation = "WHEN_SUPPORTED";
/**
* Controls when the SDK validates returned {@code x-amz-checksum-*} headers on GET
* responses. Default {@code WHEN_SUPPORTED}. Set to {@code WHEN_REQUIRED} if your
* vendor never returns these headers and you see false-positive checksum-mismatch
* errors. Invalid values fall back to {@code WHEN_SUPPORTED}.
*/
private String responseChecksumValidation = "WHEN_SUPPORTED";
}
@Data
public static class Sharing {
private boolean enabled = false;
@@ -1288,6 +1350,10 @@ public class ApplicationProperties {
public int getFfmpegSessionLimit() {
return ffmpegSessionLimit > 0 ? ffmpegSessionLimit : 2;
}
public int getPyMuPdfConvertSessionLimit() {
return 2;
}
}
@Data
@@ -1365,6 +1431,10 @@ public class ApplicationProperties {
public long getFfmpegTimeoutMinutes() {
return ffmpegTimeoutMinutes > 0 ? ffmpegTimeoutMinutes : 30;
}
public long getPyMuPdfConvertTimeoutMinutes() {
return 10;
}
}
}
}
@@ -0,0 +1,91 @@
package stirling.software.common.service;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Locale;
import java.util.concurrent.TimeUnit;
import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.util.ProcessExecutor;
/**
* Converts PDFs to Markdown by invoking the {@code pymupdf-convert} CLI tool as a separate
* subprocess.
*/
@Slf4j
@Service
public class PyMuPdfConverter {
private boolean available;
@PostConstruct
void init() {
available = probe();
if (available) {
log.info("pymupdf-convert found — PyMuPDF Markdown conversion enabled.");
} else {
log.info("pymupdf-convert not found — PyMuPDF Markdown conversion disabled.");
}
}
public boolean isAvailable() {
return available;
}
private boolean probe() {
boolean isWindows =
System.getProperty("os.name").toLowerCase(Locale.ROOT).contains("windows");
List<String> cmd =
isWindows
? List.of("where", "pymupdf-convert")
: List.of("which", "pymupdf-convert");
try {
Process p = new ProcessBuilder(cmd).redirectErrorStream(true).start();
boolean done = p.waitFor(5, TimeUnit.SECONDS);
return done && p.exitValue() == 0;
} catch (Exception e) {
log.debug("pymupdf-convert availability check failed: {}", e.getMessage());
return false;
}
}
/**
* Convert a PDF to Markdown by invoking {@code pymupdf-convert} as a subprocess.
*
* @throws IOException on process failure or if the tool is not installed
*/
public String convertToMarkdown(byte[] pdfBytes, String filename) throws IOException {
String safeName =
(filename == null || filename.isBlank())
? "document.pdf"
: filename.replace("\"", "");
Path tempDir = Files.createTempDirectory("stirling-pymupdf-");
Path inputPdf = tempDir.resolve(safeName);
Path outputMd = tempDir.resolve("output.md");
try {
Files.write(inputPdf, pdfBytes);
ProcessExecutor.getInstance(ProcessExecutor.Processes.PYMUPDF_CONVERT)
.runCommandWithOutputHandling(
List.of(
"pymupdf-convert",
inputPdf.toAbsolutePath().toString(),
outputMd.toAbsolutePath().toString()));
return Files.readString(outputMd, StandardCharsets.UTF_8);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("PyMuPDF conversion interrupted", e);
} finally {
Files.deleteIfExists(inputPdf);
Files.deleteIfExists(outputMd);
Files.deleteIfExists(tempDir);
}
}
}
@@ -4,6 +4,7 @@ import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
@@ -30,6 +31,7 @@ import io.github.pixee.security.Filenames;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.configuration.RuntimePathConfig;
import stirling.software.common.service.PyMuPdfConverter;
import stirling.software.common.util.ProcessExecutor.ProcessExecutorResult;
@Slf4j
@@ -159,6 +161,57 @@ public class PDFToFile {
finalOut, fileName, MediaType.APPLICATION_OCTET_STREAM);
}
/**
* PDF-&gt;Markdown with optional PyMuPDF acceleration.
*
* <p>When {@code pymupdf-convert} is installed and on PATH, conversion is delegated to it as a
* subprocess. On any failure — or when the tool is absent — this transparently falls back to
* the bundled {@code pdftohtml}-based converter.
*/
public ResponseEntity<Resource> processPdfToMarkdown(
MultipartFile inputFile, PyMuPdfConverter pyMuPdfConverter)
throws IOException, InterruptedException {
if (!MediaType.APPLICATION_PDF_VALUE.equals(inputFile.getContentType())) {
return new ResponseEntity<>(HttpStatus.BAD_REQUEST);
}
if (pyMuPdfConverter != null && pyMuPdfConverter.isAvailable()) {
try {
String originalName = Filenames.toSimpleFileName(inputFile.getOriginalFilename());
String baseName = originalName;
if (originalName != null && originalName.contains(".")) {
baseName = originalName.substring(0, originalName.lastIndexOf('.'));
}
String markdown =
pyMuPdfConverter.convertToMarkdown(inputFile.getBytes(), originalName);
return buildMarkdownZipResponse(markdown, baseName);
} catch (IOException e) {
log.warn(
"PyMuPDF conversion failed; falling back to pdftohtml converter: {}",
e.getMessage());
}
}
return processPdfToMarkdown(inputFile);
}
private ResponseEntity<Resource> buildMarkdownZipResponse(String markdown, String pdfBaseName)
throws IOException {
String fileName = pdfBaseName + "ToMarkdown.zip";
TempFile finalOut = tempFileManager.createManagedTempFile(".zip");
try (OutputStream fos = Files.newOutputStream(finalOut.getPath());
ZipOutputStream zipOutputStream = new ZipOutputStream(fos)) {
ZipEntry mdEntry = new ZipEntry(pdfBaseName + ".md");
zipOutputStream.putNextEntry(mdEntry);
zipOutputStream.write(markdown.getBytes(StandardCharsets.UTF_8));
zipOutputStream.closeEntry();
} catch (Exception e) {
finalOut.close();
throw e;
}
return WebResponseUtils.fileToWebResponse(
finalOut, fileName, MediaType.APPLICATION_OCTET_STREAM);
}
/**
* Updates image references in markdown to point to the images/ folder. Matches patterns like
* ![alt](filename.png) and converts to ![alt](images/filename.png)
@@ -115,6 +115,11 @@ public class ProcessExecutor {
.getProcessExecutor()
.getSessionLimit()
.getFfmpegSessionLimit();
case PYMUPDF_CONVERT ->
applicationProperties
.getProcessExecutor()
.getSessionLimit()
.getPyMuPdfConvertSessionLimit();
};
long timeoutMinutes =
@@ -180,6 +185,11 @@ public class ProcessExecutor {
.getProcessExecutor()
.getTimeoutMinutes()
.getFfmpegTimeoutMinutes();
case PYMUPDF_CONVERT ->
applicationProperties
.getProcessExecutor()
.getTimeoutMinutes()
.getPyMuPdfConvertTimeoutMinutes();
};
return new ProcessExecutor(
processType, semaphoreLimit, liveUpdates, timeoutMinutes);
@@ -550,7 +560,8 @@ public class ProcessExecutor {
GHOSTSCRIPT,
OCR_MY_PDF,
CFF_CONVERTER,
FFMPEG
FFMPEG,
PYMUPDF_CONVERT
}
@Setter
@@ -83,7 +83,16 @@ public class RequestUriUtils {
return false;
}
// Blocklist of backend/non-frontend paths that should still go through filters
// Blocklist of backend/non-frontend paths that should still go through filters.
//
// `/files` was historically a backend route; it is now a frontend route
// owned by HomePage / FileManagerView. Direct-nav or refresh on /files
// (or /files/<folder-uuid>) was returning the Spring auth filter's 401
// JSON instead of serving index.html, so the SPA never got a chance to
// mount and the user saw a raw error response. There are no `/files`
// backend mappings at the servlet root - the real storage endpoints
// live under `/api/v1/storage/files`, which is filtered out a few lines
// up by the `startsWith("/api/")` guard.
String[] backendOnlyPrefixes = {
"/register",
"/pipeline",
@@ -91,7 +100,6 @@ public class RequestUriUtils {
"/pdfjs-legacy",
"/fonts",
"/images",
"/files",
"/css",
"/js",
"/swagger",
@@ -181,7 +189,7 @@ public class RequestUriUtils {
|| trimmedUri.startsWith(
"/api/v1/mobile-scanner/") // Mobile scanner endpoints (no auth)
|| trimmedUri.startsWith("/v1/api-docs")
// Workflow participant endpoints access controlled by share tokens, not login
// Workflow participant endpoints - access controlled by share tokens, not login
|| trimmedUri.startsWith("/api/v1/workflow/participant/")
// Share-link SPA bootstrap; data APIs remain protected
|| trimmedUri.matches("^/share/[^/]+/?$");
@@ -0,0 +1,95 @@
package stirling.software.common.configuration;
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
import org.snakeyaml.engine.v2.api.LoadSettings;
import stirling.software.common.util.YamlHelper;
class ConfigInitializerTest {
private static final LoadSettings LOAD_SETTINGS =
LoadSettings.builder()
.setUseMarks(true)
.setMaxAliasesForCollections(Integer.MAX_VALUE)
.setAllowRecursiveKeys(true)
.setParseComments(true)
.build();
// Mirrors the proFeatures block of settings.yml.template after the camelCase rename.
private static final String CAMEL_CASE_TEMPLATE =
"""
premium:
proFeatures:
ssoAutoLogin: false
customMetadata:
autoUpdateMetadata: false
author: username
creator: Stirling-PDF
producer: Stirling-PDF
""";
@Test
void migrateProFeaturesKeyCasing_carriesForwardLegacyPascalCaseValues() {
// An existing install whose settings.yml still uses the old PascalCase keys.
String legacy =
"""
premium:
proFeatures:
SSOAutoLogin: true
CustomMetadata:
autoUpdateMetadata: true
author: alice
creator: bob
producer: carol
""";
YamlHelper template = new YamlHelper(LOAD_SETTINGS, CAMEL_CASE_TEMPLATE);
YamlHelper existing = new YamlHelper(LOAD_SETTINGS, legacy);
new ConfigInitializer().migrateProFeaturesKeyCasing(existing, template);
assertEquals(
"true", template.getValueByExactKeyPath("premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"true",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "autoUpdateMetadata"));
assertEquals(
"alice",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "author"));
assertEquals(
"bob",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "creator"));
assertEquals(
"carol",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "producer"));
}
@Test
void migrateProFeaturesKeyCasing_withoutLegacyKeys_keepsTemplateDefaults() {
// No PascalCase keys present -> this migration step must be a no-op.
String alreadyCamel =
"""
premium:
proFeatures:
ssoAutoLogin: true
customMetadata:
author: dave
""";
YamlHelper template = new YamlHelper(LOAD_SETTINGS, CAMEL_CASE_TEMPLATE);
YamlHelper existing = new YamlHelper(LOAD_SETTINGS, alreadyCamel);
new ConfigInitializer().migrateProFeaturesKeyCasing(existing, template);
assertEquals(
"false", template.getValueByExactKeyPath("premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"username",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "author"));
}
}
@@ -2,6 +2,8 @@ package stirling.software.common.util;
import static org.junit.jupiter.api.Assertions.*;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
@@ -12,9 +14,47 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
import stirling.software.common.configuration.InstallationPathConfig;
public class GeneralUtilsTest {
// Regression guard for the SSO auto-login persistence bug: the admin UI writes camelCase
// proFeatures keys, so saveKeyToSettings must match (and persist) them against the camelCase
// settings.yml.template. A case mismatch makes YamlHelper.updateValue silently no-op.
@Test
void saveKeyToSettings_persistsCamelCaseProFeatureKeys(@TempDir Path tempDir) throws Exception {
Path settings = tempDir.resolve("settings.yml");
Files.writeString(
settings,
"""
premium:
proFeatures:
ssoAutoLogin: false
customMetadata:
author: username
""");
try (MockedStatic<InstallationPathConfig> mocked =
Mockito.mockStatic(InstallationPathConfig.class)) {
mocked.when(InstallationPathConfig::getSettingsPath).thenReturn(settings.toString());
GeneralUtils.saveKeyToSettings("premium.proFeatures.ssoAutoLogin", true);
GeneralUtils.saveKeyToSettings("premium.proFeatures.customMetadata.author", "alice");
}
YamlHelper reloaded = new YamlHelper(settings);
assertEquals(
"true", reloaded.getValueByExactKeyPath("premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"alice",
reloaded.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "author"));
}
@Test
void testParsePageListWithAll() {
List<Integer> result = GeneralUtils.parsePageList(new String[] {"all"}, 5, false);
@@ -98,6 +98,17 @@ class RequestUriUtilsTest {
assertTrue(RequestUriUtils.isFrontendRoute("", "/split-pdf"));
}
@Test
void testIsFrontendRoute_filesRouteOwnedByFrontend() {
// /files and /files/<folder-uuid> are FileManagerView routes - they
// must fall through to the SPA index.html, not get blocked by the
// backend auth filter. Regression test for direct-nav/refresh on
// the file manager returning a 401 JSON.
assertTrue(RequestUriUtils.isFrontendRoute("", "/files"));
assertTrue(
RequestUriUtils.isFrontendRoute("", "/files/3331910a-4155-4f71-8111-e38c896bc458"));
}
@Test
void testIsFrontendRoute_pathWithExtension() {
assertFalse(RequestUriUtils.isFrontendRoute("", "/some/file.pdf"));
@@ -183,7 +194,7 @@ class RequestUriUtilsTest {
@Test
void testIsPublicAuthEndpoint_shareRootNotPublic() {
// Avoid matching bare "/share" or "/share/" must have a token segment
// Avoid matching bare "/share" or "/share/" - must have a token segment
assertFalse(RequestUriUtils.isPublicAuthEndpoint("/share", ""));
assertFalse(RequestUriUtils.isPublicAuthEndpoint("/share/", ""));
}
@@ -197,7 +208,7 @@ class RequestUriUtilsTest {
@Test
void testIsPublicAuthEndpoint_shareApiStillProtected() {
// Share-link data APIs must NOT be public they enforce auth + access checks
// Share-link data APIs must NOT be public - they enforce auth + access checks
assertFalse(RequestUriUtils.isPublicAuthEndpoint("/api/v1/storage/share-links/abc123", ""));
assertFalse(
RequestUriUtils.isPublicAuthEndpoint(
@@ -21,6 +21,13 @@ public class EndpointInterceptor implements HandlerInterceptor {
HttpServletRequest request, HttpServletResponse response, Object handler)
throws Exception {
String requestURI = request.getRequestURI();
// Prevent API responses from being stored by browsers or intermediary caches by default
String servletPath = request.getServletPath();
if (servletPath != null && servletPath.startsWith("/api/")) {
response.setHeader("Cache-Control", "private, no-store");
}
boolean isEnabled = endpointConfiguration.isEndpointEnabledForUri(requestURI);
if (!isEnabled) {
response.sendError(HttpServletResponse.SC_FORBIDDEN, "This endpoint is disabled");
@@ -101,6 +101,9 @@ public class ExternalAppDepConfig {
// Python / OpenCV special handling
checkPythonAndOpenCV();
// PyMuPDF optional acceleration
checkPyMuPdf();
dependenciesChecked = true;
} finally {
endpointConfiguration.logDisabledEndpointsSummary();
@@ -236,6 +239,15 @@ public class ExternalAppDepConfig {
}
}
private void checkPyMuPdf() {
if (isCommandAvailable("pymupdf-convert")) {
log.warn("pymupdf-convert detected — PDF->Markdown will use PyMuPDF acceleration.");
} else {
log.info(
"pymupdf-convert not found — PDF->Markdown will use the bundled pdftohtml converter.");
}
}
private void disablePythonAndOpenCV(String reason) {
List<String> pythonFeatures = getAffectedFeatures("Python");
List<String> openCVFeatures = getAffectedFeatures("OpenCV");
@@ -1,5 +1,8 @@
package stirling.software.SPDF.config;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
@@ -24,6 +27,10 @@ public class WebMvcConfig implements WebMvcConfigurer {
private static final Logger logger = LoggerFactory.getLogger(WebMvcConfig.class);
private static final CacheControl NO_CACHE = CacheControl.noCache();
private static final CacheControl IMMUTABLE_ONE_YEAR =
CacheControl.maxAge(365, TimeUnit.DAYS).cachePublic().immutable();
@Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(endpointInterceptor);
@@ -31,37 +38,95 @@ public class WebMvcConfig implements WebMvcConfigurer {
@Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
// Cache hashed assets (JS/CSS with content hashes) for 1 year
// These files have names like index-ChAS4tCC.js that change when content changes
// Check customFiles/static first, then fall back to classpath
String staticPath =
"file:"
+ stirling.software.common.configuration.InstallationPathConfig
.getStaticPath();
// 1. Service worker and PWA metadata (never store)
// Browsers revalidate SW bytes anyway; no-store is the safest for atomic updates.
registry.addResourceHandler(
"/sw.js", "/manifest.json", "/site.webmanifest", "/browserconfig.xml")
.addResourceLocations(staticPath, "classpath:/static/")
.setCacheControl(CacheControl.noStore())
.resourceChain(true);
// 2. Vite fingerprinted assets (immutable)
// These already have content hashes in filenames (e.g. index-ChAS4tCC.js)
registry.addResourceHandler("/assets/**")
.addResourceLocations(
"file:"
+ stirling.software.common.configuration.InstallationPathConfig
.getStaticPath()
+ "assets/",
"classpath:/static/assets/")
.setCacheControl(CacheControl.maxAge(365, TimeUnit.DAYS).cachePublic());
.addResourceLocations(staticPath + "assets/", "classpath:/static/assets/")
.setCacheControl(IMMUTABLE_ONE_YEAR)
.resourceChain(true);
// Don't cache index.html - it needs to be fresh to reference latest hashed assets
// Note: index.html is handled by ReactRoutingController for dynamic processing
registry.addResourceHandler("/index.html")
// 3. Media and fonts (immutable)
registry.addResourceHandler("/images/**", "/fonts/**")
.addResourceLocations(
"file:"
+ stirling.software.common.configuration.InstallationPathConfig
.getStaticPath(),
"classpath:/static/")
.setCacheControl(CacheControl.noCache().mustRevalidate());
staticPath + "images/",
"classpath:/static/images/",
staticPath + "fonts/",
"classpath:/static/fonts/")
.setCacheControl(IMMUTABLE_ONE_YEAR)
.resourceChain(true);
// Handle all other static resources (js, css, images, fonts, etc.)
// Check customFiles/static first for user overrides
// 4. Branding and stable non-fingerprinted assets (1 day + SWR)
// Use stale-while-revalidate to improve perceived performance.
registry.addResourceHandler(
"/favicon.*",
"/apple-touch-icon.png",
"/android-chrome-*.png",
"/mstile-*.png",
"/safari-pinned-tab.svg",
"/icons/**",
"/modern-logo/**",
"/classic-logo/**",
"/robots.txt",
"/3rdPartyLicenses.json",
"/pdfjs/**",
"/pdfjs-legacy/**",
"/pdfium/**",
"/locales/**",
"/css/**",
"/js/**",
"/vendor/**",
"/samples/**",
"/og_images/**",
"/Login/**",
"/manifest-classic.json")
.addResourceLocations(
staticPath,
"classpath:/static/",
staticPath + "pdfjs/",
"classpath:/static/pdfjs/",
staticPath + "pdfjs-legacy/",
"classpath:/static/pdfjs-legacy/",
staticPath + "pdfium/",
"classpath:/static/pdfium/",
staticPath + "locales/",
"classpath:/static/locales/",
staticPath + "css/",
"classpath:/static/css/",
staticPath + "js/",
"classpath:/static/js/",
staticPath + "vendor/",
"classpath:/static/vendor/",
staticPath + "samples/",
"classpath:/static/samples/",
staticPath + "og_images/",
"classpath:/static/og_images/",
staticPath + "Login/",
"classpath:/static/Login/")
.setCacheControl(
CacheControl.maxAge(Duration.ofDays(1))
.cachePublic()
.staleWhileRevalidate(Duration.ofDays(7)))
.resourceChain(true);
// 5. Catch-all (SPA fallback)
// Must check with server to ensure index.html is always fresh.
registry.addResourceHandler("/**")
.addResourceLocations(
"file:"
+ stirling.software.common.configuration.InstallationPathConfig
.getStaticPath(),
"classpath:/static/")
.setCacheControl(CacheControl.maxAge(1, TimeUnit.HOURS));
.addResourceLocations(staticPath, "classpath:/static/")
.setCacheControl(NO_CACHE)
.resourceChain(true);
}
@Override
@@ -115,9 +180,8 @@ public class WebMvcConfig implements WebMvcConfigurer {
applicationProperties.getSystem().getCorsAllowedOrigins());
// Combine user-configured origins with Tauri origins
java.util.List<String> allOrigins =
new java.util.ArrayList<>(
applicationProperties.getSystem().getCorsAllowedOrigins());
List<String> allOrigins =
new ArrayList<>(applicationProperties.getSystem().getCorsAllowedOrigins());
// Always include Tauri origins for desktop app compatibility
// Tauri v1 uses tauri://localhost, v2 uses http(s)://tauri.localhost
@@ -158,7 +222,8 @@ public class WebMvcConfig implements WebMvcConfigurer {
} else {
// Default to allowing all origins when nothing is configured
logger.debug(
"No CORS allowed origins configured in settings.yml (system.corsAllowedOrigins); WebMvcConfig allowing all origins.");
"No CORS allowed origins configured in settings.yml"
+ " (system.corsAllowedOrigins); WebMvcConfig allowing all origins.");
registry.addMapping("/**")
.allowedOriginPatterns("*")
.allowedMethods("GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS")
@@ -32,6 +32,7 @@ import stirling.software.SPDF.model.json.PdfJsonTextElement;
import stirling.software.SPDF.service.PdfJsonConversionService;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.annotations.api.GeneralApi;
import stirling.software.common.enumeration.ResourceWeight;
import stirling.software.common.model.api.general.EditTextOperation;
import stirling.software.common.util.ExceptionUtils;
import stirling.software.common.util.GeneralUtils;
@@ -75,7 +76,10 @@ public class EditTextController {
new StringToArrayListPropertyEditor<>(EditTextOperation.class));
}
@AutoJobPostMapping(consumes = "multipart/form-data", value = "/edit-text")
@AutoJobPostMapping(
consumes = "multipart/form-data",
value = "/edit-text",
resourceWeight = ResourceWeight.LARGE_WEIGHT)
@StandardPdfResponse
@Operation(
summary = "Edit text in a PDF via find and replace",
@@ -40,7 +40,8 @@ public class ScalePagesController {
private final CustomPDFDocumentFactory pdfDocumentFactory;
private final TempFileManager tempFileManager;
private static PDRectangle getTargetSize(String targetPDRectangle, PDDocument sourceDocument) {
private static PDRectangle getTargetSize(
String targetPDRectangle, String orientation, PDDocument sourceDocument) {
if ("KEEP".equals(targetPDRectangle)) {
if (sourceDocument.getNumberOfPages() == 0) {
throw ExceptionUtils.createInvalidPageSizeException("KEEP");
@@ -57,18 +58,19 @@ public class ScalePagesController {
}
Map<String, PDRectangle> sizeMap = getSizeMap();
if (sizeMap.containsKey(targetPDRectangle)) {
return sizeMap.get(targetPDRectangle);
PDRectangle base = sizeMap.get(targetPDRectangle);
if (base == null) {
throw ExceptionUtils.createInvalidPageSizeException(targetPDRectangle);
}
throw ExceptionUtils.createInvalidPageSizeException(targetPDRectangle);
if ("LANDSCAPE".equalsIgnoreCase(orientation)) {
return new PDRectangle(base.getHeight(), base.getWidth());
}
return base;
}
private static Map<String, PDRectangle> getSizeMap() {
Map<String, PDRectangle> sizeMap = new HashMap<>();
// Portrait sizes (A0-A6)
sizeMap.put("A0", PDRectangle.A0);
sizeMap.put("A1", PDRectangle.A1);
sizeMap.put("A2", PDRectangle.A2);
@@ -76,42 +78,8 @@ public class ScalePagesController {
sizeMap.put("A4", PDRectangle.A4);
sizeMap.put("A5", PDRectangle.A5);
sizeMap.put("A6", PDRectangle.A6);
// Landscape sizes (A0-A6)
sizeMap.put(
"A0_LANDSCAPE",
new PDRectangle(PDRectangle.A0.getHeight(), PDRectangle.A0.getWidth()));
sizeMap.put(
"A1_LANDSCAPE",
new PDRectangle(PDRectangle.A1.getHeight(), PDRectangle.A1.getWidth()));
sizeMap.put(
"A2_LANDSCAPE",
new PDRectangle(PDRectangle.A2.getHeight(), PDRectangle.A2.getWidth()));
sizeMap.put(
"A3_LANDSCAPE",
new PDRectangle(PDRectangle.A3.getHeight(), PDRectangle.A3.getWidth()));
sizeMap.put(
"A4_LANDSCAPE",
new PDRectangle(PDRectangle.A4.getHeight(), PDRectangle.A4.getWidth()));
sizeMap.put(
"A5_LANDSCAPE",
new PDRectangle(PDRectangle.A5.getHeight(), PDRectangle.A5.getWidth()));
sizeMap.put(
"A6_LANDSCAPE",
new PDRectangle(PDRectangle.A6.getHeight(), PDRectangle.A6.getWidth()));
// Portrait US sizes
sizeMap.put("LETTER", PDRectangle.LETTER);
sizeMap.put("LEGAL", PDRectangle.LEGAL);
// Landscape US sizes
sizeMap.put(
"LETTER_LANDSCAPE",
new PDRectangle(PDRectangle.LETTER.getHeight(), PDRectangle.LETTER.getWidth()));
sizeMap.put(
"LEGAL_LANDSCAPE",
new PDRectangle(PDRectangle.LEGAL.getHeight(), PDRectangle.LEGAL.getWidth()));
return sizeMap;
}
@@ -128,13 +96,14 @@ public class ScalePagesController {
throws IOException {
MultipartFile file = request.getFileInput();
String targetPDRectangle = request.getPageSize();
String orientation = request.getOrientation();
float scaleFactor = request.getScaleFactor();
try (PDDocument sourceDocument = pdfDocumentFactory.load(file);
PDDocument outputDocument =
pdfDocumentFactory.createNewDocumentBasedOnOldDocument(sourceDocument)) {
PDRectangle targetSize = getTargetSize(targetPDRectangle, sourceDocument);
PDRectangle targetSize = getTargetSize(targetPDRectangle, orientation, sourceDocument);
// Create LayerUtility once outside the loop for better performance
LayerUtility layerUtility = new LayerUtility(outputDocument);
@@ -275,7 +275,10 @@ public class ConvertImgPDFController {
GeneralUtils.generateFilename(file[0].getOriginalFilename(), "_converted.pdf"));
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/cbz/pdf")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/cbz/pdf",
resourceWeight = ResourceWeight.MEDIUM_WEIGHT)
@Operation(
summary = "Convert CBZ comic book archive to PDF",
description =
@@ -301,7 +304,10 @@ public class ConvertImgPDFController {
return WebResponseUtils.pdfFileToWebResponse(pdfFile, filename);
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/pdf/cbz")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/pdf/cbz",
resourceWeight = ResourceWeight.LARGE_WEIGHT)
@Operation(
summary = "Convert PDF to CBZ comic book archive",
description =
@@ -324,7 +330,10 @@ public class ConvertImgPDFController {
return WebResponseUtils.zipFileToWebResponse(cbzFile, filename);
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/cbr/pdf")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/cbr/pdf",
resourceWeight = ResourceWeight.MEDIUM_WEIGHT)
@Operation(
summary = "Convert CBR comic book archive to PDF",
description =
@@ -350,7 +359,10 @@ public class ConvertImgPDFController {
return WebResponseUtils.bytesToWebResponse(pdfBytes, filename);
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/pdf/cbr")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/pdf/cbr",
resourceWeight = ResourceWeight.LARGE_WEIGHT)
@Operation(
summary = "Convert PDF to CBR comic book archive",
description =
@@ -141,7 +141,8 @@ public class AttachmentController {
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/extract-attachments")
value = "/extract-attachments",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@Operation(
summary = "Extract attachments from PDF",
description =
@@ -176,7 +177,10 @@ public class AttachmentController {
}
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/list-attachments")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/list-attachments",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@Operation(
summary = "List attachments in PDF",
description =
@@ -193,7 +197,8 @@ public class AttachmentController {
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/rename-attachment")
value = "/rename-attachment",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@StandardPdfResponse
@Operation(
summary = "Rename attachment in PDF",
@@ -228,7 +233,8 @@ public class AttachmentController {
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/delete-attachment")
value = "/delete-attachment",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@StandardPdfResponse
@Operation(
summary = "Delete attachment from PDF",
@@ -12,6 +12,8 @@ import org.springframework.web.bind.annotation.RequestParam;
import io.swagger.v3.oas.annotations.Hidden;
import jakarta.servlet.http.HttpServletRequest;
import lombok.extern.slf4j.Slf4j;
import stirling.software.SPDF.config.EndpointConfiguration;
@@ -91,6 +93,63 @@ public class ConfigController {
return null;
}
/**
* Resolve the frontend URL the client should advertise to phones / share-link recipients.
* Priority: explicit system.frontendUrl, then the Host the user is already using to reach this
* server (works for Docker, reverse proxies, and bare-metal LANs), then a detected site-local
* IPv4, then empty.
*/
// visible for testing
String resolveFrontendUrl(HttpServletRequest request, AppConfig appConfig) {
String configured = applicationProperties.getSystem().getFrontendUrl();
if (configured != null && !configured.isBlank()) {
return configured;
}
if (request != null) {
String host = request.getServerName();
if (host != null && !host.isBlank() && !isLoopbackHost(host)) {
String scheme = request.getScheme();
int port = request.getServerPort();
boolean defaultPort =
("http".equals(scheme) && port == 80)
|| ("https".equals(scheme) && port == 443);
return defaultPort ? scheme + "://" + host : scheme + "://" + host + ":" + port;
}
}
String localIp = GeneralUtils.getLocalNetworkIp();
if (localIp != null) {
String scheme = appConfig.getBackendUrl().startsWith("https") ? "https" : "http";
return scheme + "://" + localIp + ":" + resolveEffectiveServerPort(appConfig);
}
return "";
}
/**
* The port the embedded server is actually listening on. With {@code server.port=0} (an
* ephemeral port, which the desktop bundle uses to dodge port clashes) the configured value
* stays {@code "0"} while Spring publishes the real bound port as {@code local.server.port}
* once the server is up. Advertised URLs (the mobile-scanner QR, share links) must carry the
* real port - a literal {@code :0} is unreachable and browsers reject it as ERR_UNSAFE_PORT.
*/
// visible for testing
String resolveEffectiveServerPort(AppConfig appConfig) {
String configured = appConfig.getServerPort();
if (configured == null || "0".equals(configured.trim())) {
String actual = applicationContext.getEnvironment().getProperty("local.server.port");
if (actual != null && !actual.isBlank()) {
return actual;
}
}
return configured;
}
private static boolean isLoopbackHost(String host) {
return "localhost".equalsIgnoreCase(host)
|| "127.0.0.1".equals(host)
|| "::1".equals(host)
|| "0:0:0:0:0:0:0:1".equals(host);
}
/** Check if running Enterprise edition dynamically. */
private Boolean isRunningEE() {
// Use LicenseService for fresh license status if available
@@ -107,7 +166,7 @@ public class ConfigController {
}
@GetMapping("/app-config")
public ResponseEntity<Map<String, Object>> getAppConfig() {
public ResponseEntity<Map<String, Object>> getAppConfig(HttpServletRequest request) {
Map<String, Object> configData = new HashMap<>();
try {
@@ -121,20 +180,10 @@ public class ConfigController {
// Note: Frontend expects "baseUrl" field name for compatibility
configData.put("baseUrl", appConfig.getBackendUrl());
configData.put("contextPath", appConfig.getContextPath());
configData.put("serverPort", appConfig.getServerPort());
configData.put("serverPort", resolveEffectiveServerPort(appConfig));
String frontendUrl = applicationProperties.getSystem().getFrontendUrl();
if ((frontendUrl == null || frontendUrl.isBlank())
&& Boolean.parseBoolean(
System.getProperty("STIRLING_PDF_TAURI_MODE", "false"))) {
String localIp = GeneralUtils.getLocalNetworkIp();
if (localIp != null) {
String scheme =
appConfig.getBackendUrl().startsWith("https") ? "https" : "http";
frontendUrl = scheme + "://" + localIp + ":" + appConfig.getServerPort();
}
}
configData.put("frontendUrl", frontendUrl != null ? frontendUrl : "");
configData.put("frontendUrl", resolveFrontendUrl(request, appConfig));
// Add mobile scanner settings
configData.put(
@@ -277,6 +326,9 @@ public class ConfigController {
// Premium/Enterprise settings
configData.put("premiumEnabled", applicationProperties.getPremium().isEnabled());
// AI Engine settings
configData.put("aiEngineEnabled", applicationProperties.getAiEngine().isEnabled());
// Timestamp TSA settings — single source of truth for presets + admin URLs
ApplicationProperties.Security.Timestamp tsConfig =
applicationProperties.getSecurity().getTimestamp();
@@ -12,6 +12,7 @@ import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.Resource;
import org.springframework.http.CacheControl;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
@@ -134,13 +135,22 @@ public class ReactRoutingController {
public ResponseEntity<String> serveIndexHtml(HttpServletRequest request) {
try {
if (indexHtmlExists && cachedIndexHtml != null) {
return ResponseEntity.ok().contentType(MediaType.TEXT_HTML).body(cachedIndexHtml);
return ResponseEntity.ok()
.cacheControl(CacheControl.noCache().mustRevalidate())
.contentType(MediaType.TEXT_HTML)
.body(cachedIndexHtml);
}
// Fallback: process on each request (dev mode or cache failed)
return ResponseEntity.ok().contentType(MediaType.TEXT_HTML).body(processIndexHtml());
return ResponseEntity.ok()
.cacheControl(CacheControl.noCache().mustRevalidate())
.contentType(MediaType.TEXT_HTML)
.body(processIndexHtml());
} catch (Exception ex) {
log.error("Failed to serve index.html, returning fallback", ex);
return ResponseEntity.ok().contentType(MediaType.TEXT_HTML).body(buildFallbackHtml());
return ResponseEntity.ok()
.cacheControl(CacheControl.noCache().mustRevalidate())
.contentType(MediaType.TEXT_HTML)
.body(buildFallbackHtml());
}
}
@@ -160,14 +170,19 @@ public class ReactRoutingController {
return ResponseEntity.ok().contentType(MediaType.TEXT_HTML).body(cachedCallbackHtml);
}
// `files` was historically a backend static-asset directory and was therefore
// in the exclusion list - removing it lets /files and /files/<folder-uuid>
// forward to the SPA index.html, which is what FileManagerView expects.
// (Real storage endpoints live under /api/v1/storage/files, already
// excluded by the leading `api` token in the same regex.)
@GetMapping(
"/{path:^(?!api|static|robots\\.txt|favicon\\.ico|manifest.*\\.json|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|files|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*$}")
"/{path:^(?!api|static|robots\\.txt|favicon\\.ico|manifest.*\\.json|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*$}")
public ResponseEntity<String> forwardRootPaths(HttpServletRequest request) throws IOException {
return serveIndexHtml(request);
}
@GetMapping(
"/{path:^(?!api|static|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|files|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*}/{subpath:^(?!.*\\.).*$}")
"/{path:^(?!api|static|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*}/{subpath:^(?!.*\\.).*$}")
public ResponseEntity<String> forwardNestedPaths(HttpServletRequest request)
throws IOException {
return serveIndexHtml(request);
@@ -22,6 +22,7 @@ import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.multipart.MaxUploadSizeExceededException;
import org.springframework.web.multipart.support.MissingServletRequestPartException;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.NoHandlerFoundException;
import jakarta.servlet.http.HttpServletRequest;
@@ -196,12 +197,12 @@ public class GlobalExceptionHandler {
/**
* Checks whether the given IOException indicates that the client disconnected before the
* response could be written (broken pipe, connection reset, etc.). When this happens there is
* no point in serialising a {@link ProblemDetail} body because the socket is already closed
* no point in serialising a {@link ProblemDetail} body because the socket is already closed -
* and attempting to do so may trigger a secondary {@code HttpMessageNotWritableException} if
* the response Content-Type was already committed as a non-JSON type (e.g. image/png).
*/
private static boolean isClientDisconnectException(IOException ex) {
// Walk the causal chain Jetty/Tomcat may wrap the low-level SocketException
// Walk the causal chain - Jetty/Tomcat may wrap the low-level SocketException
Throwable current = ex;
while (current != null) {
String msg = current.getMessage();
@@ -1040,6 +1041,43 @@ public class GlobalExceptionHandler {
* @param request the HTTP servlet request
* @return ProblemDetail with appropriate HTTP status
*/
/**
* Handle ResponseStatusException explicitly so its embedded HTTP status reaches the client
* instead of being swallowed by the {@code RuntimeException} catch-all (which would downgrade
* every controller-thrown 400/404/409 to a generic 500). Folder/file storage controllers and
* any other code that throws {@code ResponseStatusException} relies on this handler taking
* precedence.
*/
@ExceptionHandler(ResponseStatusException.class)
public ResponseEntity<ProblemDetail> handleResponseStatusException(
ResponseStatusException ex, HttpServletRequest request) {
HttpStatus status =
HttpStatus.resolve(ex.getStatusCode().value()) != null
? HttpStatus.valueOf(ex.getStatusCode().value())
: HttpStatus.INTERNAL_SERVER_ERROR;
String reason = ex.getReason() != null ? ex.getReason() : status.getReasonPhrase();
ProblemDetail problemDetail = createBaseProblemDetail(status, reason, request);
problemDetail.setType(URI.create("/errors/" + status.value()));
problemDetail.setTitle(status.getReasonPhrase());
problemDetail.setProperty("title", status.getReasonPhrase());
// 5xx is operator-relevant; 4xx is a normal client-rejection - log at the right level.
if (status.is5xxServerError()) {
log.error(
"ResponseStatusException {} at {}: {}",
status.value(),
request.getRequestURI(),
reason,
ex);
} else {
log.debug(
"ResponseStatusException {} at {}: {}",
status.value(),
request.getRequestURI(),
reason);
}
return ResponseEntity.status(status).contentType(PROBLEM_JSON).body(problemDetail);
}
@ExceptionHandler(RuntimeException.class)
public ResponseEntity<ProblemDetail> handleRuntimeException(
RuntimeException ex, HttpServletRequest request) {
@@ -18,4 +18,11 @@ public class PDFWithPageSize extends PDFFile {
requiredMode = Schema.RequiredMode.REQUIRED,
allowableValues = {"A0", "A1", "A2", "A3", "A4", "A5", "A6", "LETTER", "LEGAL", "KEEP"})
private String pageSize;
@Schema(
description =
"Orientation to apply to the target page size. Ignored when pageSize is KEEP.",
defaultValue = "PORTRAIT",
allowableValues = {"PORTRAIT", "LANDSCAPE"})
private String orientation = "PORTRAIT";
}
@@ -13,7 +13,9 @@ import lombok.RequiredArgsConstructor;
import stirling.software.SPDF.config.swagger.MarkdownConversionResponse;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.annotations.api.ConvertApi;
import stirling.software.common.enumeration.ResourceWeight;
import stirling.software.common.model.api.PDFFile;
import stirling.software.common.service.PyMuPdfConverter;
import stirling.software.common.util.PDFToFile;
import stirling.software.common.util.TempFileManager;
@@ -22,8 +24,12 @@ import stirling.software.common.util.TempFileManager;
public class ConvertPDFToMarkdown {
private final TempFileManager tempFileManager;
private final PyMuPdfConverter pyMuPdfConverter;
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/pdf/markdown")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/pdf/markdown",
resourceWeight = ResourceWeight.MEDIUM_WEIGHT)
@MarkdownConversionResponse
@Operation(
summary = "Convert PDF to Markdown",
@@ -33,6 +39,6 @@ public class ConvertPDFToMarkdown {
throws Exception {
MultipartFile inputFile = file.getFileInput();
PDFToFile pdfToFile = new PDFToFile(tempFileManager);
return pdfToFile.processPdfToMarkdown(inputFile);
return pdfToFile.processPdfToMarkdown(inputFile, pyMuPdfConverter);
}
}
@@ -33,7 +33,7 @@ spring.security.filter.dispatcher-types=REQUEST,ERROR
# Response compression
server.compression.enabled=true
server.compression.min-response-size=1024
server.compression.mime-types=application/json,application/xml,text/html,text/plain,text/css,application/javascript
server.compression.mime-types=application/json,application/xml,text/html,text/plain,text/css,application/javascript,image/svg+xml,application/x-font-ttf,font/opentype,application/vnd.ms-fontobject,font/woff,font/woff2,application/font-woff,application/font-woff2
spring.web.error.path=/error
spring.web.error.whitelabel.enabled=false
@@ -20,6 +20,7 @@ security:
password: "" # initial password for the first login
oauth2:
enabled: false # set to 'true' to enable login (Note: enableLogin must also be 'true' for this to work)
debugLogging: false # set to 'true' to log full ID token and UserInfo claims during OAuth2/OIDC login. Use this to diagnose claim issues (e.g. "Attribute value for 'email' cannot be null" with ADFS). WARNING: writes PII (sub, email, name) to logs; disable after troubleshooting.
client:
keycloak:
issuer: "" # URL of the Keycloak realm's OpenID Connect Discovery endpoint
@@ -93,8 +94,8 @@ premium:
key: 00000000-0000-0000-0000-000000000000
enabled: false # Enable license key checks for pro/enterprise features
proFeatures:
SSOAutoLogin: false
CustomMetadata:
ssoAutoLogin: false
customMetadata:
autoUpdateMetadata: false
author: username
creator: Stirling-PDF
@@ -245,6 +246,39 @@ storage:
provider: local # storage provider: 'local' for filesystem storage, 'database' for DB-backed storage
local:
basePath: './storage' # base directory for stored files
# ====================================================================================
# S3-COMPATIBLE OBJECT STORAGE - PRO / ENTERPRISE LICENSE REQUIRED
# storage.provider=s3, storage.provider=database, and cluster.artifactStore=s3 all
# require a valid Pro or Enterprise license.
# ====================================================================================
# Used when provider=s3 (persistent user uploads) and/or cluster.artifactStore=s3
# (transient cluster artifacts). The two consumers share this block.
# Vendor cheat sheet (set the highlighted flags to taste):
# AWS S3 -> endpoint='' region='<your-region>' pathStyleAccess=false
# Cloudflare R2 -> endpoint='https://<acct>.r2.cloudflarestorage.com' region='auto'
# pathStyleAccess=false; if uploads fail with 'unsupported header
# x-amz-checksum-*' set requestChecksumCalculation=WHEN_REQUIRED
# Supabase Storage -> endpoint='https://<project>.supabase.co/storage/v1/s3'
# region='<project-region>' pathStyleAccess=true
# (filenames with non-ASCII display fine - the storage key is opaque)
# MinIO (in-cluster) -> endpoint='http://minio:9000' region='us-east-1'
# pathStyleAccess=true allowPrivateEndpoints=true
# Backblaze B2 -> endpoint='https://s3.<region>.backblazeb2.com'
# If on a B2 deployment older than July-2025 and uploads return
# 'Unsupported header x-amz-checksum-crc32', set
# requestChecksumCalculation=WHEN_REQUIRED
# DigitalOcean Spaces -> endpoint='https://<region>.digitaloceanspaces.com'
# Note: 5GB per-object cap (regardless of multipart)
s3:
endpoint: "" # blank = use AWS regional default; otherwise full URL incl. https://
bucket: "" # required when provider=s3 or cluster.artifactStore=s3
region: us-east-1
accessKey: "" # blank = fall back to AWS DefaultCredentialsProvider (env / profile / IMDS)
secretKey: ""
pathStyleAccess: false # true for MinIO and Supabase; false for AWS/R2/most CDNs
allowPrivateEndpoints: false # true required when endpoint resolves to a private/loopback IP (e.g. in-cluster MinIO). SSRF guard - leave false for any internet-facing vendor.
requestChecksumCalculation: WHEN_SUPPORTED # WHEN_SUPPORTED|WHEN_REQUIRED|DISABLED. Set WHEN_REQUIRED if your vendor rejects auto-added x-amz-checksum-* headers (older Backblaze B2, some R2 corner cases).
responseChecksumValidation: WHEN_SUPPORTED # WHEN_SUPPORTED|WHEN_REQUIRED|DISABLED. Set WHEN_REQUIRED if you see false-positive checksum-mismatch errors on GET from a vendor that never returns checksum headers.
quotas:
maxStorageMbPerUser: -1 # Max storage per user in MB; -1 disables per-user cap
maxStorageMbTotal: -1 # Max storage across all users in MB; -1 disables total cap
@@ -335,6 +369,8 @@ cluster:
enabled: false # Master switch. 'false' (default) wires the in-process backplane and skips all cluster checks. Single-instance installs do not need to change anything here.
backplane: inprocess # Backplane implementation: 'inprocess' (single JVM only) or 'valkey' (multi-node via Valkey/Redis)
artifactStore: local # Transient cluster job-artifact backend: 'local' (per-node disk; single-node only) or 's3' (shared object store; required for multi-node). Distinct from 'storage.provider' which controls persistent user uploads - when both are 's3' they share the storage.s3.* credentials block. Multi-node deployments MUST set this to 's3'.
s3:
keyPrefix: transient/ # Bucket key prefix used by the cluster artifact store when artifactStore=s3. Trailing slash recommended. Lets a single bucket host both persistent uploads (storage.s3.*) and transient job artifacts under separate prefixes.
valkey:
url: "" # Valkey/Redis URL, e.g. 'redis://valkey:6379' or 'rediss://...' for TLS. Required when enabled=true and backplane=valkey.
tls:
@@ -0,0 +1,132 @@
package stirling.software.SPDF.config;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import org.springframework.core.type.classreading.CachingMetadataReaderFactory;
import org.springframework.core.type.classreading.MetadataReader;
import org.springframework.core.type.classreading.MetadataReaderFactory;
import org.springframework.core.type.filter.TypeFilter;
import stirling.software.common.annotations.AutoJobPostMapping;
/**
* Build-time guardrail: every {@link AutoJobPostMapping} method must declare an explicit {@code
* resourceWeight}.
*
* <p>The credits interceptor multiplies {@code resourceWeight} into the per-call charge. An
* endpoint that falls through to the annotation default produces a charge derived from a value
* nobody chose — silently under- or over-billing depending on the endpoint's true cost. Forcing
* each method to pick a value from {@link stirling.software.common.enumeration.ResourceWeight}
* keeps the choice deliberate.
*
* <p>The annotation's default is {@link Integer#MIN_VALUE} (a sentinel). Runtime readers clamp the
* value into {@code [1, 100]}, so a missed declaration can't crash production — this test is the
* contract, the clamp is the safety net.
*
* <p>Lives in {@code :stirling-pdf} (core) because that's the module whose compile classpath
* transitively sees every other module's controllers ({@code :common}, {@code :proprietary}, and
* {@code :saas} when enabled).
*/
class AutoJobPostMappingWeightTest {
private static final String SCAN_BASE_PACKAGE = "stirling.software";
@Test
void everyAutoJobPostMappingDeclaresExplicitResourceWeight() throws Exception {
List<String> offenders = findOffendingMethods();
assertTrue(
offenders.isEmpty(),
() ->
"The following @AutoJobPostMapping methods do not declare an explicit"
+ " resourceWeight. Pick a value from"
+ " stirling.software.common.enumeration.ResourceWeight (SMALL,"
+ " MEDIUM, LARGE, XLARGE) and add it to the annotation:\n - "
+ String.join("\n - ", offenders));
}
private List<String> findOffendingMethods() throws IOException, ClassNotFoundException {
List<String> offenders = new ArrayList<>();
for (Class<?> candidate : scanForCandidateClasses()) {
for (Method method : candidate.getDeclaredMethods()) {
AutoJobPostMapping annotation = method.getAnnotation(AutoJobPostMapping.class);
if (annotation == null) {
continue;
}
if (annotation.resourceWeight() == Integer.MIN_VALUE) {
offenders.add(candidate.getName() + "#" + method.getName());
}
}
}
return offenders;
}
/**
* Returns every class under {@link #SCAN_BASE_PACKAGE} that has an @AutoJobPostMapping method.
*/
private List<Class<?>> scanForCandidateClasses() throws IOException, ClassNotFoundException {
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
MetadataReaderFactory metadataReaderFactory = new CachingMetadataReaderFactory(resolver);
String pattern = "classpath*:" + SCAN_BASE_PACKAGE.replace('.', '/') + "/**/*.class";
Resource[] resources = resolver.getResources(pattern);
// Pre-filter by reading annotation metadata from the class file so we don't have to load
// every class on the test classpath just to find the few that are annotated.
TypeFilter mentionsAutoJobPostMapping =
(reader, factory) ->
reader.getAnnotationMetadata()
.getAnnotatedMethods(AutoJobPostMapping.class.getName())
.size()
> 0;
List<Class<?>> matches = new ArrayList<>();
for (Resource resource : resources) {
if (!resource.isReadable()) {
continue;
}
MetadataReader reader = metadataReaderFactory.getMetadataReader(resource);
if (!mentionsAutoJobPostMapping.match(reader, metadataReaderFactory)) {
continue;
}
matches.add(Class.forName(reader.getClassMetadata().getClassName()));
}
return matches;
}
/**
* Sanity check that the classpath scan returns non-empty; otherwise the main test passes
* vacuously.
*/
@Test
void scannerFindsAtLeastOneAutoJobPostMapping() throws Exception {
long count =
scanForCandidateClasses().stream()
.flatMap(c -> java.util.Arrays.stream(c.getDeclaredMethods()))
.filter(m -> m.isAnnotationPresent(AutoJobPostMapping.class))
.count();
assertTrue(
count > 10,
() ->
"Expected the classpath scan to find many @AutoJobPostMapping methods but"
+ " found only "
+ count
+ ". Scanner regression?");
}
@SuppressWarnings("unused")
private static String describeCandidates(List<Class<?>> candidates) {
return candidates.stream().map(Class::getName).collect(Collectors.joining(", "));
}
}
@@ -237,7 +237,8 @@ class ScalePagesControllerTest {
ScalePagesRequest request = new ScalePagesRequest();
request.setFileInput(file);
request.setPageSize("A4_LANDSCAPE");
request.setPageSize("A4");
request.setOrientation("LANDSCAPE");
request.setScaleFactor(1.0f);
setupFactory();
@@ -15,10 +15,14 @@ import org.springframework.context.ApplicationContext;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import jakarta.servlet.http.HttpServletRequest;
import stirling.software.SPDF.config.EndpointConfiguration;
import stirling.software.SPDF.config.EndpointConfiguration.DisableReason;
import stirling.software.SPDF.config.EndpointConfiguration.EndpointAvailability;
import stirling.software.common.configuration.AppConfig;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.ApplicationProperties.System;
import stirling.software.common.service.LicenseServiceInterface;
import stirling.software.common.service.ServerCertificateServiceInterface;
import stirling.software.common.service.UserServiceInterface;
@@ -173,4 +177,119 @@ class ConfigControllerTest {
assertEquals(HttpStatus.OK, response.getStatusCode());
verify(endpointConfiguration).getAllEndpoints();
}
@Test
void resolveFrontendUrl_prefersExplicitConfiguredValue() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn("https://pdf.example.com");
// Request would say something else, but configured wins.
HttpServletRequest req = mock(HttpServletRequest.class);
AppConfig appConfig = mock(AppConfig.class);
assertEquals(
"https://pdf.example.com", configController.resolveFrontendUrl(req, appConfig));
}
@Test
void resolveFrontendUrl_usesRequestHostWhenNotConfigured() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn(null);
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("192.168.1.100");
when(req.getScheme()).thenReturn("http");
when(req.getServerPort()).thenReturn(8080);
assertEquals(
"http://192.168.1.100:8080",
configController.resolveFrontendUrl(req, mock(AppConfig.class)));
}
@Test
void resolveFrontendUrl_elidesDefaultHttpsPort() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn("");
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("pdf.example.com");
when(req.getScheme()).thenReturn("https");
when(req.getServerPort()).thenReturn(443);
assertEquals(
"https://pdf.example.com",
configController.resolveFrontendUrl(req, mock(AppConfig.class)));
}
@Test
void resolveFrontendUrl_fallsThroughOnLoopbackHost() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn(null);
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("localhost");
AppConfig appConfig = mock(AppConfig.class);
when(appConfig.getBackendUrl()).thenReturn("http://localhost:8080");
when(appConfig.getServerPort()).thenReturn("8080");
// Detected IP (if any) wins over loopback request host. We can't assert the
// exact value (depends on the host running the test) but we can assert it
// never returns "localhost".
String result = configController.resolveFrontendUrl(req, appConfig);
assertNotNull(result);
assertFalse(result.contains("localhost"));
}
@Test
void resolveFrontendUrl_usesActualPortWhenServerPortIsEphemeral() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn(null);
// Loopback host forces the detected-LAN-IP branch, which is where an
// ephemeral server.port=0 would otherwise leak through as ":0".
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("localhost");
AppConfig appConfig = mock(AppConfig.class);
when(appConfig.getBackendUrl()).thenReturn("http://localhost");
when(appConfig.getServerPort()).thenReturn("0");
org.springframework.core.env.Environment environment =
mock(org.springframework.core.env.Environment.class);
when(applicationContext.getEnvironment()).thenReturn(environment);
when(environment.getProperty("local.server.port")).thenReturn("54321");
String result = configController.resolveFrontendUrl(req, appConfig);
assertNotNull(result);
assertTrue(result.endsWith(":54321"));
assertFalse(result.contains(":0"));
}
@Test
void resolveEffectiveServerPort_prefersActualBoundPortWhenConfiguredZero() {
AppConfig appConfig = mock(AppConfig.class);
when(appConfig.getServerPort()).thenReturn("0");
org.springframework.core.env.Environment environment =
mock(org.springframework.core.env.Environment.class);
when(applicationContext.getEnvironment()).thenReturn(environment);
when(environment.getProperty("local.server.port")).thenReturn("54321");
assertEquals("54321", configController.resolveEffectiveServerPort(appConfig));
}
@Test
void resolveEffectiveServerPort_keepsConfiguredNonZeroPort() {
AppConfig appConfig = mock(AppConfig.class);
when(appConfig.getServerPort()).thenReturn("8080");
// Non-zero configured port is authoritative; the runtime env is never consulted.
assertEquals("8080", configController.resolveEffectiveServerPort(appConfig));
}
}
@@ -2,6 +2,7 @@ package stirling.software.SPDF.model.api.converters;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.nullable;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.multipart;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
@@ -23,12 +24,13 @@ import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.multipart.MultipartFile;
import stirling.software.common.service.PyMuPdfConverter;
import stirling.software.common.util.PDFToFile;
class ConvertPDFToMarkdownTest {
private MockMvc mockMvc() {
return MockMvcBuilders.standaloneSetup(new ConvertPDFToMarkdown(null))
return MockMvcBuilders.standaloneSetup(new ConvertPDFToMarkdown(null, null))
.setControllerAdvice(new GlobalErrorHandler())
.build();
}
@@ -52,7 +54,9 @@ class ConvertPDFToMarkdownTest {
Mockito.mockConstruction(
PDFToFile.class,
(mock, ctx) -> {
when(mock.processPdfToMarkdown(any(MultipartFile.class)))
when(mock.processPdfToMarkdown(
any(MultipartFile.class),
nullable(PyMuPdfConverter.class)))
.thenAnswer(
inv ->
ResponseEntity.ok()
@@ -83,7 +87,8 @@ class ConvertPDFToMarkdownTest {
// And that the uploaded file was passed to processPdfToMarkdown()
PDFToFile created = construction.constructed().get(0);
ArgumentCaptor<MultipartFile> captor = ArgumentCaptor.forClass(MultipartFile.class);
verify(created, times(1)).processPdfToMarkdown(captor.capture());
verify(created, times(1))
.processPdfToMarkdown(captor.capture(), nullable(PyMuPdfConverter.class));
MultipartFile passed = captor.getValue();
// Minimal plausibility checks
@@ -98,7 +103,9 @@ class ConvertPDFToMarkdownTest {
Mockito.mockConstruction(
PDFToFile.class,
(mock, ctx) -> {
when(mock.processPdfToMarkdown(any(MultipartFile.class)))
when(mock.processPdfToMarkdown(
any(MultipartFile.class),
nullable(PyMuPdfConverter.class)))
.thenThrow(new RuntimeException("boom"));
})) {
@@ -0,0 +1,93 @@
package stirling.software.common.configuration;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.mockStatic;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.MockedStatic;
import stirling.software.common.util.GeneralUtils;
import stirling.software.common.util.YamlHelper;
/**
* End-to-end check of the container-restart path. {@link ConfigInitializer#ensureConfigExists()} is
* what runs on every startup, merging the on-disk settings.yml with the bundled
* settings.yml.template. These tests exercise it against the real template on the classpath to
* prove admin-saved proFeatures values survive a restart - the bug behind "the SSO auto-login
* button resets every time the container resets".
*/
class ConfigInitializerRestartTest {
private static String read(Path settings, String... keyPath) throws IOException {
return String.valueOf(new YamlHelper(settings).getValueByExactKeyPath(keyPath));
}
@Test
void ssoAutoLoginAndCustomMetadata_persistAcrossRestart(@TempDir Path tmp) throws Exception {
Path settings = tmp.resolve("settings.yml");
Path custom = tmp.resolve("custom_settings.yml");
try (MockedStatic<InstallationPathConfig> paths =
mockStatic(InstallationPathConfig.class)) {
paths.when(InstallationPathConfig::getSettingsPath).thenReturn(settings.toString());
paths.when(InstallationPathConfig::getCustomSettingsPath).thenReturn(custom.toString());
ConfigInitializer init = new ConfigInitializer();
// First boot: settings.yml created from the bundled template (camelCase, default off).
init.ensureConfigExists();
assertEquals("false", read(settings, "premium", "proFeatures", "ssoAutoLogin"));
// Admin enables SSO auto-login and edits custom metadata via the exact save path the
// admin settings controller uses.
GeneralUtils.saveKeyToSettings("premium.proFeatures.ssoAutoLogin", true);
GeneralUtils.saveKeyToSettings("premium.proFeatures.customMetadata.author", "acme");
// Container restart: ensureConfigExists merges the saved file with the template again.
init.ensureConfigExists();
assertEquals("true", read(settings, "premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"acme", read(settings, "premium", "proFeatures", "customMetadata", "author"));
}
}
@Test
void legacyPascalCaseConfig_isMigratedAndPreservedOnRestart(@TempDir Path tmp)
throws Exception {
Path settings = tmp.resolve("settings.yml");
Path custom = tmp.resolve("custom_settings.yml");
try (MockedStatic<InstallationPathConfig> paths =
mockStatic(InstallationPathConfig.class)) {
paths.when(InstallationPathConfig::getSettingsPath).thenReturn(settings.toString());
paths.when(InstallationPathConfig::getCustomSettingsPath).thenReturn(custom.toString());
ConfigInitializer init = new ConfigInitializer();
// Seed a full settings.yml as an OLD install would have written it: PascalCase keys
// with
// SSO auto-login enabled.
init.ensureConfigExists();
String legacy =
Files.readString(settings)
.replace("ssoAutoLogin: false", "SSOAutoLogin: true")
.replace("customMetadata:", "CustomMetadata:");
Files.writeString(settings, legacy);
// Upgrade restart.
init.ensureConfigExists();
// Value carried forward onto the new camelCase key; the legacy PascalCase key is gone.
assertEquals("true", read(settings, "premium", "proFeatures", "ssoAutoLogin"));
assertNull(
new YamlHelper(settings)
.getValueByExactKeyPath("premium", "proFeatures", "SSOAutoLogin"));
}
}
}
+2
View File
@@ -123,6 +123,8 @@ SwaggerDoc.json
*.tar.gz
*.rar
*.db
# Whitelist the H2 fixtures that feed the version-migration CI smoke test.
!src/test/resources/db-migration-fixtures/*.mv.db
/build
/app/proprietary/build/
+9
View File
@@ -5,6 +5,8 @@ repositories {
ext {
jwtVersion = '0.13.0'
awsSdkVersion = '2.44.12'
testcontainersMinioVersion = '1.21.4'
}
bootRun {
@@ -71,6 +73,13 @@ dependencies {
implementation('com.coveo:saml-client:5.0.0') {
exclude group: 'org.opensaml', module: 'opensaml-core'
}
implementation "software.amazon.awssdk:s3:$awsSdkVersion"
implementation "software.amazon.awssdk:url-connection-client:$awsSdkVersion"
testImplementation "org.testcontainers:minio:$testcontainersMinioVersion"
testImplementation "org.testcontainers:junit-jupiter:$testcontainersMinioVersion"
testImplementation "org.testcontainers:localstack:$testcontainersMinioVersion"
}
tasks.register('prepareKotlinBuildScriptModel') {}
@@ -0,0 +1,200 @@
package stirling.software.proprietary.cluster.s3;
import java.net.InetAddress;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3ClientBuilder;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
/**
* Shared factory for {@link S3Client} and {@link S3Presigner} instances used by both {@code
* S3StorageProvider} and {@code S3FileStore}, so endpoint/region/credentials wiring lives in
* exactly one place.
*/
@Slf4j
public final class S3Clients {
private S3Clients() {}
/** Paired client and presigner with coordinated lifecycle. */
public record Bundle(S3Client client, S3Presigner presigner) implements AutoCloseable {
@Override
public void close() {
try {
presigner.close();
} catch (Exception e) {
log.warn("Error closing S3 presigner", e);
}
try {
client.close();
} catch (Exception e) {
log.warn("Error closing S3 client", e);
}
}
}
/** Build a client+presigner pair from the shared S3 config block. */
public static Bundle build(ApplicationProperties.Storage.S3 cfg, String usage) {
if (cfg == null) {
throw new IllegalStateException(
usage + " requires storage.s3.* configuration to be set");
}
if (cfg.getBucket() == null || cfg.getBucket().isBlank()) {
throw new IllegalStateException(usage + " requires storage.s3.bucket to be set");
}
String region =
cfg.getRegion() == null || cfg.getRegion().isBlank()
? "us-east-1"
: cfg.getRegion();
S3Configuration s3Configuration =
S3Configuration.builder().pathStyleAccessEnabled(cfg.isPathStyleAccess()).build();
RequestChecksumCalculation requestChecksum =
parseRequestChecksum(cfg.getRequestChecksumCalculation());
ResponseChecksumValidation responseChecksum =
parseResponseChecksum(cfg.getResponseChecksumValidation());
S3ClientBuilder clientBuilder =
S3Client.builder()
.httpClient(UrlConnectionHttpClient.create())
.region(Region.of(region))
.serviceConfiguration(s3Configuration)
.requestChecksumCalculation(requestChecksum)
.responseChecksumValidation(responseChecksum);
S3Presigner.Builder presignerBuilder =
S3Presigner.builder()
.region(Region.of(region))
.serviceConfiguration(s3Configuration);
if (cfg.getEndpoint() != null && !cfg.getEndpoint().isBlank()) {
URI endpoint;
try {
endpoint = new URI(cfg.getEndpoint());
} catch (URISyntaxException e) {
throw new IllegalStateException(
"Invalid storage.s3.endpoint: " + cfg.getEndpoint(), e);
}
validateEndpointHost(endpoint, cfg.isAllowPrivateEndpoints());
clientBuilder.endpointOverride(endpoint);
presignerBuilder.endpointOverride(endpoint);
}
boolean hasStaticCreds =
cfg.getAccessKey() != null
&& !cfg.getAccessKey().isBlank()
&& cfg.getSecretKey() != null
&& !cfg.getSecretKey().isBlank();
if (hasStaticCreds) {
AwsBasicCredentials credentials =
AwsBasicCredentials.create(cfg.getAccessKey(), cfg.getSecretKey());
StaticCredentialsProvider provider = StaticCredentialsProvider.create(credentials);
clientBuilder.credentialsProvider(provider);
presignerBuilder.credentialsProvider(provider);
} else {
clientBuilder.credentialsProvider(DefaultCredentialsProvider.create());
presignerBuilder.credentialsProvider(DefaultCredentialsProvider.create());
}
log.debug(
"Configured S3 {}: bucket={}, region={}, endpoint={}, pathStyle={}",
usage,
cfg.getBucket(),
region,
cfg.getEndpoint() == null || cfg.getEndpoint().isBlank()
? "<aws-default>"
: cfg.getEndpoint(),
cfg.isPathStyleAccess());
return new Bundle(clientBuilder.build(), presignerBuilder.build());
}
/**
* Block SSRF via the S3 endpoint setting. An admin who can edit config could otherwise point
* the SDK at the cloud metadata service (e.g. {@code http://169.254.169.254/}) and exfiltrate
* instance-role credentials. Reject any endpoint whose host resolves to a loopback, link-local,
* or RFC1918 private address unless the operator has explicitly opted in via {@code
* storage.s3.allow-private-endpoints=true}.
*/
static void validateEndpointHost(URI endpoint, boolean allowPrivate) {
if (allowPrivate) {
return;
}
String host = endpoint.getHost();
if (host == null || host.isBlank()) {
throw new IllegalStateException("storage.s3.endpoint must include a host: " + endpoint);
}
InetAddress[] addresses;
try {
addresses = InetAddress.getAllByName(host);
} catch (UnknownHostException e) {
throw new IllegalStateException(
"Unable to resolve storage.s3.endpoint host '" + host + "'", e);
}
for (InetAddress address : addresses) {
if (isPrivateOrLocal(address)) {
throw new IllegalStateException(
"storage.s3.endpoint host '"
+ host
+ "' resolves to private/link-local address "
+ address.getHostAddress()
+ "; set storage.s3.allow-private-endpoints=true to opt in"
+ " (e.g. for MinIO or in-cluster S3).");
}
}
}
private static boolean isPrivateOrLocal(InetAddress address) {
return address.isLoopbackAddress()
|| address.isLinkLocalAddress()
|| address.isSiteLocalAddress()
|| address.isAnyLocalAddress()
|| address.isMulticastAddress();
}
static RequestChecksumCalculation parseRequestChecksum(String value) {
if (value == null || value.isBlank()) {
return RequestChecksumCalculation.WHEN_SUPPORTED;
}
try {
return RequestChecksumCalculation.valueOf(
value.trim().toUpperCase(java.util.Locale.ROOT));
} catch (IllegalArgumentException ex) {
log.warn(
"Unknown storage.s3.request-checksum-calculation value '{}', falling back to WHEN_SUPPORTED",
value);
return RequestChecksumCalculation.WHEN_SUPPORTED;
}
}
static ResponseChecksumValidation parseResponseChecksum(String value) {
if (value == null || value.isBlank()) {
return ResponseChecksumValidation.WHEN_SUPPORTED;
}
try {
return ResponseChecksumValidation.valueOf(
value.trim().toUpperCase(java.util.Locale.ROOT));
} catch (IllegalArgumentException ex) {
log.warn(
"Unknown storage.s3.response-checksum-validation value '{}', falling back to WHEN_SUPPORTED",
value);
return ResponseChecksumValidation.WHEN_SUPPORTED;
}
}
}
@@ -0,0 +1,226 @@
package stirling.software.proprietary.cluster.s3;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.util.Optional;
import java.util.UUID;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.FileStore;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
/**
* S3-backed {@link FileStore} for transient job-result files. Objects are namespaced under a
* configurable key prefix (default {@code transient/}) and can coexist in the same bucket as {@code
* S3StorageProvider}.
*/
@Slf4j
public class S3FileStore implements FileStore, AutoCloseable {
public static final String DEFAULT_KEY_PREFIX = "transient/";
private final S3Client s3Client;
private final String bucket;
private final String keyPrefix;
private final boolean ownsClient;
public S3FileStore(S3Client s3Client, String bucket) {
this(s3Client, bucket, DEFAULT_KEY_PREFIX, true);
}
public S3FileStore(S3Client s3Client, String bucket, String keyPrefix) {
this(s3Client, bucket, keyPrefix, true);
}
/**
* @param ownsClient when true, {@link #close()} will close the supplied client. Set to false in
* tests that share the client with another consumer.
*/
public S3FileStore(S3Client s3Client, String bucket, String keyPrefix, boolean ownsClient) {
if (bucket == null || bucket.isBlank()) {
throw new IllegalArgumentException("S3 bucket must be configured");
}
this.s3Client = s3Client;
this.bucket = bucket;
this.keyPrefix = normalizePrefix(keyPrefix);
this.ownsClient = ownsClient;
}
@Override
public Stored store(InputStream in, String originalName) throws IOException {
String fileId = UUID.randomUUID().toString();
// S3 PUT requires a known content-length; spool to a temp file first so memory stays
// bounded for large payloads, then stream the file to S3 via RequestBody.fromFile.
Path tempFile = Files.createTempFile("s3-upload-", ".bin");
long size;
try {
try (InputStream src = in) {
Files.copy(src, tempFile, StandardCopyOption.REPLACE_EXISTING);
}
size = Files.size(tempFile);
PutObjectRequest request =
PutObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
s3Client.putObject(request, RequestBody.fromFile(tempFile));
} catch (SdkException e) {
throw new IOException("Failed to upload object to S3", e);
}
} finally {
try {
Files.deleteIfExists(tempFile);
} catch (IOException cleanupError) {
log.warn("Failed to delete S3 upload temp file: {}", tempFile, cleanupError);
}
}
return new Stored(fileId, size);
}
@Override
public InputStream retrieve(String fileId) throws IOException {
validateFileId(fileId);
GetObjectRequest request =
GetObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
ResponseInputStream<GetObjectResponse> stream = s3Client.getObject(request);
return new BufferedInputStream(stream);
} catch (NoSuchKeyException e) {
throw new IOException("File not found with ID: " + fileId, e);
} catch (SdkException e) {
throw new IOException("Failed to load object from S3", e);
}
}
@Override
public byte[] retrieveBytes(String fileId) throws IOException {
validateFileId(fileId);
GetObjectRequest request =
GetObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try (ResponseInputStream<GetObjectResponse> stream = s3Client.getObject(request)) {
return stream.readAllBytes();
} catch (NoSuchKeyException e) {
throw new IOException("File not found with ID: " + fileId, e);
} catch (SdkException e) {
throw new IOException("Failed to load object from S3", e);
}
}
@Override
public long size(String fileId) throws IOException {
validateFileId(fileId);
HeadObjectRequest request =
HeadObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
HeadObjectResponse response = s3Client.headObject(request);
return Optional.ofNullable(response.contentLength()).orElse(0L);
} catch (NoSuchKeyException e) {
throw new IOException("File not found with ID: " + fileId, e);
} catch (S3Exception e) {
if (e.statusCode() == 404) {
throw new IOException("File not found with ID: " + fileId, e);
}
throw new IOException("Failed to head object in S3", e);
} catch (SdkException e) {
throw new IOException("Failed to head object in S3", e);
}
}
@Override
public boolean delete(String fileId) {
try {
validateFileId(fileId);
} catch (IllegalArgumentException e) {
log.warn("Refusing to delete invalid file id: {}", fileId);
return false;
}
try {
s3Client.deleteObject(
DeleteObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build());
return true;
} catch (SdkException e) {
log.error("Error deleting file with ID: {}", fileId, e);
return false;
}
}
@Override
public boolean exists(String fileId) {
try {
validateFileId(fileId);
} catch (IllegalArgumentException e) {
return false;
}
HeadObjectRequest request =
HeadObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
s3Client.headObject(request);
return true;
} catch (NoSuchKeyException e) {
return false;
} catch (S3Exception e) {
if (e.statusCode() == 404) {
return false;
}
log.warn("Error checking existence for file ID: {}", fileId, e);
return false;
} catch (SdkException e) {
log.warn("Error checking existence for file ID: {}", fileId, e);
return false;
}
}
@Override
public void close() {
if (!ownsClient) {
return;
}
try {
s3Client.close();
} catch (Exception e) {
log.warn("Error closing S3 client", e);
}
}
String resolveKey(String fileId) {
return keyPrefix + fileId;
}
private static void validateFileId(String fileId) {
if (fileId == null || fileId.isBlank()) {
throw new IllegalArgumentException("File ID must not be blank");
}
if (fileId.contains("..") || fileId.contains("/") || fileId.contains("\\")) {
throw new IllegalArgumentException("Invalid file ID");
}
}
private static String normalizePrefix(String prefix) {
if (prefix == null || prefix.isBlank()) {
return "";
}
String trimmed = prefix.trim();
if (trimmed.startsWith("/")) {
trimmed = trimmed.substring(1);
}
if (!trimmed.isEmpty() && !trimmed.endsWith("/")) {
trimmed = trimmed + "/";
}
return trimmed;
}
}
@@ -0,0 +1,37 @@
package stirling.software.proprietary.cluster.s3;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.FileStore;
import stirling.software.common.model.ApplicationProperties;
/** Activates the S3-backed transient {@link FileStore} when {@code cluster.artifactStore=s3}. */
@Slf4j
@Configuration
@RequiredArgsConstructor
@ConditionalOnProperty(prefix = "cluster", name = "artifactStore", havingValue = "s3")
public class S3FileStoreConfiguration {
private final ApplicationProperties applicationProperties;
@Bean(destroyMethod = "close")
@ConditionalOnMissingBean
public FileStore fileStore(@Value("${cluster.s3.keyPrefix:transient/}") String keyPrefix) {
ApplicationProperties.Storage.S3 cfg = applicationProperties.getStorage().getS3();
S3Clients.Bundle bundle = S3Clients.build(cfg, "cluster file store");
// FileStore has no signed-URL contract; close the unused presigner immediately.
try {
bundle.presigner().close();
} catch (Exception ignored) {
}
log.info("Cluster FileStore: s3 (bucket={}, keyPrefix={})", cfg.getBucket(), keyPrefix);
return new S3FileStore(bundle.client(), cfg.getBucket(), keyPrefix, true);
}
}
@@ -32,7 +32,10 @@ public enum AiPdfContentType {
// Heavy content
COMPLIANCE("compliance"),
IMAGES("images");
IMAGES("images"),
// PyMuPDF worker — pre-rendered Markdown
PYMUPDF_MARKDOWN("pymupdf_markdown");
private final String value;
@@ -11,7 +11,7 @@ import jakarta.validation.constraints.NotNull;
import lombok.Data;
@Data
@Schema(description = "Run an AI workflow against one or more PDF files")
@Schema(description = "Run an AI workflow")
public class AiWorkflowRequest {
@NotNull
@@ -49,7 +49,11 @@ public class EEAppConfig {
@Profile("security & !saas")
@Bean(name = "SSOAutoLogin")
public boolean ssoAutoLogin() {
return applicationProperties.getPremium().getProFeatures().isSsoAutoLogin();
boolean enabled = applicationProperties.getPremium().getProFeatures().isSsoAutoLogin();
if (enabled) {
licenseKeyChecker.requireProOrEnterprise("premium.proFeatures.ssoAutoLogin=true");
}
return enabled;
}
// TODO: Remove post migration
@@ -32,7 +32,10 @@ public class LicenseKeyChecker {
private final UserLicenseSettingsService licenseSettingsService;
private License premiumEnabledResult = License.NORMAL;
// volatile: written by evaluateLicense() on the @Scheduled refresh thread, read by request
// threads via getPremiumLicenseEnabledResult() / requireProOrEnterprise(). Ensures readers see
// the latest tier rather than a stale cached value.
private volatile License premiumEnabledResult = License.NORMAL;
public LicenseKeyChecker(
KeygenLicenseVerifier licenseService,
@@ -133,4 +136,16 @@ public class LicenseKeyChecker {
public License getPremiumLicenseEnabledResult() {
return premiumEnabledResult;
}
/**
* Throws {@link IllegalStateException} if the current license is not Pro or Enterprise. Used by
* boot-time gates to fail fast when an operator enables a premium-only setting without a valid
* license. {@code configuredAs} is the human-readable property path (e.g. {@code
* "storage.provider=s3"}) and appears in the exception message.
*/
public void requireProOrEnterprise(String configuredAs) {
if (premiumEnabledResult != License.SERVER && premiumEnabledResult != License.ENTERPRISE) {
throw new IllegalStateException(configuredAs + " requires a Pro or Enterprise license");
}
}
}
@@ -17,6 +17,7 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.annotations.api.GeneralApi;
import stirling.software.common.enumeration.ResourceWeight;
import stirling.software.proprietary.security.model.api.Email;
import stirling.software.proprietary.security.service.EmailService;
@@ -39,7 +40,10 @@ public class EmailController {
* attachment.
* @return ResponseEntity with success or error message.
*/
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/send-email")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/send-email",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@Operation(
summary = "Send an email with an attachment",
description =
@@ -1,6 +1,10 @@
package stirling.software.proprietary.security.service;
import java.util.Collections;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.TreeSet;
import org.springframework.security.authentication.LockedException;
import org.springframework.security.oauth2.client.oidc.userinfo.OidcUserRequest;
@@ -8,6 +12,8 @@ import org.springframework.security.oauth2.client.oidc.userinfo.OidcUserService;
import org.springframework.security.oauth2.client.userinfo.OAuth2UserService;
import org.springframework.security.oauth2.core.OAuth2AuthenticationException;
import org.springframework.security.oauth2.core.OAuth2Error;
import org.springframework.security.oauth2.core.oidc.OidcIdToken;
import org.springframework.security.oauth2.core.oidc.OidcUserInfo;
import org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser;
import org.springframework.security.oauth2.core.oidc.user.OidcUser;
@@ -39,20 +45,37 @@ public class CustomOAuth2UserService implements OAuth2UserService<OidcUserReques
@Override
public OidcUser loadUser(OidcUserRequest userRequest) throws OAuth2AuthenticationException {
String registrationId = userRequest.getClientRegistration().getRegistrationId();
boolean debugLogging = Boolean.TRUE.equals(oauth2Properties.getDebugLogging());
// Resolved inside the try so a bad/null useAsUsername (IllegalArgumentException from
// valueOf, or NPE on toUpperCase) is caught and wrapped as OAuth2AuthenticationException
// by the existing handlers below, matching the pre-debugLogging behaviour.
String usernameAttributeKey = null;
try {
OidcUser user = delegate.loadUser(userRequest);
String usernameAttributeKey =
usernameAttributeKey =
UsernameAttribute.valueOf(oauth2Properties.getUseAsUsername().toUpperCase())
.getName();
OidcUser user = delegate.loadUser(userRequest);
if (debugLogging) {
logClaimDump(
"OAuth2/OIDC login claims received",
registrationId,
usernameAttributeKey,
user.getIdToken(),
user.getUserInfo(),
user.getAttributes(),
false);
}
// Extract SSO provider information
String ssoProviderId = user.getSubject(); // Standard OIDC 'sub' claim
String ssoProvider = userRequest.getClientRegistration().getRegistrationId();
String username = user.getAttribute(usernameAttributeKey);
log.debug(
"OAuth2 login - Provider: {}, ProviderId: {}, Username: {}",
ssoProvider,
registrationId,
ssoProviderId,
username);
@@ -79,10 +102,154 @@ public class CustomOAuth2UserService implements OAuth2UserService<OidcUserReques
usernameAttributeKey);
} catch (IllegalArgumentException e) {
log.error("Error loading OIDC user: {}", e.getMessage());
// Only emit the claim dump if we successfully resolved usernameAttributeKey. A null
// value here means UsernameAttribute.valueOf rejected the configured useAsUsername
// before delegate.loadUser ran — that error message is self-explanatory and a claim
// dump would have no resolved-key to compare against.
if (debugLogging && usernameAttributeKey != null) {
// The DefaultOidcUser constructor (or our own checks) rejected the chosen
// username attribute. Dump the claims we DID receive so the operator can pick
// a different value for security.oauth2.useAsUsername.
logClaimDump(
"OAuth2/OIDC login FAILED - dumping received claims",
registrationId,
usernameAttributeKey,
userRequest.getIdToken(),
null,
userRequest.getIdToken() == null
? Collections.emptyMap()
: userRequest.getIdToken().getClaims(),
true);
}
throw new OAuth2AuthenticationException(new OAuth2Error(e.getMessage()), e);
} catch (Exception e) {
log.error("Unexpected error loading OIDC user", e);
if (debugLogging && usernameAttributeKey != null && userRequest.getIdToken() != null) {
logClaimDump(
"OAuth2/OIDC login FAILED (unexpected error) - dumping ID token claims",
registrationId,
usernameAttributeKey,
userRequest.getIdToken(),
null,
userRequest.getIdToken().getClaims(),
true);
}
throw new OAuth2AuthenticationException("Unexpected error during authentication");
}
}
/**
* Emits a multi-line diagnostic dump of the claims returned by the OAuth2/OIDC provider. Only
* invoked when {@code security.oauth2.debugLogging=true}.
*
* @param banner short title for the log block
* @param registrationId Spring client registration id (e.g. "demarest", "keycloak")
* @param usernameAttributeKey the claim key the application is configured to use as username
* @param idToken the decoded ID token, may be null on unexpected failures
* @param userInfo the decoded UserInfo response, may be null if the provider returned none
* @param mergedAttributes the merged attribute map Spring uses for {@code getAttribute()}
* @param failure true if logging in the error path (uses ERROR level), false for INFO
*/
private void logClaimDump(
String banner,
String registrationId,
String usernameAttributeKey,
OidcIdToken idToken,
OidcUserInfo userInfo,
Map<String, Object> mergedAttributes,
boolean failure) {
StringBuilder sb = new StringBuilder();
sb.append("\n========== [OAUTH2 DEBUG] ").append(banner).append(" ==========\n");
sb.append("Provider registrationId : ").append(registrationId).append('\n');
sb.append("Configured useAsUsername: ")
.append(oauth2Properties.getUseAsUsername())
.append(" (looks up claim key '")
.append(usernameAttributeKey)
.append("')\n");
if (idToken != null) {
Map<String, Object> idClaims = idToken.getClaims();
sb.append("\n-- ID token claims (")
.append(idClaims == null ? 0 : idClaims.size())
.append(") --\n");
appendClaims(sb, idClaims);
sb.append("ID token issued at : ").append(idToken.getIssuedAt()).append('\n');
sb.append("ID token expires at: ").append(idToken.getExpiresAt()).append('\n');
} else {
sb.append("\n-- ID token: <null> --\n");
}
if (userInfo != null && userInfo.getClaims() != null) {
sb.append("\n-- UserInfo endpoint claims (")
.append(userInfo.getClaims().size())
.append(") --\n");
appendClaims(sb, userInfo.getClaims());
} else {
sb.append("\n-- UserInfo endpoint claims: none returned --\n");
}
if (mergedAttributes != null) {
sb.append("\n-- Merged attribute keys available to useAsUsername: ")
.append(new TreeSet<>(mergedAttributes.keySet()))
.append("\n");
Object resolved = mergedAttributes.get(usernameAttributeKey);
sb.append("-- Value at '")
.append(usernameAttributeKey)
.append("' : ")
.append(resolved == null ? "<NULL — this is why login fails>" : resolved)
.append('\n');
if (resolved == null) {
Set<String> hints = suggestUsernameClaims(mergedAttributes.keySet());
if (!hints.isEmpty()) {
sb.append(
"-- Hint: the following claim(s) are present and map to a"
+ " known UsernameAttribute value — try setting"
+ " security.oauth2.useAsUsername to one of: ")
.append(hints)
.append('\n');
}
}
}
sb.append(
"\nWARNING: this block contains PII. Set security.oauth2.debugLogging=false once"
+ " troubleshooting is complete.\n");
sb.append("========== [/OAUTH2 DEBUG] ==========");
if (failure) {
log.error(sb.toString());
} else {
log.info(sb.toString());
}
}
private static void appendClaims(StringBuilder sb, Map<String, Object> claims) {
if (claims == null || claims.isEmpty()) {
sb.append(" (no claims)\n");
return;
}
// Sort for stable, scannable output
new TreeSet<>(claims.keySet())
.forEach(
key -> {
Object value = claims.get(key);
sb.append(" ").append(key).append(" = ").append(value).append('\n');
});
}
/**
* Returns the intersection of the claim keys the provider actually returned and the keys that
* {@link UsernameAttribute} accepts — i.e. valid values the operator could put in {@code
* security.oauth2.useAsUsername} to make this login work.
*/
private static Set<String> suggestUsernameClaims(Set<String> availableClaimKeys) {
Set<String> supported = new TreeSet<>();
for (UsernameAttribute attr : UsernameAttribute.values()) {
if (availableClaimKeys.contains(attr.getName())) {
supported.add(attr.getName());
}
}
return supported;
}
}
@@ -31,6 +31,7 @@ import stirling.software.common.service.CustomPDFDocumentFactory;
import stirling.software.common.service.FileStorage;
import stirling.software.common.service.InternalApiClient;
import stirling.software.common.service.InternalApiTimeoutException;
import stirling.software.common.service.PyMuPdfConverter;
import stirling.software.common.service.ToolMetadataService;
import stirling.software.common.util.ExceptionUtils;
import stirling.software.common.util.TempFile;
@@ -74,6 +75,7 @@ public class AiWorkflowService {
private final TempFileManager tempFileManager;
private final FileIdStrategy fileIdStrategy;
private final AiEngineEndpointResolver endpointResolver;
private final PyMuPdfConverter pyMuPdfConverter;
@FunctionalInterface
public interface ProgressListener {
@@ -137,6 +139,9 @@ public class AiWorkflowService {
? new ArrayList<>()
: new ArrayList<>(request.getConversationHistory()));
initialRequest.setEnabledEndpoints(endpointResolver.getEnabledEndpointUrls());
boolean workerAvailable = pyMuPdfConverter.isAvailable();
initialRequest.setPymupdfWorkerAvailable(workerAvailable);
log.info("[pymupdf-convert] available={}", workerAvailable);
listener.onProgress(AiWorkflowProgressEvent.of(AiWorkflowPhase.ANALYZING));
@@ -183,6 +188,13 @@ public class AiWorkflowService {
cannotContinue("AI engine requested content extraction more than once."));
}
// Fast path: when the engine identifies a pdf-to-markdown task and pymupdf-convert is
// available, skip feeding content back to the engine and convert directly.
if ("pdf_to_markdown".equals(response.getResumeWith())
&& request.isPymupdfWorkerAvailable()) {
return runPyMuPdfConversion(filesById, listener);
}
List<AiWorkflowFileRequest> requestedFiles = response.getFiles();
// Validate requested file ids before loading anything
@@ -365,6 +377,31 @@ public class AiWorkflowService {
return new WorkflowState.Terminal(response);
}
private WorkflowState runPyMuPdfConversion(
Map<String, MultipartFile> filesById, ProgressListener listener) throws IOException {
listener.onProgress(AiWorkflowProgressEvent.of(AiWorkflowPhase.PROCESSING));
List<Resource> outputs = new ArrayList<>();
for (MultipartFile file : filesById.values()) {
String baseName =
file.getOriginalFilename() != null
? file.getOriginalFilename().replaceFirst("\\.[^.]+$", "")
: "document";
String markdown =
pyMuPdfConverter.convertToMarkdown(file.getBytes(), file.getOriginalFilename());
String safeFilename = Filenames.toSimpleFileName(baseName + ".md");
byte[] bytes = markdown.getBytes(java.nio.charset.StandardCharsets.UTF_8);
outputs.add(
new org.springframework.core.io.ByteArrayResource(bytes) {
@Override
public String getFilename() {
return safeFilename;
}
});
}
return new WorkflowState.Terminal(
buildCompletedResponse("Converted PDF to Markdown.", outputs, List.of(), null));
}
private WorkflowState onGenerateFile(AiWorkflowResponse response, ProgressListener listener)
throws IOException {
String content = response.getGeneratedContent();
@@ -745,5 +782,6 @@ public class AiWorkflowService {
private List<WorkflowArtifact> artifacts = new ArrayList<>();
private String resumeWith;
private List<String> enabledEndpoints = new ArrayList<>();
private boolean pymupdfWorkerAvailable;
}
}
@@ -30,6 +30,7 @@ import stirling.software.SPDF.pdf.parser.PdfModels.RawLine;
import stirling.software.SPDF.pdf.parser.PdfModels.TableFragment;
import stirling.software.SPDF.pdf.parser.PdfModels.TextFragment;
import stirling.software.SPDF.pdf.parser.TabulaTableParser;
import stirling.software.common.service.PyMuPdfConverter;
import stirling.software.common.util.ExceptionUtils;
import stirling.software.common.util.PdfUtils;
import stirling.software.proprietary.model.api.ai.AiPdfContentType;
@@ -44,6 +45,7 @@ public class PdfContentExtractor {
private final TabulaTableParser tabulaTableParser;
private final PdfIngester pdfIngester;
private final PyMuPdfConverter pyMuPdfConverter;
private static final int MAX_CHARACTERS_PER_PAGE = 4_000;
@@ -190,6 +192,8 @@ public class PdfContentExtractor {
extractText(lf, fileReq, remainingPages, remainingCharacters));
case PAGE_LAYOUT ->
Optional.<PdfContentResult>ofNullable(extractPageLayout(lf, remainingPages));
case PYMUPDF_MARKDOWN ->
Optional.<PdfContentResult>ofNullable(extractPyMuPdfMarkdown(lf));
default -> {
log.warn(
"Content type {} not yet implemented, skipping for {}",
@@ -255,6 +259,11 @@ public class PdfContentExtractor {
artifact.setFiles(results.stream().map(PageLayoutFileResult.class::cast).toList());
yield artifact;
}
case PYMUPDF_MARKDOWN -> {
PyMuPdfMarkdownArtifact artifact = new PyMuPdfMarkdownArtifact();
artifact.setFiles(results.stream().map(PyMuPdfMarkdownResult.class::cast).toList());
yield artifact;
}
case TOOL_REPORT ->
throw new IllegalArgumentException(
"TOOL_REPORT artifacts are not produced by PdfContentExtractor");
@@ -370,7 +379,8 @@ public class PdfContentExtractor {
enum ArtifactKind {
EXTRACTED_TEXT("extracted_text"),
PAGE_LAYOUT("page_layout"),
TOOL_REPORT("tool_report");
TOOL_REPORT("tool_report"),
PYMUPDF_MARKDOWN("pymupdf_markdown");
private final String value;
@@ -469,4 +479,46 @@ public class PdfContentExtractor {
private final ArtifactKind kind = ArtifactKind.PAGE_LAYOUT;
private List<PageLayoutFileResult> files = new ArrayList<>();
}
private PyMuPdfMarkdownResult extractPyMuPdfMarkdown(LoadedFile lf) {
try {
log.info("[pymupdf-convert] converting file={}", lf.fileName());
java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
lf.document().save(baos);
String markdown = pyMuPdfConverter.convertToMarkdown(baos.toByteArray(), lf.fileName());
log.info(
"[pymupdf-convert] success file={} markdown-chars={}",
lf.fileName(),
markdown.length());
PyMuPdfMarkdownResult result = new PyMuPdfMarkdownResult();
result.setFileName(lf.fileName());
result.setMarkdown(markdown);
return result;
} catch (Exception e) {
log.warn(
"[pymupdf-convert] failed for file={}, falling back to page layout: {}",
lf.fileName(),
e.getMessage());
return null;
}
}
/** PyMuPDF worker pre-rendered Markdown for one file. */
@Data
static final class PyMuPdfMarkdownResult implements PdfContentResult {
private String fileName;
private String markdown;
@Override
public ArtifactKind getArtifactKind() {
return ArtifactKind.PYMUPDF_MARKDOWN;
}
}
/** Artifact carrying PyMuPDF-rendered Markdown for all input files. */
@Data
static final class PyMuPdfMarkdownArtifact implements WorkflowArtifact {
private final ArtifactKind kind = ArtifactKind.PYMUPDF_MARKDOWN;
private List<PyMuPdfMarkdownResult> files = new ArrayList<>();
}
}
@@ -0,0 +1,95 @@
package stirling.software.proprietary.storage.config;
import java.util.Locale;
import java.util.Optional;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import jakarta.annotation.PostConstruct;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.configuration.ee.LicenseKeyChecker;
/**
* Fails fast at boot if cluster mode is enabled with node-local storage. Validates both {@code
* storage.provider} (persistent uploads) and {@code cluster.artifactStore} (transient job-result
* files): neither may be {@code local} when {@code cluster.enabled=true}. Additionally enforces
* that any S3-backed configuration ({@code storage.provider=s3} or {@code
* cluster.artifactStore=s3}) is accompanied by a valid Pro / Enterprise license.
*/
@Configuration
@RequiredArgsConstructor
@Slf4j
public class ClusterStorageGate {
private final ApplicationProperties applicationProperties;
private final LicenseKeyChecker licenseKeyChecker;
@Value("${cluster.enabled:false}")
private boolean clusterEnabled;
@Value("${cluster.artifactStore:local}")
private String clusterArtifactStore;
@PostConstruct
void validate() {
// License enforcement runs regardless of cluster.enabled: even a single-node setup that
// selects a remote backend must hold a Pro or higher license.
ApplicationProperties.Storage storage = applicationProperties.getStorage();
if (storage != null && storage.isEnabled()) {
String provider = normalize(storage.getProvider());
if ("s3".equals(provider) || "database".equals(provider)) {
licenseKeyChecker.requireProOrEnterprise("storage.provider=" + provider);
}
}
if ("s3".equals(normalize(clusterArtifactStore))) {
licenseKeyChecker.requireProOrEnterprise("cluster.artifactStore=s3");
}
if (!clusterEnabled) {
return;
}
if (storage != null && storage.isEnabled()) {
validate(
"storage.provider",
storage.getProvider(),
"Local filesystem storage cannot be shared across cluster nodes."
+ " Configure storage.provider=s3 (with storage.s3.bucket /"
+ " endpoint / credentials) or storage.provider=database before"
+ " enabling clustering.");
}
validate(
"cluster.artifactStore",
clusterArtifactStore,
"Per-node disk cannot back transient job-result files in a multi-node"
+ " deployment; downloads would 404 whenever the load balancer routes"
+ " a follow-up request to a different node. Configure"
+ " cluster.artifactStore=s3 (reuses storage.s3.* config)"
+ " before enabling clustering.");
}
private static String normalize(String value) {
return Optional.ofNullable(value).orElse("local").trim().toLowerCase(Locale.ROOT);
}
private static void validate(String propertyName, String configuredValue, String remediation) {
String normalized =
Optional.ofNullable(configuredValue)
.orElse("local")
.trim()
.toLowerCase(Locale.ROOT);
if ("local".equals(normalized)) {
throw new IllegalStateException(
"Cluster mode (cluster.enabled=true) is incompatible with "
+ propertyName
+ "=local. "
+ remediation);
}
log.info(
"Cluster storage gate: clusterEnabled=true, {}={} -> OK", propertyName, normalized);
}
}
@@ -15,8 +15,11 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.common.configuration.InstallationPathConfig;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.cluster.s3.S3Clients;
import stirling.software.proprietary.security.configuration.ee.LicenseKeyChecker;
import stirling.software.proprietary.storage.provider.DatabaseStorageProvider;
import stirling.software.proprietary.storage.provider.LocalStorageProvider;
import stirling.software.proprietary.storage.provider.S3StorageProvider;
import stirling.software.proprietary.storage.provider.StorageProvider;
import stirling.software.proprietary.storage.repository.StoredFileBlobRepository;
@@ -27,8 +30,9 @@ public class StorageProviderConfig {
private final ApplicationProperties applicationProperties;
private final StoredFileBlobRepository storedFileBlobRepository;
private final LicenseKeyChecker licenseKeyChecker;
@Bean
@Bean(destroyMethod = "close")
public StorageProvider storageProvider() {
boolean storageEnabled = applicationProperties.getStorage().isEnabled();
String providerName =
@@ -37,8 +41,13 @@ public class StorageProviderConfig {
.trim()
.toLowerCase(Locale.ROOT);
if ("database".equals(providerName)) {
licenseKeyChecker.requireProOrEnterprise("storage.provider=database");
return new DatabaseStorageProvider(storedFileBlobRepository);
}
if ("s3".equals(providerName)) {
licenseKeyChecker.requireProOrEnterprise("storage.provider=s3");
return buildS3Provider(applicationProperties.getStorage().getS3());
}
if (!"local".equals(providerName)) {
throw new IllegalStateException("Storage provider not supported: " + providerName);
}
@@ -71,4 +80,9 @@ public class StorageProviderConfig {
}
return new LocalStorageProvider(basePath);
}
private S3StorageProvider buildS3Provider(ApplicationProperties.Storage.S3 cfg) {
S3Clients.Bundle bundle = S3Clients.build(cfg, "storage provider");
return new S3StorageProvider(bundle.client(), bundle.presigner(), cfg.getBucket());
}
}
@@ -0,0 +1,91 @@
package stirling.software.proprietary.storage.controller;
import java.util.List;
import java.util.UUID;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import jakarta.validation.Valid;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
import stirling.software.proprietary.storage.service.FolderService;
/**
* Folder placement endpoints for existing stored files. Thin adapter: validates the request shape,
* delegates the transaction to {@link FolderService}, then maps the result onto the HTTP status.
* Authentication, storage-gate, ownership checks, and the bulk cap all live on the service (where
* {@code @Transactional} also lives) so the JDBC connection isn't held through JSON serialization.
*/
@RestController
@RequestMapping("/api/v1/storage/files")
@RequiredArgsConstructor
public class FileFolderPlacementController {
private static final int BULK_MOVE_MAX_FILES = 1000;
private final FolderService folderService;
/** Move a single file to a folder (or to root when folderId is null). */
@PatchMapping("/{fileId}/folder")
public ResponseEntity<Void> moveFileToFolder(
@PathVariable Long fileId, @Valid @RequestBody FolderPlacement body) {
folderService.moveFileToFolder(fileId, body.getFolderId());
return ResponseEntity.noContent().build();
}
/**
* Bulk move - fewer round-trips than calling the single endpoint N times. Returns 200 on full
* success, 207 (Multi-Status) when some files were skipped (typically because they don't belong
* to the caller).
*/
@PatchMapping("/folder")
public ResponseEntity<BulkMoveResponse> bulkMove(@Valid @RequestBody BulkMoveRequest body) {
FolderService.BulkMoveResult result =
folderService.bulkMoveFilesToFolder(body.getFolderId(), body.getFileIds());
HttpStatus status =
result.skippedFileIds().isEmpty() ? HttpStatus.OK : HttpStatus.MULTI_STATUS;
return ResponseEntity.status(status)
.body(new BulkMoveResponse(result.movedFileIds(), result.skippedFileIds()));
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class FolderPlacement {
private UUID folderId;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class BulkMoveRequest {
private UUID folderId;
@NotNull
@Size(
min = 1,
max = BULK_MOVE_MAX_FILES,
message = "fileIds must contain between 1 and 1000 entries")
private List<Long> fileIds;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class BulkMoveResponse {
private List<Long> movedFileIds;
private List<Long> skippedFileIds;
}
}
@@ -1,7 +1,11 @@
package stirling.software.proprietary.storage.controller;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
import org.springframework.http.ContentDisposition;
import org.springframework.http.HttpHeaders;
@@ -25,6 +29,7 @@ import org.springframework.web.server.ResponseStatusException;
import io.swagger.v3.oas.annotations.tags.Tag;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.FileShare;
@@ -35,17 +40,22 @@ import stirling.software.proprietary.storage.model.api.ShareLinkMetadataResponse
import stirling.software.proprietary.storage.model.api.ShareLinkResponse;
import stirling.software.proprietary.storage.model.api.ShareWithUserRequest;
import stirling.software.proprietary.storage.model.api.StoredFileResponse;
import stirling.software.proprietary.storage.provider.StorageProvider;
import stirling.software.proprietary.storage.service.FileStorageService;
@RestController
@RequestMapping("/api/v1/storage")
@RequiredArgsConstructor
@Slf4j
@Tag(
name = "File Storage",
description = "Stored file management, sharing, and share link operations")
public class FileStorageController {
private static final Duration SIGNED_URL_TTL = Duration.ofMinutes(5);
private final FileStorageService fileStorageService;
private final StorageProvider storageProvider;
@PostMapping(
value = "/files",
@@ -91,7 +101,9 @@ public class FileStorageController {
User user = fileStorageService.requireAuthenticatedUser();
StoredFile file = fileStorageService.getAccessibleFile(user, fileId);
fileStorageService.requireReadAccess(user, file);
return buildFileResponse(file, inline);
Optional<ResponseEntity<org.springframework.core.io.Resource>> redirect =
tryRedirectToSignedUrl(file, inline);
return redirect.orElseGet(() -> buildFileResponse(file, inline));
}
@DeleteMapping("/files/{fileId}")
@@ -189,7 +201,9 @@ public class FileStorageController {
fileStorageService.requireReadAccess(share);
fileStorageService.recordShareAccess(share, authentication, inline);
StoredFile file = share.getFile();
return buildFileResponse(file, inline);
Optional<ResponseEntity<org.springframework.core.io.Resource>> redirect =
tryRedirectToSignedUrl(file, inline);
return redirect.orElseGet(() -> buildFileResponse(file, inline));
}
@GetMapping("/share-links/{token}/metadata")
@@ -272,4 +286,34 @@ public class FileStorageController {
&& authentication.isAuthenticated()
&& !"anonymousUser".equals(authentication.getPrincipal());
}
private Optional<ResponseEntity<org.springframework.core.io.Resource>> tryRedirectToSignedUrl(
StoredFile file, boolean inline) {
if (file == null || file.getStorageKey() == null || file.getStorageKey().isBlank()) {
return Optional.empty();
}
try {
Optional<URI> signed =
storageProvider.signedDownloadUrl(
file.getStorageKey(),
SIGNED_URL_TTL,
inline,
file.getOriginalFilename());
if (signed.isEmpty()) {
return Optional.empty();
}
HttpHeaders headers = new HttpHeaders();
headers.setLocation(signed.get());
ResponseEntity<org.springframework.core.io.Resource> response =
ResponseEntity.status(HttpStatus.FOUND).headers(headers).build();
return Optional.of(response);
} catch (IOException e) {
log.warn(
"Failed to create signed download URL for file {} (key: {}), falling back to streaming",
file.getId(),
file.getStorageKey(),
e);
return Optional.empty();
}
}
}
@@ -0,0 +1,70 @@
package stirling.software.proprietary.storage.controller;
import java.net.URI;
import java.util.List;
import java.util.UUID;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import stirling.software.proprietary.storage.model.api.CreateFolderRequest;
import stirling.software.proprietary.storage.model.api.FolderResponse;
import stirling.software.proprietary.storage.model.api.UpdateFolderRequest;
import stirling.software.proprietary.storage.service.FolderService;
/**
* REST endpoints for user-owned folders. Phase A - no folder-level sharing yet (Phase 3).
*
* <p>All operations are scoped to the authenticated user; existing single-file storage endpoints in
* {@link FileStorageController} are left alone so the cert-signing and standard upload flows are
* unaffected.
*/
@RestController
@RequestMapping("/api/v1/storage/folders")
@RequiredArgsConstructor
public class FolderController {
private final FolderService folderService;
@GetMapping
public List<FolderResponse> listFolders() {
return folderService.listFolders();
}
@PostMapping
public ResponseEntity<FolderResponse> createFolder(
@Valid @RequestBody CreateFolderRequest request) {
FolderResponse response = folderService.createFolder(request);
// 201 Created with Location header - conventional REST. The idempotent re-return path
// (same id resubmitted) also lands here; treating it as 201 keeps wire semantics simple.
return ResponseEntity.status(HttpStatus.CREATED)
.location(URI.create("/api/v1/storage/folders/" + response.id()))
.body(response);
}
@PatchMapping("/{folderId}")
public ResponseEntity<FolderResponse> updateFolder(
@PathVariable UUID folderId, @Valid @RequestBody UpdateFolderRequest request) {
return ResponseEntity.ok(folderService.updateFolder(folderId, request));
}
@DeleteMapping("/{folderId}")
public ResponseEntity<DeleteFolderResponse> deleteFolder(@PathVariable UUID folderId) {
List<UUID> removed = folderService.deleteFolder(folderId);
return ResponseEntity.ok(new DeleteFolderResponse(removed));
}
public record DeleteFolderResponse(List<UUID> removedFolderIds) {}
}
@@ -0,0 +1,104 @@
package stirling.software.proprietary.storage.model;
import java.io.Serializable;
import java.time.LocalDateTime;
import java.util.UUID;
import org.hibernate.annotations.CreationTimestamp;
import org.hibernate.annotations.OnDelete;
import org.hibernate.annotations.OnDeleteAction;
import org.hibernate.annotations.UpdateTimestamp;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.Id;
import jakarta.persistence.Index;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.persistence.Version;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import stirling.software.proprietary.security.model.User;
/**
* A user-owned folder used by the file manager UI to organise stored files. Phase A entity - no
* folder-level sharing yet (Phase 3).
*
* <p>The id is a UUID rather than a numeric auto-increment so it round-trips with the
* client-generated {@code FolderId} and survives cross-device sync without re-keying.
*/
@Entity
@Table(
name = "folders",
indexes = {
@Index(name = "idx_folders_owner", columnList = "owner_id"),
@Index(name = "idx_folders_parent", columnList = "parent_folder_id"),
@Index(name = "idx_folders_owner_parent", columnList = "owner_id, parent_folder_id")
})
@NoArgsConstructor
@Getter
@Setter
public class Folder implements Serializable {
private static final long serialVersionUID = 1L;
/**
* Dialect-portable UUID column. The previous {@code columnDefinition = "uuid"} was
* Postgres-specific and broke on H2/MariaDB. Hibernate's {@code UUID} mapping picks the right
* native type per dialect (BINARY(16) on H2/MariaDB, uuid on Postgres) when no explicit
* columnDefinition is set.
*/
@Id
@Column(name = "folder_id", nullable = false)
private UUID id;
/**
* {@code OnDeleteAction.CASCADE} so deleting the owning {@code User} cascades to this row at
* the DB level - UserService.deleteUserRelatedData doesn't enumerate folders today, and leaving
* the FK without an action throws a constraint violation on user delete.
*/
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "owner_id", nullable = false)
@OnDelete(action = OnDeleteAction.CASCADE)
private User owner;
/**
* Parent folder; null = root. {@code OnDeleteAction.CASCADE} so a backend-side parent delete
* cleans children automatically, matching the service-layer recursive-delete contract.
*/
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "parent_folder_id")
@OnDelete(action = OnDeleteAction.CASCADE)
private Folder parent;
@Column(name = "name", nullable = false, length = 255)
private String name;
@Column(name = "color", length = 32)
private String color;
@Column(name = "icon", length = 64)
private String icon;
/**
* Optimistic-locking version. Cross-PC sync without this lets last-write-win silently. The
* column is nullable so existing rows from a pre-version deployment can be backfilled by
* Hibernate's update-on-write rather than failing the ddl-auto upgrade.
*/
@Version
@Column(name = "version")
private Long version;
@CreationTimestamp
@Column(name = "created_at", updatable = false)
private LocalDateTime createdAt;
@UpdateTimestamp
@Column(name = "updated_at")
private LocalDateTime updatedAt;
}
@@ -6,6 +6,8 @@ import java.util.HashSet;
import java.util.Set;
import org.hibernate.annotations.CreationTimestamp;
import org.hibernate.annotations.OnDelete;
import org.hibernate.annotations.OnDeleteAction;
import org.hibernate.annotations.UpdateTimestamp;
import jakarta.persistence.CascadeType;
@@ -35,7 +37,8 @@ import stirling.software.proprietary.workflow.model.WorkflowSession;
name = "stored_files",
indexes = {
@Index(name = "idx_stored_files_owner", columnList = "owner_id"),
@Index(name = "idx_stored_files_workflow", columnList = "workflow_session_id")
@Index(name = "idx_stored_files_workflow", columnList = "workflow_session_id"),
@Index(name = "idx_stored_files_folder", columnList = "folder_id")
})
@NoArgsConstructor
@Getter
@@ -106,6 +109,20 @@ public class StoredFile implements Serializable {
orphanRemoval = true)
private Set<FileShare> shares = new HashSet<>();
/**
* Optional folder placement for the file manager UI. Null = root. Hibernate ddl-auto will add
* this as a nullable column on upgrade so existing records continue to work untouched.
*
* <p>{@code OnDeleteAction.SET_NULL} so any backend that drops a folder row (admin script,
* future cleanup job, cascading user delete) cleanly orphans files to root rather than leaving
* dangling FK references. The application path ({@code FolderRepository.clearFolderForFiles})
* still runs first as a belt-and-braces.
*/
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "folder_id")
@OnDelete(action = OnDeleteAction.SET_NULL)
private Folder folder;
@CreationTimestamp
@Column(name = "created_at", updatable = false)
private LocalDateTime createdAt;
@@ -0,0 +1,43 @@
package stirling.software.proprietary.storage.model.api;
import java.util.UUID;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Pattern;
import jakarta.validation.constraints.Size;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
@Data
@NoArgsConstructor
@AllArgsConstructor
public class CreateFolderRequest {
/**
* Client-generated UUID - lets the caller round-trip the same id it stored locally. Optional;
* the server generates one when missing.
*/
private UUID id;
@NotBlank
@Size(max = 255)
private String name;
private UUID parentFolderId;
/** Hex colour string (#rrggbb or #rrggbbaa) - matches the frontend palette format. */
@Size(max = 32)
@Pattern(
regexp = "^#[0-9a-fA-F]{6}([0-9a-fA-F]{2})?$",
message = "color must be a #RRGGBB or #RRGGBBAA hex value")
private String color;
/** Icon identifier - lowercase alphanumerics, hyphens, underscores only. */
@Size(max = 64)
@Pattern(
regexp = "^[a-z0-9_-]+$",
message = "icon must be a lowercase id (a-z, 0-9, '-' or '_')")
private String icon;
}
@@ -0,0 +1,38 @@
package stirling.software.proprietary.storage.model.api;
import java.time.LocalDateTime;
import java.util.UUID;
import stirling.software.proprietary.storage.model.Folder;
/**
* Outbound DTO for folder responses. Records are immutable, value-equality-based, and far less
* accident-prone than a {@code @Data} class with public setters.
*/
public record FolderResponse(
UUID id,
String name,
UUID parentFolderId,
String color,
String icon,
Long version,
LocalDateTime createdAt,
LocalDateTime updatedAt) {
public static FolderResponse from(Folder folder) {
// {@code folder.getParent().getId()} on a lazy proxy returns the FK value cached at the
// join column WITHOUT initialising the proxy under standard Hibernate, so this does
// not N+1. If a future Hibernate update changes that, switch the JPQL list query to a
// constructor projection.
UUID parentId = folder.getParent() == null ? null : folder.getParent().getId();
return new FolderResponse(
folder.getId(),
folder.getName(),
parentId,
folder.getColor(),
folder.getIcon(),
folder.getVersion(),
folder.getCreatedAt(),
folder.getUpdatedAt());
}
}
@@ -2,6 +2,7 @@ package stirling.software.proprietary.storage.model.api;
import java.time.LocalDateTime;
import java.util.List;
import java.util.UUID;
import lombok.Builder;
import lombok.Getter;
@@ -22,4 +23,10 @@ public class StoredFileResponse {
private final List<SharedUserResponse> sharedUsers;
private final List<ShareLinkResponse> shareLinks;
private final String filePurpose;
/**
* Optional folder placement (Phase A). Null when the file lives at the root or when the server
* build doesn't have the folders feature enabled - existing clients should treat null as root.
*/
private final UUID folderId;
}
@@ -0,0 +1,58 @@
package stirling.software.proprietary.storage.model.api;
import java.util.UUID;
import jakarta.validation.constraints.Pattern;
import jakarta.validation.constraints.Size;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* PATCH-style update - every field is optional. Send only the fields you want to change.
*
* <p>The {@code reparent} flag distinguishes "do not change parent" from "move to root" since
* {@code parentFolderId == null} alone is ambiguous in a sparse body. We use a boxed {@link
* Boolean} so a missing field deserialises to {@code null} (= "do not reparent") rather than to
* primitive {@code false}, removing a class of "I PATCHed only the name but the server reset my
* parent" footguns.
*
* <p>When the trimmed name is empty (e.g. {@code " "}) the service rejects the request with HTTP
* 400 - silent drops are too easy to mistake for a successful rename.
*/
@Data
@NoArgsConstructor
@AllArgsConstructor
public class UpdateFolderRequest {
/** When provided, must contain at least one non-whitespace character. */
@Size(max = 255)
@Pattern(regexp = "\\S.*", message = "name must not be blank")
private String name;
private Boolean reparent;
private UUID parentFolderId;
@Size(max = 32)
@Pattern(
regexp = "^(|#[0-9a-fA-F]{6}([0-9a-fA-F]{2})?)$",
message = "color must be empty or a #RRGGBB / #RRGGBBAA hex value")
private String color;
@Size(max = 64)
@Pattern(
regexp = "^([a-z0-9_-]+)?$",
message = "icon must be a lowercase id (a-z, 0-9, '-' or '_') or empty")
private String icon;
/**
* Convenience accessor - treats null as "do not reparent". Named differently from the
* Lombok-generated {@code getReparent()} so callers don't accidentally use one for the other
* (the getter is nullable {@code Boolean}; this method collapses to primitive).
*/
@com.fasterxml.jackson.annotation.JsonIgnore
public boolean shouldReparent() {
return Boolean.TRUE.equals(reparent);
}
}
@@ -24,6 +24,9 @@ public class LocalStorageProvider implements StorageProvider {
@Override
public StoredObject store(User owner, MultipartFile file) throws IOException {
if (owner == null || owner.getId() == null) {
throw new IllegalArgumentException("owner.id is required for local storage key");
}
String originalFilename = sanitizeFilename(file.getOriginalFilename());
String storageKey =
owner.getId()
@@ -77,6 +80,7 @@ public class LocalStorageProvider implements StorageProvider {
if (filename == null || filename.isBlank()) {
return "file";
}
return Paths.get(filename).getFileName().toString();
String stripped = Paths.get(filename).getFileName().toString().replaceAll("\\p{Cntrl}", "");
return stripped.isBlank() ? "file" : stripped;
}
}
@@ -0,0 +1,190 @@
package stirling.software.proprietary.storage.provider;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.Optional;
import java.util.UUID;
import org.springframework.core.io.InputStreamResource;
import org.springframework.core.io.Resource;
import org.springframework.web.multipart.MultipartFile;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.security.model.User;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
/** {@link StorageProvider} backed by an S3-compatible object store. */
@Slf4j
public class S3StorageProvider implements StorageProvider, AutoCloseable {
private final S3Client s3Client;
private final S3Presigner s3Presigner;
private final String bucket;
public S3StorageProvider(S3Client s3Client, S3Presigner s3Presigner, String bucket) {
if (bucket == null || bucket.isBlank()) {
throw new IllegalArgumentException("S3 bucket must be configured");
}
this.s3Client = s3Client;
this.s3Presigner = s3Presigner;
this.bucket = bucket;
}
@Override
public StoredObject store(User owner, MultipartFile file) throws IOException {
if (owner == null || owner.getId() == null) {
throw new IllegalArgumentException("owner.id is required for S3 storage key");
}
String originalFilename = sanitizeFilename(file.getOriginalFilename());
// Key is opaque ({ownerId}/{uuid}) so non-ASCII filenames don't break vendors that
// restrict key charset (e.g. Supabase Storage returns 400 Invalid key on unicode).
// The display name is preserved in StoredObject.originalFilename and the DB row.
String storageKey = owner.getId() + "/" + UUID.randomUUID();
PutObjectRequest.Builder request =
PutObjectRequest.builder().bucket(bucket).key(storageKey);
if (file.getContentType() != null && !file.getContentType().isBlank()) {
request.contentType(file.getContentType());
}
try (InputStream inputStream = file.getInputStream()) {
s3Client.putObject(
request.build(), RequestBody.fromInputStream(inputStream, file.getSize()));
} catch (SdkException e) {
throw new IOException("Failed to upload object to S3", e);
}
return StoredObject.builder()
.storageKey(storageKey)
.originalFilename(originalFilename)
.contentType(file.getContentType())
.sizeBytes(file.getSize())
.build();
}
@Override
public Resource load(String storageKey) throws IOException {
GetObjectRequest request =
GetObjectRequest.builder().bucket(bucket).key(storageKey).build();
try {
ResponseInputStream<GetObjectResponse> stream = s3Client.getObject(request);
long contentLength =
stream.response().contentLength() != null
? stream.response().contentLength()
: -1;
return new InputStreamResource(stream) {
@Override
public long contentLength() {
return contentLength;
}
};
} catch (NoSuchKeyException e) {
throw new IOException("File not found", e);
} catch (SdkException e) {
throw new IOException("Failed to load object from S3", e);
}
}
@Override
public void delete(String storageKey) throws IOException {
try {
s3Client.deleteObject(
DeleteObjectRequest.builder().bucket(bucket).key(storageKey).build());
} catch (SdkException e) {
throw new IOException("Failed to delete object from S3", e);
}
}
@Override
public Optional<URI> signedDownloadUrl(String storageKey, Duration ttl) throws IOException {
return signedDownloadUrl(storageKey, ttl, false, null);
}
@Override
public Optional<URI> signedDownloadUrl(
String storageKey, Duration ttl, boolean inline, String originalFilename)
throws IOException {
if (storageKey == null || storageKey.isBlank()) {
return Optional.empty();
}
Duration effectiveTtl =
ttl == null || ttl.isZero() || ttl.isNegative() ? Duration.ofMinutes(5) : ttl;
try {
GetObjectRequest.Builder getBuilder =
GetObjectRequest.builder().bucket(bucket).key(storageKey);
String disposition = buildContentDisposition(inline, originalFilename);
if (disposition != null) {
getBuilder.responseContentDisposition(disposition);
}
GetObjectPresignRequest presignRequest =
GetObjectPresignRequest.builder()
.signatureDuration(effectiveTtl)
.getObjectRequest(getBuilder.build())
.build();
PresignedGetObjectRequest presigned = s3Presigner.presignGetObject(presignRequest);
return Optional.of(presigned.url().toURI());
} catch (SdkException | URISyntaxException e) {
log.warn("Failed to create presigned S3 GET URL for key {}", storageKey, e);
return Optional.empty();
}
}
// Returns null when originalFilename is blank; S3 falls back to its own default in that case.
static String buildContentDisposition(boolean inline, String originalFilename) {
if (originalFilename == null || originalFilename.isBlank()) {
return null;
}
// Strip CR/LF and other control chars before path parsing (Paths.get throws on them on
// Windows, and they defeat header parsers).
String stripped = originalFilename.replaceAll("\\p{Cntrl}", "");
// Use only the basename to avoid leaking directory structure into the header.
int lastSeparator = Math.max(stripped.lastIndexOf('/'), stripped.lastIndexOf('\\'));
if (lastSeparator >= 0) {
stripped = stripped.substring(lastSeparator + 1);
}
if (stripped.isBlank()) {
return null;
}
// Escape per RFC 6266 quoted-string rules.
String escaped = stripped.replace("\\", "\\\\").replace("\"", "\\\"");
return (inline ? "inline" : "attachment") + "; filename=\"" + escaped + "\"";
}
@Override
public void close() {
try {
s3Presigner.close();
} catch (Exception e) {
log.warn("Error closing S3 presigner", e);
}
try {
s3Client.close();
} catch (Exception e) {
log.warn("Error closing S3 client", e);
}
}
private String sanitizeFilename(String filename) {
if (filename == null || filename.isBlank()) {
return "file";
}
String stripped = Paths.get(filename).getFileName().toString().replaceAll("\\p{Cntrl}", "");
return stripped.isBlank() ? "file" : stripped;
}
}
@@ -1,16 +1,45 @@
package stirling.software.proprietary.storage.provider;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.util.Optional;
import org.springframework.core.io.Resource;
import org.springframework.web.multipart.MultipartFile;
import stirling.software.proprietary.security.model.User;
public interface StorageProvider {
public interface StorageProvider extends AutoCloseable {
StoredObject store(User owner, MultipartFile file) throws IOException;
Resource load(String storageKey) throws IOException;
void delete(String storageKey) throws IOException;
/**
* Releases any backend-specific resources. Default no-op so {@link LocalStorageProvider} and
* {@link DatabaseStorageProvider} (which hold no closeable handles) satisfy Spring's
* {@code @Bean(destroyMethod = "close")} signature requirement without ceremony. {@code
* S3StorageProvider} overrides this to close the underlying SDK client + presigner.
*/
@Override
default void close() {}
/**
* Returns a presigned download URL valid for {@code ttl}, or {@link Optional#empty()} if the
* provider does not support signed URLs (callers fall back to {@link #load(String)}).
*/
default Optional<URI> signedDownloadUrl(String storageKey, Duration ttl) throws IOException {
return signedDownloadUrl(storageKey, ttl, false, null);
}
/**
* Like {@link #signedDownloadUrl(String, Duration)} with explicit Content-Disposition control.
*/
default Optional<URI> signedDownloadUrl(
String storageKey, Duration ttl, boolean inline, String originalFilename)
throws IOException {
return Optional.empty();
}
}
@@ -0,0 +1,35 @@
package stirling.software.proprietary.storage.repository;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.Folder;
public interface FolderRepository extends JpaRepository<Folder, UUID> {
Optional<Folder> findByIdAndOwner(UUID id, User owner);
List<Folder> findAllByOwnerOrderByName(User owner);
long countByOwner(User owner);
/**
* Clear the folder reference on every file currently inside any of the given folders. Used when
* a folder subtree is deleted - files fall back to the root rather than dangling.
*
* <p>{@code flushAutomatically + clearAutomatically} forces Hibernate to flush any cached dirty
* {@code StoredFile} entities before the bulk UPDATE runs, and clears the persistence context
* afterwards so a subsequent {@code deleteAllByIdInBatch} on the parent folders doesn't see
* stale entity state referencing the about-to-be-deleted folder.
*/
@Modifying(flushAutomatically = true, clearAutomatically = true)
@Query("UPDATE StoredFile sf SET sf.folder = null WHERE sf.folder.id IN :folderIds")
void clearFolderForFiles(@Param("folderIds") List<UUID> folderIds);
}
@@ -59,6 +59,13 @@ public interface StoredFileRepository extends JpaRepository<StoredFile, Long> {
List<StoredFile> findAllByOwner(User owner);
/**
* Bulk lookup used by the folder-placement controller. Returns only files owned by {@code
* owner}; ids that don't exist or that belong to another user are silently dropped so the
* caller can compute the "skipped" set by subtraction.
*/
List<StoredFile> findAllByIdInAndOwner(List<Long> ids, User owner);
@Modifying
@Transactional
@Query(
@@ -459,6 +459,7 @@ public class FileStorageService {
file.getPurpose() != null
? file.getPurpose().name().toLowerCase(Locale.ROOT)
: null)
.folderId(file.getFolder() != null ? file.getFolder().getId() : null)
.build();
}
@@ -0,0 +1,417 @@
package stirling.software.proprietary.storage.service;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.http.HttpStatus;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.server.ResponseStatusException;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.Folder;
import stirling.software.proprietary.storage.model.StoredFile;
import stirling.software.proprietary.storage.model.api.CreateFolderRequest;
import stirling.software.proprietary.storage.model.api.FolderResponse;
import stirling.software.proprietary.storage.model.api.UpdateFolderRequest;
import stirling.software.proprietary.storage.repository.FolderRepository;
import stirling.software.proprietary.storage.repository.StoredFileRepository;
/**
* Phase A folder operations. Each call is scoped to the authenticated user - folders are private to
* their owner. Folder-level sharing is a Phase 3 feature.
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class FolderService {
/**
* Hard cap on folders per user. Beyond this {@link #createFolder} rejects with 409 - guards
* against per-account folder-explosion DoS and bounds the in-memory subtree walk in {@link
* #deleteFolder}.
*/
private static final long MAX_FOLDERS_PER_USER = 5_000L;
/**
* Hard cap on chain depth from the root to any folder. Bounds the lazy-proxy walk in {@link
* #enforceDepthAndCycle} - otherwise a user could build a chain up to MAX_FOLDERS_PER_USER deep
* and force one Hibernate SELECT per ancestor on every reparent (5,000+ SELECTs == seconds of
* DB time per request, per-account weaponizable as DoS).
*/
private static final int MAX_FOLDER_DEPTH = 64;
/**
* Hard cap on bulk-move payload size, mirroring the request-validation cap on {@code
* FileFolderPlacementController.BulkMoveRequest.fileIds}. Re-asserted at the service layer
* because controller-level @Valid bounds aren't enforced when the service is called directly
* (e.g. by future internal callers or tests).
*/
private static final int BULK_MOVE_MAX_FILES = 1000;
private final FolderRepository folderRepository;
private final StoredFileRepository storedFileRepository;
private final ApplicationProperties applicationProperties;
/**
* Gate every public method on storage being enabled, mirroring {@code
* FileStorageService.ensureStorageEnabled}. Without this, folder CRUD still works when {@code
* storage.enabled=false} or {@code security.enableLogin=false}, defeating the operator's intent
* to disable storage end-to-end.
*/
private void ensureStorageEnabled() {
if (!applicationProperties.getSecurity().isEnableLogin()) {
throw new ResponseStatusException(
HttpStatus.FORBIDDEN, "Storage requires login to be enabled");
}
if (!applicationProperties.getStorage().isEnabled()) {
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "Storage is disabled");
}
}
/** List every folder owned by the current user, alphabetical. */
@Transactional(readOnly = true)
public List<FolderResponse> listFolders() {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
return folderRepository.findAllByOwnerOrderByName(user).stream()
.map(FolderResponse::from)
.toList();
}
@Transactional
public FolderResponse createFolder(CreateFolderRequest request) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
// Reject self-parenting up-front. Without this, a client posting
// {id: X, parentFolderId: X} for a folder X they already own would silently
// get the existing folder back (idempotent path) and never learn that the
// parentFolderId they sent was ignored. For new ids the parent lookup would
// 404, but the message is misleading.
if (request.getId() != null && request.getId().equals(request.getParentFolderId())) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "A folder cannot be its own parent");
}
Folder parent = resolveParent(request.getParentFolderId(), user, null);
UUID id = request.getId() != null ? request.getId() : UUID.randomUUID();
// Idempotent: if this user already owns a folder with the supplied id, return it
// unchanged. Single fetch (the previous code did findByIdAndOwner twice with a race
// window between the two lookups).
java.util.Optional<Folder> existing = folderRepository.findByIdAndOwner(id, user);
if (existing.isPresent()) {
return FolderResponse.from(existing.get());
}
// The id is a global primary key. If the id exists for a *different* user, surfacing 500
// with a constraint-violation stack trace leaks far too much; convert to 409 Conflict so
// the caller can pick a fresh id.
if (folderRepository.existsById(id)) {
throw new ResponseStatusException(
HttpStatus.CONFLICT,
"A folder with this id already exists; choose a different id");
}
if (folderRepository.countByOwner(user) >= MAX_FOLDERS_PER_USER) {
throw new ResponseStatusException(
HttpStatus.CONFLICT,
"Folder limit reached (max " + MAX_FOLDERS_PER_USER + " per user)");
}
Folder folder = new Folder();
folder.setId(id);
folder.setOwner(user);
folder.setParent(parent);
folder.setName(request.getName().trim());
folder.setColor(request.getColor());
folder.setIcon(request.getIcon());
// saveAndFlush forces the INSERT now so @CreationTimestamp populates
// createdAt/updatedAt before we build the response. Plain save defers
// the SQL until @Transactional commit, and the response would carry
// null timestamps that the frontend trust-boundary parser then rejects.
Folder saved = folderRepository.saveAndFlush(folder);
log.info(
"Folder created: user={} id={} parent={}",
user.getId(),
saved.getId(),
parent == null ? "root" : parent.getId());
return FolderResponse.from(saved);
}
@Transactional
public FolderResponse updateFolder(UUID id, UpdateFolderRequest request) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
Folder folder = requireOwnedFolder(id, user);
if (request.getName() != null) {
String trimmed = request.getName().trim();
if (trimmed.isEmpty()) {
// Bean validation should already catch this via @Pattern, but be explicit so
// an empty-after-trim payload reaches the user as a 400 instead of being
// silently dropped.
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "Folder name cannot be blank");
}
folder.setName(trimmed);
}
if (request.shouldReparent()) {
Folder newParent = resolveParent(request.getParentFolderId(), user, folder.getId());
folder.setParent(newParent);
}
if (request.getColor() != null) {
folder.setColor(request.getColor().isEmpty() ? null : request.getColor());
}
if (request.getIcon() != null) {
folder.setIcon(request.getIcon().isEmpty() ? null : request.getIcon());
}
// saveAndFlush so @UpdateTimestamp populates updatedAt before the
// response is serialized (same reason as createFolder).
return FolderResponse.from(folderRepository.saveAndFlush(folder));
}
/**
* Recursive delete. Returns the ids of every folder that was removed so the caller can purge
* them from its local cache. Files inside those folders are detached (folder_id set to null) -
* never deleted.
*/
@Transactional
public List<UUID> deleteFolder(UUID id) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
Folder folder = requireOwnedFolder(id, user);
// Build the parent → children map once. Project to id-only via the
// existing entity list (Hibernate already has the column loaded -
// we only access f.getParent().getId() on a managed proxy, which
// does NOT initialize the proxy because Hibernate has the FK
// value cached at the join column).
Map<UUID, List<UUID>> childIdsByParent = new HashMap<>();
for (Folder f : folderRepository.findAllByOwnerOrderByName(user)) {
UUID parentId = f.getParent() == null ? null : f.getParent().getId();
childIdsByParent.computeIfAbsent(parentId, k -> new ArrayList<>()).add(f.getId());
}
// Iterative subtree collection - prior recursive form blew the JVM
// stack on deeply nested chains a malicious caller could create.
List<UUID> removed = new ArrayList<>();
Set<UUID> seen = new HashSet<>();
Deque<UUID> stack = new ArrayDeque<>();
stack.push(folder.getId());
while (!stack.isEmpty()) {
UUID cur = stack.pop();
if (!seen.add(cur)) continue;
removed.add(cur);
List<UUID> children = childIdsByParent.get(cur);
if (children != null) {
for (UUID childId : children) stack.push(childId);
}
}
if (!removed.isEmpty()) {
folderRepository.clearFolderForFiles(removed);
folderRepository.deleteAllByIdInBatch(removed);
log.info(
"Folder subtree deleted: user={} root={} count={}",
user.getId(),
folder.getId(),
removed.size());
}
return removed;
}
/**
* Move a single owned file to a folder (or root when {@code folderId} is null). Owns its
* own @Transactional rather than relying on the caller so the JDBC connection is released as
* soon as the writes commit, not held through controller-side JSON serialization.
*/
@Transactional
public void moveFileToFolder(Long fileId, UUID folderId) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
StoredFile file =
storedFileRepository
.findByIdAndOwner(fileId, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.NOT_FOUND,
"File not found or not owned by current user"));
file.setFolder(resolveOwnedFolder(folderId, user));
storedFileRepository.save(file);
}
/**
* Bulk move that returns the moved + skipped split. Skipped == file ids the caller doesn't own
* (or that no longer exist); the controller surfaces this as 207 Multi-Status.
*/
@Transactional
public BulkMoveResult bulkMoveFilesToFolder(UUID folderId, List<Long> fileIds) {
ensureStorageEnabled();
if (fileIds == null || fileIds.isEmpty()) {
return new BulkMoveResult(List.of(), List.of());
}
if (fileIds.size() > BULK_MOVE_MAX_FILES) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"fileIds must contain between 1 and " + BULK_MOVE_MAX_FILES + " entries");
}
User user = requireAuthenticatedUser();
Folder target = resolveOwnedFolder(folderId, user);
List<StoredFile> owned = storedFileRepository.findAllByIdInAndOwner(fileIds, user);
Set<Long> ownedIds = new HashSet<>(owned.size());
for (StoredFile f : owned) {
f.setFolder(target);
ownedIds.add(f.getId());
}
// If the target folder was deleted concurrently between resolveOwnedFolder and the
// flush, the FK constraint fires as DataIntegrityViolationException. Surface that as
// 409 Conflict so the caller sees an actionable error instead of a 500 stack.
try {
storedFileRepository.saveAll(owned);
storedFileRepository.flush();
} catch (DataIntegrityViolationException ex) {
throw new ResponseStatusException(
HttpStatus.CONFLICT,
"Target folder no longer exists; refresh and try again",
ex);
}
List<Long> moved = owned.stream().map(StoredFile::getId).toList();
List<Long> skipped = fileIds.stream().filter(id -> !ownedIds.contains(id)).toList();
if (!skipped.isEmpty()) {
log.warn(
"bulkMove: user {} skipped {} of {} files (not owned or missing)",
user.getId(),
skipped.size(),
fileIds.size());
}
return new BulkMoveResult(moved, skipped);
}
/** Result of {@link #bulkMoveFilesToFolder}. Records are immutable + auto-serializable. */
public record BulkMoveResult(List<Long> movedFileIds, List<Long> skippedFileIds) {}
// ─── helpers ────────────────────────────────────────────────────
/**
* Resolve a placement-target folder. Distinct from {@link #resolveParent} because move targets
* don't carry the parent-cycle semantics - we only need the folder to exist AND belong to the
* caller. Returns null for null input (root).
*/
private Folder resolveOwnedFolder(UUID folderId, User user) {
if (folderId == null) return null;
return folderRepository
.findByIdAndOwner(folderId, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Folder does not exist or is not owned by you"));
}
private Folder requireOwnedFolder(UUID id, User user) {
return folderRepository
.findByIdAndOwner(id, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.NOT_FOUND,
"Folder not found or not owned by current user"));
}
private Folder resolveParent(UUID parentId, User user, UUID forbidId) {
if (parentId == null) return null;
if (forbidId != null && parentId.equals(forbidId)) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "A folder cannot be its own parent");
}
Folder parent =
folderRepository
.findByIdAndOwner(parentId, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Parent folder does not exist or is not owned by you"));
// Reject before the child is created/moved if attaching it would push the chain past the
// depth cap. Done in one pass that also returns the cycle answer so we don't walk the
// lazy-proxy chain twice.
enforceDepthAndCycle(parent, user, forbidId);
return parent;
}
/**
* Single pass that walks the parent chain to root and (a) rejects if attaching a child here
* would exceed MAX_FOLDER_DEPTH, (b) rejects if {@code forbidId} appears in the chain (cycle on
* reparent), (c) rejects on a broken graph, and (d) rejects if any ancestor is owned by a
* different user (defense-in-depth: callers always pass a parent already ownership-checked, but
* the parent chain is followed via lazy proxy without re-checking ownership at each hop, so any
* stray cross-owner edge in the database would otherwise leak ancestor folder ids through the
* cycle error message). The walk is hard-bounded at MAX_FOLDER_DEPTH so a corrupted database
* (chain longer than the API would allow) can never produce an unbounded SELECT loop.
*/
private void enforceDepthAndCycle(Folder candidateParent, User user, UUID forbidId) {
Folder cursor = candidateParent;
Set<UUID> seen = new HashSet<>();
int depth = 0;
while (cursor != null) {
if (cursor.getOwner() == null || !cursor.getOwner().getId().equals(user.getId())) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "Folder hierarchy is corrupted; contact support");
}
if (forbidId != null && cursor.getId().equals(forbidId)) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Cannot move a folder inside one of its descendants");
}
if (!seen.add(cursor.getId())) {
// broken graph (cycle in stored data)
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "Folder hierarchy is corrupted; contact support");
}
depth += 1;
// candidateParent is at depth 1 from the new child's perspective. After the walk,
// `depth` equals the number of ancestors including candidateParent, which is the
// depth at which the new child would live. Reject before exceeding the cap.
if (depth >= MAX_FOLDER_DEPTH) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Folder nesting limit reached (max " + MAX_FOLDER_DEPTH + " levels)");
}
cursor = cursor.getParent();
}
}
private User requireAuthenticatedUser() {
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
if (authentication == null
|| !authentication.isAuthenticated()
|| !(authentication.getPrincipal() instanceof User user)) {
throw new ResponseStatusException(HttpStatus.UNAUTHORIZED, "Authentication required");
}
return user;
}
}
@@ -0,0 +1,147 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.net.URI;
import org.junit.jupiter.api.Test;
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
class S3ClientsTest {
@Test
void validateEndpointHost_publicAwsHost_passes() {
assertThatCode(
() ->
S3Clients.validateEndpointHost(
URI.create("https://s3.us-east-1.amazonaws.com"), false))
.doesNotThrowAnyException();
}
@Test
void validateEndpointHost_metadataServiceIp_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("http://169.254.169.254/"), false))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("allow-private-endpoints");
}
@Test
void validateEndpointHost_loopback_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("http://127.0.0.1:9000/"), false))
.isInstanceOf(IllegalStateException.class);
}
@Test
void validateEndpointHost_rfc1918Private_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("http://10.0.0.5:9000/"), false))
.isInstanceOf(IllegalStateException.class);
}
@Test
void validateEndpointHost_allowPrivateOptIn_bypassesCheck() {
assertThatCode(
() ->
S3Clients.validateEndpointHost(
URI.create("http://169.254.169.254/"), true))
.doesNotThrowAnyException();
assertThatCode(
() ->
S3Clients.validateEndpointHost(
URI.create("http://127.0.0.1:9000/"), true))
.doesNotThrowAnyException();
}
@Test
void validateEndpointHost_missingHost_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("file:///etc/passwd"), false))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("must include a host");
}
@Test
void validateEndpointHost_errorMessageNamesTheFlag() {
assertThat(
catchMessage(
() ->
S3Clients.validateEndpointHost(
URI.create("http://192.168.1.10:9000/"), false)))
.contains("storage.s3.allow-private-endpoints");
}
// ----- requestChecksumCalculation parsing -----
@Test
void parseRequestChecksum_nullOrBlank_defaultsToWhenSupported() {
assertThat(S3Clients.parseRequestChecksum(null))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
assertThat(S3Clients.parseRequestChecksum(""))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
assertThat(S3Clients.parseRequestChecksum(" "))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
}
@Test
void parseRequestChecksum_caseInsensitive_andTrimmed() {
assertThat(S3Clients.parseRequestChecksum("when_required"))
.isEqualTo(RequestChecksumCalculation.WHEN_REQUIRED);
assertThat(S3Clients.parseRequestChecksum(" WHEN_REQUIRED "))
.isEqualTo(RequestChecksumCalculation.WHEN_REQUIRED);
assertThat(S3Clients.parseRequestChecksum("When_Supported"))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
}
@Test
void parseRequestChecksum_unknownValue_fallsBackToDefault() {
assertThat(S3Clients.parseRequestChecksum("yes-please"))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
assertThat(S3Clients.parseRequestChecksum("disabled-completely"))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
}
// ----- responseChecksumValidation parsing -----
@Test
void parseResponseChecksum_nullOrBlank_defaultsToWhenSupported() {
assertThat(S3Clients.parseResponseChecksum(null))
.isEqualTo(ResponseChecksumValidation.WHEN_SUPPORTED);
assertThat(S3Clients.parseResponseChecksum(""))
.isEqualTo(ResponseChecksumValidation.WHEN_SUPPORTED);
}
@Test
void parseResponseChecksum_explicitWhenRequired_returnedAsEnum() {
assertThat(S3Clients.parseResponseChecksum("WHEN_REQUIRED"))
.isEqualTo(ResponseChecksumValidation.WHEN_REQUIRED);
}
@Test
void parseResponseChecksum_unknownValue_fallsBackToDefault() {
assertThat(S3Clients.parseResponseChecksum("nope"))
.isEqualTo(ResponseChecksumValidation.WHEN_SUPPORTED);
}
private static String catchMessage(Runnable r) {
try {
r.run();
return "";
} catch (RuntimeException e) {
return e.getMessage() == null ? "" : e.getMessage();
}
}
}
@@ -0,0 +1,253 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.MinIOContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import stirling.software.common.cluster.FileStore;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
@Testcontainers(disabledWithoutDocker = true)
class S3FileStoreTest {
private static final String BUCKET = "stirling-test-filestore";
private static final String ACCESS_KEY = "minioadmin";
private static final String SECRET_KEY = "minioadmin";
@Container
static MinIOContainer minio =
new MinIOContainer("minio/minio:latest")
.withUserName(ACCESS_KEY)
.withPassword(SECRET_KEY);
private static S3Client s3Client;
private static S3FileStore store;
@BeforeAll
static void setUp() {
URI endpoint = URI.create(minio.getS3URL());
AwsBasicCredentials creds = AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY);
S3Configuration s3Config = S3Configuration.builder().pathStyleAccessEnabled(true).build();
s3Client =
S3Client.builder()
.endpointOverride(endpoint)
.httpClient(UrlConnectionHttpClient.create())
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(creds))
.serviceConfiguration(s3Config)
.build();
s3Client.createBucket(CreateBucketRequest.builder().bucket(BUCKET).build());
store = new S3FileStore(s3Client, BUCKET, "transient/", false);
}
@AfterAll
static void tearDown() {
if (store != null) {
store.close();
}
if (s3Client != null) {
s3Client.close();
}
}
@Test
void blankBucket_constructorRejects() {
assertThatThrownBy(() -> new S3FileStore(s3Client, ""))
.isInstanceOf(IllegalArgumentException.class);
assertThatThrownBy(() -> new S3FileStore(s3Client, null))
.isInstanceOf(IllegalArgumentException.class);
}
@Test
void store_thenRetrieve_roundTripsContent() throws IOException {
byte[] payload = "hello cluster s3".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = store.store(new ByteArrayInputStream(payload), "foo.txt");
assertThat(stored.fileId()).isNotBlank();
assertThat(stored.size()).isEqualTo(payload.length);
assertThat(store.exists(stored.fileId())).isTrue();
assertThat(store.size(stored.fileId())).isEqualTo(payload.length);
assertThat(store.retrieveBytes(stored.fileId())).isEqualTo(payload);
try (InputStream in = store.retrieve(stored.fileId())) {
assertThat(in.readAllBytes()).isEqualTo(payload);
}
}
@Test
void store_keysUseConfiguredPrefix() throws IOException {
byte[] payload = "prefixed".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = store.store(new ByteArrayInputStream(payload), "p.txt");
String prefixed = store.resolveKey(stored.fileId());
assertThat(prefixed).startsWith("transient/");
s3Client.headObject(HeadObjectRequest.builder().bucket(BUCKET).key(prefixed).build());
assertThatThrownBy(
() ->
s3Client.headObject(
HeadObjectRequest.builder()
.bucket(BUCKET)
.key(stored.fileId())
.build()))
.isInstanceOfAny(
NoSuchKeyException.class,
software.amazon.awssdk.services.s3.model.S3Exception.class);
}
@Test
void emptyPrefix_writesAtBucketRoot() throws IOException {
S3FileStore rootStore = new S3FileStore(s3Client, BUCKET, "", false);
byte[] payload = "no-prefix".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = rootStore.store(new ByteArrayInputStream(payload), "r.txt");
assertThat(rootStore.resolveKey(stored.fileId())).isEqualTo(stored.fileId());
assertThat(rootStore.retrieveBytes(stored.fileId())).isEqualTo(payload);
assertThat(rootStore.delete(stored.fileId())).isTrue();
}
@Test
void delete_removesObject_andReturnsTrue() throws IOException {
FileStore.Stored stored =
store.store(new ByteArrayInputStream(new byte[] {1, 2, 3}), "d.bin");
assertThat(store.delete(stored.fileId())).isTrue();
assertThat(store.exists(stored.fileId())).isFalse();
assertThatThrownBy(() -> store.retrieveBytes(stored.fileId()))
.isInstanceOf(IOException.class);
}
@Test
void delete_unknownKey_isIdempotentReturnsTrue() {
// S3 DeleteObject is idempotent (returns 204 whether or not the object existed).
// The store reflects S3's behaviour rather than racing a HEAD before each DELETE.
assertThat(store.delete("00000000-0000-0000-0000-000000000000")).isTrue();
}
@Test
void retrieve_missingKey_throwsIOException() {
assertThatThrownBy(() -> store.retrieveBytes("does-not-exist"))
.isInstanceOf(IOException.class);
assertThatThrownBy(() -> store.retrieve("does-not-exist")).isInstanceOf(IOException.class);
assertThatThrownBy(() -> store.size("does-not-exist")).isInstanceOf(IOException.class);
}
@Test
void exists_returnsFalseForBlankOrTraversalIds() {
assertThat(store.exists(null)).isFalse();
assertThat(store.exists("")).isFalse();
assertThat(store.exists("..")).isFalse();
assertThat(store.exists("a/b")).isFalse();
assertThat(store.exists("a\\b")).isFalse();
}
@Test
void delete_traversalId_returnsFalseWithoutCall() {
assertThat(store.delete("../etc/passwd")).isFalse();
assertThat(store.delete("foo/bar")).isFalse();
}
@Test
void store_largePayload_streamsViaTempFileWithoutBufferingInMemory() throws IOException {
long payloadSize = 16L * 1024 * 1024;
Path tempDir = Path.of(System.getProperty("java.io.tmpdir"));
long uploadTempsBefore = countS3UploadTemps(tempDir);
FileStore.Stored stored;
try (InputStream large = new RepeatingInputStream((byte) 0x42, payloadSize)) {
stored = store.store(large, "big.bin");
}
assertThat(stored.size()).isEqualTo(payloadSize);
assertThat(store.size(stored.fileId())).isEqualTo(payloadSize);
assertThat(countS3UploadTemps(tempDir)).isEqualTo(uploadTempsBefore);
store.delete(stored.fileId());
}
@Test
void store_uploadFailure_stillDeletesTempFile() {
Path tempDir = Path.of(System.getProperty("java.io.tmpdir"));
long uploadTempsBefore = countS3UploadTemps(tempDir);
// Non-existent bucket causes putObject to fail after the temp file is written, exercising
// the failure-path cleanup in the finally block.
S3FileStore brokenStore =
new S3FileStore(s3Client, "bucket-that-does-not-exist", "transient/", false);
assertThatThrownBy(
() ->
brokenStore.store(
new ByteArrayInputStream(
"payload".getBytes(StandardCharsets.UTF_8)),
"x.bin"))
.isInstanceOf(IOException.class);
assertThat(countS3UploadTemps(tempDir)).isEqualTo(uploadTempsBefore);
}
private static long countS3UploadTemps(Path tempDir) {
try (Stream<Path> entries = Files.list(tempDir)) {
return entries.filter(p -> p.getFileName().toString().startsWith("s3-upload-")).count();
} catch (IOException e) {
return 0L;
}
}
/** Generates {@code length} bytes of a single value without buffering them in memory. */
private static final class RepeatingInputStream extends InputStream {
private final byte value;
private long remaining;
RepeatingInputStream(byte value, long length) {
this.value = value;
this.remaining = length;
}
@Override
public int read() {
if (remaining <= 0) {
return -1;
}
remaining--;
return value & 0xFF;
}
@Override
public int read(byte[] b, int off, int len) {
if (remaining <= 0) {
return -1;
}
int toWrite = (int) Math.min(len, remaining);
for (int i = 0; i < toWrite; i++) {
b[off + i] = value;
}
remaining -= toWrite;
return toWrite;
}
}
}
@@ -0,0 +1,779 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.springframework.mock.web.MockMultipartFile;
import stirling.software.common.cluster.FileStore;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.provider.S3StorageProvider;
import stirling.software.proprietary.storage.provider.StoredObject;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.model.NoSuchBucketException;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
/**
* Comprehensive live-vendor test against a real S3-compatible endpoint specified via {@code
* S3_SMOKE_*} env vars. Skipped automatically when {@code S3_SMOKE_ENDPOINT} is not set, so CI is
* not affected. Covers:
*
* <ul>
* <li>{@code S3StorageProvider} CRUD: store / load / delete / presigned URL
* <li>{@code S3FileStore} CRUD (cluster artifact path)
* <li>Folder semantics simulated via key prefixes (matches production usage)
* <li>Negative paths: wrong secret, missing bucket, missing key, traversal IDs
* <li>Edge cases: zero-byte, unicode filename, multi-megabyte streaming
* <li>Configuration guards: SSRF endpoint rejection, bucket validation
* </ul>
*
* Every uploaded key is tracked and removed in {@link #cleanUp} so re-running against the same
* bucket leaves no residue.
*/
@EnabledIfEnvironmentVariable(named = "S3_SMOKE_ENDPOINT", matches = ".+")
class S3VendorComprehensiveTest {
private static final String PREFIX = "stirling-comprehensive/" + UUID.randomUUID() + "/";
private static ApplicationProperties.Storage.S3 cfg;
private static S3Clients.Bundle bundle;
private static S3StorageProvider provider;
private static String bucket;
private static String vendorLabel;
private static User owner;
private static final List<String> keysToCleanup =
Collections.synchronizedList(new ArrayList<>());
@BeforeAll
static void setUp() {
cfg = configFromEnv();
bucket = cfg.getBucket();
vendorLabel = System.getenv().getOrDefault("S3_SMOKE_LABEL", "external");
bundle = S3Clients.build(cfg, "comprehensive[" + vendorLabel + "]");
provider = new S3StorageProvider(bundle.client(), bundle.presigner(), bucket);
owner = new User();
owner.setId(7L);
owner.setUsername("comprehensive-tester");
}
@AfterAll
static void cleanUp() {
if (bundle != null) {
for (String key : keysToCleanup) {
try {
bundle.client().deleteObject(d -> d.bucket(bucket).key(key));
} catch (Exception e) {
// Best-effort cleanup; ignore.
}
}
try {
provider.close();
} catch (Exception ignored) {
}
bundle.close();
}
}
private static String track(String key) {
keysToCleanup.add(key);
return key;
}
private static ApplicationProperties.Storage.S3 configFromEnv() {
ApplicationProperties.Storage.S3 c = new ApplicationProperties.Storage.S3();
c.setEndpoint(System.getenv("S3_SMOKE_ENDPOINT"));
c.setBucket(requireEnv("S3_SMOKE_BUCKET"));
c.setRegion(System.getenv().getOrDefault("S3_SMOKE_REGION", "us-east-1"));
c.setAccessKey(requireEnv("S3_SMOKE_KEY"));
c.setSecretKey(requireEnv("S3_SMOKE_SECRET"));
c.setPathStyleAccess(
Boolean.parseBoolean(System.getenv().getOrDefault("S3_SMOKE_PATHSTYLE", "false")));
c.setAllowPrivateEndpoints(false);
return c;
}
private static String requireEnv(String name) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
throw new IllegalStateException(name + " env var must be set");
}
return value;
}
// ==========================================================================================
// FILE CRUD via S3StorageProvider (user-uploaded files)
// ==========================================================================================
@Test
void provider_store_thenLoad_matchesBytes() throws IOException {
byte[] payload = ("provider-roundtrip-" + vendorLabel).getBytes(StandardCharsets.UTF_8);
MockMultipartFile file = new MockMultipartFile("file", "doc.txt", "text/plain", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
assertThat(obj.getStorageKey()).isNotBlank();
assertThat(obj.getSizeBytes()).isEqualTo(payload.length);
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(payload);
}
@Test
void provider_delete_removesObject() throws IOException {
byte[] payload = "delete-me".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file = new MockMultipartFile("file", "x.txt", "text/plain", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
provider.delete(obj.getStorageKey());
assertThatThrownBy(() -> provider.load(obj.getStorageKey()))
.isInstanceOf(IOException.class);
}
@Test
void provider_load_missingKey_throws() {
assertThatThrownBy(() -> provider.load(PREFIX + "does-not-exist"))
.isInstanceOf(IOException.class);
}
@Test
void provider_presignedDownload_returnsBytesOverHttp() throws Exception {
byte[] payload = "presign me".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file = new MockMultipartFile("file", "p.txt", "text/plain", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
java.util.Optional<java.net.URI> url =
provider.signedDownloadUrl(obj.getStorageKey(), Duration.ofMinutes(5));
assertThat(url).isPresent();
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(url.get()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.body()).isEqualTo(payload);
}
@Test
void provider_store_zeroBytes_isAccepted() throws IOException {
MockMultipartFile empty =
new MockMultipartFile("file", "empty.txt", "text/plain", new byte[0]);
StoredObject obj = provider.store(owner, empty);
track(obj.getStorageKey());
assertThat(obj.getSizeBytes()).isZero();
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(new byte[0]);
}
@Test
void provider_store_unicodeFilename_yieldsOpaqueAsciiKey_andPreservesNameForDisplay()
throws IOException {
// Regression: pre-fix, the storage key embedded the filename verbatim, which Supabase
// rejected with 400 Invalid key. Post-fix, the key is {ownerId}/{uuid} (ASCII-only)
// and the original unicode name lives on StoredObject.originalFilename.
String unicodeName = "résumé-日本語-é.pdf";
byte[] payload = "u".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", unicodeName, "application/pdf", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
assertThat(obj.getStorageKey()).matches("[0-9]+/[0-9a-fA-F-]+");
assertThat(obj.getStorageKey())
.isEqualTo(
new String(
obj.getStorageKey().getBytes(StandardCharsets.US_ASCII),
StandardCharsets.US_ASCII));
assertThat(obj.getOriginalFilename()).isEqualTo(unicodeName);
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(payload);
}
// ==========================================================================================
// Concurrency, overwrite, TTL expiry (added after initial run surfaced the unicode bug)
// ==========================================================================================
@Test
void provider_concurrent10Uploads_allSucceedWithDistinctKeys() throws Exception {
int n = 10;
java.util.concurrent.ExecutorService pool =
java.util.concurrent.Executors.newFixedThreadPool(n);
try {
List<java.util.concurrent.Future<StoredObject>> futures = new ArrayList<>();
for (int i = 0; i < n; i++) {
final int idx = i;
futures.add(
pool.submit(
() -> {
byte[] payload =
("concurrent-" + idx).getBytes(StandardCharsets.UTF_8);
MockMultipartFile f =
new MockMultipartFile(
"file",
"c-" + idx + ".txt",
"text/plain",
payload);
StoredObject obj = provider.store(owner, f);
track(obj.getStorageKey());
return obj;
}));
}
java.util.Set<String> keys = new java.util.HashSet<>();
for (java.util.concurrent.Future<StoredObject> fut : futures) {
StoredObject obj = fut.get(30, java.util.concurrent.TimeUnit.SECONDS);
assertThat(keys.add(obj.getStorageKey()))
.as("distinct key for each parallel upload")
.isTrue();
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isNotEmpty();
}
} finally {
pool.shutdownNow();
}
}
@Test
void sameKey_overwrite_returnsLatestPayload() {
String key = PREFIX + "overwrite-" + UUID.randomUUID() + ".txt";
track(key);
byte[] first = "FIRST".getBytes(StandardCharsets.UTF_8);
byte[] second = "SECOND".getBytes(StandardCharsets.UTF_8);
bundle.client().putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(first));
bundle.client().putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(second));
assertThat(getRaw(key)).isEqualTo(second);
}
@Test
void presignedDownload_afterTtlExpiry_returns403() throws Exception {
String key = PREFIX + "presign-expiry-" + UUID.randomUUID() + ".txt";
byte[] payload = "presign expiry".getBytes(StandardCharsets.UTF_8);
track(putRaw(key, "presign expiry"));
// 2-second TTL, then wait long enough that any vendor clock skew tolerance is also past.
PresignedGetObjectRequest presigned =
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofSeconds(2))
.getObjectRequest(g -> g.bucket(bucket).key(key))
.build());
// Confirm it works while valid - rules out unrelated failures.
HttpResponse<byte[]> ok =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(ok.statusCode()).isEqualTo(200);
assertThat(ok.body()).isEqualTo(payload);
Thread.sleep(5_000);
HttpResponse<byte[]> expired =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(expired.statusCode())
.as("presigned URL must be rejected after TTL expires")
.isIn(400, 403);
}
@Test
void provider_store_4MBPayload_streams() throws IOException {
byte[] payload = new byte[4 * 1024 * 1024];
java.util.Arrays.fill(payload, (byte) 0x42);
MockMultipartFile file =
new MockMultipartFile("file", "big.bin", "application/octet-stream", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
assertThat(obj.getSizeBytes()).isEqualTo(payload.length);
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(payload);
}
// ==========================================================================================
// FILE CRUD via S3FileStore (cluster artifact path)
// ==========================================================================================
@Test
void fileStore_storeAndRetrieve_roundTrip() throws IOException {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
byte[] payload = "filestore round trip".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = store.store(new ByteArrayInputStream(payload), "rt.txt");
track(store.resolveKey(stored.fileId()));
assertThat(store.size(stored.fileId())).isEqualTo(payload.length);
assertThat(store.retrieveBytes(stored.fileId())).isEqualTo(payload);
assertThat(store.exists(stored.fileId())).isTrue();
}
@Test
void fileStore_delete_returnsTrue_andExistsFalseAfter() throws IOException {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
FileStore.Stored stored = store.store(new ByteArrayInputStream("x".getBytes()), "del.txt");
assertThat(store.delete(stored.fileId())).isTrue();
assertThat(store.exists(stored.fileId())).isFalse();
}
@Test
void fileStore_retrieveBytes_missingKey_throws() {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
assertThatThrownBy(() -> store.retrieveBytes("does-not-exist"))
.isInstanceOf(IOException.class);
}
@Test
void fileStore_rejectsTraversalId() {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
assertThat(store.exists("..")).isFalse();
assertThat(store.delete("../etc/passwd")).isFalse();
assertThat(store.exists("a/b")).isFalse();
assertThat(store.exists("a\\b")).isFalse();
}
// ==========================================================================================
// Folder semantics simulated via key prefixes
// ==========================================================================================
@Test
void folderPrefix_isolatesObjects_andDeleteByPrefixDoesNotTouchRoot() throws IOException {
// Two "folders" + a root object - all reuse the test PREFIX so cleanup catches them.
String folderA = PREFIX + "folder-A/";
String folderB = PREFIX + "folder-B/";
String rootObj = PREFIX + "root-" + UUID.randomUUID() + ".txt";
track(putRaw(folderA + "file-1.txt", "in-A"));
track(putRaw(folderA + "file-2.txt", "in-A2"));
track(putRaw(folderB + "file-1.txt", "in-B"));
track(putRaw(rootObj, "at-root"));
// "Delete folder A": delete every key under folderA prefix
deleteAllUnderPrefix(folderA);
// Verify A is empty, B and root untouched
assertThat(headOrNull(folderA + "file-1.txt")).isNull();
assertThat(headOrNull(folderB + "file-1.txt")).isNotNull();
assertThat(headOrNull(rootObj)).isNotNull();
}
@Test
void moveBetweenFolders_viaCopyAndDelete_preservesContent() throws Exception {
String oldKey = PREFIX + "move-old/" + UUID.randomUUID() + ".txt";
String newKey = PREFIX + "move-new/" + UUID.randomUUID() + ".txt";
byte[] payload = "moveable".getBytes(StandardCharsets.UTF_8);
track(oldKey);
track(newKey);
bundle.client()
.putObject(p -> p.bucket(bucket).key(oldKey), RequestBody.fromBytes(payload));
// Simulate move: server-side copy + delete original.
bundle.client()
.copyObject(
c ->
c.sourceBucket(bucket)
.sourceKey(oldKey)
.destinationBucket(bucket)
.destinationKey(newKey));
bundle.client().deleteObject(d -> d.bucket(bucket).key(oldKey));
assertThat(headOrNull(oldKey)).isNull();
assertThat(getRaw(newKey)).isEqualTo(payload);
}
// ==========================================================================================
// Negative: wrong settings / wrong creds
// ==========================================================================================
@Test
void wrongSecret_throwsOnFirstOperation() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setSecretKey("definitely-not-the-real-secret-" + UUID.randomUUID());
try (S3Clients.Bundle badBundle = S3Clients.build(bad, "wrong-secret")) {
assertThatThrownBy(() -> badBundle.client().headBucket(h -> h.bucket(bucket)))
.isInstanceOf(S3Exception.class)
.satisfies(e -> assertThat(((S3Exception) e).statusCode()).isIn(401, 403, 400));
}
}
@Test
void nonExistentBucket_throwsOnHeadOrPut() {
String fakeBucket = "stirling-no-such-bucket-" + UUID.randomUUID();
assertThatThrownBy(() -> bundle.client().headBucket(h -> h.bucket(fakeBucket)))
.isInstanceOfAny(NoSuchBucketException.class, S3Exception.class);
}
@Test
void blankBucket_atBuildTime_throwsIllegalState() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setBucket("");
assertThatThrownBy(() -> S3Clients.build(bad, "blank-bucket"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("bucket");
}
@Test
void invalidEndpointUri_atBuildTime_throwsIllegalState() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setEndpoint("not a valid uri ::::");
assertThatThrownBy(() -> S3Clients.build(bad, "bad-uri"))
.isInstanceOf(IllegalStateException.class);
}
@Test
void privateEndpoint_withoutOptIn_atBuildTime_throwsIllegalState() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setEndpoint("http://127.0.0.1:9000");
bad.setAllowPrivateEndpoints(false);
assertThatThrownBy(() -> S3Clients.build(bad, "loopback"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("private");
}
@Test
void getMissingKey_returnsNoSuchKey() {
String missing = PREFIX + "missing-" + UUID.randomUUID();
assertThatThrownBy(() -> bundle.client().getObject(g -> g.bucket(bucket).key(missing)))
.isInstanceOfAny(NoSuchKeyException.class, S3Exception.class);
}
// ==========================================================================================
// Bundle lifecycle
// ==========================================================================================
@Test
void bundleClose_isIdempotent() {
ApplicationProperties.Storage.S3 c = configFromEnv();
S3Clients.Bundle b = S3Clients.build(c, "lifecycle");
b.close();
b.close(); // should not throw
}
// ==========================================================================================
// Internal helpers (using the bundle directly for prefix/folder simulation)
// ==========================================================================================
private String putRaw(String key, String body) {
bundle.client()
.putObject(
p -> p.bucket(bucket).key(key),
RequestBody.fromBytes(body.getBytes(StandardCharsets.UTF_8)));
return key;
}
private byte[] getRaw(String key) {
return bundle.client().getObjectAsBytes(g -> g.bucket(bucket).key(key)).asByteArray();
}
private Object headOrNull(String key) {
try {
return bundle.client().headObject(h -> h.bucket(bucket).key(key));
} catch (Exception e) {
return null;
}
}
private byte[] tryGetBytes(String key) {
try {
return bundle.client().getObjectAsBytes(g -> g.bucket(bucket).key(key)).asByteArray();
} catch (Exception e) {
return null;
}
}
private void deleteAllUnderPrefix(String prefix) {
var listing = bundle.client().listObjectsV2(l -> l.bucket(bucket).prefix(prefix));
for (var obj : listing.contents()) {
bundle.client().deleteObject(d -> d.bucket(bucket).key(obj.key()));
}
}
// ==========================================================================================
// Key edge cases: leading/trailing/double slash, length, URL-special chars
// ==========================================================================================
@Test
void key_trailingSlash_storesAsZeroByteFolderMarker() {
String key = PREFIX + "folder-marker-" + UUID.randomUUID() + "/";
track(key);
// S3 spec: trailing slash is legal and creates a 0-byte "folder marker" object.
// Some vendors normalize it away; capture either behavior.
bundle.client()
.putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(new byte[0]));
Object head = headOrNull(key);
// Either: vendor accepts the marker (head is non-null) or normalizes to bare key.
assertThat(head != null || headOrNull(key.substring(0, key.length() - 1)) != null)
.as("vendor should either accept trailing-slash marker or normalize to bare key")
.isTrue();
}
@Test
void key_doubleSlash_normalizedOrStoredVerbatim() {
String key = PREFIX + "double//slash-" + UUID.randomUUID() + ".txt";
track(key);
bundle.client()
.putObject(
p -> p.bucket(bucket).key(key),
RequestBody.fromBytes("ds".getBytes(StandardCharsets.UTF_8)));
// Either GET-with-the-exact-key works, or vendor normalized -> single-slash form works.
String alt = key.replace("//", "/");
track(alt);
byte[] viaExact = tryGetBytes(key);
byte[] viaNormalized = tryGetBytes(alt);
assertThat(viaExact != null || viaNormalized != null)
.as("either exact double-slash key or normalized single-slash form must return")
.isTrue();
}
@Test
void key_200Chars_isStoredAndRetrievable() {
// Stirling production keys are ~45 chars ({ownerId}/{uuid}). 200 chars exceeds that by
// ~5x but stays inside every vendor's documented limit. The S3 spec max is 1024 bytes
// but some vendors (Supabase) impose stricter caps (~250-byte total path including
// bucket prefix - 1000 chars fails with KeyTooLongError).
StringBuilder sb = new StringBuilder(PREFIX + "long/");
while (sb.length() < 200) {
sb.append("abcdefghij");
}
String longKey = sb.substring(0, 200);
track(longKey);
byte[] payload = "long-key".getBytes(StandardCharsets.UTF_8);
bundle.client()
.putObject(p -> p.bucket(bucket).key(longKey), RequestBody.fromBytes(payload));
assertThat(getRaw(longKey)).isEqualTo(payload);
}
@Test
void key_safeSpecialChars_areSignedAndRetrievableViaSdk() {
// Restrict to chars every S3-compatible vendor accepts: dot, dash, underscore.
// Stirling's production key format ({ownerId}/{uuid}) is even narrower; this test
// confirms the SDK SigV4 signer copes with slightly more exotic ASCII-safe keys.
// Note: Supabase rejects keys containing space / + / ? / & / # ("400 Invalid key"),
// see documentsVendorKeyRestrictions_tolerantTest for that documentation.
String key =
PREFIX
+ "safe-special/"
+ UUID.randomUUID()
+ "_segment.with-dots.and_underscores.txt";
track(key);
bundle.client()
.putObject(
p -> p.bucket(bucket).key(key),
RequestBody.fromBytes("safe".getBytes(StandardCharsets.UTF_8)));
assertThat(getRaw(key)).isEqualTo("safe".getBytes(StandardCharsets.UTF_8));
}
@Test
void documentsVendorKeyRestrictions_tolerantTest() {
// Documents - rather than enforces - which key characters cause vendor rejection.
// Stirling production code is safe because S3StorageProvider always emits an
// ASCII-safe UUID-only key. If you ever change that, this test becomes a canary.
// AWS S3 and MinIO accept all of these; Supabase rejects all of them with 400.
String[] suspiciousKeys = {
PREFIX + "with space.txt",
PREFIX + "with+plus.txt",
PREFIX + "with#hash.txt",
PREFIX + "with?question.txt",
PREFIX + "with&amp.txt",
};
int accepted = 0;
int rejected = 0;
for (String k : suspiciousKeys) {
track(k);
try {
bundle.client()
.putObject(
p -> p.bucket(bucket).key(k),
RequestBody.fromBytes("x".getBytes(StandardCharsets.UTF_8)));
accepted++;
} catch (S3Exception e) {
assertThat(e.statusCode())
.as("vendor rejection must be a clean 4xx, not a signature mismatch")
.isBetween(400, 499);
rejected++;
}
}
assertThat(accepted + rejected).isEqualTo(suspiciousKeys.length);
}
// ==========================================================================================
// Presigned-URL: TTL bounds + Content-Disposition behavior (Stirling uses this for shares)
// ==========================================================================================
@Test
void presignedGet_ttlExceeding7Days_isRejectedAtSigningTime() {
String key = PREFIX + "ttl-overflow-" + UUID.randomUUID() + ".txt";
track(putRaw(key, "x"));
// SigV4 caps presigned URL TTL at 7 days. SDK should refuse to sign anything larger.
assertThatThrownBy(
() ->
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofDays(8))
.getObjectRequest(
g -> g.bucket(bucket).key(key))
.build()))
.isInstanceOfAny(IllegalArgumentException.class, RuntimeException.class);
}
@Test
void provider_signedDownloadUrl_attachmentDisposition_endsWithAttachmentHeader()
throws Exception {
byte[] payload = "attach me".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", "report.pdf", "application/pdf", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
java.util.Optional<java.net.URI> url =
provider.signedDownloadUrl(
obj.getStorageKey(), Duration.ofMinutes(2), false, "report.pdf");
assertThat(url).isPresent();
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(url.get()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
// Supabase + AWS both honor response-content-disposition query param.
assertThat(resp.headers().firstValue("content-disposition").orElse(""))
.as("vendor must honor response-content-disposition override in presigned URL")
.startsWith("attachment");
}
@Test
void provider_signedDownloadUrl_inlineDisposition_endsWithInlineHeader() throws Exception {
byte[] payload = "inline".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", "preview.pdf", "application/pdf", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
java.util.Optional<java.net.URI> url =
provider.signedDownloadUrl(
obj.getStorageKey(), Duration.ofMinutes(2), true, "preview.pdf");
assertThat(url).isPresent();
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(url.get()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.headers().firstValue("content-disposition").orElse(""))
.as("inline=true must set 'inline' disposition")
.startsWith("inline");
}
// ==========================================================================================
// List pagination + HEAD missing semantics
// ==========================================================================================
@Test
void listObjectsV2_paginationWithMaxKeys_returnsContinuationToken() {
// Stage 3 objects under a unique sub-prefix.
String prefix = PREFIX + "page-" + UUID.randomUUID() + "/";
for (int i = 0; i < 3; i++) {
track(putRaw(prefix + "obj-" + i, "p" + i));
}
var first = bundle.client().listObjectsV2(l -> l.bucket(bucket).prefix(prefix).maxKeys(1));
assertThat(first.contents()).hasSize(1);
assertThat(first.isTruncated()).isTrue();
assertThat(first.nextContinuationToken()).isNotBlank();
var second =
bundle.client()
.listObjectsV2(
l ->
l.bucket(bucket)
.prefix(prefix)
.maxKeys(2)
.continuationToken(first.nextContinuationToken()));
assertThat(second.contents()).hasSize(2);
assertThat(second.isTruncated()).isFalse();
}
@Test
void headObject_missingKey_throwsNoSuchKeyOr404() {
String missing = PREFIX + "head-missing-" + UUID.randomUUID();
assertThatThrownBy(() -> bundle.client().headObject(h -> h.bucket(bucket).key(missing)))
.isInstanceOf(S3Exception.class)
.satisfies(e -> assertThat(((S3Exception) e).statusCode()).isEqualTo(404));
}
/**
* Presigned-URL test scaffolding for parity with the smoke test (covers the SDK presign path).
*/
@Test
void presignGetObject_independentOfProvider_returnsBytes() throws Exception {
String key = PREFIX + "presign-direct-" + UUID.randomUUID() + ".txt";
byte[] payload = "direct presign".getBytes(StandardCharsets.UTF_8);
track(putRaw(key, "direct presign"));
PresignedGetObjectRequest presigned =
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(2))
.getObjectRequest(g -> g.bucket(bucket).key(key))
.build());
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.body()).isEqualTo(payload);
}
}
@@ -0,0 +1,161 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import java.io.ByteArrayInputStream;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.localstack.LocalStackContainer;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import stirling.software.common.cluster.FileStore;
import stirling.software.common.model.ApplicationProperties;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
/**
* End-to-end smoke against the full {@link S3Clients#build} path. Defaults to a LocalStack
* container so it runs in CI; if {@code S3_SMOKE_ENDPOINT} is set, swaps in a real vendor (AWS /
* Supabase / R2 / MinIO over network) to validate live signing + DNS.
*/
@Testcontainers(disabledWithoutDocker = true)
class S3VendorSmokeTest {
private static LocalStackContainer localstack;
private static S3Clients.Bundle bundle;
private static String bucket;
private static String vendorLabel;
@BeforeAll
static void setUp() {
ApplicationProperties.Storage.S3 cfg = new ApplicationProperties.Storage.S3();
String envEndpoint = System.getenv("S3_SMOKE_ENDPOINT");
if (envEndpoint != null && !envEndpoint.isBlank()) {
vendorLabel = System.getenv().getOrDefault("S3_SMOKE_LABEL", "external");
cfg.setEndpoint(envEndpoint);
cfg.setBucket(requireEnv("S3_SMOKE_BUCKET"));
cfg.setRegion(System.getenv().getOrDefault("S3_SMOKE_REGION", "us-east-1"));
cfg.setAccessKey(requireEnv("S3_SMOKE_KEY"));
cfg.setSecretKey(requireEnv("S3_SMOKE_SECRET"));
cfg.setPathStyleAccess(
Boolean.parseBoolean(
System.getenv().getOrDefault("S3_SMOKE_PATHSTYLE", "false")));
cfg.setAllowPrivateEndpoints(
Boolean.parseBoolean(
System.getenv().getOrDefault("S3_SMOKE_ALLOWPRIVATE", "false")));
} else {
vendorLabel = "localstack";
localstack =
new LocalStackContainer(DockerImageName.parse("localstack/localstack:3.8"))
.withServices(LocalStackContainer.Service.S3);
localstack.start();
cfg.setEndpoint(
localstack.getEndpointOverride(LocalStackContainer.Service.S3).toString());
cfg.setBucket("stirling-smoke");
cfg.setRegion(localstack.getRegion());
cfg.setAccessKey(localstack.getAccessKey());
cfg.setSecretKey(localstack.getSecretKey());
// Exercise virtual-hosted addressing where possible. LocalStack supports both;
// path-style remains covered by the MinIO suite.
cfg.setPathStyleAccess(false);
// Required: localhost is a loopback address and would otherwise be rejected.
cfg.setAllowPrivateEndpoints(true);
}
bundle = S3Clients.build(cfg, "vendor-smoke[" + vendorLabel + "]");
bucket = cfg.getBucket();
ensureBucketExists(bucket);
}
@AfterAll
static void tearDown() {
if (bundle != null) {
bundle.close();
}
if (localstack != null) {
localstack.stop();
}
}
@Test
void s3FileStore_roundTripsContentAgainstVendor() throws Exception {
S3FileStore store = new S3FileStore(bundle.client(), bucket, "smoke/", false);
byte[] payload = ("hello from " + vendorLabel).getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored =
store.store(new ByteArrayInputStream(payload), "smoke-payload.txt");
try {
assertThat(stored.size()).isEqualTo(payload.length);
assertThat(store.exists(stored.fileId())).isTrue();
assertThat(store.size(stored.fileId())).isEqualTo(payload.length);
assertThat(store.retrieveBytes(stored.fileId())).isEqualTo(payload);
} finally {
assertThat(store.delete(stored.fileId())).isTrue();
assertThat(store.exists(stored.fileId())).isFalse();
}
}
@Test
void presignedGet_downloadsContentOverHttp() throws Exception {
String key = "smoke/presign-" + System.currentTimeMillis() + ".txt";
byte[] payload = ("presigned by " + vendorLabel).getBytes(StandardCharsets.UTF_8);
bundle.client().putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(payload));
try {
PresignedGetObjectRequest presigned =
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(5))
.getObjectRequest(g -> g.bucket(bucket).key(key))
.build());
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.body()).isEqualTo(payload);
} finally {
bundle.client().deleteObject(d -> d.bucket(bucket).key(key));
}
}
private static String requireEnv(String name) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
throw new IllegalStateException(
name + " env var must be set when S3_SMOKE_ENDPOINT is set");
}
return value;
}
private static void ensureBucketExists(String b) {
try {
bundle.client().headBucket(h -> h.bucket(b));
} catch (S3Exception e) {
if (e.statusCode() == 404 || e.statusCode() == 301 || e.statusCode() == 400) {
try {
bundle.client().createBucket(c -> c.bucket(b));
} catch (S3Exception ignored) {
// Bucket already exists or vendor disallows runtime create (Supabase/R2 often
// require pre-create). Caller is expected to have pre-created it in that case.
}
}
}
}
}
@@ -0,0 +1,59 @@
package stirling.software.proprietary.security.configuration.ee;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
import org.junit.jupiter.api.Test;
import stirling.software.common.model.ApplicationProperties;
class EEAppConfigTest {
@Test
void ssoAutoLogin_disabled_returnsFalse_andDoesNotConsultLicense() {
ApplicationProperties props = new ApplicationProperties();
props.getPremium().getProFeatures().setSsoAutoLogin(false);
LicenseKeyChecker checker = mock(LicenseKeyChecker.class);
EEAppConfig cfg = new EEAppConfig(props, checker);
assertThat(cfg.ssoAutoLogin()).isFalse();
verifyNoInteractions(checker);
}
@Test
void ssoAutoLogin_enabled_withProLicense_returnsTrue() {
ApplicationProperties props = new ApplicationProperties();
props.getPremium().getProFeatures().setSsoAutoLogin(true);
LicenseKeyChecker checker = mock(LicenseKeyChecker.class);
when(checker.getPremiumLicenseEnabledResult())
.thenReturn(KeygenLicenseVerifier.License.SERVER);
EEAppConfig cfg = new EEAppConfig(props, checker);
assertThat(cfg.ssoAutoLogin()).isTrue();
}
@Test
void ssoAutoLogin_enabled_withoutLicense_throwsAtBootTime() {
ApplicationProperties props = new ApplicationProperties();
props.getPremium().getProFeatures().setSsoAutoLogin(true);
LicenseKeyChecker checker = mock(LicenseKeyChecker.class);
// Real LicenseKeyChecker.requireProOrEnterprise throws on NORMAL; mock that behavior here.
org.mockito.Mockito.doThrow(
new IllegalStateException(
"premium.proFeatures.ssoAutoLogin=true requires a Pro or Enterprise license"))
.when(checker)
.requireProOrEnterprise("premium.proFeatures.ssoAutoLogin=true");
EEAppConfig cfg = new EEAppConfig(props, checker);
assertThatThrownBy(cfg::ssoAutoLogin)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"premium.proFeatures.ssoAutoLogin=true requires a Pro or Enterprise license");
}
}
@@ -1,5 +1,7 @@
package stirling.software.proprietary.security.configuration.ee;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
@@ -86,4 +88,43 @@ class LicenseKeyCheckerTest {
assertEquals(License.NORMAL, checker.getPremiumLicenseEnabledResult());
verifyNoInteractions(verifier);
}
// ----- requireProOrEnterprise: shared boot-time gate for premium features -----
@Test
void requireProOrEnterprise_normalLicense_throwsWithFeatureName() {
LicenseKeyChecker checker = checkerWithLicense(License.NORMAL);
assertThatThrownBy(() -> checker.requireProOrEnterprise("storage.provider=s3"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("storage.provider=s3 requires a Pro or Enterprise license");
}
@Test
void requireProOrEnterprise_serverLicense_passes() {
LicenseKeyChecker checker = checkerWithLicense(License.SERVER);
assertThatCode(() -> checker.requireProOrEnterprise("any.feature=true"))
.doesNotThrowAnyException();
}
@Test
void requireProOrEnterprise_enterpriseLicense_passes() {
LicenseKeyChecker checker = checkerWithLicense(License.ENTERPRISE);
assertThatCode(() -> checker.requireProOrEnterprise("any.feature=true"))
.doesNotThrowAnyException();
}
private LicenseKeyChecker checkerWithLicense(License level) {
ApplicationProperties props = new ApplicationProperties();
if (level == License.NORMAL) {
props.getPremium().setEnabled(false);
} else {
props.getPremium().setEnabled(true);
props.getPremium().setKey("any");
when(verifier.verifyLicense("any")).thenReturn(level);
}
LicenseKeyChecker checker =
new LicenseKeyChecker(verifier, props, userLicenseSettingsService);
checker.init();
return checker;
}
}
@@ -0,0 +1,221 @@
package stirling.software.proprietary.security.service;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.when;
import java.lang.reflect.Field;
import java.time.Instant;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.slf4j.LoggerFactory;
import org.springframework.security.oauth2.client.oidc.userinfo.OidcUserRequest;
import org.springframework.security.oauth2.client.oidc.userinfo.OidcUserService;
import org.springframework.security.oauth2.client.registration.ClientRegistration;
import org.springframework.security.oauth2.core.AuthorizationGrantType;
import org.springframework.security.oauth2.core.OAuth2AuthenticationException;
import org.springframework.security.oauth2.core.oidc.IdTokenClaimNames;
import org.springframework.security.oauth2.core.oidc.OidcIdToken;
import org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser;
import stirling.software.common.model.ApplicationProperties;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.read.ListAppender;
/**
* Verifies the opt-in OAuth2/OIDC claim-dump diagnostic logging added to {@link
* CustomOAuth2UserService} for troubleshooting provider misconfiguration (e.g. ADFS not emitting an
* {@code email} claim).
*/
@ExtendWith(MockitoExtension.class)
class CustomOAuth2UserServiceDebugLoggingTest {
@Mock private UserService userService;
@Mock private LoginAttemptService loginAttemptService;
@Mock private OidcUserRequest userRequest;
private ListAppender<ILoggingEvent> appender;
private Logger serviceLogger;
@BeforeEach
void attachLogCapture() {
serviceLogger = (Logger) LoggerFactory.getLogger(CustomOAuth2UserService.class);
appender = new ListAppender<>();
appender.start();
serviceLogger.addAppender(appender);
// Make sure INFO-level dumps reach the appender even if the default config is WARN+.
serviceLogger.setLevel(Level.DEBUG);
}
@AfterEach
void detachLogCapture() {
serviceLogger.detachAppender(appender);
appender.stop();
}
@Test
void whenDebugLoggingOff_failureProducesNoClaimDump() throws Exception {
ApplicationProperties.Security.OAUTH2 props = oauthProps("email", false);
CustomOAuth2UserService service =
new CustomOAuth2UserService(props, userService, loginAttemptService);
// Provider gave us claims, but no "email" — same shape as the ADFS bug report.
Map<String, Object> claims = baseClaims();
claims.put("upn", "jdoe@demarest.com.br");
replaceDelegateWithStub(service, claims);
lenient()
.when(userRequest.getIdToken())
.thenReturn(new OidcIdToken("token", Instant.now(), Instant.MAX, claims));
lenient().when(userRequest.getClientRegistration()).thenReturn(stubRegistration());
assertThrows(OAuth2AuthenticationException.class, () -> service.loadUser(userRequest));
assertThat(appender.list)
.as("no debug dump should appear when debugLogging=false")
.noneMatch(e -> e.getFormattedMessage().contains("[OAUTH2 DEBUG]"));
}
@Test
void whenDebugLoggingOn_failureDumpsClaimsAndSuggestsAlternative() throws Exception {
ApplicationProperties.Security.OAUTH2 props = oauthProps("email", true);
CustomOAuth2UserService service =
new CustomOAuth2UserService(props, userService, loginAttemptService);
Map<String, Object> claims = adfsStyleClaims();
// ADFS-style: no `email`, but `preferred_username` IS a valid UsernameAttribute value.
claims.put("preferred_username", "jdoe@demarest.com.br");
// `upn` is NOT in UsernameAttribute, so it must NOT appear in the suggestion hint.
claims.put("upn", "jdoe@demarest.com.br");
replaceDelegateWithStub(service, claims);
lenient()
.when(userRequest.getIdToken())
.thenReturn(new OidcIdToken("token", Instant.now(), Instant.MAX, claims));
lenient().when(userRequest.getClientRegistration()).thenReturn(stubRegistration());
assertThrows(OAuth2AuthenticationException.class, () -> service.loadUser(userRequest));
List<ILoggingEvent> dumps =
appender.list.stream()
.filter(e -> e.getFormattedMessage().contains("[OAUTH2 DEBUG]"))
.toList();
assertThat(dumps).as("expected at least one debug-dump log line").isNotEmpty();
String combined =
String.join("\n", dumps.stream().map(ILoggingEvent::getFormattedMessage).toList());
assertThat(combined)
.contains("Provider registrationId : demarest")
.contains("Configured useAsUsername: email")
.contains("preferred_username")
.contains("upn = jdoe@demarest.com.br")
.contains("<NULL — this is why login fails>");
// The hint must include 'preferred_username' (a valid UsernameAttribute value present
// in the claims) and MUST NOT include 'upn' (not in the UsernameAttribute enum).
String hintLine =
combined.lines()
.filter(l -> l.contains("Hint:"))
.findFirst()
.orElseThrow(() -> new AssertionError("no Hint: line in dump"));
assertThat(hintLine).contains("preferred_username").doesNotContain("upn");
}
@Test
void invalidUseAsUsername_isWrappedAsOAuth2AuthenticationException() {
// Regression: an earlier draft moved UsernameAttribute.valueOf(...) outside the try/catch,
// so a typo'd or null useAsUsername leaked as a raw IllegalArgumentException instead of
// being wrapped, breaking Spring's authentication exception handling. This test pins the
// post-fix behaviour: valueOf() failures stay inside the guarded section.
ApplicationProperties.Security.OAUTH2 props = oauthProps("not_a_real_attribute", true);
CustomOAuth2UserService service =
new CustomOAuth2UserService(props, userService, loginAttemptService);
lenient().when(userRequest.getClientRegistration()).thenReturn(stubRegistration());
// No need to stub the OIDC delegate — control flow shouldn't reach it.
OAuth2AuthenticationException thrown =
assertThrows(
OAuth2AuthenticationException.class, () -> service.loadUser(userRequest));
assertThat(thrown.getCause()).isInstanceOf(IllegalArgumentException.class);
// We deliberately do NOT emit the claim dump in this case (we have no resolved
// usernameAttributeKey to compare against, and the IllegalArgumentException message
// already explains the misconfiguration).
assertThat(appender.list)
.as("no claim dump when useAsUsername itself is invalid")
.noneMatch(e -> e.getFormattedMessage().contains("[OAUTH2 DEBUG]"));
}
// ---------- helpers ----------
private static ApplicationProperties.Security.OAUTH2 oauthProps(
String useAsUsername, boolean debugLogging) {
ApplicationProperties.Security.OAUTH2 p = new ApplicationProperties.Security.OAUTH2();
p.setEnabled(true);
p.setUseAsUsername(useAsUsername);
p.setDebugLogging(debugLogging);
return p;
}
private static Map<String, Object> baseClaims() {
Map<String, Object> claims = new LinkedHashMap<>();
claims.put(IdTokenClaimNames.SUB, "abc-123");
claims.put(IdTokenClaimNames.ISS, "https://sts.example.com/adfs");
claims.put(IdTokenClaimNames.AUD, Collections.singletonList("client-id"));
claims.put(IdTokenClaimNames.IAT, Instant.now());
claims.put(IdTokenClaimNames.EXP, Instant.now().plusSeconds(3600));
claims.put("given_name", "Jane");
claims.put("family_name", "Doe");
return claims;
}
/**
* ADFS-style claim set with {@code given_name}/{@code family_name} removed, so the suggestion
* hint test isolates a single expected UsernameAttribute value.
*/
private static Map<String, Object> adfsStyleClaims() {
Map<String, Object> claims = baseClaims();
claims.remove("given_name");
claims.remove("family_name");
return claims;
}
private static ClientRegistration stubRegistration() {
return ClientRegistration.withRegistrationId("demarest")
.clientId("client-id")
.clientSecret("client-secret")
.authorizationGrantType(AuthorizationGrantType.AUTHORIZATION_CODE)
.redirectUri("https://app.example.com/login/oauth2/code/demarest")
.authorizationUri("https://sts.example.com/adfs/oauth2/authorize")
.tokenUri("https://sts.example.com/adfs/oauth2/token")
.jwkSetUri("https://sts.example.com/adfs/discovery/keys")
.build();
}
/**
* Swap the private {@code delegate} field on {@link CustomOAuth2UserService} for a stub that
* returns a {@link DefaultOidcUser} built from the supplied claims. Lets us drive the test
* without standing up a real OIDC provider.
*/
private void replaceDelegateWithStub(
CustomOAuth2UserService service, Map<String, Object> claims) throws Exception {
OidcIdToken idToken =
new OidcIdToken("raw-token", Instant.now(), Instant.MAX, new HashMap<>(claims));
DefaultOidcUser delegateUser =
new DefaultOidcUser(Collections.emptyList(), idToken, IdTokenClaimNames.SUB);
OidcUserService delegateMock = org.mockito.Mockito.mock(OidcUserService.class);
when(delegateMock.loadUser(any())).thenReturn(delegateUser);
Field f = CustomOAuth2UserService.class.getDeclaredField("delegate");
f.setAccessible(true);
f.set(service, delegateMock);
}
}
@@ -49,6 +49,7 @@ import stirling.software.common.service.FileStorage;
import stirling.software.common.service.FileStorage.StoredFile;
import stirling.software.common.service.InternalApiClient;
import stirling.software.common.service.InternalApiTimeoutException;
import stirling.software.common.service.PyMuPdfConverter;
import stirling.software.common.service.ToolMetadataService;
import stirling.software.common.util.TempFileManager;
import stirling.software.common.util.TempFileRegistry;
@@ -85,6 +86,7 @@ class AiWorkflowServiceTest {
@Mock private ToolMetadataService toolMetadataService;
@Mock private FileIdStrategy fileIdStrategy;
@Mock private AiEngineEndpointResolver endpointResolver;
@Mock private PyMuPdfConverter pyMuPdfConverter;
@TempDir Path tempDir;
@@ -117,7 +119,8 @@ class AiWorkflowServiceTest {
toolMetadataService,
tempFileManager,
fileIdStrategy,
endpointResolver);
endpointResolver,
pyMuPdfConverter);
when(endpointResolver.getEnabledEndpointUrls()).thenReturn(List.of());
}
@@ -0,0 +1,250 @@
package stirling.software.proprietary.storage.config;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.lang.reflect.Field;
import org.junit.jupiter.api.Test;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.configuration.ee.KeygenLicenseVerifier.License;
import stirling.software.proprietary.security.configuration.ee.LicenseKeyChecker;
class ClusterStorageGateTest {
@Test
void clusterDisabled_localStorage_passes() {
ClusterStorageGate gate = newGate(false, true, "local", "local");
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void clusterDisabled_s3Storage_passes() {
ClusterStorageGate gate = newGate(false, true, "s3", "local");
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void clusterEnabled_storageDisabled_butArtifactStoreLocal_fails() {
ClusterStorageGate gate = newGate(true, false, "local", "local");
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("cluster.artifactStore=local");
}
@Test
void clusterEnabled_storageDisabled_artifactStoreS3_passes() {
ClusterStorageGate gate = newGate(true, false, "local", "s3");
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void clusterEnabled_localStorage_fails() {
ClusterStorageGate gate = newGate(true, true, "local", "s3");
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("storage.provider=local")
.hasMessageContaining("storage.provider=s3")
.hasMessageContaining("storage.provider=database");
}
@Test
void clusterEnabled_localStorage_caseInsensitive_fails() {
ClusterStorageGate gate = newGate(true, true, "LOCAL", "s3");
assertThatThrownBy(gate::validate).isInstanceOf(IllegalStateException.class);
}
@Test
void clusterEnabled_nullProvider_treatedAsLocal_fails() {
ClusterStorageGate gate = newGate(true, true, null, "s3");
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("storage.provider=local");
}
@Test
void clusterEnabled_s3Storage_andArtifactStoreS3_passes() {
ClusterStorageGate gate = newGate(true, true, "s3", "s3");
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void clusterEnabled_databaseStorage_andArtifactStoreS3_passes() {
ClusterStorageGate gate = newGate(true, true, "database", "s3");
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void clusterEnabled_s3Storage_butLocalArtifactStore_fails() {
ClusterStorageGate gate = newGate(true, true, "s3", "local");
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("cluster.artifactStore=local");
}
@Test
void clusterEnabled_localArtifactStore_caseInsensitive_fails() {
ClusterStorageGate gate = newGate(true, true, "s3", "LOCAL");
assertThatThrownBy(gate::validate).isInstanceOf(IllegalStateException.class);
}
@Test
void clusterEnabled_nullArtifactStore_treatedAsLocal_fails() {
ClusterStorageGate gate = newGate(true, true, "s3", null);
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("cluster.artifactStore=local");
}
@Test
void clusterEnabled_nullStorageObject_passesProviderCheck_butArtifactStoreStillEvaluated() {
ApplicationProperties props = new ApplicationProperties();
props.setStorage(null);
ClusterStorageGate gate = new ClusterStorageGate(props, mockLicenseChecker(License.SERVER));
setClusterEnabled(gate, true);
setClusterArtifactStore(gate, "s3");
assertThatCode(gate::validate).doesNotThrowAnyException();
}
// ----- License gating for premium storage backends -----
@Test
void storageProviderS3_withoutProLicense_throws() {
ClusterStorageGate gate = newGate(false, true, "s3", "local", License.NORMAL);
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("storage.provider=s3 requires a Pro or Enterprise license");
}
@Test
void storageProviderDatabase_withoutProLicense_throws() {
ClusterStorageGate gate = newGate(false, true, "database", "local", License.NORMAL);
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"storage.provider=database requires a Pro or Enterprise license");
}
@Test
void storageProviderS3_withServerLicense_passes() {
ClusterStorageGate gate = newGate(false, true, "s3", "local", License.SERVER);
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void storageProviderS3_withEnterpriseLicense_passes() {
ClusterStorageGate gate = newGate(false, true, "s3", "local", License.ENTERPRISE);
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void storageProviderDatabase_withServerLicense_passes() {
ClusterStorageGate gate = newGate(false, true, "database", "local", License.SERVER);
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void clusterArtifactStoreS3_withoutProLicense_throws() {
ClusterStorageGate gate = newGate(false, false, "local", "s3", License.NORMAL);
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"cluster.artifactStore=s3 requires a Pro or Enterprise license");
}
@Test
void clusterArtifactStoreS3_withServerLicense_passes() {
ClusterStorageGate gate = newGate(false, false, "local", "s3", License.SERVER);
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void localOnly_normalLicense_passes_licenseNotChecked() {
ClusterStorageGate gate = newGate(false, true, "local", "local", License.NORMAL);
assertThatCode(gate::validate).doesNotThrowAnyException();
}
@Test
void storageDisabled_butArtifactStoreS3_withoutLicense_stillThrows() {
ClusterStorageGate gate = newGate(false, false, "local", "s3", License.NORMAL);
assertThatThrownBy(gate::validate)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("cluster.artifactStore=s3");
}
private static ClusterStorageGate newGate(
boolean clusterEnabled,
boolean storageEnabled,
String provider,
String clusterArtifactStore) {
// Default to a SERVER license so existing tests (which assert clustering / artifact-store
// rules independently of license) continue to pass. License-specific tests below build
// gates with explicit license tiers.
return newGate(
clusterEnabled, storageEnabled, provider, clusterArtifactStore, License.SERVER);
}
private static ClusterStorageGate newGate(
boolean clusterEnabled,
boolean storageEnabled,
String provider,
String clusterArtifactStore,
License license) {
ApplicationProperties props = new ApplicationProperties();
ApplicationProperties.Storage storage = new ApplicationProperties.Storage();
storage.setEnabled(storageEnabled);
storage.setProvider(provider);
props.setStorage(storage);
LicenseKeyChecker checker = mockLicenseChecker(license);
ClusterStorageGate gate = new ClusterStorageGate(props, checker);
setClusterEnabled(gate, clusterEnabled);
setClusterArtifactStore(gate, clusterArtifactStore);
return gate;
}
private static LicenseKeyChecker mockLicenseChecker(License license) {
LicenseKeyChecker checker = mock(LicenseKeyChecker.class);
when(checker.getPremiumLicenseEnabledResult()).thenReturn(license);
if (license == License.SERVER || license == License.ENTERPRISE) {
doNothing().when(checker).requireProOrEnterprise(anyString());
} else {
// Mirror real LicenseKeyChecker.requireProOrEnterprise so message assertions match.
org.mockito.Mockito.doAnswer(
inv -> {
throw new IllegalStateException(
inv.getArgument(0)
+ " requires a Pro or Enterprise license");
})
.when(checker)
.requireProOrEnterprise(anyString());
}
return checker;
}
private static void setClusterEnabled(ClusterStorageGate gate, boolean enabled) {
try {
Field f = ClusterStorageGate.class.getDeclaredField("clusterEnabled");
f.setAccessible(true);
f.setBoolean(gate, enabled);
assertThat(f.getBoolean(gate)).isEqualTo(enabled);
} catch (ReflectiveOperationException e) {
throw new AssertionError("Failed to set clusterEnabled via reflection", e);
}
}
private static void setClusterArtifactStore(ClusterStorageGate gate, String value) {
try {
Field f = ClusterStorageGate.class.getDeclaredField("clusterArtifactStore");
f.setAccessible(true);
f.set(gate, value);
} catch (ReflectiveOperationException e) {
throw new AssertionError("Failed to set clusterArtifactStore via reflection", e);
}
}
}
@@ -0,0 +1,116 @@
package stirling.software.proprietary.storage.config;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import org.junit.jupiter.api.Test;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.configuration.ee.KeygenLicenseVerifier.License;
import stirling.software.proprietary.security.configuration.ee.LicenseKeyChecker;
import stirling.software.proprietary.storage.provider.LocalStorageProvider;
import stirling.software.proprietary.storage.provider.StorageProvider;
import stirling.software.proprietary.storage.repository.StoredFileBlobRepository;
/**
* Verifies the Pro/Enterprise license gate on the S3 storage backend without touching real S3
* clients (and without needing Docker). Provider-specific construction is delegated to the existing
* provider tests.
*/
class StorageProviderConfigTest {
@Test
void provider_local_normalLicense_buildsLocalProviderWithoutLicenseCheck() {
StorageProviderConfig cfg = newConfig("local", License.NORMAL);
StorageProvider provider = cfg.storageProvider();
assertThat(provider).isInstanceOf(LocalStorageProvider.class);
}
@Test
void provider_s3_normalLicense_throwsBeforeBuildingClient() {
StorageProviderConfig cfg = newConfig("s3", License.NORMAL);
// License check must throw BEFORE S3Clients.build tries to validate endpoint / bucket.
// Otherwise an empty config would surface as a confusing "bucket must be set" error.
assertThatThrownBy(cfg::storageProvider)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("storage.provider=s3 requires a Pro or Enterprise license");
}
@Test
void provider_database_normalLicense_throws() {
StorageProviderConfig cfg = newConfig("database", License.NORMAL);
assertThatThrownBy(cfg::storageProvider)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"storage.provider=database requires a Pro or Enterprise license");
}
@Test
void provider_database_serverLicense_buildsDatabaseProvider() {
StorageProviderConfig cfg = newConfig("database", License.SERVER);
assertThatCode(cfg::storageProvider).doesNotThrowAnyException();
}
@Test
void provider_s3_serverLicense_passesLicenseCheck_thenFailsOnEmptyConfig() {
StorageProviderConfig cfg = newConfig("s3", License.SERVER);
// Valid license, but no bucket/endpoint configured - so we expect a CONFIG error,
// not a license error. The error message must not mention the license.
assertThatThrownBy(cfg::storageProvider)
.isInstanceOf(IllegalStateException.class)
.hasMessageNotContaining("Pro or Enterprise license");
}
@Test
void provider_s3_enterpriseLicense_passesLicenseCheck_thenFailsOnEmptyConfig() {
StorageProviderConfig cfg = newConfig("s3", License.ENTERPRISE);
assertThatThrownBy(cfg::storageProvider)
.isInstanceOf(IllegalStateException.class)
.hasMessageNotContaining("Pro or Enterprise license");
}
@Test
void provider_unknown_normalLicense_throwsUnsupportedProvider_notLicense() {
StorageProviderConfig cfg = newConfig("magic", License.NORMAL);
assertThatThrownBy(cfg::storageProvider)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("Storage provider not supported: magic")
.hasMessageNotContaining("license");
}
private static StorageProviderConfig newConfig(String provider, License license) {
ApplicationProperties props = new ApplicationProperties();
props.getStorage().setProvider(provider);
props.getStorage()
.setEnabled(false); // local-fallback path skips dir creation when disabled
StoredFileBlobRepository repo = mock(StoredFileBlobRepository.class);
LicenseKeyChecker checker = mock(LicenseKeyChecker.class);
when(checker.getPremiumLicenseEnabledResult()).thenReturn(license);
if (license == License.SERVER || license == License.ENTERPRISE) {
doNothing().when(checker).requireProOrEnterprise(anyString());
} else {
// Mirror real LicenseKeyChecker.requireProOrEnterprise so message assertions match.
doAnswer(
inv -> {
throw new IllegalStateException(
inv.getArgument(0)
+ " requires a Pro or Enterprise license");
})
.when(checker)
.requireProOrEnterprise(anyString());
}
return new StorageProviderConfig(props, repo, checker);
}
}
@@ -0,0 +1,130 @@
package stirling.software.proprietary.storage.controller;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyBoolean;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.header;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.redirectedUrl;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
import java.net.URI;
import java.time.Duration;
import java.util.Optional;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.MvcResult;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.StoredFile;
import stirling.software.proprietary.storage.provider.StorageProvider;
import stirling.software.proprietary.storage.service.FileStorageService;
@ExtendWith(MockitoExtension.class)
class FileStorageControllerTest {
private static final String SIGNED_URL =
"https://test-bucket.s3.example.com/signed-blob?X-Amz-Signature=abc";
@Mock private FileStorageService fileStorageService;
@Mock private StorageProvider storageProvider;
private MockMvc mockMvc;
@BeforeEach
void setUp() {
FileStorageController controller =
new FileStorageController(fileStorageService, storageProvider);
mockMvc = MockMvcBuilders.standaloneSetup(controller).build();
}
@Test
void downloadFile_whenProviderReturnsSignedUrl_returns302RedirectWithoutSessionCredentials()
throws Exception {
StoredFile file = newStoredFile();
when(fileStorageService.requireAuthenticatedUser()).thenReturn(file.getOwner());
when(fileStorageService.getAccessibleFile(file.getOwner(), 77L)).thenReturn(file);
when(storageProvider.signedDownloadUrl(
eq("11/abc-doc.pdf"), any(Duration.class), anyBoolean(), anyString()))
.thenReturn(Optional.of(URI.create(SIGNED_URL)));
MvcResult result =
mockMvc.perform(get("/api/v1/storage/files/{fileId}/download", 77L))
.andExpect(status().is(HttpStatus.FOUND.value()))
.andExpect(header().string(HttpHeaders.LOCATION, SIGNED_URL))
.andExpect(redirectedUrl(SIGNED_URL))
.andReturn();
// Regression fence: signed URLs delegate auth to the URL itself, so the redirect
// response must NOT carry any session credentials forward.
assertThat(result.getResponse().getHeader(HttpHeaders.AUTHORIZATION)).isNull();
assertThat(result.getResponse().getHeader(HttpHeaders.COOKIE)).isNull();
assertThat(result.getResponse().getHeader(HttpHeaders.SET_COOKIE)).isNull();
}
@Test
void downloadFile_inlineFalse_forwardsAttachmentDispositionToSignedUrl() throws Exception {
StoredFile file = newStoredFile();
when(fileStorageService.requireAuthenticatedUser()).thenReturn(file.getOwner());
when(fileStorageService.getAccessibleFile(file.getOwner(), 77L)).thenReturn(file);
when(storageProvider.signedDownloadUrl(
eq("11/abc-doc.pdf"), any(Duration.class), eq(false), eq("doc.pdf")))
.thenReturn(Optional.of(URI.create(SIGNED_URL)));
mockMvc.perform(get("/api/v1/storage/files/{fileId}/download", 77L))
.andExpect(status().is(HttpStatus.FOUND.value()))
.andExpect(header().string(HttpHeaders.LOCATION, SIGNED_URL));
verify(storageProvider)
.signedDownloadUrl(
eq("11/abc-doc.pdf"), any(Duration.class), eq(false), eq("doc.pdf"));
}
@Test
void downloadFile_inlineTrue_forwardsInlineDispositionToSignedUrl() throws Exception {
StoredFile file = newStoredFile();
when(fileStorageService.requireAuthenticatedUser()).thenReturn(file.getOwner());
when(fileStorageService.getAccessibleFile(file.getOwner(), 77L)).thenReturn(file);
when(storageProvider.signedDownloadUrl(
eq("11/abc-doc.pdf"), any(Duration.class), eq(true), eq("doc.pdf")))
.thenReturn(Optional.of(URI.create(SIGNED_URL)));
mockMvc.perform(get("/api/v1/storage/files/{fileId}/download", 77L).param("inline", "true"))
.andExpect(status().is(HttpStatus.FOUND.value()))
.andExpect(header().string(HttpHeaders.LOCATION, SIGNED_URL));
verify(storageProvider)
.signedDownloadUrl(
eq("11/abc-doc.pdf"), any(Duration.class), eq(true), eq("doc.pdf"));
}
private static StoredFile newStoredFile() {
User user = new User();
user.setId(11L);
user.setUsername("alice");
StoredFile file = new StoredFile();
file.setId(77L);
file.setOwner(user);
file.setOriginalFilename("doc.pdf");
file.setContentType("application/pdf");
file.setSizeBytes(123L);
file.setStorageKey("11/abc-doc.pdf");
return file;
}
}
@@ -0,0 +1,282 @@
package stirling.software.proprietary.storage.provider;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URI;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.Optional;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.Resource;
import org.springframework.mock.web.MockMultipartFile;
import org.testcontainers.containers.MinIOContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import stirling.software.proprietary.security.model.User;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
@Testcontainers(disabledWithoutDocker = true)
class S3StorageProviderTest {
private static final String BUCKET = "stirling-test-bucket";
private static final String ACCESS_KEY = "minioadmin";
private static final String SECRET_KEY = "minioadmin";
@Container
static MinIOContainer minio =
new MinIOContainer("minio/minio:latest")
.withUserName(ACCESS_KEY)
.withPassword(SECRET_KEY);
private static S3Client s3Client;
private static S3Presigner s3Presigner;
private static S3StorageProvider provider;
@BeforeAll
static void setUp() {
URI endpoint = URI.create(minio.getS3URL());
AwsBasicCredentials creds = AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY);
S3Configuration s3Config = S3Configuration.builder().pathStyleAccessEnabled(true).build();
s3Client =
S3Client.builder()
.endpointOverride(endpoint)
.httpClient(UrlConnectionHttpClient.create())
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(creds))
.serviceConfiguration(s3Config)
.build();
s3Presigner =
S3Presigner.builder()
.endpointOverride(endpoint)
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(creds))
.serviceConfiguration(s3Config)
.build();
s3Client.createBucket(CreateBucketRequest.builder().bucket(BUCKET).build());
provider = new S3StorageProvider(s3Client, s3Presigner, BUCKET);
}
@AfterAll
static void tearDown() {
if (provider != null) {
provider.close();
}
}
@Test
void blankBucket_constructorRejects() {
assertThatThrownBy(() -> new S3StorageProvider(s3Client, s3Presigner, ""))
.isInstanceOf(IllegalArgumentException.class);
assertThatThrownBy(() -> new S3StorageProvider(s3Client, s3Presigner, null))
.isInstanceOf(IllegalArgumentException.class);
}
@Test
void store_thenLoad_roundTripsContent() throws Exception {
User owner = new User();
owner.setId(42L);
byte[] content = "hello s3 round trip".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", "sample.pdf", "application/pdf", content);
StoredObject stored = provider.store(owner, file);
// Key is intentionally opaque ({ownerId}/{uuid}) - the filename is preserved on
// StoredObject.originalFilename for display, never in the S3 key, so vendors that
// restrict key charset (e.g. Supabase: ASCII only) accept any filename.
assertThat(stored.getStorageKey())
.matches(
"42/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}");
assertThat(stored.getStorageKey()).doesNotContain("sample.pdf");
assertThat(stored.getOriginalFilename()).isEqualTo("sample.pdf");
assertThat(stored.getContentType()).isEqualTo("application/pdf");
assertThat(stored.getSizeBytes()).isEqualTo(content.length);
Resource loaded = provider.load(stored.getStorageKey());
try (InputStream in = loaded.getInputStream()) {
assertThat(in.readAllBytes()).isEqualTo(content);
}
}
@Test
void load_unknownKey_throwsIOException() {
assertThatThrownBy(() -> provider.load("does/not/exist.txt"))
.isInstanceOf(IOException.class);
}
@Test
void store_unicodeFilename_yieldsAsciiOnlyKey_andPreservesOriginalName() throws Exception {
// Regression: Supabase Storage rejects S3 keys containing non-ASCII chars (400
// Invalid key). Locking in that the storage key never embeds the filename so any
// unicode display name still uploads successfully.
User owner = new User();
owner.setId(99L);
String unicodeName = "résumé-日本語-é.pdf";
byte[] payload = "u".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", unicodeName, "application/pdf", payload);
StoredObject stored = provider.store(owner, file);
assertThat(stored.getStorageKey())
.matches(
"99/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}");
assertThat(stored.getOriginalFilename()).isEqualTo(unicodeName);
try (InputStream in = provider.load(stored.getStorageKey()).getInputStream()) {
assertThat(in.readAllBytes()).isEqualTo(payload);
}
}
@Test
void delete_removesObject() throws Exception {
User owner = new User();
owner.setId(7L);
MockMultipartFile file =
new MockMultipartFile(
"file", "todelete.bin", "application/octet-stream", new byte[] {1, 2, 3});
StoredObject stored = provider.store(owner, file);
provider.delete(stored.getStorageKey());
assertThatThrownBy(() -> provider.load(stored.getStorageKey()))
.isInstanceOf(IOException.class);
}
@Test
void delete_unknownKey_isNoOp() {
assertThat(catchIOException(() -> provider.delete("never-existed"))).isNull();
}
@Test
void signedDownloadUrl_returnsWorkingPresignedGet() throws Exception {
User owner = new User();
owner.setId(99L);
byte[] content = "presigned payload".getBytes(StandardCharsets.UTF_8);
StoredObject stored =
provider.store(
owner, new MockMultipartFile("file", "presign.txt", "text/plain", content));
Optional<URI> signed =
provider.signedDownloadUrl(stored.getStorageKey(), Duration.ofMinutes(2));
assertThat(signed).isPresent();
URI uri = signed.get();
assertThat(uri.getScheme()).isIn("http", "https");
assertThat(uri.getRawQuery()).contains("X-Amz-Signature");
HttpURLConnection conn = (HttpURLConnection) new URL(uri.toString()).openConnection();
try {
assertThat(conn.getResponseCode()).isEqualTo(200);
try (InputStream in = conn.getInputStream()) {
assertThat(in.readAllBytes()).isEqualTo(content);
}
} finally {
conn.disconnect();
}
}
@Test
void signedDownloadUrl_nullKey_returnsEmpty() throws Exception {
assertThat(provider.signedDownloadUrl(null, Duration.ofMinutes(1))).isEmpty();
assertThat(provider.signedDownloadUrl(" ", Duration.ofMinutes(1))).isEmpty();
}
@Test
void signedDownloadUrl_nullOrZeroTtl_appliesDefault() throws Exception {
User owner = new User();
owner.setId(3L);
StoredObject stored =
provider.store(
owner,
new MockMultipartFile(
"file",
"ttl.txt",
"text/plain",
"x".getBytes(StandardCharsets.UTF_8)));
assertThat(provider.signedDownloadUrl(stored.getStorageKey(), null)).isPresent();
assertThat(provider.signedDownloadUrl(stored.getStorageKey(), Duration.ZERO)).isPresent();
assertThat(provider.signedDownloadUrl(stored.getStorageKey(), Duration.ofSeconds(-5)))
.isPresent();
}
@Test
void signedDownloadUrl_inlineFlagEncodesResponseContentDispositionInQuery() throws Exception {
User owner = new User();
owner.setId(55L);
StoredObject stored =
provider.store(
owner,
new MockMultipartFile(
"file",
"stored-name.pdf",
"application/pdf",
"payload".getBytes(StandardCharsets.UTF_8)));
URI attached =
provider.signedDownloadUrl(
stored.getStorageKey(), Duration.ofMinutes(2), false, "report.pdf")
.orElseThrow();
String attachedQuery =
java.net.URLDecoder.decode(attached.getRawQuery(), StandardCharsets.UTF_8);
assertThat(attachedQuery)
.contains("response-content-disposition=attachment; filename=\"report.pdf\"");
URI inline =
provider.signedDownloadUrl(
stored.getStorageKey(), Duration.ofMinutes(2), true, "report.pdf")
.orElseThrow();
String inlineQuery =
java.net.URLDecoder.decode(inline.getRawQuery(), StandardCharsets.UTF_8);
assertThat(inlineQuery)
.contains("response-content-disposition=inline; filename=\"report.pdf\"");
URI bare =
provider.signedDownloadUrl(
stored.getStorageKey(), Duration.ofMinutes(2), false, null)
.orElseThrow();
assertThat(bare.getRawQuery()).doesNotContain("response-content-disposition");
}
@Test
void buildContentDisposition_escapesQuotesAndStripsControlChars() {
assertThat(S3StorageProvider.buildContentDisposition(true, "ev\"il\r\nname.pdf"))
.isEqualTo("inline; filename=\"ev\\\"ilname.pdf\"");
assertThat(S3StorageProvider.buildContentDisposition(false, null)).isNull();
assertThat(S3StorageProvider.buildContentDisposition(false, " ")).isNull();
}
private static IOException catchIOException(IOAction action) {
try {
action.run();
return null;
} catch (IOException e) {
return e;
}
}
@FunctionalInterface
private interface IOAction {
void run() throws IOException;
}
}
@@ -0,0 +1,298 @@
package stirling.software.proprietary.storage.service;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.util.Optional;
import java.util.UUID;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.context.SecurityContext;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.web.server.ResponseStatusException;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.Folder;
import stirling.software.proprietary.storage.model.api.CreateFolderRequest;
import stirling.software.proprietary.storage.repository.FolderRepository;
import stirling.software.proprietary.storage.repository.StoredFileRepository;
/**
* Unit tests for {@link FolderService}. Covers the regressions Connor flagged in PR #6383:
*
* <ul>
* <li>storage-enabled gate must be enforced (added in the same PR)
* <li>cross-user folder access must 404, not leak existence
* <li>cycle detection on reparent must 400
* <li>depth cap must reject chains past MAX_FOLDER_DEPTH
* <li>per-user folder count cap must 409
* </ul>
*
* Hibernate is mocked: this is a pure-Mockito unit test, not a slice test. Adequate for the
* service-layer behaviors above; full DB integration belongs in a separate {@code @DataJpaTest}.
*/
@ExtendWith(MockitoExtension.class)
class FolderServiceTest {
@Mock private FolderRepository folderRepository;
@Mock private StoredFileRepository storedFileRepository;
@Mock private ApplicationProperties applicationProperties;
@Mock private ApplicationProperties.Security security;
@Mock private ApplicationProperties.Storage storage;
private FolderService service;
private User user;
@BeforeEach
void setUp() {
// Default to "storage enabled" so the unrelated tests don't have to repeat the wiring.
// Individual tests override with disabled state.
lenient().when(applicationProperties.getSecurity()).thenReturn(security);
lenient().when(applicationProperties.getStorage()).thenReturn(storage);
lenient().when(security.isEnableLogin()).thenReturn(true);
lenient().when(storage.isEnabled()).thenReturn(true);
service = new FolderService(folderRepository, storedFileRepository, applicationProperties);
user = new User();
user.setId(42L);
user.setUsername("alice");
SecurityContext ctx = SecurityContextHolder.createEmptyContext();
ctx.setAuthentication(
new UsernamePasswordAuthenticationToken(user, null, java.util.List.of()));
SecurityContextHolder.setContext(ctx);
}
@AfterEach
void tearDown() {
SecurityContextHolder.clearContext();
}
@Test
void listFolders_rejects_when_login_disabled() {
when(security.isEnableLogin()).thenReturn(false);
assertThatThrownBy(() -> service.listFolders())
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e ->
assertThat(((ResponseStatusException) e).getStatusCode().value())
.isEqualTo(403));
}
@Test
void listFolders_rejects_when_storage_disabled() {
when(storage.isEnabled()).thenReturn(false);
assertThatThrownBy(() -> service.listFolders())
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e ->
assertThat(((ResponseStatusException) e).getStatusCode().value())
.isEqualTo(403));
}
@Test
void createFolder_under_unknown_parent_returns_400_without_leaking_existence() {
// Parent UUID exists for ANOTHER user; current-user lookup misses it. The repository
// returns Optional.empty() and the service must surface a generic 400, not a 404 that
// could be used to probe for existence by id-guessing.
UUID foreignParentId = UUID.randomUUID();
when(folderRepository.findByIdAndOwner(eq(foreignParentId), eq(user)))
.thenReturn(Optional.empty());
CreateFolderRequest req = new CreateFolderRequest();
req.setName("Child");
req.setParentFolderId(foreignParentId);
assertThatThrownBy(() -> service.createFolder(req))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e -> {
ResponseStatusException rse = (ResponseStatusException) e;
assertThat(rse.getStatusCode().value()).isEqualTo(400);
assertThat(rse.getReason()).doesNotContain(foreignParentId.toString());
});
}
@Test
void createFolder_409_when_user_at_folder_cap() {
// Stub out an existing-id miss so we reach the cap check (no Mockito unnecessary-stub
// warnings from the OTHER paths because we exit at the cap before the existsById call).
UUID newId = UUID.randomUUID();
when(folderRepository.findByIdAndOwner(eq(newId), eq(user))).thenReturn(Optional.empty());
when(folderRepository.existsById(eq(newId))).thenReturn(false);
when(folderRepository.countByOwner(eq(user))).thenReturn(5_000L);
CreateFolderRequest req = new CreateFolderRequest();
req.setName("Overflow");
req.setId(newId);
assertThatThrownBy(() -> service.createFolder(req))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e ->
assertThat(((ResponseStatusException) e).getStatusCode().value())
.isEqualTo(409));
}
@Test
void resolveParent_rejects_when_chain_exceeds_depth_cap() {
// Build a chain Hibernate-proxy-style: 64 ancestor stubs reachable via getParent(). The
// 65th createFolder attempt under the deepest existing folder should be rejected with
// 400 before any further work.
Folder root = makeFolder(UUID.randomUUID(), null);
Folder cursor = root;
for (int i = 0; i < 63; i++) {
Folder child = makeFolder(UUID.randomUUID(), cursor);
cursor = child;
}
// cursor is at depth 64 from root. Attempting to add another folder under cursor pushes
// the new child to depth 65 - past the cap. resolveParent walks cursor->root counting
// ancestors, which is exactly 64, and rejects.
Folder deepest = cursor;
when(folderRepository.findByIdAndOwner(eq(deepest.getId()), eq(user)))
.thenReturn(Optional.of(deepest));
CreateFolderRequest req = new CreateFolderRequest();
req.setName("Too deep");
req.setParentFolderId(deepest.getId());
assertThatThrownBy(() -> service.createFolder(req))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e -> {
ResponseStatusException rse = (ResponseStatusException) e;
assertThat(rse.getStatusCode().value()).isEqualTo(400);
assertThat(rse.getReason()).containsIgnoringCase("nesting limit");
});
}
@Test
void updateFolder_rejects_cycle_on_reparent() {
// A -> B -> C. Attempt to reparent A under C (i.e. set A.parent = C). C's chain to root
// includes B which includes A, so the cycle check must fire with 400.
Folder a = makeFolder(UUID.randomUUID(), null);
Folder b = makeFolder(UUID.randomUUID(), a);
Folder c = makeFolder(UUID.randomUUID(), b);
when(folderRepository.findByIdAndOwner(eq(a.getId()), eq(user))).thenReturn(Optional.of(a));
when(folderRepository.findByIdAndOwner(eq(c.getId()), eq(user))).thenReturn(Optional.of(c));
stirling.software.proprietary.storage.model.api.UpdateFolderRequest req =
new stirling.software.proprietary.storage.model.api.UpdateFolderRequest();
req.setParentFolderId(c.getId());
// shouldReparent() requires the explicit reparent flag - without it the
// parent change is silently skipped (PATCH-style semantics).
req.setReparent(true);
assertThatThrownBy(() -> service.updateFolder(a.getId(), req))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e -> {
ResponseStatusException rse = (ResponseStatusException) e;
assertThat(rse.getStatusCode().value()).isEqualTo(400);
assertThat(rse.getReason()).containsIgnoringCase("descendants");
});
}
@Test
void updateFolder_rejects_when_folder_not_owned() {
// Owner mismatch surfaces as 404, NOT 403 - 403 would confirm the folder exists, leaking
// ids to probing users. Stays consistent with the createFolder-under-unknown-parent test
// above.
UUID foreignId = UUID.randomUUID();
when(folderRepository.findByIdAndOwner(eq(foreignId), eq(user)))
.thenReturn(Optional.empty());
stirling.software.proprietary.storage.model.api.UpdateFolderRequest req =
new stirling.software.proprietary.storage.model.api.UpdateFolderRequest();
req.setName("Renamed");
assertThatThrownBy(() -> service.updateFolder(foreignId, req))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e ->
assertThat(((ResponseStatusException) e).getStatusCode().value())
.isEqualTo(404));
}
@Test
void moveFileToFolder_rejects_when_target_folder_not_owned() {
// File belongs to current user but target folder belongs to someone else. Service must
// 400, not move the file.
UUID foreignFolderId = UUID.randomUUID();
stirling.software.proprietary.storage.model.StoredFile file =
mock(stirling.software.proprietary.storage.model.StoredFile.class);
when(storedFileRepository.findByIdAndOwner(eq(100L), eq(user)))
.thenReturn(Optional.of(file));
when(folderRepository.findByIdAndOwner(eq(foreignFolderId), eq(user)))
.thenReturn(Optional.empty());
assertThatThrownBy(() -> service.moveFileToFolder(100L, foreignFolderId))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e ->
assertThat(((ResponseStatusException) e).getStatusCode().value())
.isEqualTo(400));
}
@Test
void bulkMove_rejects_oversized_payload() {
// Bypass the @Valid bound by calling the service directly - the cap must hold here too,
// not just at the controller's request validator.
java.util.List<Long> tooMany = new java.util.ArrayList<>();
for (int i = 0; i < 1001; i++) tooMany.add((long) i);
assertThatThrownBy(() -> service.bulkMoveFilesToFolder(null, tooMany))
.isInstanceOf(ResponseStatusException.class)
.satisfies(
e ->
assertThat(((ResponseStatusException) e).getStatusCode().value())
.isEqualTo(400));
}
@Test
void bulkMove_returns_moved_and_skipped_split() {
// Ownership filter on the repository returns a subset; the rest land in skippedFileIds.
Folder target = makeFolder(UUID.randomUUID(), null);
when(folderRepository.findByIdAndOwner(eq(target.getId()), eq(user)))
.thenReturn(Optional.of(target));
stirling.software.proprietary.storage.model.StoredFile fileA =
mock(stirling.software.proprietary.storage.model.StoredFile.class);
when(fileA.getId()).thenReturn(1L);
stirling.software.proprietary.storage.model.StoredFile fileB =
mock(stirling.software.proprietary.storage.model.StoredFile.class);
when(fileB.getId()).thenReturn(2L);
when(storedFileRepository.findAllByIdInAndOwner(any(), eq(user)))
.thenReturn(java.util.List.of(fileA, fileB));
FolderService.BulkMoveResult result =
service.bulkMoveFilesToFolder(target.getId(), java.util.List.of(1L, 2L, 3L, 4L));
assertThat(result.movedFileIds()).containsExactly(1L, 2L);
assertThat(result.skippedFileIds()).containsExactly(3L, 4L);
}
// ─── helpers ────────────────────────────────────────────────────────────────
private Folder makeFolder(UUID id, Folder parent) {
Folder f = new Folder();
f.setId(id);
f.setOwner(user);
f.setName("f-" + id.toString().substring(0, 8));
f.setParent(parent);
return f;
}
}
@@ -0,0 +1,110 @@
# DB migration test fixtures
These `.mv.db` files are H2 databases captured from past Stirling-PDF releases.
They feed the CI smoke test that verifies a fresh build can still boot and
authenticate against a database created by an older version.
| File | Source release | Tables | Notes |
|---|---|---|---|
| `stirling-pdf-v2.0.0.mv.db` | [v2.0.0](https://github.com/Stirling-Tools/Stirling-PDF/releases/tag/v2.0.0) | users, authorities, teams, sessions, audit_events, persistent_logins, invite_tokens, user_license_settings, user_settings | Pre-storage/workflow schema. |
| `stirling-pdf-v2.5.0.mv.db` | [v2.5.0](https://github.com/Stirling-Tools/Stirling-PDF/releases/tag/v2.5.0) | same as v2.0.0 | Schema unchanged from v2.0.0; intentionally kept as a separate fixture to exercise the "skip every other minor" upgrade path. |
| `stirling-pdf-v2.10.0.mv.db` | [v2.10.0](https://github.com/Stirling-Tools/Stirling-PDF/releases/tag/v2.10.0) | v2.5.0 tables + file_shares, file_share_accesses, stored_files, stored_file_blobs, storage_cleanup_entries, user_server_certificates, workflow_sessions, workflow_participants, participant_notifications | Adds the file-sharing and workflow signing schema. |
All three were generated against H2 `2.3.232` and use the same on-disk file
format, so the runtime driver can open any of them without conversion.
## What's in each fixture
* `admin` user with the default password `stirling` (BCrypt `$2a$10$...`).
* The internal API user `STIRLING-PDF-BACKEND-API-USER`.
* `ROLE_ADMIN` authority row for the admin user.
* `Default` and `Internal` teams.
* `user_license_settings` row (singleton).
`audit_events`, `sessions`, and `user_settings` are empty in the OSS-flavored
fixtures: those tables are written only on Enterprise builds (audit) or
require an HTTP-session-creating flow (sessions / settings) that the OSS form
login no longer exposes. The migration test only depends on the admin user
existing, so leaving these empty is intentional.
## What the CI test checks
`.github/workflows/db-migration-test.yml` runs `scripts/db-migration/run-migration-test.sh`,
which for each fixture:
1. Copies the fixture into `configs/stirling-pdf-DB-2.3.232.mv.db` of a clean
working directory.
2. Boots the current `:stirling-pdf:bootJar` against it on a free port.
3. Waits for Spring to start (no `SchemaManagementException` in the log).
4. POSTs `{"username":"admin","password":"stirling"}` to `/api/v1/auth/login`
and asserts the response is `200 OK`.
A red CI on this job means a schema change in the PR is not backwards
compatible with an existing user database. Common causes:
* Adding a non-nullable column without a default.
* Renaming a column (Hibernate's `update` strategy adds the new column and
leaves the old one orphaned with the data still in it).
* Changing a column type in an incompatible way.
* Dropping or renaming a foreign-key target.
## Regenerating fixtures
There's no automated regenerator script - fixtures are rare to refresh and the
manual steps are short. For each version you want to capture:
```bash
# 1. Download the JAR for that release (requires `gh` authenticated against
# github.com/Stirling-Tools/Stirling-PDF).
gh release download v2.10.0 \
--repo Stirling-Tools/Stirling-PDF \
--pattern 'Stirling-PDF-with-login.jar' \
--output /tmp/stirling-v2.10.0.jar
# 2. Boot the JAR in a clean working directory. DB_CLOSE_ON_EXIT=TRUE is
# the only override that matters - it makes the H2 file flush on JVM exit
# even if you Ctrl-C instead of going through a graceful shutdown.
workdir=$(mktemp -d)
mkdir -p "$workdir/configs"
cd "$workdir"
java -jar /tmp/stirling-v2.10.0.jar \
--server.port=8089 \
--spring.datasource.url='jdbc:h2:file:./configs/stirling-pdf-DB-2.3.232;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=TRUE;MODE=PostgreSQL' \
&
# 3. Wait until http://localhost:8089/login responds, then log in once to
# materialize whatever rows the app writes on first boot.
curl -sf -X POST -H 'Content-Type: application/json' \
-d '{"username":"admin","password":"stirling"}' \
http://localhost:8089/api/v1/auth/login
# 4. Shut it down (any kill works - DB_CLOSE_ON_EXIT=TRUE handles the flush).
kill -TERM %1 && wait %1
# 5. Copy the .mv.db here, renamed for the version.
cp "$workdir/configs/stirling-pdf-DB-2.3.232.mv.db" \
app/proprietary/src/test/resources/db-migration-fixtures/stirling-pdf-v2.10.0.mv.db
```
Requirements: Java 21+ (the historical JARs target Java 17 / 21).
## Adding a new fixture
When a new minor release ships, repeat the steps above for the new tag and
add a row to the table at the top of this file. Keep the historical fixtures -
the test gets stronger with each schema generation it covers.
## Inspecting a fixture by hand
The H2 driver bundled with the build ships an interactive shell:
```bash
h2_jar=$(find ~/.gradle/caches/modules-2 -name 'h2-2.3.232.jar' | head -1)
cd app/proprietary/src/test/resources/db-migration-fixtures
java -cp "$h2_jar" org.h2.tools.Shell \
-url 'jdbc:h2:file:./stirling-pdf-v2.10.0;ACCESS_MODE_DATA=r;MODE=PostgreSQL' \
-user sa
```
`ACCESS_MODE_DATA=r` keeps the inspection read-only so you can't accidentally
mutate a committed fixture.
@@ -5,18 +5,24 @@ import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
/** Registers the {@code :saas} module's entities and repositories with Spring Data JPA. */
/**
* Registers the {@code :saas} module's entities and repositories with Spring Data JPA. Any new
* package holding {@code @Repository} or {@code @Entity} classes must be added here, or the beans
* won't wire at startup.
*/
@Configuration
@Profile("saas")
@EnableJpaRepositories(
basePackages = {
"stirling.software.saas.repository",
"stirling.software.saas.billing.repository",
"stirling.software.saas.ai.repository"
"stirling.software.saas.ai.repository",
"stirling.software.saas.payg.repository"
})
@EntityScan({
"stirling.software.saas.model",
"stirling.software.saas.billing.model",
"stirling.software.saas.ai.model"
"stirling.software.saas.ai.model",
"stirling.software.saas.payg"
})
public class SaasJpaConfig {}
@@ -73,6 +73,13 @@ public class TeamMembership implements Serializable {
@Column(name = "updated_at", nullable = false)
private LocalDateTime updatedAt;
/**
* Optional per-member spend cap inside the team's wallet, in doc units. NULL means the member
* is bounded only by the team-wide cap.
*/
@Column(name = "cap_units")
private Long capUnits;
public boolean isLeader() {
return role == TeamRole.LEADER;
}
@@ -0,0 +1,173 @@
package stirling.software.saas.payg.docs;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.util.List;
import java.util.Objects;
import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.util.TempFile;
import stirling.software.common.util.TempFileManager;
import stirling.software.jpdfium.PdfDocument;
import stirling.software.saas.payg.policy.PricingPolicy;
/**
* Reads pages via jpdfium for PDF inputs; treats every other content type as bytes-only.
*
* <p>For PDFs, units are the larger of {@code ceil(pages / docPagesPerUnit)} and {@code ceil(bytes
* / docBytesPerUnit)}. For non-PDFs, only the bytes axis contributes. A single file is clamped to
* {@code [1, policy.fileUnitCap]}; a multi-file group is clamped to {@code [1, policy.fileUnitCap *
* file_count]} applied to the sum of raw per-file units. Malformed/encrypted PDFs fall back to
* bytes-only.
*
* <p>{@code policy.minChargeUnits} is applied by the charge service, not here. The classifier only
* enforces an absolute floor of {@link #MIN_UNITS_PER_NONEMPTY_FILE} so callers can rely on
* "non-empty input → at least 1 unit".
*/
@Slf4j
@Component
@Profile("saas")
@RequiredArgsConstructor
public class DefaultDocumentClassifier implements DocumentClassifier {
private static final String PDF_CONTENT_TYPE = "application/pdf";
private static final String DEFAULT_CONTENT_TYPE = "application/octet-stream";
/** Floor for non-empty input. Distinct from {@code policy.minChargeUnits} (applied later). */
private static final int MIN_UNITS_PER_NONEMPTY_FILE = 1;
private final TempFileManager tempFileManager;
@Override
public DocumentMetrics classify(MultipartFile file, PricingPolicy policy) {
Objects.requireNonNull(file, "file");
Objects.requireNonNull(policy, "policy");
FileFacts facts = inspect(file);
long rawUnits = computeRawUnits(facts.pages, facts.bytes, policy);
// toIntExact: fail loud on overflow rather than silently wrapping a billing number.
int units =
Math.toIntExact(
Math.max(
MIN_UNITS_PER_NONEMPTY_FILE,
Math.min(policy.getFileUnitCap(), rawUnits)));
return new DocumentMetrics(facts.pages, facts.bytes, facts.contentType, units);
}
@Override
public DocumentMetrics classify(List<MultipartFile> files, PricingPolicy policy) {
Objects.requireNonNull(files, "files");
Objects.requireNonNull(policy, "policy");
if (files.isEmpty()) {
throw new IllegalArgumentException("files must not be empty");
}
int totalPages = 0;
long totalBytes = 0;
long rawUnitsSum = 0;
String firstContentType = null;
for (MultipartFile file : files) {
FileFacts facts = inspect(file);
// Sum the *raw* (unclamped) per-file units so the group cap below can actually bind.
// Per-file clamping in this loop would make the group cap a no-op.
rawUnitsSum =
saturatedAdd(rawUnitsSum, computeRawUnits(facts.pages, facts.bytes, policy));
totalPages = saturatedAdd(totalPages, facts.pages);
totalBytes = saturatedAdd(totalBytes, facts.bytes);
if (firstContentType == null) {
firstContentType = facts.contentType;
}
}
long groupCap = (long) policy.getFileUnitCap() * files.size();
// toIntExact: fail loud on overflow rather than silently wrapping.
int totalUnits =
Math.toIntExact(
Math.max(
(long) MIN_UNITS_PER_NONEMPTY_FILE,
Math.min(groupCap, rawUnitsSum)));
return new DocumentMetrics(
totalPages,
totalBytes,
firstContentType != null ? firstContentType : DEFAULT_CONTENT_TYPE,
totalUnits);
}
private FileFacts inspect(MultipartFile file) {
long bytes = file.getSize();
String contentType =
file.getContentType() != null ? file.getContentType() : DEFAULT_CONTENT_TYPE;
int pages = isPdf(contentType, file.getOriginalFilename()) ? readPageCount(file) : 0;
return new FileFacts(pages, bytes, contentType);
}
private static long computeRawUnits(int pages, long bytes, PricingPolicy policy) {
long pageUnits = pages > 0 ? ceilDiv(pages, policy.getDocPagesPerUnit()) : 0L;
long byteUnits = ceilDiv(bytes, policy.getDocBytesPerUnit());
return Math.max(pageUnits, byteUnits);
}
private static long ceilDiv(long numerator, long divisor) {
if (numerator <= 0) {
return 0;
}
return (numerator + divisor - 1) / divisor;
}
private static boolean isPdf(String contentType, String filename) {
if (PDF_CONTENT_TYPE.equalsIgnoreCase(contentType)) {
return true;
}
return filename != null && filename.toLowerCase().endsWith(".pdf");
}
/**
* Materialises the upload to a managed temp file and asks jpdfium for the page count. Returns 0
* if the file can't be parsed — the byte-derived axis still produces a charge.
*/
private int readPageCount(MultipartFile file) {
try (TempFile temp = tempFileManager.createManagedTempFile(".pdf")) {
try (InputStream in = file.getInputStream();
OutputStream out = Files.newOutputStream(temp.getPath())) {
in.transferTo(out);
}
try (PdfDocument doc = PdfDocument.open(temp.getPath())) {
return doc.pageCount();
}
} catch (IOException | RuntimeException e) {
log.debug(
"Could not read PDF page count for {} ({}); falling back to bytes-only units",
file.getOriginalFilename(),
e.getClass().getSimpleName());
return 0;
}
}
private static int saturatedAdd(int a, int b) {
long sum = (long) a + b;
if (sum > Integer.MAX_VALUE) {
return Integer.MAX_VALUE;
}
return (int) sum;
}
private static long saturatedAdd(long a, long b) {
try {
return Math.addExact(a, b);
} catch (ArithmeticException e) {
return Long.MAX_VALUE;
}
}
private record FileFacts(int pages, long bytes, String contentType) {}
}
@@ -0,0 +1,25 @@
package stirling.software.saas.payg.docs;
import java.util.List;
import org.springframework.web.multipart.MultipartFile;
import stirling.software.saas.payg.policy.PricingPolicy;
/**
* Computes the doc-unit cost of an uploaded file (or multi-file input) under a given policy.
*
* <p>Returns {@code docUnits} with an absolute floor of 1 for non-empty input. {@code
* policy.minChargeUnits} is applied at charge time, not here.
*/
public interface DocumentClassifier {
/** Classify a single uploaded file. Returns at least 1 unit, capped at {@code fileUnitCap}. */
DocumentMetrics classify(MultipartFile file, PricingPolicy policy);
/**
* Classify a multi-file input (e.g. a merge or overlay). Returns the sum of each file's raw
* units, capped at {@code fileUnitCap × files.size()} and floored at 1.
*/
DocumentMetrics classify(List<MultipartFile> files, PricingPolicy policy);
}
@@ -0,0 +1,12 @@
package stirling.software.saas.payg.docs;
/**
* Output of {@link DocumentClassifier#classify}. {@code pages} is {@code 0} for non-PDF inputs.
*
* @param pages page count (0 for non-PDFs and for files whose page count couldn't be read)
* @param bytes raw byte length of the file
* @param contentType MIME type as reported by the upload, or {@code "application/octet-stream"}
* when unknown
* @param docUnits computed unit cost, clamped to the policy's {@code fileUnitCap}
*/
public record DocumentMetrics(int pages, long bytes, String contentType, int docUnits) {}

Some files were not shown because too many files have changed in this diff Show More