Stirling-Tools/Stirling-PDF
1
0
Fork 0
mirror of https://github.com/Stirling-Tools/Stirling-PDF.git synced 2026-09-03 05:10:16 +03:00
Code Issues Packages Projects Releases Wiki Activity

merge main

Browse Source
This commit is contained in:
Frooodle
2026-07-10 10:27:03 +01:00
parent e8f76aa2d9 68ec176719
commit e7a81092c5
1295 changed files with 143086 additions and 19331 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/feature-walkthrough/SKILL.md
+97
View File
@@ -0,0 +1,97 @@
---
name: feature-walkthrough
description: >-
Explain the full logic and process of the current branch end-to-end so someone
with no prior knowledge of the task can understand, review, and reproduce it.
Scopes the change from the branch diff, traces the flow across every layer it
touches (frontend tool/hook/component, Java controller/service/endpoint, Python
engine, config, i18n, tests), and produces a self-contained walkthrough document
with Mermaid diagrams (sequence/flow/architecture), annotated file map with
clickable references, before/after behavior, screenshots where a UI is involved,
a "try it locally" section, and edge cases/risks. Use when asked for a feature or
branch walkthrough, "explain what this branch does", a design/logic writeup, PR
reviewer onboarding, or a hand-off doc. Pass --html to also emit a rendered HTML
version; --no-screens to skip screenshots.
argument-hint: "[branch-or-area] [--html] [--no-screens]"
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
---
# Feature / Branch Walkthrough
Turn the current branch into a walkthrough a newcomer can follow. Audience:
**someone who has never seen this task**. Explain the *why*, the *flow*, and *how to
try it* - not just a diff summary.
`$ARGUMENTS` may name a branch or area to focus on; default is the current branch
vs `main`. Flags: `--html` (also emit a rendered HTML twin), `--no-screens`.
## Process
### 1. Scope the change
- `git log --oneline main..HEAD` and `git diff --stat main...HEAD` for the shape.
- Read the PR description / commit messages for stated intent. Do **not** invent
history or motivation that isn't evidenced (state current behavior in present tense).
- Classify touched files by layer:
- **Frontend**: tools (`frontend/editor/src/core/components/tools/*` or `.../core/tools/*`),
hooks (`core/hooks/tools/*`, `useToolOperation`), contexts, routes, i18n
(`public/locales/en-US`).
- **Java backend**: controllers (`.../controller/api/...`), services, models, config.
- **Engine**: `engine/src/stirling/{agents,contracts,api,services}`.
- **Config / build / docker / tests.**
### 2. Trace the flow end-to-end
Follow one real path from user action to result. For a typical PDF tool that's:
UI control → `useToolOperation` hook → `POST /api/v1/...` → Spring controller →
service (PDFBox / LibreOffice / engine call) → response → review panel → download.
Read the actual files so the narrative is true to the code, and collect the exact
file:line anchors you'll cite.
### 3. Draw the diagrams (Mermaid)
Pick what fits; usually 2-3 of:
- **Sequence diagram** - request/response across frontend → backend → engine.
- **Flowchart** - the core decision/branching logic of the feature.
- **Architecture/component** - new pieces and how they wire to existing ones.
- **State** - if the feature has modes/steps.
Keep nodes labeled in plain language. Validate the Mermaid parses before shipping.
### 4. Screenshots (unless --no-screens)
If a UI is involved, capture key states with the stubbed Playwright harness
(see the **ui-walkthrough** skill and `files-page-screenshots.spec.ts` for the
pattern) or, for before/after, capture `main` then the branch. Drop PNGs in
`walkthrough/<feature>/` and reference them from the doc. For backend-only
changes, show request/response examples (curl + JSON) instead.
### 5. Write the walkthrough
Create `walkthrough/<feature>/FEATURE-WALKTHROUGH.md` with:
1. **TL;DR** - what the branch does and who it's for, in 3-4 sentences.
2. **Problem & approach** - what wasn't possible before; the chosen solution.
3. **Architecture diagram** + 1-paragraph orientation.
4. **End-to-end flow** - the sequence diagram + a numbered walk of each step,
each citing the real file (clickable `path:line`).
5. **Key files** - annotated map (path → one line on its role).
6. **Logic deep-dive** - the flowchart + prose for the non-obvious decisions.
7. **Behavior** - before vs after; screenshots or request/response examples.
8. **Try it locally** - exact steps (`task dev` / `task dev:all`, the route to
open or the curl to run, any env like `DOCKER_ENABLE_SECURITY` or a test
license key). Make it copy-pasteable.
9. **Edge cases, risks, follow-ups** - what's untested, known limits, gotchas.
Markdown is the primary deliverable - it renders with diagrams in GitHub PRs and
IDEs, no build step, ideal for review.
### 6. If `--html`
Also emit `walkthrough/<feature>/walkthrough.html`: the same content with Mermaid
rendered via `mermaid.initialize({startOnLoad:true})` (script from CDN; note in
the file that rendering diagrams needs network, the `.md` is the offline copy) and
screenshots inline. Keep it self-contained otherwise.
### 7. Deliver
Give the doc path and a short chat summary. Offer to `SendUserFile` it.
## Principles
- **True to the code.** Every claim traces to a file you read; cite `path:line`.
No fabricated migration/version history.
- **Newcomer-first.** Define repo-specific terms (FileContext, `useToolOperation`,
the `@app/*` layer cascade, stubbed vs live tests) on first use.
- **Show, don't assert.** Prefer a diagram + a real example over adjectives.
- Don't commit the `walkthrough/` output unless asked.
.claude/skills/ui-before-after/SKILL.md
+122
View File
@@ -0,0 +1,122 @@
---
name: ui-before-after
description: >-
Analyse a branch or PR and automatically capture before/after screenshots of
every UI surface its changes touch, then pixel-diff the pairs to surface what
actually changed and assemble PR-ready before/after montage images. Generic and
diff-driven: it derives the capture targets from the diff (changed tools/routes →
URLs) instead of hand-listing screens, captures "before" from the base branch and
"after" from the head, then keeps only the views that visually differ. Each
comparison is auto-cropped to the region that actually changed (the bounding box of
differing pixels), falling back to the full page only when the change spans most of
it. Use for before/after shots, a visual diff of a branch/PR, "screenshots for the
PR description", "show what changed in the UI", or a side-by-side of UI changes.
Takes a PR number/URL (resolved via gh) or a branch; defaults to the current branch
vs its base. Flags: --scope <selector>, --base <ref|merge-base>, --theme
light|dark|both, --all (capture every route, not just changed), --no-autocrop,
--pagewide <n>, --threshold <n>.
argument-hint: "[PR# | PR-url | branch] [--scope <sel>] [--base <ref>] [--theme both] [--all] [--no-autocrop]"
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
---
# UI Before / After (generic visual diff)
Point it at a branch or PR; it figures out which UI changed, screenshots every
affected surface **before** (base) and **after** (head), pixel-diffs the pairs, and
montages the ones that actually changed into images for the PR description.
`$ARGUMENTS`: a PR number/URL, a branch, or nothing (current branch vs base).
By default it captures the full viewport and auto-crops each comparison to the region
that changed. Flags: `--scope <css>` (narrow the *capture* to a container, e.g.
`[data-sidebar="tool-panel"]`, when you already know where the change is),
`--no-autocrop` (keep full frames), `--pagewide <fraction>` (above this share of the
page, skip cropping; default 0.6), `--base <ref|merge-base>`,
`--theme light|dark|both`, `--all` (walk every route, not just changed),
`--threshold <fraction>` (diff sensitivity, default 0.001).
Shares the capture harness with **ui-walkthrough** - read its SKILL.md for the
stubbed-Playwright setup, worktree node_modules + `generate-icons`, the
stale-`:5173` gotcha, and the dark-mode init-script. Bundled helpers:
[capture-spec.template.ts](capture-spec.template.ts), [diff-shots.mjs](diff-shots.mjs),
[montage-template.html](montage-template.html), [shoot-sections.mjs](shoot-sections.mjs).
## Process
### 1. Resolve target + base
```
gh pr view <pr> --json number,title,headRefName,baseRefName,url,files # PR
# or branch: base = merge-base(main, HEAD); head = HEAD
gh pr diff <pr> --name-only # or: git diff --name-only <base>...HEAD
```
### 2. Derive capture targets from the diff (the "analyse" step - no hand-listing)
Map changed frontend files to URLs generically:
- **Tools**: a changed `components/tools/<toolDir>/…` or `hooks/tools/<tool>/…` →
toolId → URL via the repo's own rule `getToolUrlPath` in
[toolsTaxonomy.ts:200](frontend/editor/src/core/data/toolsTaxonomy.ts): `/` + the
id kebab-cased (`addPageNumbers` → `/add-page-numbers`).
- **Pages/routes**: changed `filesPage/*` → `/files`, etc.
- `--all`: enumerate every tool in the registry instead of just changed ones.
Write `frontend/editor/screenshots/ui-diff/targets.json` =
`[{ "id":"compress", "url":"/compress", "name":"Compress" }]`. This is what makes
it generic - the spec never names a tool.
### 3. Capture AFTER (head) then BEFORE (base)
Copy [capture-spec.template.ts](capture-spec.template.ts) →
`src/core/tests/stubbed/ui-before-after.spec.ts` (it loops `targets.json`, seeds a
sample PDF so file-dependent panels render, navigates to each URL, and screenshots
the full viewport - or the `--scope` container if given). Ensure the harness is ready
(node_modules + icons).
```
# after = current head
cd frontend/editor && PR_SHOT_SIDE=after PR_SHOT_THEME=light \
npx playwright test --project=stubbed ui-before-after.spec.ts
# before = base, in an isolated worktree (copy the spec + targets.json in)
git worktree add ../ba-base origin/<baseRefName> # or the merge-base
# set up its frontend, copy spec + screenshots/ui-diff/targets.json across, then:
cd ../ba-base/frontend/editor && PR_SHOT_SIDE=before PR_SHOT_THEME=light \
npx playwright test --project=stubbed ui-before-after.spec.ts
# copy its screenshots/ui-diff/before/ back next to after/. Repeat with
# PR_SHOT_THEME=dark if --theme includes dark. Remove worktree when done.
```
### 4. Auto-diff (surface what changed)
```
cd frontend/editor && node <skill>/diff-shots.mjs \
screenshots/ui-diff/before screenshots/ui-diff/after screenshots/ui-diff
```
Produces `diff-report.json` classifying each view `unchanged | changed | added |
removed`. For each changed view it computes the bounding box of differing pixels and
writes cropped `__before_crop.png` / `__after_crop.png` / `__diff.png` to that region
(+ padding) - **unless** the change covers more than `--pagewide` of the frame, where
it keeps the full frame (`pageWide:true`). Drop `unchanged` - that's the noise the
user doesn't want.
### 5. Montage the changes
Build the manifest from the non-unchanged entries (group by tab/tool; each becomes a
state row with before/after). For changed views use the cropped `cropBefore` /
`cropAfter` from `diff-report.json` (tight on the affected region; full frame when
`pageWide`); `added`/`removed` render the "not present" placeholder. Fill
[montage-template.html](montage-template.html) (replace the `window.__BA__` data
block; base64-inline the PNGs for portability), then render one PNG per section with
[shoot-sections.mjs](shoot-sections.mjs). Optionally include the `__diff.png` overlay
as a third column.
### 6. Deliver
Output the `montage_<tab>.png` files + a short summary (N changed / added / removed,
M unchanged skipped) and a paste-ready Markdown block. GitHub has no PR-body image
API, so tell the user to drag the PNGs into the description. Do **not** post to the
PR.
## Gotchas
- Two installs (base worktree + head); junction main's node_modules only if its deps
match that ref, else `npm ci` (see ui-walkthrough's stale-dep note).
- A view that errors on one side (refactored/removed) → that side is missing; the
diff marks it added/removed rather than failing the run.
- Pixel diff needs equal dimensions, so capture at a fixed viewport (the template
does); a view whose size changed is reported as "changed (dimensions differ)",
uncropped.
- Auto-crop uses a single bounding box, so two far-apart changes give one large crop
(or trip `--pagewide`); narrow with `--scope` if that happens.
- `getToolUrlPath` is the source of truth for tool URLs - use it, don't guess slugs.
- Don't commit `screenshots/`, the throwaway spec, or the base worktree.
.claude/skills/ui-before-after/capture-spec.template.ts
+67
View File
@@ -0,0 +1,67 @@
// Generic before/after capturer. NOT app-specific: it walks a targets.json that
// the ui-before-after skill generates from the branch/PR diff, so nothing here is
// hand-listed. Copy to src/core/tests/stubbed/ui-before-after.spec.ts, then run
// once per (side, theme):
// PR_SHOT_SIDE=after PR_SHOT_THEME=light \
// npx playwright test --project=stubbed ui-before-after.spec.ts
//
// targets.json shape: [{ "id":"compress", "url":"/compress", "name":"Compress",
// "needsFile": true }]
import { test } from "@app/tests/helpers/stub-test-base";
import type { Page } from "@playwright/test";
import fs from "node:fs";
import path from "node:path";
const SIDE = process.env.PR_SHOT_SIDE ?? "after";
const THEME = process.env.PR_SHOT_THEME ?? "light";
// Capture the full viewport by default so the affected region is in frame
// wherever it is; diff-shots.mjs crops each comparison to what actually changed.
// Set PR_SHOT_SCOPE to a selector to narrow the capture to one container.
const SCOPE = process.env.PR_SHOT_SCOPE ?? "";
const ROOT = path.resolve(process.cwd(), "screenshots", "ui-diff");
const OUT = path.join(ROOT, SIDE);
// A tiny sample PDF so file-dependent tool panels render. Point at a real fixture.
const SAMPLE_PDF = process.env.PR_SHOT_SAMPLE ?? "src/core/tests/test-fixtures/sample.pdf";
type Target = { id: string; url: string; name?: string; needsFile?: boolean };
const targets: Target[] = JSON.parse(fs.readFileSync(path.join(ROOT, "targets.json"), "utf-8"));
test.use({ autoGoto: false, viewport: { width: 1600, height: 900 }, seedJwt: true });
async function applyTheme(page: Page): Promise<void> {
if (THEME !== "dark") return;
await page.addInitScript(() => {
localStorage.setItem("mantine-color-scheme", "dark");
localStorage.setItem("mantine-color-scheme-value", "dark");
});
await page.emulateMedia({ colorScheme: "dark" });
}
async function seedFile(page: Page): Promise<void> {
if (!fs.existsSync(SAMPLE_PDF)) return;
await page.goto("/", { waitUntil: "domcontentloaded" });
await page.getByTestId("files-button").click().catch(() => {});
await page.locator('[data-testid="file-input"]').setInputFiles(SAMPLE_PDF).catch(() => {});
await page.locator(".file-sidebar-file-item").first().isVisible({ timeout: 8_000 }).catch(() => {});
}
for (const t of targets) {
// One test per target so a single failure doesn't drop the rest.
test(`${SIDE}/${THEME} ${t.id}`, async ({ page }) => {
fs.mkdirSync(OUT, { recursive: true });
await applyTheme(page);
if (t.needsFile !== false) await seedFile(page);
await page.goto(t.url, { waitUntil: "domcontentloaded" });
await page.waitForTimeout(400); // settle Mantine portals/transitions
const shot = path.join(OUT, `${t.id}__${THEME}.png`);
if (SCOPE) {
const scope = page.locator(SCOPE).first();
if (await scope.isVisible({ timeout: 8_000 }).catch(() => false)) {
await scope.screenshot({ path: shot });
return;
}
}
// Full viewport (fixed size → stable dimensions for pixel diffing).
await page.screenshot({ path: shot });
});
}
.claude/skills/ui-before-after/diff-shots.mjs
+106
View File
@@ -0,0 +1,106 @@
// Auto-diff before/ vs after/ screenshots, classify each as
// unchanged | changed | added | removed, and CROP each changed pair to the
// affected region (bounding box of differing pixels + padding) - unless the
// change spans most of the page, in which case the full frame is kept.
// Run from frontend/editor (so deps resolve):
// node <skill>/diff-shots.mjs <beforeDir> <afterDir> [outDir]
// Env:
// DIFF_THRESHOLD min fraction of differing pixels to count as changed (default 0.001)
// DIFF_PAD padding px around the affected region (default 24)
// DIFF_PAGEWIDE if affected bbox area / image area exceeds this, keep full frame (default 0.6)
import fs from "node:fs";
import path from "node:path";
import { createRequire } from "node:module";
const require = createRequire(path.join(process.cwd(), "noop.js"));
const pm = require("pixelmatch");
const pixelmatch = pm.default || pm;
const { PNG } = require("pngjs");
const beforeDir = path.resolve(process.argv[2]);
const afterDir = path.resolve(process.argv[3]);
const outDir = path.resolve(process.argv[4] || afterDir);
const THRESHOLD = Number(process.env.DIFF_THRESHOLD ?? "0.001");
const PAD = Number(process.env.DIFF_PAD ?? "24");
const PAGEWIDE = Number(process.env.DIFF_PAGEWIDE ?? "0.6");
const read = (p) => PNG.sync.read(fs.readFileSync(p));
const isShot = (f) => f.endsWith(".png") && !/__(diff|before_crop|after_crop)\.png$/.test(f);
const list = (d) => (fs.existsSync(d) ? fs.readdirSync(d).filter(isShot) : []);
const names = [...new Set([...list(beforeDir), ...list(afterDir)])].sort();
fs.mkdirSync(outDir, { recursive: true });
function cropPNG(src, x, y, w, h) {
const out = new PNG({ width: w, height: h });
PNG.bitblt(src, out, x, y, w, h, 0, 0);
return out;
}
const writePNG = (p, png) => fs.writeFileSync(p, PNG.sync.write(png));
// Bounding box of differing pixels using a diff mask (alpha>0 where changed).
function changedBBox(before, after, w, h) {
const mask = new PNG({ width: w, height: h });
pixelmatch(before.data, after.data, mask.data, w, h, { threshold: 0.1, diffMask: true });
let minX = w, minY = h, maxX = -1, maxY = -1, count = 0;
for (let y = 0; y < h; y++) {
for (let x = 0; x < w; x++) {
if (mask.data[(y * w + x) * 4 + 3] > 0) {
count++;
if (x < minX) minX = x; if (x > maxX) maxX = x;
if (y < minY) minY = y; if (y > maxY) maxY = y;
}
}
}
return maxX < 0 ? null : { minX, minY, maxX, maxY, count };
}
const report = [];
for (const name of names) {
const id = name.replace(/\.png$/, "");
const bp = path.join(beforeDir, name), ap = path.join(afterDir, name);
const hasB = fs.existsSync(bp), hasA = fs.existsSync(ap);
if (hasB && !hasA) { report.push({ id, status: "removed", before: bp }); continue; }
if (!hasB && hasA) { report.push({ id, status: "added", after: ap }); continue; }
const before = read(bp), after = read(ap);
if (before.width !== after.width || before.height !== after.height) {
report.push({ id, status: "changed", note: "dimensions differ", before: bp, after: ap });
continue;
}
const w = after.width, h = after.height;
const overlay = new PNG({ width: w, height: h });
const px = pixelmatch(before.data, after.data, overlay.data, w, h, { threshold: 0.1 });
const ratio = px / (w * h);
if (ratio <= THRESHOLD) { report.push({ id, status: "unchanged", ratio: Number(ratio.toFixed(5)), before: bp, after: ap }); continue; }
const box = changedBBox(before, after, w, h);
// Pad + clamp the affected region.
const x = Math.max(0, box.minX - PAD), y = Math.max(0, box.minY - PAD);
const x2 = Math.min(w, box.maxX + 1 + PAD), y2 = Math.min(h, box.maxY + 1 + PAD);
const bw = x2 - x, bh = y2 - y;
const pageWide = (bw * bh) / (w * h) > PAGEWIDE;
const entry = { id, status: "changed", ratio: Number(ratio.toFixed(5)), before: bp, after: ap, pageWide };
if (pageWide) {
// Change spans most of the page - keep the full frame, full overlay.
const dp = path.join(outDir, `${id}__diff.png`); writePNG(dp, overlay);
entry.diff = dp;
} else {
entry.bbox = { x, y, w: bw, h: bh };
const cb = path.join(outDir, `${id}__before_crop.png`); writePNG(cb, cropPNG(before, x, y, bw, bh));
const ca = path.join(outDir, `${id}__after_crop.png`); writePNG(ca, cropPNG(after, x, y, bw, bh));
const dp = path.join(outDir, `${id}__diff.png`); writePNG(dp, cropPNG(overlay, x, y, bw, bh));
entry.cropBefore = cb; entry.cropAfter = ca; entry.diff = dp;
}
report.push(entry);
}
fs.writeFileSync(path.join(outDir, "diff-report.json"), JSON.stringify(report, null, 2));
const changed = report.filter((r) => r.status !== "unchanged");
console.log(`diffed ${report.length} view(s): ${changed.length} changed/added/removed, ${report.length - changed.length} unchanged`);
for (const r of changed) {
const tail = r.status !== "changed" ? ""
: r.pageWide ? " (page-wide → full frame)"
: ` (${(r.ratio * 100).toFixed(2)}%, cropped to ${r.bbox.w}×${r.bbox.h})`;
console.log(` ${r.status.padEnd(9)} ${r.id}${tail}${r.note ? " - " + r.note : ""}`);
}
.claude/skills/ui-before-after/make_example.py
+48
View File
@@ -0,0 +1,48 @@
"""Build EXAMPLE.html from montage-template.html using REAL files-page shots as
stand-in before/after pairs (layout demo, not an actual PR diff). Inlines PNGs as
data URIs so the HTML is portable. Run: python make_example.py"""
import base64
import json
import pathlib
import re
HERE = pathlib.Path(__file__).parent
SHOTS = pathlib.Path(
r"C:\Users\systo\git\Stirling-PDFNew\.claude\worktrees\kind-faraday-522a30"
r"\frontend\editor\screenshots\files-page"
)
def uri(fname):
p = SHOTS / fname
return "data:image/png;base64," + base64.b64encode(p.read_bytes()).decode() if p.exists() else None
data = {
"pr": "DEMO",
"title": "EXAMPLE — before/after montage (layout demo, real Files-page shots; not a real PR diff)",
"base": "main", "head": "demo-branch",
"cropSelector": "[data-sidebar=\"tool-panel\"] (real runs crop to the side; these demo shots are full-page)",
"tabs": [
{"id": "files", "title": "Files page", "ctx": "Each row = one flow state; left = base branch, right = this PR.",
"states": [
{"name": "Empty folder", "before": uri("01_empty_state_ctas.png"), "after": uri("02_empty_state_storage_off.png")},
{"name": "Files + details panel", "before": uri("03_subtoolbar_with_files.png"), "after": uri("06_details_panel_save_to_server.png")},
{"name": "Delete folder confirm", "before": None, "after": uri("19_delete_folder_dialog.png"), "note": "New in this PR"},
]},
{"id": "move", "title": "Move-to-folder dialog",
"states": [
{"name": "Dialog opened", "before": uri("07_move_dialog_collapsed.png"), "after": uri("08_move_dialog_create_folder_expanded.png")},
{"name": "After folder created", "before": None, "after": uri("08b_move_dialog_after_create_folder.png"), "note": "New flow"},
]},
],
}
tpl = (HERE / "montage-template.html").read_text(encoding="utf-8")
out = re.sub(
r"/\*__DATA__\*/.*?/\*__END__\*/",
lambda _m: "/*__DATA__*/" + json.dumps(data) + "/*__END__*/",
tpl, count=1, flags=re.S,
)
(HERE / "EXAMPLE.html").write_text(out, encoding="utf-8")
print("wrote", HERE / "EXAMPLE.html", "(", (HERE / "EXAMPLE.html").stat().st_size // 1024, "KB )")
.claude/skills/ui-before-after/montage-template.html
+106
View File
@@ -0,0 +1,106 @@
<!doctype html>
<!--
Before/After montage for a PR description. The ui-before-after skill replaces
the JSON in the window.__BA__ data block below with the captured manifest, then
screenshots each .tab-section (id="section-<tabId>") into a PNG to drag into the
PR description. Self-contained; images may be relative paths or data URIs.
Data shape:
{
"pr":"6552","title":"...","base":"main","head":"feat/x",
"cropSelector":"[data-sidebar=\"tool-panel\"]",
"tabs":[
{ "id":"sign","title":"Sign tool","states":[
{"name":"Initial","before":"before/sign__initial.png","after":"after/sign__initial.png"},
{"name":"Cert selected","before":null,"after":"after/sign__cert.png","note":"New in this PR"}
]}
]
}
-->
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Before / After</title>
<style>
:root { --bg:#ffffff; --ink:#0b0c0e; --muted:#6b7280; --line:#e5e7eb;
--before:#6b7280; --after:#1f883d; --frame:#f3f4f6; --note:#b45309; }
* { box-sizing: border-box; }
body { margin:0; background:var(--bg); color:var(--ink);
font:14px/1.5 -apple-system,"Segoe UI",Roboto,system-ui,sans-serif; }
.wrap { max-width:1100px; margin:0 auto; padding:24px; }
.doc-head { margin-bottom:8px; }
.doc-head h1 { font-size:18px; margin:0 0 2px; }
.doc-head .sub { color:var(--muted); font-size:12.5px; }
.legend { display:flex; gap:14px; align-items:center; margin:10px 0 4px; font-size:12px; color:var(--muted); }
.chip { font-size:10px; font-weight:700; letter-spacing:.04em; text-transform:uppercase;
padding:2px 8px; border-radius:999px; color:#fff; }
.chip.before { background:var(--before); } .chip.after { background:var(--after); }
.tab-section { border:1px solid var(--line); border-radius:14px; padding:18px 18px 8px;
margin:18px 0; background:var(--bg); }
.tab-section > h2 { font-size:16px; margin:0 0 2px; }
.tab-section > .ctx { color:var(--muted); font-size:12px; margin-bottom:14px; }
.state { margin-bottom:18px; }
.state .name { font-weight:600; font-size:13.5px; margin-bottom:8px; display:flex; gap:8px; align-items:center; }
.state .name .note { font-weight:500; color:var(--note); font-size:12px; }
.pair { display:grid; grid-template-columns:1fr 1fr; gap:14px; align-items:start; }
.cell { border:1px solid var(--line); border-radius:10px; overflow:hidden; background:var(--frame); }
.cell .cap { display:flex; align-items:center; gap:8px; padding:7px 10px; border-bottom:1px solid var(--line);
background:var(--bg); }
.cell .cap .meta { color:var(--muted); font-size:11px; }
.cell img { display:block; width:100%; height:auto; background:#fff; }
.cell.empty .ph { display:flex; align-items:center; justify-content:center; height:160px; color:var(--muted);
font-size:12.5px; text-align:center; padding:0 16px; }
.single .pair { grid-template-columns:1fr; }
.empty-doc { color:var(--muted); padding:40px; text-align:center; }
@media (max-width:760px){ .pair{ grid-template-columns:1fr; } }
</style>
</head>
<body>
<div class="wrap" id="root"></div>
<script id="data">
window.__BA__ = /*__DATA__*/{"pr":"","title":"No data","base":"","head":"","cropSelector":"","tabs":[]}/*__END__*/;
</script>
<script>
(function(){
var D = window.__BA__ || { tabs: [] };
var root = document.getElementById("root");
function el(html){ var t=document.createElement("template"); t.innerHTML=html.trim(); return t.content.firstChild; }
function esc(s){ return (s==null?"":String(s)).replace(/[&<>]/g, function(c){return {"&":"&amp;","<":"&lt;",">":"&gt;"}[c];}); }
function cell(kind, src){
if (src) {
return '<div class="cell"><div class="cap"><span class="chip '+kind+'">'+kind+'</span></div>'+
'<img src="'+esc(src)+'" alt="'+kind+'"/></div>';
}
return '<div class="cell empty"><div class="cap"><span class="chip '+kind+'">'+kind+'</span>'+
'<span class="meta">not present</span></div><div class="ph">No '+kind+' screenshot for this state</div></div>';
}
var head = '<div class="doc-head"><h1>'+esc(D.title || ("PR #"+D.pr))+'</h1>'+
'<div class="sub">Before / after &nbsp;·&nbsp; base <code>'+esc(D.base)+'</code> → head <code>'+esc(D.head)+'</code>'+
(D.cropSelector ? ' &nbsp;·&nbsp; cropped to <code>'+esc(D.cropSelector)+'</code>' : '')+'</div></div>'+
'<div class="legend"><span class="chip before">Before</span> base branch'+
'<span class="chip after">After</span> this PR</div>';
root.appendChild(el('<div>'+head+'</div>'));
if (!D.tabs || !D.tabs.length){ root.appendChild(el('<div class="empty-doc">No tabs captured yet.</div>')); return; }
D.tabs.forEach(function(tab){
var states = (tab.states||[]).map(function(s){
var onlyOne = (!s.before || !s.after);
return '<div class="state'+(onlyOne?' ':'')+'">'+
'<div class="name">'+esc(s.name)+(s.note?'<span class="note">'+esc(s.note)+'</span>':'')+'</div>'+
'<div class="pair">'+cell("before", s.before)+cell("after", s.after)+'</div></div>';
}).join("");
var sec = '<section class="tab-section" id="section-'+esc(tab.id)+'">'+
'<h2>'+esc(tab.title)+'</h2>'+
(tab.ctx?'<div class="ctx">'+esc(tab.ctx)+'</div>':'')+
states+'</section>';
root.appendChild(el(sec));
});
})();
</script>
</body>
</html>
.claude/skills/ui-before-after/shoot-sections.mjs
+25
View File
@@ -0,0 +1,25 @@
// Render each .tab-section of a montage HTML into its own PNG (the PR-ready image).
// Run from frontend/editor (so @playwright/test resolves):
// node <skill>/shoot-sections.mjs <montage.html> <outDir>
import path from "node:path";
import { pathToFileURL } from "node:url";
import { createRequire } from "node:module";
const require = createRequire(path.join(process.cwd(), "noop.js"));
const { chromium } = require("@playwright/test");
const htmlPath = path.resolve(process.argv[2]);
const outDir = path.resolve(process.argv[3] || path.dirname(htmlPath));
const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1200, height: 1200 }, deviceScaleFactor: 2 });
await page.goto(pathToFileURL(htmlPath).href, { waitUntil: "load" });
await page.waitForTimeout(250); // let images/fonts paint
const ids = await page.$$eval(".tab-section", (els) => els.map((e) => e.id));
if (!ids.length) { console.error("no .tab-section found"); process.exit(1); }
for (const id of ids) {
const name = id.replace(/^section-/, "");
await page.locator("#" + id).screenshot({ path: path.join(outDir, `montage_${name}.png`) });
console.log("wrote montage_" + name + ".png");
}
await browser.close();
.claude/skills/ui-walkthrough/SKILL.md
+120
View File
@@ -0,0 +1,120 @@
---
name: ui-walkthrough
description: >-
Full UI investigation of the current branch's feature. Enumerates every view
and state (empty, populated, loading, error, each dialog/menu/panel, responsive
breakpoints, light + dark + RTL), captures them with the stubbed Playwright
harness, assembles a single-image HTML walkthrough with a global light/dark
toggle slider, then runs two review passes: visual/consistency (alignment,
spacing, professionalism, dark/light parity, contrast, truncation) and
UX/ease-of-use (flow, discoverability, affordances, empty/error states,
expectations). Use when asked for a UI walkthrough, screenshot review, design
or QA pass, "find anywhere to make it easier/better for users", or before
merging frontend work. Pass --fix to auto-apply safe frontend fixes and
re-capture; --theme to limit themes; --no-rtl to skip RTL.
argument-hint: "[feature/area] [--fix] [--theme light|dark|both] [--no-rtl] [--breakpoints]"
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
---
# UI Walkthrough
Produce a reviewable HTML walkthrough of a feature's UI in every state and theme,
then critique it. Optionally auto-fix and re-capture.
`$ARGUMENTS` may name the feature/area to focus on. If empty, scope from the
current branch diff. Flags: `--fix`, `--theme light|dark|both` (default both),
`--no-rtl`, `--breakpoints` (also capture phone/narrow widths).
## What this repo gives you (use it, don't reinvent)
- **Stubbed Playwright project** = backend-free screenshots via `page.route()` mocks.
Reference implementation: `frontend/editor/src/core/tests/stubbed/files-page-screenshots.spec.ts`.
It already shows the light / **dark** / **RTL** passes, JWT seeding, IndexedDB
seeding, and dumping PNGs to a `screenshots/<area>/` folder. Copy its shape.
- Helpers: `frontend/editor/src/core/tests/helpers/ui-helpers.ts`
(`uploadFiles`, `openSettings`, `waitForModalOpen`, `dismissTourTooltip`, …)
and the `stub-test-base` fixtures (`autoGoto`, `seedJwt`, `viewport`).
- Config: `frontend/editor/playwright.config.ts` (run from `frontend/editor/`).
- Report template: [report-template.html](report-template.html) - self-contained,
one big image at a time, a global light/dark slider that flips every shot,
thumbnail rail, prev/next + arrow keys, and a Findings tab.
## Process
### 1. Scope the feature
- If `$ARGUMENTS` is empty: `git diff --name-only main...HEAD` and read the PR/commits.
Identify changed pages, tools (`core/components/tools/<tool>` or `core/tools/<tool>`),
dialogs, panels, and routes.
- Enumerate **every view and state** to capture, e.g.:
empty / populated / loading / error / disabled; each dialog, menu, popover, tooltip;
each tab or step; selection + multi-select; success/result panel; and (if relevant)
permission/role variants. Write the list down before capturing - it's the report's spine.
### 2. Prepare the harness (worktree-safe)
Worktrees have no `node_modules` and no generated icons. From repo root:
```
cd frontend && npm ci # or junction main's node_modules (see memory)
cd frontend/editor && node scripts/generate-icons.js
```
Kill any stale dev server first (it serves old modules):
`Get-NetTCPConnection -LocalPort 5173 -State Listen | %{ Stop-Process -Id $_.OwningProcess -Force }`
### 3. Write the capture spec
Create `frontend/editor/src/core/tests/stubbed/<feature>-walkthrough.spec.ts`,
modeled on `files-page-screenshots.spec.ts`. For each enumerated view:
- stub the APIs it needs, drive the UI to that state, wait on a real locator
(not a fixed sleep), `await settle(page)` for Mantine portals, then
`page.screenshot({ path: shotPath("NN_name_<theme>") })`.
- Capture each view in **light and dark** (and RTL unless `--no-rtl`). Reuse the
`enableDarkMode` / `enableRtl` init-script pattern from the reference spec
(`localStorage["mantine-color-scheme"]="dark"` + `emulateMedia({colorScheme:"dark"})`).
- Name shots `NN_<view>_<theme>.png` so light/dark pair up by suffix.
- Prefer **stable test-ids** over translated accessible names (RTL/i18n breaks text locators).
Run it: `cd frontend/editor && npx playwright test --project=stubbed <feature>-walkthrough.spec.ts`.
Add `--project=stubbed-firefox`/`-webkit` only if cross-browser layout matters.
### 4. Build the report
- Copy `report-template.html` to `screenshots/<feature>/walkthrough.html` (so the
relative `screenshots/...` image paths resolve, or rewrite paths to sit beside it).
- Build the manifest and inject it: replace the JSON between the
`/*__DATA__*/` … `/*__END__*/` markers with one `views[]` entry per view
(`{id,title,light,dark,viewport,notes}`) and an empty `findings` object you'll
fill in step 5. Keep `light`/`dark` as relative paths.
- The toggle slider answers the "one big image + flip light/dark for all" request:
it shows a single large screenshot, and switching the slider re-themes every view.
### 5. Review pass 1 - visual & consistency
Open each screenshot (Read the PNG) and judge against the others:
alignment & spacing rhythm, control placement, button hierarchy, typography,
**light/dark parity** (contrast, invisible borders, washed-out text, wrong tokens),
truncation/overflow, RTL mirroring, focus states, icon consistency, professional polish.
Record each issue as a finding `{severity:high|med|low, view, title, detail, fix}`.
### 6. Review pass 2 - UX & ease of use
Walk the flow as a first-time user: discoverability, number of steps, affordance
clarity, empty-state guidance, error recovery, destructive-action confirmation,
defaults, loading feedback, mobile reachability, accessible names, and whether the
UI matches user expectations for this kind of tool. Record findings the same way.
Write both finding lists into the report's `findings.visual` / `findings.ux`,
and add short per-view `notes`. Re-inject the manifest.
### 7. If `--fix`
Only safe, self-contained frontend fixes (spacing, alignment, tokens, missing
dark-mode colors, labels, aria, obvious copy). For each: edit the component/CSS,
mark the finding `fixed:true` with what changed, then **re-run the spec** to
re-capture the affected shots and regenerate the report. Run `task frontend:check`.
Leave anything risky or ambiguous as a finding, not a change.
### 8. Deliver
Tell the user the report path and give a tight chat summary: N views ×
themes captured, top findings by severity, and (if `--fix`) what changed.
Optionally `SendUserFile` the `walkthrough.html`.
## Gotchas
- Stale `:5173` server serves old bundles - kill it before capturing (see step 2).
- Missing `material-symbols-icons.json` → blank app → every shot times out. Run
`generate-icons.js` first.
- `await settle(page)` before shots or portals/transitions tear mid-capture.
- Don't commit the generated `screenshots/` or the throwaway spec unless asked.
.claude/skills/ui-walkthrough/make_example.py
+116
View File
@@ -0,0 +1,116 @@
"""Build a self-contained EXAMPLE.html from report-template.html with mock
light/dark screenshots, so the viewer + global theme slider can be demoed
without a real capture run. Run: python make_example.py"""
import base64
import json
import pathlib
import re
HERE = pathlib.Path(__file__).parent
def svg(bg, fg, panel, accent, muted, label, kind):
"""A simple fake 'screen' SVG: title bar, sidebar, content varies by kind."""
parts = [
f'<svg xmlns="http://www.w3.org/2000/svg" width="1600" height="900" viewBox="0 0 1600 900">',
f'<rect width="1600" height="900" fill="{bg}"/>',
# top bar
f'<rect width="1600" height="64" fill="{panel}"/>',
f'<circle cx="40" cy="32" r="12" fill="{accent}"/>',
f'<rect x="64" y="24" width="160" height="16" rx="6" fill="{muted}"/>',
f'<rect x="1430" y="20" width="130" height="24" rx="12" fill="{accent}"/>',
# left sidebar
f'<rect x="0" y="64" width="220" height="836" fill="{panel}"/>',
]
for i in range(6):
y = 100 + i * 56
parts.append(f'<rect x="24" y="{y}" width="172" height="32" rx="8" fill="{bg}"/>')
if kind == "empty":
parts += [
f'<rect x="700" y="360" width="200" height="120" rx="16" fill="none" stroke="{muted}" stroke-width="3" stroke-dasharray="10 8"/>',
f'<rect x="690" y="510" width="220" height="44" rx="10" fill="{accent}"/>',
f'<text x="800" y="600" fill="{muted}" font-family="sans-serif" font-size="26" text-anchor="middle">{label}</text>',
]
elif kind == "form":
for i in range(4):
y = 140 + i * 90
parts.append(f'<rect x="280" y="{y}" width="160" height="16" rx="6" fill="{muted}"/>')
parts.append(f'<rect x="280" y="{y+26}" width="900" height="44" rx="8" fill="{panel}" stroke="{muted}" stroke-width="1"/>')
parts.append(f'<rect x="280" y="560" width="200" height="50" rx="10" fill="{accent}"/>')
parts.append(f'<text x="800" y="850" fill="{muted}" font-family="sans-serif" font-size="24" text-anchor="middle">{label}</text>')
else: # dialog
parts += [
f'<rect width="1600" height="900" fill="{fg}" opacity="0.45"/>',
f'<rect x="520" y="280" width="560" height="360" rx="18" fill="{panel}"/>',
f'<rect x="556" y="320" width="280" height="22" rx="8" fill="{fg}"/>',
f'<rect x="556" y="372" width="488" height="14" rx="6" fill="{muted}"/>',
f'<rect x="556" y="398" width="420" height="14" rx="6" fill="{muted}"/>',
f'<rect x="820" y="560" width="110" height="44" rx="9" fill="{bg}" stroke="{muted}"/>',
f'<rect x="946" y="560" width="98" height="44" rx="9" fill="{accent}"/>',
f'<text x="800" y="700" fill="#fff" font-family="sans-serif" font-size="24" text-anchor="middle">{label}</text>',
]
parts.append("</svg>")
return "".join(parts)
def data_uri(s):
return "data:image/svg+xml;base64," + base64.b64encode(s.encode()).decode()
LIGHT = dict(bg="#ffffff", fg="#111418", panel="#f1f3f6", accent="#2f6fed", muted="#c2c8d0")
DARK = dict(bg="#16181c", fg="#000000", panel="#1f232a", accent="#5b8cff", muted="#3a414b")
def pair(kind, label):
return (
data_uri(svg(LIGHT["bg"], LIGHT["fg"], LIGHT["panel"], LIGHT["accent"], LIGHT["muted"], label, kind)),
data_uri(svg(DARK["bg"], DARK["fg"], DARK["panel"], DARK["accent"], DARK["muted"], label, kind)),
)
views = []
for idx, (kind, title, label) in enumerate([
("empty", "Empty state", "Drop a PDF to start"),
("form", "Tool options panel", "Compress options"),
("dialog", "Confirm dialog", "Replace original file?"),
], start=1):
light, dark = pair(kind, label)
views.append({
"id": f"{idx:02d}_{kind}",
"title": title,
"light": light,
"dark": dark,
"viewport": "1600x900",
"notes": ["This is mock data to demo the viewer."],
})
data = {
"feature": "EXAMPLE - Compress PDF (mock data)",
"branch": "demo",
"generated": "example",
"views": views,
"findings": {
"visual": [
{"severity": "high", "view": "03_dialog", "title": "Dialog buttons too close",
"detail": "Cancel/Confirm have only 8px gap; easy to misclick.",
"fix": "Increase gap to var(--mantine-spacing-md)."},
{"severity": "low", "view": "02_form", "title": "Field labels low contrast in dark mode",
"detail": "Muted token fails WCAG AA on the dark panel.",
"fix": "Use --mantine-color-dimmed instead of a hard-coded grey."},
],
"ux": [
{"severity": "med", "view": "01_empty", "title": "Primary CTA below the dropzone",
"detail": "Users expect the action button adjacent to the dropzone.",
"fix": "Move the button directly under the dashed zone."},
],
},
}
tpl = (HERE / "report-template.html").read_text(encoding="utf-8")
out = re.sub(
r"/\*__DATA__\*/.*?/\*__END__\*/",
lambda _m: "/*__DATA__*/" + json.dumps(data) + "/*__END__*/",
tpl, count=1, flags=re.S,
)
(HERE / "EXAMPLE.html").write_text(out, encoding="utf-8")
print("wrote", (HERE / "EXAMPLE.html"))
.claude/skills/ui-walkthrough/report-template.html
+298
View File
@@ -0,0 +1,298 @@
<!doctype html>
<!--
UI Walkthrough report template (self-contained, works from file://).
The ui-walkthrough skill replaces the JSON in the window.__WALKTHROUGH__ data
block below with the captured manifest. Do not add external CDN deps - it must open offline.
Data shape:
{
"feature": "Compress PDF tool",
"branch": "claude/...",
"generated": "2026-06-21",
"views": [
{ "id": "01_empty", "title": "Empty state",
"light": "screenshots/compress/01_empty_light.png",
"dark": "screenshots/compress/01_empty_dark.png",
"viewport": "1600x900",
"notes": ["Heading is centered", "Primary CTA below the fold on mobile"] }
],
"findings": {
"visual": [ { "severity":"high", "view":"01_empty", "title":"...", "detail":"...", "fix":"..." } ],
"ux": [ { "severity":"med", "view":"03_dialog", "title":"...", "detail":"...", "fix":"..." } ]
}
}
-->
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>UI Walkthrough</title>
<style>
:root {
--bg: #f6f7f9; --panel: #ffffff; --panel-2: #f0f2f5; --text: #1a1b1e;
--muted: #6b7280; --border: #e2e5ea; --accent: #2f6fed; --accent-weak: #e8f0fe;
--shadow: 0 1px 3px rgba(0,0,0,.08), 0 8px 24px rgba(0,0,0,.06);
--hi: #d92d20; --med: #d98e00; --low: #2f6fed; --stage: #0b0c0e;
}
html[data-theme="dark"] {
--bg: #0d0e10; --panel: #16181c; --panel-2: #1d2024; --text: #e6e8eb;
--muted: #9aa3ad; --border: #2a2e35; --accent: #5b8cff; --accent-weak: #1a2336;
--shadow: 0 1px 3px rgba(0,0,0,.5), 0 8px 24px rgba(0,0,0,.4); --stage: #000;
}
* { box-sizing: border-box; }
body { margin: 0; font: 14px/1.5 -apple-system, "Segoe UI", Roboto, system-ui, sans-serif;
background: var(--bg); color: var(--text); }
header { display: flex; align-items: center; gap: 16px; padding: 12px 20px;
background: var(--panel); border-bottom: 1px solid var(--border); position: sticky; top: 0; z-index: 5; }
header h1 { font-size: 15px; margin: 0; font-weight: 650; }
header .sub { color: var(--muted); font-size: 12px; }
.spacer { flex: 1; }
.counter { color: var(--muted); font-variant-numeric: tabular-nums; font-size: 13px; }
.tabs { display: flex; gap: 4px; }
.tab { border: 1px solid var(--border); background: var(--panel-2); color: var(--text);
padding: 6px 12px; border-radius: 8px; cursor: pointer; font-size: 13px; }
.tab.active { background: var(--accent); color: #fff; border-color: var(--accent); }
/* Light/Dark slider */
.theme-toggle { display: flex; align-items: center; gap: 9px; user-select: none; }
.theme-toggle .lbl { font-size: 12px; color: var(--muted); }
.theme-toggle .lbl.on { color: var(--text); font-weight: 600; }
.switch { position: relative; width: 52px; height: 28px; }
.switch input { opacity: 0; width: 0; height: 0; }
.slider { position: absolute; inset: 0; cursor: pointer; background: var(--panel-2);
border: 1px solid var(--border); border-radius: 999px; transition: .2s; }
.slider:before { content: ""; position: absolute; height: 20px; width: 20px; left: 3px; top: 3px;
background: #fbbf24; border-radius: 50%; transition: .2s; box-shadow: 0 1px 2px rgba(0,0,0,.3); }
.switch input:checked + .slider { background: var(--accent); }
.switch input:checked + .slider:before { transform: translateX(24px); background: #c7d2fe; }
main { display: grid; grid-template-columns: 240px 1fr; height: calc(100vh - 53px); }
.rail { border-right: 1px solid var(--border); overflow-y: auto; background: var(--panel); padding: 8px; }
.rail .group-label { font-size: 11px; text-transform: uppercase; letter-spacing: .05em;
color: var(--muted); padding: 10px 8px 4px; }
.thumb { display: flex; gap: 9px; align-items: center; padding: 7px; border-radius: 8px;
cursor: pointer; border: 1px solid transparent; }
.thumb:hover { background: var(--panel-2); }
.thumb.active { background: var(--accent-weak); border-color: var(--accent); }
.thumb img { width: 64px; height: 40px; object-fit: cover; border-radius: 4px; border: 1px solid var(--border); background: var(--stage); }
.thumb .t { font-size: 12.5px; line-height: 1.3; }
.thumb .badge { font-size: 10px; color: var(--muted); }
.thumb .dot { width: 7px; height: 7px; border-radius: 50%; margin-left: auto; flex: none; }
.stagewrap { display: flex; flex-direction: column; min-width: 0; }
.stage { flex: 1; display: flex; align-items: center; justify-content: center; padding: 22px;
background: var(--stage); position: relative; min-height: 0; }
.stage img { max-width: 100%; max-height: 100%; object-fit: contain; border-radius: 8px;
box-shadow: 0 4px 30px rgba(0,0,0,.4); background: #fff; }
html[data-theme="dark"] .stage img { background: #16181c; }
.nav-btn { position: absolute; top: 50%; transform: translateY(-50%); width: 42px; height: 42px;
border-radius: 50%; border: 1px solid var(--border); background: var(--panel);
color: var(--text); cursor: pointer; font-size: 18px; opacity: .85; }
.nav-btn:hover { opacity: 1; } .nav-btn.prev { left: 16px; } .nav-btn.next { right: 16px; }
.nav-btn:disabled { opacity: .25; cursor: default; }
.missing { color: var(--muted); font-size: 13px; text-align: center; }
.detail { border-top: 1px solid var(--border); background: var(--panel); padding: 14px 20px;
max-height: 38vh; overflow-y: auto; }
.detail h2 { margin: 0 0 4px; font-size: 15px; }
.detail .meta { color: var(--muted); font-size: 12px; margin-bottom: 10px; }
.notes { list-style: none; padding: 0; margin: 0; display: grid; gap: 6px; }
.notes li { display: flex; gap: 8px; align-items: flex-start; }
.sev { font-size: 10px; font-weight: 700; text-transform: uppercase; padding: 2px 7px; border-radius: 999px;
color: #fff; flex: none; margin-top: 1px; }
.sev.high { background: var(--hi); } .sev.med { background: var(--med); } .sev.low { background: var(--low); }
.finding .fix { color: var(--muted); font-size: 12.5px; }
.finding .fix b { color: var(--text); font-weight: 600; }
/* Summary tab */
.summary { padding: 20px 28px; overflow-y: auto; }
.summary h2 { font-size: 16px; margin: 22px 0 8px; }
.summary .empty { color: var(--muted); }
.card { background: var(--panel); border: 1px solid var(--border); border-radius: 10px;
padding: 12px 14px; margin-bottom: 8px; box-shadow: var(--shadow); }
.card .head { display: flex; gap: 8px; align-items: center; }
.card a { color: var(--accent); text-decoration: none; cursor: pointer; }
.hide { display: none !important; }
kbd { font: 11px ui-monospace, monospace; background: var(--panel-2); border: 1px solid var(--border);
border-radius: 4px; padding: 1px 5px; }
</style>
</head>
<body>
<header>
<div>
<h1 id="feature-title">UI Walkthrough</h1>
<div class="sub" id="feature-sub"></div>
</div>
<div class="spacer"></div>
<div class="tabs">
<button class="tab active" data-tab="viewer">Walkthrough</button>
<button class="tab" data-tab="summary">Findings</button>
</div>
<div class="counter" id="counter"></div>
<label class="theme-toggle" title="Toggle light / dark for every screenshot">
<span class="lbl" id="lbl-light">Light</span>
<span class="switch"><input type="checkbox" id="theme-switch" /><span class="slider"></span></span>
<span class="lbl" id="lbl-dark">Dark</span>
</label>
</header>
<main id="viewer-pane">
<aside class="rail" id="rail"></aside>
<section class="stagewrap">
<div class="stage">
<button class="nav-btn prev" id="prev" aria-label="Previous">&#8249;</button>
<img id="stage-img" alt="" />
<div class="missing hide" id="missing"></div>
<button class="nav-btn next" id="next" aria-label="Next">&#8250;</button>
</div>
<div class="detail">
<h2 id="view-title"></h2>
<div class="meta" id="view-meta"></div>
<ul class="notes" id="view-notes"></ul>
</div>
</section>
</main>
<section class="summary hide" id="summary-pane"></section>
<script id="data">
window.__WALKTHROUGH__ = /*__DATA__*/{"feature":"No data","branch":"","generated":"","views":[],"findings":{"visual":[],"ux":[]}}/*__END__*/;
</script>
<script>
(function () {
var D = window.__WALKTHROUGH__ || { views: [], findings: { visual: [], ux: [] } };
var views = D.views || [];
var state = { i: 0, theme: localStorage.getItem("ui-wt-theme") || "light", tab: "viewer" };
var $ = function (id) { return document.getElementById(id); };
function sevClass(s) { return s === "high" ? "high" : s === "med" || s === "medium" ? "med" : "low"; }
function applyChrome() {
document.documentElement.setAttribute("data-theme", state.theme);
$("theme-switch").checked = state.theme === "dark";
$("lbl-light").classList.toggle("on", state.theme === "light");
$("lbl-dark").classList.toggle("on", state.theme === "dark");
}
function srcFor(v) { return state.theme === "dark" ? (v.dark || v.light) : (v.light || v.dark); }
function findingsForView(id) {
var all = (D.findings && D.findings.visual || []).concat(D.findings && D.findings.ux || []);
return all.filter(function (f) { return f.view === id; });
}
function renderRail() {
var rail = $("rail");
rail.innerHTML = "";
if (!views.length) { rail.innerHTML = '<div class="group-label">No views captured</div>'; return; }
views.forEach(function (v, idx) {
var fs = findingsForView(v.id);
var worst = fs.some(function (f){return sevClass(f.severity)==="high";}) ? "var(--hi)"
: fs.some(function (f){return sevClass(f.severity)==="med";}) ? "var(--med)"
: fs.length ? "var(--low)" : "transparent";
var el = document.createElement("div");
el.className = "thumb" + (idx === state.i ? " active" : "");
el.innerHTML = '<img src="' + srcFor(v) + '" alt="" />' +
'<div><div class="t">' + (v.title || v.id) + '</div>' +
'<div class="badge">' + (v.viewport || "") + '</div></div>' +
'<span class="dot" style="background:' + worst + '"></span>';
el.onclick = function () { state.i = idx; render(); };
rail.appendChild(el);
});
}
function render() {
applyChrome();
if (!views.length) {
$("missing").classList.remove("hide"); $("stage-img").classList.add("hide");
$("missing").textContent = "No screenshots in this report yet.";
$("counter").textContent = ""; return;
}
var v = views[state.i];
var src = srcFor(v);
var img = $("stage-img");
if (src) {
img.classList.remove("hide"); $("missing").classList.add("hide");
img.src = src; img.alt = v.title || v.id;
} else {
img.classList.add("hide"); $("missing").classList.remove("hide");
$("missing").textContent = "No " + state.theme + " screenshot for this view.";
}
$("counter").textContent = (state.i + 1) + " / " + views.length;
$("view-title").textContent = v.title || v.id;
$("view-meta").textContent = [v.viewport, state.theme + " mode"].filter(Boolean).join(" · ");
var notes = $("view-notes"); notes.innerHTML = "";
var fs = findingsForView(v.id);
(v.notes || []).forEach(function (n) {
var li = document.createElement("li"); li.textContent = "· " + n; notes.appendChild(li);
});
fs.forEach(function (f) {
var li = document.createElement("li"); li.className = "finding";
li.innerHTML = '<span class="sev ' + sevClass(f.severity) + '">' + (f.severity || "note") + '</span>' +
'<span><b>' + (f.title || "") + '</b> — ' + (f.detail || "") +
(f.fix ? ' <span class="fix"><b>Fix:</b> ' + f.fix + '</span>' : '') + '</span>';
notes.appendChild(li);
});
$("prev").disabled = state.i === 0;
$("next").disabled = state.i === views.length - 1;
renderRail();
}
function renderSummary() {
var pane = $("summary-pane");
function block(title, arr) {
var h = '<h2>' + title + ' (' + arr.length + ')</h2>';
if (!arr.length) return h + '<div class="empty">None found.</div>';
return h + arr.map(function (f) {
return '<div class="card"><div class="head">' +
'<span class="sev ' + sevClass(f.severity) + '">' + (f.severity || "note") + '</span>' +
'<b>' + (f.title || "") + '</b>' +
(f.view ? ' <a data-jump="' + f.view + '">' + f.view + '</a>' : '') + '</div>' +
'<div style="margin-top:6px">' + (f.detail || "") + '</div>' +
(f.fix ? '<div class="finding" style="margin-top:6px"><span class="fix"><b>Fix:</b> ' + f.fix + '</span></div>' : '') +
'</div>';
}).join("");
}
pane.innerHTML = block("Visual & consistency", (D.findings && D.findings.visual) || []) +
block("UX & ease of use", (D.findings && D.findings.ux) || []);
pane.querySelectorAll("[data-jump]").forEach(function (a) {
a.onclick = function () {
var id = a.getAttribute("data-jump");
var idx = views.findIndex(function (v) { return v.id === id; });
if (idx >= 0) { state.i = idx; setTab("viewer"); }
};
});
}
function setTab(t) {
state.tab = t;
document.querySelectorAll(".tab").forEach(function (b) { b.classList.toggle("active", b.dataset.tab === t); });
$("viewer-pane").classList.toggle("hide", t !== "viewer");
$("summary-pane").classList.toggle("hide", t !== "summary");
if (t === "viewer") $("viewer-pane").style.display = "grid";
if (t === "summary") renderSummary();
}
// wiring
$("feature-title").textContent = D.feature || "UI Walkthrough";
$("feature-sub").textContent = [D.branch, D.generated].filter(Boolean).join(" · ");
$("theme-switch").onchange = function () {
state.theme = this.checked ? "dark" : "light";
localStorage.setItem("ui-wt-theme", state.theme);
render();
};
$("prev").onclick = function () { if (state.i > 0) { state.i--; render(); } };
$("next").onclick = function () { if (state.i < views.length - 1) { state.i++; render(); } };
document.addEventListener("keydown", function (e) {
if (state.tab !== "viewer") return;
if (e.key === "ArrowLeft") $("prev").click();
if (e.key === "ArrowRight") $("next").click();
if (e.key.toLowerCase() === "t") $("theme-switch").click();
});
document.querySelectorAll(".tab").forEach(function (b) { b.onclick = function () { setTab(b.dataset.tab); }; });
render();
})();
</script>
</body>
</html>
.github/aur/stirling-pdf-desktop/PKGBUILD
+1 -1
View File
@@ -1,6 +1,6 @@
# Maintainer: Stirling PDF Inc <contact@stirlingpdf.com>
pkgname=stirling-pdf-desktop
pkgver=2.14.0
pkgver=2.14.1
pkgrel=1
pkgdesc="Locally hosted, web-based PDF manipulation tool (Tauri desktop app, official Stirling PDF Inc build)"
arch=('x86_64')
.github/aur/stirling-pdf-server-bin/PKGBUILD
+1 -1
View File
@@ -1,6 +1,6 @@
# Maintainer: Stirling PDF Inc <contact@stirlingpdf.com>
pkgname=stirling-pdf-server-bin
pkgver=2.14.0
pkgver=2.14.1
pkgrel=1
pkgdesc="Locally hosted, web-based PDF manipulation tool (server JAR, prebuilt)"
arch=('any')
.github/config/.files.yaml
+15
View File
@@ -87,6 +87,21 @@ engine: &engine
- Taskfile.yml
- .taskfiles/engine.yml
# Files that can make the committed generated API models (frontend tool API
# types + engine tool models) go stale: the Java tool surfaces they derive from,
# the generators, the generated files themselves (to catch a hand-edit), and the
# tasks that drive generation. Deliberately excludes the broad frontend/docker/
# testing globs, so a CSS-only PR does not boot the backend to rebuild the spec.
generated-models: &generated-models
- *openapi
- frontend/editor/scripts/generate-tool-api-types.mts
- frontend/editor/src/core/types/toolApiTypes.ts
- engine/scripts/generate_tool_models.py
- engine/src/stirling/models/tool_models.py
- .taskfiles/frontend.yml
- .taskfiles/engine.yml
- .github/workflows/check-generated-models.yml
licenses-frontend: &licenses-frontend
- ".github/workflows/frontend-backend-licenses-update.yml"
- "frontend/package.json"
.github/workflows/PR-Auto-Deploy-V2.yml
+108 -6
View File
@@ -116,6 +116,9 @@ jobs:
env:
USE_DEPOT: ${{ needs.pick.outputs.is_fork != 'true' }}
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
# Single source of truth for whether this preview embeds the admin portal:
# drives the image build-arg and the deployment comment.
BUILD_PORTAL: "true"
steps:
- name: Harden Runner
@@ -246,7 +249,9 @@ jobs:
file: ./docker/embedded/Dockerfile
push: true
tags: ${{ secrets.DOCKER_HUB_USERNAME }}/test:v2-${{ steps.commit-hash.outputs.app_short }}
build-args: VERSION_TAG=v2-alpha
build-args: |
VERSION_TAG=v2-alpha
BUILD_PORTAL=${{ env.BUILD_PORTAL }}
platforms: linux/amd64
- name: Build and push V2 image (Docker fork fallback)
@@ -259,7 +264,9 @@ jobs:
cache-from: type=gha,scope=stirling-pdf-latest
cache-to: type=gha,mode=max,scope=stirling-pdf-latest
tags: ${{ secrets.DOCKER_HUB_USERNAME }}/test:v2-${{ steps.commit-hash.outputs.app_short }}
build-args: VERSION_TAG=v2-alpha
build-args: |
VERSION_TAG=v2-alpha
BUILD_PORTAL=${{ env.BUILD_PORTAL }}
platforms: linux/amd64
- name: Set up SSH
@@ -290,6 +297,8 @@ jobs:
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/storage:/storage:rw
environment:
DISABLE_ADDITIONAL_FEATURES: "false"
POLICIES_ENABLED: "true"
STIRLING_BILLING_ACCOUNT_LINK_ENABLED: "true"
SECURITY_ENABLELOGIN: "true"
SECURITY_INITIALLOGIN_USERNAME: "${{ secrets.TEST_LOGIN_USERNAME }}"
SECURITY_INITIALLOGIN_PASSWORD: "${{ secrets.TEST_LOGIN_PASSWORD }}"
@@ -333,9 +342,70 @@ jobs:
# Set port for output
echo "v2_port=${V2_PORT}" >> $GITHUB_OUTPUT
# ---- Storybook preview (only when this PR touches stories/.storybook) ----
# Runs inside the same approved-contributor-gated deploy job, so it deploys
# under the exact same access rules as the app preview.
- name: Detect Storybook changes
id: sb-changes
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
with:
list-files: json
filters: |
storybook:
- 'frontend/**/*.stories.@(ts|tsx|mdx)'
- 'frontend/**/*.mdx'
- 'frontend/.storybook/**'
- name: Set up Node.js for Storybook
if: steps.sb-changes.outputs.storybook == 'true'
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "22"
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install Task for Storybook
if: steps.sb-changes.outputs.storybook == 'true'
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
- name: Build and deploy Storybook
id: storybook
if: steps.sb-changes.outputs.storybook == 'true'
env:
VPS_HOST: ${{ secrets.NEW_VPS_HOST }}
VPS_USER: ${{ secrets.NEW_VPS_USERNAME }}
run: |
set -euo pipefail
# `prepare` generates the icon set stories import (not committed).
task frontend:prepare
task frontend:storybook:build
PR=${{ needs.check-pr.outputs.pr_number }}
# Served at the ROOT of its own port so Storybook's global MSW worker
# (/mockServiceWorker.js) resolves. Port = PR + 20000 (bijective, offset
# from the app preview's bare-PR-number port).
SB_PORT=$((PR + 20000))
DIR=/stirling/SB-PR-$PR
tar czf storybook.tgz -C frontend/storybook-static .
scp -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
storybook.tgz "$VPS_USER@$VPS_HOST:/tmp/storybook-$PR.tgz"
ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -T \
"$VPS_USER@$VPS_HOST" << ENDSSH
set -e
rm -rf "$DIR" && mkdir -p "$DIR"
tar xzf /tmp/storybook-$PR.tgz -C "$DIR"
rm -f /tmp/storybook-$PR.tgz
docker rm -f storybook-pr-$PR 2>/dev/null || true
docker run -d --name storybook-pr-$PR --restart unless-stopped \
-p $SB_PORT:80 -v "$DIR":/usr/share/nginx/html:ro nginx:alpine
ENDSSH
echo "url=http://$VPS_HOST:$SB_PORT/" >> "$GITHUB_OUTPUT"
- name: Post V2 deployment URL to PR
if: success()
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
SB_URL: ${{ steps.storybook.outputs.url }}
SB_FILES: ${{ steps.sb-changes.outputs.storybook_files }}
with:
github-token: ${{ steps.setup-bot.outputs.token }}
script: |
@@ -359,12 +429,40 @@ jobs:
}
const deploymentUrl = `http://${{ secrets.NEW_VPS_HOST }}:${v2Port}`;
const httpsUrl = `https://${v2Port}.ssl.stirlingpdf.cloud`;
// Only mention the portal when this image actually embeds it.
// Use the direct IP URL - the SSL hostname isn't supported yet.
const withPortal = "${{ env.BUILD_PORTAL }}" === "true";
const portalNote = withPortal
? `🧩 **Admin portal** included - try it at [${deploymentUrl}/portal](${deploymentUrl}/portal).\n\n`
: ``;
// Storybook preview: only present when this PR changed stories/config.
const sbUrl = process.env.SB_URL;
let storybookNote = "";
if (sbUrl) {
const files = JSON.parse(process.env.SB_FILES || "[]");
const stories = files.filter((f) => /\.stories\.(ts|tsx|mdx)$/.test(f));
const config = files.filter((f) => f.startsWith("frontend/.storybook/"));
const shorten = (f) =>
f.replace(/^frontend\/editor\/src\//, "").replace(/^frontend\//, "");
const storyList = stories.map((f) => `- \`${shorten(f)}\``).join("\n");
const configList = config.map((f) => `- \`${shorten(f)}\``).join("\n");
const summary =
`${stories.length} stor${stories.length === 1 ? "y" : "ies"} changed` +
(config.length ? ` (+${config.length} config file${config.length === 1 ? "" : "s"})` : "");
storybookNote =
`📚 **Storybook:** [${sbUrl}](${sbUrl})\n\n` +
`<details>\n<summary>${summary}</summary>\n\n` +
(storyList ? `**Stories**\n${storyList}\n\n` : "") +
(configList ? `**Config**\n${configList}\n` : "") +
`</details>\n\n`;
}
const commentBody = `## 🚀 V2 Auto-Deployment Complete!\n\n` +
`Your V2 PR with embedded architecture has been deployed!\n\n` +
`🔗 **Direct Test URL (non-SSL)** [${deploymentUrl}](${deploymentUrl})\n\n` +
`🔐 **Secure HTTPS URL**: [${httpsUrl}](${httpsUrl})\n\n` +
portalNote +
storybookNote +
`_This deployment will be automatically cleaned up when the PR is closed._\n\n` +
`🔄 **Auto-deployed** for approved V2 contributors.`;
@@ -460,7 +558,11 @@ jobs:
else
echo "V2 PR directory not found, nothing to clean up"
fi
# Remove this PR's Storybook preview (container + files), if any.
docker rm -f storybook-pr-${{ github.event.pull_request.number }} 2>/dev/null || true
rm -rf /stirling/SB-PR-${{ github.event.pull_request.number }}
# Clean up old unused images (older than 2 weeks) but keep recent ones for reuse
docker image prune -af --filter "until=336h" --filter "label!=keep=true" || true
.github/workflows/ai-engine.yml
+4 -99
View File
@@ -1,9 +1,9 @@
name: AI Engine CI
# Validates the Python AI engine: regenerates tool models and runs the
# engine quality gate (lint, type-check, format-check, tests). Called from
# build.yml on PRs and merge_group; also runs directly on push to main as
# a post-merge safety net.
# Runs the engine quality gate (lint, type-check, format-check, tests). Called
# from build.yml on PRs and merge_group; also runs directly on push to main as
# a post-merge safety net. Freshness of the generated tool_models.py is checked
# by the shared check-generated-models workflow.
on:
workflow_call:
push:
@@ -34,104 +34,9 @@ jobs:
with:
enable-cache: true
- name: Set up JDK 25
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
java-version: "25"
distribution: "temurin"
- name: Setup Gradle
uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6.1.0
with:
gradle-version: 9.6.0
- name: Install Task
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
- name: Regenerate tool models
run: task engine:tool-models
- name: Verify tool models are up to date
id: tool-models-check
continue-on-error: true
run: git diff --exit-code engine/src/stirling/models/tool_models.py
- name: Comment on tool models check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.tool-models-check.outcome == 'failure' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- tool-models-check -->';
const body = [
marker,
'### Tool Models Check Failed',
'',
'The generated `engine/src/stirling/models/tool_models.py` is out of date with the Java OpenAPI spec and will need to be regenerated before it can be merged in.',
'',
'Run `task engine:tool-models` to regenerate, then commit the updated file.',
].join('\n');
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});
}
- name: Fail if tool models check failed
if: steps.tool-models-check.outcome == 'failure'
run: |
echo "============================================"
echo " Tool Models Check Failed"
echo "============================================"
echo ""
echo "The generated engine/src/stirling/models/tool_models.py"
echo "is out of date with the Java OpenAPI spec and will"
echo "need to be regenerated before it can be merged in."
echo ""
echo "Run 'task engine:tool-models' to regenerate, then"
echo "commit the updated file."
echo "============================================"
exit 1
- name: Remove tool models check comment on success
if: steps.tool-models-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- tool-models-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
- name: Quality-check engine
id: engine-check
run: task engine:check
.github/workflows/build-enterprise.yml
+19
View File
@@ -167,6 +167,8 @@ jobs:
wait_for_backend
- name: Run enterprise OAuth Playwright tests
id: oauth-tests
env:
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results-oauth.json
run: task e2e:enterprise -- --grep "OAuth"
- name: Stop backend + tear down OAuth Keycloak
if: always()
@@ -240,6 +242,8 @@ jobs:
wait_for_backend
- name: Run enterprise SAML Playwright tests
id: saml-tests
env:
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results-saml.json
run: task e2e:enterprise -- --grep "SAML"
- name: Stop backend + tear down SAML Keycloak
if: always()
@@ -270,6 +274,8 @@ jobs:
wait_for_backend
- name: Run enterprise feature Playwright tests
id: feature-tests
env:
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results-feature.json
run: task e2e:enterprise -- --grep "Enterprise license"
- name: Print backend log on failure
if: failure()
@@ -282,6 +288,19 @@ jobs:
run: |
source /tmp/helpers.sh
stop_backend
- name: Flag flaky tests
# Runs regardless of the test outcomes: a flaky test (passed on retry)
# leaves its step green, so this is the only place it surfaces. Merges
# all three phase reports (some may be absent if an earlier phase hard-
# failed and skipped the rest). Emits ::warning:: annotations + a job
# summary; never fails the job.
if: always()
working-directory: frontend
run: >
npx tsx editor/scripts/report-flaky-tests.mts
"${{ github.workspace }}/frontend/playwright-report/results-oauth.json"
"${{ github.workspace }}/frontend/playwright-report/results-saml.json"
"${{ github.workspace }}/frontend/playwright-report/results-feature.json"
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
.github/workflows/build.yml
+17
View File
@@ -43,6 +43,7 @@ jobs:
docker-base: ${{ steps.changes.outputs.docker-base }}
tauri: ${{ steps.changes.outputs.tauri }}
engine: ${{ steps.changes.outputs.engine }}
generated-models: ${{ steps.changes.outputs.generated-models }}
proprietary: ${{ steps.changes.outputs.proprietary }}
steps:
- name: Harden the runner (Audit all outbound calls)
@@ -171,6 +172,20 @@ jobs:
uses: ./.github/workflows/ai-engine.yml
secrets: inherit
# The generated frontend types and engine tool models are both derived from
# the Java OpenAPI spec. This job regenerates and diffs them; it boots the
# backend, so it is gated on the narrow generated-models filter (spec source,
# generators, generated files, generation tasks) rather than the broad
# frontend filter, so a CSS-only PR does not pay for a backend build.
generated-models:
if: needs.files-changed.outputs.generated-models == 'true'
needs: [files-changed]
permissions:
contents: read
pull-requests: write
uses: ./.github/workflows/check-generated-models.yml
secrets: inherit
pre-commit:
needs: [files-changed]
permissions:
@@ -228,6 +243,7 @@ jobs:
- test-build-docker-images
- tauri-build
- ai-engine
- generated-models
- pre-commit
- dependency-review
runs-on: ubuntu-latest
@@ -253,6 +269,7 @@ jobs:
test-build-docker-images=${{ needs.test-build-docker-images.result }}
tauri-build=${{ needs.tauri-build.result }}
ai-engine=${{ needs.ai-engine.result }}
generated-models=${{ needs.generated-models.result }}
pre-commit=${{ needs.pre-commit.result }}
dependency-review=${{ needs.dependency-review.result }}
run: |
.github/workflows/check-generated-models.yml
+148
View File
@@ -0,0 +1,148 @@
name: Check generated models
# Verifies the committed generated API models are still in sync with the Java
# OpenAPI spec: the frontend tool API types
# (frontend/editor/src/core/types/toolApiTypes.ts) and the engine tool
# models (engine/src/stirling/models/tool_models.py). Regenerates both with the
# single top-level `task tool-models` and fails if either committed file is
# out of date. Called from build.yml when the backend Java, frontend, or engine
# changes; also runs on push to main as a post-merge safety net.
on:
workflow_call:
push:
branches: [main]
permissions:
contents: read
jobs:
generated-models:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
env:
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true
- name: Set up JDK 25
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
java-version: "25"
distribution: "temurin"
- name: Setup Gradle
uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6.1.0
with:
gradle-version: 9.6.0
- name: Set up Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "22"
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install Task
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
# Rebuilds the OpenAPI spec from the current Java and regenerates both the
# frontend types and the engine tool models from it.
- name: Regenerate generated models
run: task tool-models
- name: Verify generated models are up to date
id: models-check
continue-on-error: true
run: |
git diff --exit-code \
frontend/editor/src/core/types/toolApiTypes.ts \
engine/src/stirling/models/tool_models.py
- name: Comment on generated models check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.models-check.outcome == 'failure' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- generated-models-check -->';
const body = [
marker,
'### Generated Models Check Failed',
'',
'The generated `frontend/editor/src/core/types/toolApiTypes.ts` and/or `engine/src/stirling/models/tool_models.py` are out of date with the Java OpenAPI spec and will need to be regenerated before they can be merged in.',
'',
'Run `task tool-models` to regenerate both, then commit the updated files.',
].join('\n');
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});
}
- name: Fail if generated models check failed
if: steps.models-check.outcome == 'failure'
run: |
echo "============================================"
echo " Generated Models Check Failed"
echo "============================================"
echo ""
echo "The generated frontend API types and/or engine tool"
echo "models are out of date with the Java OpenAPI spec and"
echo "will need to be regenerated before they can be merged in."
echo ""
echo "Run 'task tool-models' to regenerate both, then"
echo "commit the updated files."
echo "============================================"
exit 1
- name: Remove generated models check comment on success
if: steps.models-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- generated-models-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
.github/workflows/e2e-live.yml
+10
View File
@@ -62,7 +62,17 @@ jobs:
# .test-state/playwright/coverage-pw/ for the post-process step
# to aggregate. Chromium-only - other engines silently skip.
PW_COVERAGE: "1"
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results.json
run: task e2e:live
- name: Flag flaky tests
# Runs regardless of the test outcome: a flaky test (passed on retry)
# leaves the step green, so this is the only place it surfaces. Emits
# ::warning:: annotations + a job summary; never fails the job.
if: always()
working-directory: frontend
run: npx tsx editor/scripts/report-flaky-tests.mts "$PLAYWRIGHT_JSON_OUTPUT_FILE"
env:
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results.json
- name: Generate JaCoCo report from e2e:live .exec
if: always()
id: live-coverage
.github/workflows/e2e-stubbed.yml
+11
View File
@@ -44,7 +44,18 @@ jobs:
VITE_BUILD_FOR_PREVIEW: "1"
run: task frontend:build
- name: Run stubbed E2E tests (chromium)
env:
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results.json
run: task e2e:stubbed -- --workers=3
- name: Flag flaky tests
# Runs regardless of the test outcome: a flaky test (passed on retry)
# leaves the step green, so this is the only place it surfaces. Emits
# ::warning:: annotations + a job summary; never fails the job.
if: always()
working-directory: frontend
run: npx tsx editor/scripts/report-flaky-tests.mts "$PLAYWRIGHT_JSON_OUTPUT_FILE"
env:
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ github.workspace }}/frontend/playwright-report/results.json
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
.taskfiles/frontend.yml
+17
View File
@@ -396,6 +396,23 @@ tasks:
# Code Generation
# ============================================================
tool-models:
desc: "Generate tool API types from the Java OpenAPI spec"
deps: [install, ":backend:swagger"]
cmds:
- npx tsx editor/scripts/generate-tool-api-types.mts --spec ../SwaggerDoc.json --output editor/src/core/types/toolApiTypes.ts
sources:
- editor/scripts/generate-tool-api-types.mts
- ../SwaggerDoc.json
generates:
- editor/src/core/types/toolApiTypes.ts
tool-models:check:
desc: "Fail if committed tool API types are out of date"
deps: [install, ":backend:swagger"]
cmds:
- npx tsx editor/scripts/generate-tool-api-types.mts --spec ../SwaggerDoc.json --output editor/src/core/types/toolApiTypes.ts --check
licenses:generate:
desc: "Generate frontend license report"
deps: [install]
AGENTS.md
+1
View File
@@ -453,6 +453,7 @@ The frontend is organized with a clear separation of concerns:
- **CRITICAL**: Always update translations in `en-US` only - all other languages (including `en-GB`) are handled separately
- Translation files are located in `frontend/editor/public/locales/`
- After changing any translation file, run `task pre-commit:fix`
## Important Notes
Taskfile.yml
+10
View File
@@ -185,6 +185,16 @@ tasks:
- task: frontend:format:check
- task: engine:format:check
# ============================================================
# Code generation
# ============================================================
tool-models:
desc: "Generate all API models from the Java OpenAPI spec"
cmds:
- task: frontend:tool-models
- task: engine:tool-models
# ============================================================
# Quality Gate
# ============================================================
app/common/src/main/java/stirling/software/common/service/PdfMetadataService.java
+14
View File
@@ -7,6 +7,7 @@ import java.time.format.DateTimeFormatter;
import java.util.Calendar;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDDocumentInformation;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;
@@ -17,6 +18,9 @@ import stirling.software.common.model.PdfMetadata;
@Service
public class PdfMetadataService {
/** ({@code {labels}}). Written by the classify-and-label tool. */
public static final String CLASSIFICATION_KEY = "StirlingPDFClassification";
private final ApplicationProperties applicationProperties;
private final String stirlingPDFLabel;
private final UserServiceInterface userService;
@@ -177,4 +181,14 @@ public class PdfMetadataService {
}
pdf.getDocumentInformation().setAuthor(author);
}
/**
* Write the document classifier's JSON result into the custom Info-dictionary field {@link
* #CLASSIFICATION_KEY}, leaving all other metadata untouched.
*/
public void setClassificationMetadata(PDDocument pdf, String classificationJson) {
PDDocumentInformation info = pdf.getDocumentInformation();
info.setCustomMetadataValue(CLASSIFICATION_KEY, classificationJson);
pdf.setDocumentInformation(info);
}
}
app/common/src/main/java/stirling/software/common/util/RequestUriUtils.java
+10
View File
@@ -57,6 +57,16 @@ public class RequestUriUtils {
return true;
}
// Admin portal SPA shell (mounted at /processor — must match the frontend
// PORTAL_BASENAME). Served publicly like the editor root so a direct nav /
// refresh to /processor loads the app (the JWT lives in localStorage, not a
// cookie, so the server can't authenticate the navigation itself). The
// portal gates access via its own auth gate + RequirePortalAccess, and its
// data APIs stay protected, so serving the shell pre-auth is safe.
if (normalizedUri.equals("/processor") || normalizedUri.startsWith("/processor/")) {
return true;
}
// Treat common static file extensions as static resources
return normalizedUri.endsWith(".svg")
|| normalizedUri.endsWith(".png")
app/common/src/test/java/stirling/software/common/util/RequestUriUtilsTest.java
+8
View File
@@ -73,6 +73,14 @@ class RequestUriUtilsTest {
assertTrue(RequestUriUtils.isStaticResource("/mobile-scanner"));
}
@Test
void testIsStaticResource_portalShell() {
// The admin portal SPA shell (/processor) is served pre-auth so it's directly navigable.
assertTrue(RequestUriUtils.isStaticResource("/processor"));
assertTrue(RequestUriUtils.isStaticResource("/processor/users"));
assertTrue(RequestUriUtils.isStaticResource("/app", "/app/processor"));
}
// --- isFrontendRoute tests ---
@Test
app/core/build.gradle
+11 -1
View File
@@ -175,6 +175,14 @@ springBoot {
// Frontend build tasks - only enabled with -PbuildWithFrontend=true
def buildWithFrontend = project.hasProperty('buildWithFrontend') && project.property('buildWithFrontend') == 'true'
def buildPrototypes = project.hasProperty('prototypesMode') && project.property('prototypesMode') == 'true'
// The admin portal ships as a lazy route inside the editor bundle (see
// proprietary/routes/adminRouteExtensions). -PbuildWithPortal=true includes that
// chunk via VITE_INCLUDE_PORTAL on the editor build; the deploy GHA sets it when
// the portal or AI layers change. Building the portal implies building the editor.
def buildWithPortal = project.hasProperty('buildWithPortal') && project.property('buildWithPortal') == 'true'
if (buildWithPortal) {
buildWithFrontend = true
}
// Workspace root holds package.json and node_modules (shared across editor /
// future portal). Editor-specific paths (src, public, dist, tauri) live one
// level deeper under frontend/editor/.
@@ -297,9 +305,11 @@ tasks.register('npmBuild', Exec) {
// Override VITE_API_BASE_URL to use relative paths for production builds
// This ensures JARs work regardless of how they're deployed (direct, proxied, etc.)
environment 'VITE_API_BASE_URL', '/'
// Include the admin portal's lazy route/chunk in the editor build when requested.
environment 'VITE_INCLUDE_PORTAL', (buildWithPortal ? 'true' : 'false')
doFirst {
println "Building editor frontend application for production (mode=${frontendMode}, VITE_API_BASE_URL=/)"
println "Building editor frontend application for production (mode=${frontendMode}, VITE_API_BASE_URL=/, portal=${buildWithPortal})"
}
}
app/core/src/main/java/stirling/software/SPDF/controller/api/security/GetInfoOnPDF.java
+29
View File
@@ -305,6 +305,23 @@ public class GetInfoOnPDF {
}
}
/**
* Info-dictionary keys exposed above via typed getters; any other key in the dictionary is
* surfaced as custom metadata (e.g. the classification policy's StirlingPDFClassification
* entry).
*/
private static final java.util.Set<String> STANDARD_INFO_KEYS =
java.util.Set.of(
"Title",
"Author",
"Subject",
"Keywords",
"Producer",
"Creator",
"CreationDate",
"ModDate",
"Trapped");
private static ObjectNode extractMetadata(PDDocument document) {
ObjectNode metadata = objectMapper.createObjectNode();
@@ -335,6 +352,18 @@ public class GetInfoOnPDF {
if (modificationDate != null) {
metadata.put("ModificationDate", modificationDate);
}
// Surface custom Info-dictionary entries (anything beyond the
// standard fields above) — e.g. StirlingPDFClassification
for (String key : info.getMetadataKeys()) {
if (STANDARD_INFO_KEYS.contains(key)) {
continue;
}
String value = info.getCustomMetadataValue(key);
if (value != null && !value.isBlank()) {
metadata.put(key, value);
}
}
}
} catch (Exception e) {
log.error("Error extracting metadata: {}", e.getMessage());
app/core/src/main/java/stirling/software/SPDF/model/api/general/PosterPdfRequest.java
+34 -12
View File
@@ -1,5 +1,7 @@
package stirling.software.SPDF.model.api.general;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Data;
@@ -17,20 +19,8 @@ public class PosterPdfRequest extends PDFFile {
allowableValues = {"A4", "Letter", "A3", "A5", "Legal", "Tabloid"})
private String pageSize = "A4";
@Schema(
description = "Horizontal decimation factor (how many columns to split into)",
requiredMode = Schema.RequiredMode.NOT_REQUIRED,
defaultValue = "2",
minimum = "1",
maximum = "10")
private int xFactor = 2;
@Schema(
description = "Vertical decimation factor (how many rows to split into)",
requiredMode = Schema.RequiredMode.NOT_REQUIRED,
defaultValue = "2",
minimum = "1",
maximum = "10")
private int yFactor = 2;
@Schema(
@@ -38,4 +28,36 @@ public class PosterPdfRequest extends PDFFile {
requiredMode = Schema.RequiredMode.NOT_REQUIRED,
defaultValue = "false")
private boolean rightToLeft = false;
@JsonProperty("xFactor")
@Schema(
description = "Horizontal decimation factor (how many columns to split into)",
requiredMode = Schema.RequiredMode.NOT_REQUIRED,
defaultValue = "2",
minimum = "1",
maximum = "10")
public int getXFactor() {
return xFactor;
}
@JsonProperty("xFactor")
public void setXFactor(int xFactor) {
this.xFactor = xFactor;
}
@JsonProperty("yFactor")
@Schema(
description = "Vertical decimation factor (how many rows to split into)",
requiredMode = Schema.RequiredMode.NOT_REQUIRED,
defaultValue = "2",
minimum = "1",
maximum = "10")
public int getYFactor() {
return yFactor;
}
@JsonProperty("yFactor")
public void setYFactor(int yFactor) {
this.yFactor = yFactor;
}
}
app/core/src/main/java/stirling/software/SPDF/model/api/security/AddPasswordRequest.java
+2 -1
View File
@@ -29,7 +29,8 @@ public class AddPasswordRequest extends PDFFile {
description = "The length of the encryption key",
type = "integer",
allowableValues = {"40", "128", "256"},
requiredMode = Schema.RequiredMode.REQUIRED)
requiredMode = Schema.RequiredMode.NOT_REQUIRED,
defaultValue = "256")
private int keyLength = 256;
@Schema(description = "Whether document assembly is prevented", defaultValue = "false")
app/proprietary/src/main/java/stirling/software/proprietary/access/config/AccessConfig.java
+9
View File
@@ -4,7 +4,9 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import stirling.software.proprietary.access.service.DefaultPrincipalResolver;
import stirling.software.proprietary.access.service.DefaultTeamLeadLookup;
import stirling.software.proprietary.access.service.PrincipalResolver;
import stirling.software.proprietary.access.service.TeamLeadLookup;
/** Access-layer bean wiring. */
@@ -17,4 +19,11 @@ public class AccessConfig {
TeamLeadLookup defaultTeamLeadLookup() {
return new DefaultTeamLeadLookup();
}
/** USER/TEAM projection unless another bean is defined (e.g. the saas resolver). */
@Bean
@ConditionalOnMissingBean(PrincipalResolver.class)
PrincipalResolver defaultPrincipalResolver() {
return new DefaultPrincipalResolver();
}
}
app/proprietary/src/main/java/stirling/software/proprietary/access/controller/ResourceGrantController.java
+33 -4
View File
@@ -25,7 +25,9 @@ import stirling.software.proprietary.access.model.PrincipalType;
import stirling.software.proprietary.access.model.ResourceGrant;
import stirling.software.proprietary.access.model.ResourceType;
import stirling.software.proprietary.access.service.ResourceAccessService;
import stirling.software.proprietary.security.database.repository.UserRepository;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.security.repository.TeamRepository;
/** Admin endpoints to grant/revoke access to gated resources (the portal, integration configs). */
@RestController
@@ -36,6 +38,8 @@ import stirling.software.proprietary.security.model.User;
public class ResourceGrantController {
private final ResourceAccessService accessService;
private final UserRepository userRepository;
private final TeamRepository teamRepository;
@GetMapping("/grants")
public ResponseEntity<?> list(
@@ -45,6 +49,14 @@ public class ResourceGrantController {
return ResponseEntity.ok(grants.stream().map(this::toDto).toList());
}
@GetMapping("/grants/by-principal")
public ResponseEntity<?> listByPrincipal(
@RequestParam PrincipalType principalType, @RequestParam Long principalId) {
List<ResourceGrant> grants =
accessService.listGrantsForPrincipal(principalType, principalId);
return ResponseEntity.ok(grants.stream().map(this::toDto).toList());
}
@PostMapping("/grants")
public ResponseEntity<?> create(
@RequestBody GrantRequest request, @AuthenticationPrincipal User admin) {
@@ -57,17 +69,26 @@ public class ResourceGrantController {
"error",
"resourceType, principalType and principalId are required"));
}
// PORTAL is a singleton (empty resourceId); every other type must name a resource.
boolean portal = request.resourceType() == ResourceType.PORTAL;
if (!portal && (request.resourceId() == null || request.resourceId().isBlank())) {
return ResponseEntity.badRequest()
.body(Map.of("error", "resourceId is required for " + request.resourceType()));
}
Long principalId = request.principalId();
String principalError = validatePrincipalExists(request.principalType(), principalId);
if (principalError != null) {
return ResponseEntity.badRequest().body(Map.of("error", principalError));
}
AccessPermission permission =
request.permission() == null ? AccessPermission.USE : request.permission();
// PORTAL is a singleton resource; its grants always target the whole type.
String resourceId =
request.resourceType() == ResourceType.PORTAL ? "" : request.resourceId();
String resourceId = portal ? "" : request.resourceId();
ResourceGrant grant =
accessService.grant(
request.resourceType(),
resourceId,
request.principalType(),
request.principalId(),
principalId,
permission,
admin);
return ResponseEntity.ok(toDto(grant));
@@ -79,6 +100,14 @@ public class ResourceGrantController {
return ResponseEntity.ok(Map.of("message", "Grant revoked"));
}
// Rejects grants to nonexistent principals (dead rows otherwise).
private String validatePrincipalExists(PrincipalType type, Long id) {
return switch (type) {
case USER -> userRepository.existsById(id) ? null : "User " + id + " does not exist";
case TEAM -> teamRepository.existsById(id) ? null : "Team " + id + " does not exist";
};
}
private Map<String, Object> toDto(ResourceGrant g) {
Map<String, Object> m = new HashMap<>();
m.put("id", g.getId());
app/proprietary/src/main/java/stirling/software/proprietary/access/model/OwnedResource.java
+11
View File
@@ -54,4 +54,15 @@ public abstract class OwnedResource {
public Long getOwnerTeamId() {
return ownerTeam != null ? ownerTeam.getId() : null;
}
/** Owner as a principal ref; null when server-owned (admin-only ownership). */
public PrincipalRef getOwnerRef() {
if (getOwnerUserId() != null) {
return PrincipalRef.user(getOwnerUserId());
}
if (getOwnerTeamId() != null) {
return PrincipalRef.team(getOwnerTeamId());
}
return null;
}
}
app/proprietary/src/main/java/stirling/software/proprietary/access/model/PrincipalRef.java
+20
View File
@@ -0,0 +1,20 @@
package stirling.software.proprietary.access.model;
import java.util.Locale;
/** A (type, id) principal pair; the atom grants and ownership are expressed in. */
public record PrincipalRef(PrincipalType type, Long id) {
public static PrincipalRef user(Long id) {
return new PrincipalRef(PrincipalType.USER, id);
}
public static PrincipalRef team(Long id) {
return new PrincipalRef(PrincipalType.TEAM, id);
}
/** Canonical engine wire form, e.g. "user:12". */
public String token() {
return type.name().toLowerCase(Locale.ROOT) + ":" + id;
}
}
app/proprietary/src/main/java/stirling/software/proprietary/access/model/PrincipalType.java
+1 -1
View File
@@ -1,6 +1,6 @@
package stirling.software.proprietary.access.model;
/** Who a {@link ResourceGrant} is granted to. Org-wide access is expressed via default policy. */
/** Who a {@link ResourceGrant} is granted to. */
public enum PrincipalType {
USER,
TEAM
app/proprietary/src/main/java/stirling/software/proprietary/access/repository/ResourceGrantRepository.java
+16
View File
@@ -3,11 +3,15 @@ package stirling.software.proprietary.access.repository;
import java.util.List;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import org.springframework.stereotype.Repository;
import stirling.software.proprietary.access.model.PrincipalType;
import stirling.software.proprietary.access.model.ResourceGrant;
import stirling.software.proprietary.access.model.ResourceType;
import stirling.software.proprietary.security.model.User;
@Repository
public interface ResourceGrantRepository extends JpaRepository<ResourceGrant, Long> {
@@ -18,8 +22,20 @@ public interface ResourceGrantRepository extends JpaRepository<ResourceGrant, Lo
List<ResourceGrant> findByResourceTypeAndPrincipalTypeAndPrincipalId(
ResourceType resourceType, PrincipalType principalType, Long principalId);
/** All grants held by a principal, across resource types (for the manage-access view). */
List<ResourceGrant> findByPrincipalTypeAndPrincipalId(
PrincipalType principalType, Long principalId);
void deleteByResourceTypeAndResourceId(ResourceType resourceType, String resourceId);
/** Removes every grant held by a principal; used when the user/team behind it is deleted. */
void deleteByPrincipalTypeAndPrincipalId(PrincipalType principalType, Long principalId);
// Detach issued grants so deleting the granting user does not hit the FK.
@Modifying
@Query("update ResourceGrant g set g.grantedBy = null where g.grantedBy = :user")
void clearGrantedBy(@Param("user") User user);
boolean existsByResourceTypeAndResourceIdAndPrincipalTypeAndPrincipalId(
ResourceType resourceType,
String resourceId,
app/proprietary/src/main/java/stirling/software/proprietary/access/security/ResourceAccessSecurity.java
+6 -1
View File
@@ -11,7 +11,12 @@ import stirling.software.proprietary.access.service.ResourceAccessService;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.security.service.UserService;
/** {@code @PreAuthorize} bean for portal-access checks. Active in self-hosted and saas. */
/**
* {@code @PreAuthorize} bean for portal-access checks. Active in self-hosted and saas. Convention:
* every portal-exclusive endpoint is gated with
* {@code @PreAuthorize("@resourceAccess.canUsePortal()")}; endpoints shared with the editor (e.g.
* the policies API) must NOT be.
*/
@Component("resourceAccess")
@RequiredArgsConstructor
public class ResourceAccessSecurity {
app/proprietary/src/main/java/stirling/software/proprietary/access/service/DefaultPrincipalResolver.java
+32
View File
@@ -0,0 +1,32 @@
package stirling.software.proprietary.access.service;
import java.util.HashSet;
import java.util.Set;
import stirling.software.proprietary.access.model.PrincipalRef;
import stirling.software.proprietary.security.model.User;
/**
* Self-hosted projection: the user and their team. One deployment = one org, so ORG_ALL is open.
*/
public class DefaultPrincipalResolver implements PrincipalResolver {
@Override
public Set<PrincipalRef> principalsOf(User user) {
if (user == null) {
return Set.of();
}
Set<PrincipalRef> principals = new HashSet<>();
principals.add(PrincipalRef.user(user.getId()));
if (user.getTeam() != null) {
principals.add(PrincipalRef.team(user.getTeam().getId()));
}
return principals;
}
// Self-hosted is a single deployment-wide org, so ORG_ALL admits every authenticated user.
@Override
public boolean allowsDeploymentWideAccess() {
return true;
}
}
app/proprietary/src/main/java/stirling/software/proprietary/access/service/MembershipTeamLeadLookup.java
+33
View File
@@ -0,0 +1,33 @@
package stirling.software.proprietary.access.service;
import org.springframework.stereotype.Component;
import lombok.RequiredArgsConstructor;
import stirling.software.common.model.enumeration.TeamRole;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
/** Real lookup backed by team_memberships LEADER rows; wins over the no-op default bean. */
@Component
@RequiredArgsConstructor
public class MembershipTeamLeadLookup implements TeamLeadLookup {
private final TeamMembershipRepository memberships;
@Override
public boolean isAnyTeamLeader(User user) {
return user != null
&& user.getId() != null
&& memberships.existsByUserIdAndRole(user.getId(), TeamRole.LEADER);
}
@Override
public boolean isLeaderOfTeam(User user, Long teamId) {
return user != null
&& user.getId() != null
&& teamId != null
&& memberships.existsByTeamIdAndUserIdAndRole(
teamId, user.getId(), TeamRole.LEADER);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/access/service/OwnershipService.java
+6 -2
View File
@@ -36,15 +36,19 @@ public class OwnershipService {
return accessService.canUseResource(
type,
String.valueOf(resource.getId()),
resource.getOwnerUserId(),
resource.getOwnerRef(),
resource.getDefaultAccess(),
user);
}
/** Whether the user may manage the resource. */
public boolean canManage(ResourceType type, OwnedResource resource, User user) {
// Disabled resources bypass grants for MANAGE too: admin/owner only.
if (!resource.isEnabled()) {
return isAdmin(user) || isOwner(resource, user);
}
return accessService.canManageResource(
type, String.valueOf(resource.getId()), resource.getOwnerUserId(), user);
type, String.valueOf(resource.getId()), resource.getOwnerRef(), user);
}
/**
app/proprietary/src/main/java/stirling/software/proprietary/access/service/PrincipalResolver.java
+28
View File
@@ -0,0 +1,28 @@
package stirling.software.proprietary.access.service;
import java.util.Set;
import java.util.stream.Collectors;
import stirling.software.proprietary.access.model.PrincipalRef;
import stirling.software.proprietary.security.model.User;
/** Projects a user onto the set of principals they act as. */
public interface PrincipalResolver {
/** Every principal the user acts as; empty for a null user. */
Set<PrincipalRef> principalsOf(User user);
/**
* Whether this deployment treats every authenticated user as one org, so the {@code ORG_ALL}
* default policy admits anyone. Self-hosted: true. Multi-tenant saas: false, so an {@code
* ORG_ALL} resource can't leak across tenants. Defaults to false (deny) for safety.
*/
default boolean allowsDeploymentWideAccess() {
return false;
}
/** Canonical wire tokens for the engine, e.g. "user:12". */
default Set<String> principalTokens(User user) {
return principalsOf(user).stream().map(PrincipalRef::token).collect(Collectors.toSet());
}
}
app/proprietary/src/main/java/stirling/software/proprietary/access/service/ResourceAccessService.java
+44 -27
View File
@@ -14,6 +14,7 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.enumeration.Role;
import stirling.software.proprietary.access.model.AccessPermission;
import stirling.software.proprietary.access.model.DefaultAccessPolicy;
import stirling.software.proprietary.access.model.PrincipalRef;
import stirling.software.proprietary.access.model.PrincipalType;
import stirling.software.proprietary.access.model.ResourceGrant;
import stirling.software.proprietary.access.model.ResourceType;
@@ -29,6 +30,7 @@ public class ResourceAccessService {
private final ResourceGrantRepository grantRepository;
private final TeamLeadLookup teamLeadLookup;
private final PrincipalResolver principalResolver;
@Value("${security.portal.defaultAccess:ADMINS_AND_TEAM_LEADS}")
private DefaultAccessPolicy portalDefaultPolicy;
@@ -44,28 +46,28 @@ public class ResourceAccessService {
public boolean canUseResource(
ResourceType type,
String resourceId,
Long ownerUserId,
PrincipalRef owner,
DefaultAccessPolicy defaultPolicy,
User user) {
if (user == null) {
return false;
}
if (isOwner(ownerUserId, user) || isAdmin(user)) {
if (isOwner(owner, user) || isAdmin(user)) {
return true;
}
if (hasGrant(type, normalize(resourceId), user, AccessPermission.USE)) {
return true;
}
return matchesDefault(defaultPolicy, user);
return matchesDefault(defaultPolicy, owner, user);
}
/** Whether the user may manage (edit/delete/share) a resource. No default-policy fallback. */
public boolean canManageResource(
ResourceType type, String resourceId, Long ownerUserId, User user) {
ResourceType type, String resourceId, PrincipalRef owner, User user) {
if (user == null) {
return false;
}
if (isOwner(ownerUserId, user) || isAdmin(user)) {
if (isOwner(owner, user) || isAdmin(user)) {
return true;
}
return hasGrant(type, normalize(resourceId), user, AccessPermission.MANAGE);
@@ -110,21 +112,22 @@ public class ResourceAccessService {
return grantRepository.findByResourceTypeAndResourceId(type, normalize(resourceId));
}
/** Resource ids of the given type that this user (or their team) holds any grant on. */
/** Every grant a principal holds, for the per-user/per-team manage-access view. */
public List<ResourceGrant> listGrantsForPrincipal(
PrincipalType principalType, Long principalId) {
return grantRepository.findByPrincipalTypeAndPrincipalId(principalType, principalId);
}
/** Resource ids of the given type that any of the user's principals holds a grant on. */
public Set<String> grantedResourceIds(ResourceType type, User user) {
if (user == null) {
return Set.of();
}
Set<String> ids = new HashSet<>();
for (ResourceGrant g :
grantRepository.findByResourceTypeAndPrincipalTypeAndPrincipalId(
type, PrincipalType.USER, user.getId())) {
ids.add(g.getResourceId());
}
if (user.getTeam() != null) {
for (PrincipalRef principal : principalResolver.principalsOf(user)) {
for (ResourceGrant g :
grantRepository.findByResourceTypeAndPrincipalTypeAndPrincipalId(
type, PrincipalType.TEAM, user.getTeam().getId())) {
type, principal.type(), principal.id())) {
ids.add(g.getResourceId());
}
}
@@ -135,18 +138,12 @@ public class ResourceAccessService {
private boolean hasGrant(
ResourceType type, String resourceId, User user, AccessPermission required) {
Long teamId = user.getTeam() != null ? user.getTeam().getId() : null;
Set<PrincipalRef> principals = principalResolver.principalsOf(user);
for (ResourceGrant g : grantRepository.findByResourceTypeAndResourceId(type, resourceId)) {
if (!permissionSatisfies(g.getPermission(), required)) {
continue;
}
if (g.getPrincipalType() == PrincipalType.USER
&& g.getPrincipalId().equals(user.getId())) {
return true;
}
if (g.getPrincipalType() == PrincipalType.TEAM
&& teamId != null
&& g.getPrincipalId().equals(teamId)) {
if (principals.contains(new PrincipalRef(g.getPrincipalType(), g.getPrincipalId()))) {
return true;
}
}
@@ -161,20 +158,40 @@ public class ResourceAccessService {
return held == AccessPermission.MANAGE;
}
private boolean matchesDefault(DefaultAccessPolicy policy, User user) {
private boolean matchesDefault(DefaultAccessPolicy policy, PrincipalRef owner, User user) {
if (policy == null) {
return false;
}
return switch (policy) {
case ORG_ALL -> true;
// Admins already pass above; only team leads here.
case ADMINS_AND_TEAM_LEADS -> teamLeadLookup.isAnyTeamLeader(user);
// Deployment-wide only where the resolver treats everyone as one org; saas resolvers
// return false, so ORG_ALL cannot leak a tenant's resource to another tenant's users.
case ORG_ALL -> principalResolver.allowsDeploymentWideAccess();
// Admins already pass above; only team leads here, scoped to the owning team.
case ADMINS_AND_TEAM_LEADS -> matchesTeamLeadDefault(owner, user);
case EXPLICIT_ONLY -> false;
};
}
private boolean isOwner(Long ownerUserId, User user) {
return ownerUserId != null && ownerUserId.equals(user.getId());
// Portal (no owner) admits any team lead; a team-owned resource admits only that team's
// leads; a user-owned resource admits no extra leads.
private boolean matchesTeamLeadDefault(PrincipalRef owner, User user) {
if (owner == null) {
return teamLeadLookup.isAnyTeamLeader(user);
}
return owner.type() == PrincipalType.TEAM
&& owner.id() != null
&& teamLeadLookup.isLeaderOfTeam(user, owner.id());
}
// Team owners are the owning team's leaders; plain members are not.
private boolean isOwner(PrincipalRef owner, User user) {
if (owner == null || owner.id() == null) {
return false;
}
return switch (owner.type()) {
case USER -> owner.id().equals(user.getId());
case TEAM -> teamLeadLookup.isLeaderOfTeam(user, owner.id());
};
}
private boolean isAdmin(User user) {
app/proprietary/src/main/java/stirling/software/proprietary/access/service/SecretMasker.java
+22 -5
View File
@@ -18,15 +18,26 @@ public class SecretMasker {
// Cap recursion so a pathologically nested payload cannot overflow the stack.
private static final int MAX_DEPTH = 32;
// Key-name substrings that mark a value sensitive. Over-masking a non-secret is
// safe; leaking a secret is not, so this errs broad - but a per-type schema
// whitelist would be a stronger boundary for free-form config (follow-up).
private static final Set<String> SENSITIVE_HINTS =
Set.of(
"secret",
"password",
"passphrase",
"pwd",
"token",
"apikey",
"accesskey",
"credential",
"privatekey");
"privatekey",
"authorization",
"cookie",
"session",
"connectionstring",
"bearer",
"signature");
/** Replace sensitive values with the mask (recursively) for safe display. */
public Map<String, Object> mask(Map<String, Object> config) {
@@ -73,18 +84,24 @@ public class SecretMasker {
private Map<String, Object> merge(
Map<String, Object> stored, Map<String, Object> incoming, int depth) {
Map<String, Object> out = new LinkedHashMap<>(stored);
// Replace semantics (PUT): the result is the incoming document, except a redacted secret
// keeps its stored value. Keys absent from incoming are dropped, so edits can remove them.
Map<String, Object> out = new LinkedHashMap<>();
for (Map.Entry<String, Object> e : incoming.entrySet()) {
String key = e.getKey();
Object value = e.getValue();
if (isSensitive(key)) {
if (!isRedacted(value, depth)) {
if (isRedacted(value, depth)) {
if (stored.containsKey(key)) {
out.put(key, stored.get(key)); // keep the stored secret
}
} else {
out.put(key, value); // a real new secret replaces the stored one
}
continue; // redacted (blank / mask) -> keep stored
continue;
}
if (depth < MAX_DEPTH
&& out.get(key) instanceof Map<?, ?> s
&& stored.get(key) instanceof Map<?, ?> s
&& value instanceof Map<?, ?> i) {
out.put(key, merge(castMap(s), castMap(i), depth + 1));
} else {
app/proprietary/src/main/java/stirling/software/proprietary/audit/ControllerAuditAspect.java
+10
View File
@@ -130,6 +130,7 @@ public class ControllerAuditAspect {
String previousPrincipal = MDC.get("auditPrincipal");
String previousOrigin = MDC.get("auditOrigin");
String previousSource = MDC.get("auditSource");
String previousIp = MDC.get("auditIp");
// EARLY CAPTURE: Capture from SecurityContext on request thread, store in MDC for async
@@ -161,6 +162,14 @@ public class ControllerAuditAspect {
return joinPoint.proceed();
}
// Stamp the free-UI source only for non-@Audited controller traffic — an actual
// tool / UI action. @Audited events (login, settings) return above without a source,
// so they never count as an "active editor" or a free UI run. The finally block
// restores auditSource, so a pooled thread can't leak a stale "WEB" into them.
if (previousSource == null) {
MDC.put("auditSource", auditService.captureCurrentSource());
}
long start = System.currentTimeMillis();
// Use auditService to create the base audit data
@@ -247,6 +256,7 @@ public class ControllerAuditAspect {
} finally {
restoreMdcValue("auditPrincipal", previousPrincipal);
restoreMdcValue("auditOrigin", previousOrigin);
restoreMdcValue("auditSource", previousSource);
restoreMdcValue("auditIp", previousIp);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/audit/DefaultPortalAuditScopeResolver.java
+15
View File
@@ -0,0 +1,15 @@
package stirling.software.proprietary.audit;
import org.springframework.stereotype.Component;
/** Self-hosted default: admins see the whole-server audit log, everyone else is denied. */
@Component
public class DefaultPortalAuditScopeResolver implements PortalAuditScopeResolver {
@Override
public PortalAuditScope resolve() {
return PortalAuditScopeResolver.hasAdminAuthority()
? PortalAuditScope.server()
: PortalAuditScope.denied();
}
}
app/proprietary/src/main/java/stirling/software/proprietary/audit/PortalAuditEventRow.java
+7
View File
@@ -0,0 +1,7 @@
package stirling.software.proprietary.audit;
import java.time.Instant;
/** Immutable, cacheable projection of an {@code audit_events} row, shared across portal views. */
public record PortalAuditEventRow(
long id, String principal, String type, String data, Instant timestamp) {}
app/proprietary/src/main/java/stirling/software/proprietary/audit/PortalAuditScope.java
+21
View File
@@ -0,0 +1,21 @@
package stirling.software.proprietary.audit;
import java.util.List;
/** Resolved audit visibility: fullServer (admin), principals-scoped (team lead), or !allowed. */
public record PortalAuditScope(
boolean allowed, boolean fullServer, List<String> principals, String cacheKey) {
public static PortalAuditScope denied() {
return new PortalAuditScope(false, false, List.of(), "denied");
}
// Named server()/team() to avoid colliding with the record's fullServer() accessor.
public static PortalAuditScope server() {
return new PortalAuditScope(true, true, List.of(), "server");
}
public static PortalAuditScope team(String cacheKey, List<String> principals) {
return new PortalAuditScope(true, false, List.copyOf(principals), cacheKey);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/audit/PortalAuditScopeResolver.java
+18
View File
@@ -0,0 +1,18 @@
package stirling.software.proprietary.audit;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
/** Resolves which slice of the audit log the caller may see. */
public interface PortalAuditScopeResolver {
PortalAuditScope resolve();
/** True when the current authentication carries {@code ROLE_ADMIN}. */
static boolean hasAdminAuthority() {
Authentication auth = SecurityContextHolder.getContext().getAuthentication();
return auth != null
&& auth.getAuthorities().stream()
.anyMatch(a -> "ROLE_ADMIN".equals(a.getAuthority()));
}
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/ClassificationLabelsController.java
+136
View File
@@ -0,0 +1,136 @@
package stirling.software.proprietary.classification;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import io.swagger.v3.oas.annotations.Hidden;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import lombok.RequiredArgsConstructor;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.service.UserServiceInterface;
import stirling.software.proprietary.classification.model.ClassificationLabels;
import stirling.software.proprietary.classification.model.LabelsValidator;
import stirling.software.proprietary.classification.store.ClassificationLabelStore;
import stirling.software.proprietary.classification.store.TeamLabelsEntity;
import stirling.software.proprietary.policy.config.PolicyManagementAuthority;
/**
* Read/write the team's classification label set — the flat vocabulary the document classifier runs
* against. Shared and team-scoped exactly like policies: every user reads their own team's labels,
* and only a user who may edit policies (a team leader on SaaS, the global admin self-hosted; see
* {@link PolicyManagementAuthority}) may change it — gated only when login is enabled, since
* single-user deployments trust the local operator. A team with no stored labels reads as {@code
* 204}; that team has no vocabulary, so its documents are not classified (there is no built-in
* default on the backend or the engine — the label data lives only in the frontend).
*/
@RestController
@RequestMapping("/api/v1/classification/labels")
@Hidden
@RequiredArgsConstructor
@Tag(name = "Classification", description = "Team-scoped document-classification labels")
@ConditionalOnBooleanProperty(name = "policies.enabled")
public class ClassificationLabelsController {
private final ClassificationLabelStore labelStore;
private final PolicyManagementAuthority policyManagementAuthority;
private final ApplicationProperties applicationProperties;
private final UserServiceInterface userService;
@GetMapping
@Operation(
summary = "Get the team's classification labels",
description =
"Returns the caller's team label set, or 204 when the team has none (its"
+ " documents are then not classified).")
public ResponseEntity<ClassificationLabels> getTeamLabels() {
return labelStore
.findByTeam(currentTeamId())
.map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.noContent().build());
}
@PutMapping(consumes = MediaType.APPLICATION_JSON_VALUE)
@Operation(
summary = "Save the team's classification labels",
description =
"Validates and stores the label set for the caller's team, shared by everyone"
+ " on the team. Requires the policy-editor role for the team.")
public ResponseEntity<ClassificationLabels> saveTeamLabels(
@RequestBody ClassificationLabels labels) {
requireEditingAllowed();
validate(labels);
ClassificationLabels saved = labelStore.save(currentTeamId(), labels, currentUsername());
return ResponseEntity.ok(saved);
}
@DeleteMapping
@Operation(
summary = "Reset the team's classification labels",
description =
"Removes the team's stored label set; its documents are then not classified"
+ " until labels are saved again. Requires the policy-editor role for the"
+ " team.")
public ResponseEntity<Void> resetTeamLabels() {
requireEditingAllowed();
labelStore.deleteByTeam(currentTeamId());
return ResponseEntity.noContent().build();
}
private static void validate(ClassificationLabels labels) {
try {
LabelsValidator.validate(labels);
} catch (IllegalArgumentException e) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, e.getMessage());
}
}
/**
* Editing the team labels requires the editor role for the caller's team — the same gate
* policies use (team leader on SaaS, global admin self-hosted). Single-user deployments (login
* disabled) have no such role, so they trust the local operator.
*/
private void requireEditingAllowed() {
if (!applicationProperties.getSecurity().isEnableLogin()) {
return;
}
if (!policyManagementAuthority.canEditPolicies()) {
throw new ResponseStatusException(
HttpStatus.FORBIDDEN,
"The team classification labels may only be changed by a team leader");
}
}
/**
* The caller's team key. With login disabled the single operator owns the {@link
* TeamLabelsEntity#NO_TEAM} sentinel row; with login enabled a caller with no resolvable team
* is an error rather than being dropped into the shared sentinel bucket (which would let
* unteamed users read and overwrite each other's "team" labels).
*/
private Long currentTeamId() {
Long teamId = policyManagementAuthority.currentUserTeamId();
if (teamId != null) {
return teamId;
}
if (!applicationProperties.getSecurity().isEnableLogin()) {
return TeamLabelsEntity.NO_TEAM;
}
throw new ResponseStatusException(
HttpStatus.UNAUTHORIZED, "Could not resolve the current user's team");
}
private String currentUsername() {
return userService == null ? null : userService.getCurrentUsername();
}
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/model/ClassificationLabel.java
+9
View File
@@ -0,0 +1,9 @@
package stirling.software.proprietary.classification.model;
/**
* One entry in the classification vocabulary. {@code id} is the label's stable identity (a slug,
* unique within a set): it is what the engine returns and what is stored on the document. {@code
* name} is the human display text the classifier model reasons over. {@code icon} is an optional
* presentational key (a Material Symbols name shown in the file sidebar); the engine never sees it.
*/
public record ClassificationLabel(String id, String name, String icon) {}
app/proprietary/src/main/java/stirling/software/proprietary/classification/model/ClassificationLabels.java
+16
View File
@@ -0,0 +1,16 @@
package stirling.software.proprietary.classification.model;
import java.util.List;
/**
* A flat multi-label classification vocabulary — the set of labels a document may be assigned.
* Stored per team (admin-edited, shared by everyone on the team); the classifier runs against these
* label names. A team with no stored set has no vocabulary, so its documents are not classified —
* neither the backend nor the engine holds a default of its own.
*/
public record ClassificationLabels(List<ClassificationLabel> labels) {
public ClassificationLabels {
labels = labels == null ? List.of() : List.copyOf(labels);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/model/LabelsValidator.java
+71
View File
@@ -0,0 +1,71 @@
package stirling.software.proprietary.classification.model;
import java.util.HashSet;
import java.util.Locale;
import java.util.Set;
import java.util.regex.Pattern;
/**
* Structural validation for a user- or admin-supplied label set, run before it is stored so a
* malformed vocabulary can never reach the classifier. Mirrors the invariants the engine relies on:
* non-blank ids and names, each unique within the set (ids exactly, names case-insensitively).
*/
public final class LabelsValidator {
private LabelsValidator() {}
// Generous upper bounds so a legitimate label set is never blocked, but a single team or user
// can't store an unbounded blob that would bloat the row, balloon the classifier prompt, or
// exhaust memory on deserialize.
static final int MAX_LABELS = 500;
static final int MAX_TEXT_LENGTH = 128;
// Icon is a Material Symbols key (lowercase, digits, hyphens). Enforce the SHAPE server-side —
// the exact allowlist lives in the frontend — so a client bypassing the UI can't store
// arbitrary
// text that would render as garbage (or worse) in every teammate's sidebar.
private static final Pattern ICON_KEY = Pattern.compile("^[a-z0-9-]+$");
/**
* @throws IllegalArgumentException with a human-readable message when the label set is invalid.
*/
public static void validate(ClassificationLabels labels) {
if (labels == null || labels.labels() == null) {
throw new IllegalArgumentException("Labels are required");
}
if (labels.labels().size() > MAX_LABELS) {
throw new IllegalArgumentException("Too many labels (max " + MAX_LABELS + ")");
}
Set<String> ids = new HashSet<>();
Set<String> names = new HashSet<>();
for (ClassificationLabel label : labels.labels()) {
requireText(label.id(), "Label id");
requireText(label.name(), "Label name");
if (label.icon() != null && !label.icon().isEmpty()) {
if (label.icon().length() > MAX_TEXT_LENGTH) {
throw new IllegalArgumentException(
"Label icon is too long (max " + MAX_TEXT_LENGTH + " characters)");
}
if (!ICON_KEY.matcher(label.icon()).matches()) {
throw new IllegalArgumentException("Invalid label icon: " + label.icon());
}
}
if (!ids.add(label.id().trim())) {
throw new IllegalArgumentException("Duplicate label id: " + label.id());
}
if (!names.add(label.name().trim().toLowerCase(Locale.ROOT))) {
throw new IllegalArgumentException("Duplicate label name: " + label.name());
}
}
}
private static void requireText(String value, String field) {
if (value == null || value.isBlank()) {
throw new IllegalArgumentException(field + " must not be blank");
}
if (value.trim().length() > MAX_TEXT_LENGTH) {
throw new IllegalArgumentException(
field + " is too long (max " + MAX_TEXT_LENGTH + " characters)");
}
}
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/store/ClassificationLabelStore.java
+22
View File
@@ -0,0 +1,22 @@
package stirling.software.proprietary.classification.store;
import java.util.Optional;
import stirling.software.proprietary.classification.model.ClassificationLabels;
/**
* Stores one {@link ClassificationLabels} set per team. A {@code null} teamId addresses the
* unteamed set (login disabled / no resolvable team), mirroring how the policy store treats a null
* team.
*/
public interface ClassificationLabelStore {
/** The team's stored labels, or empty when it has none (callers then skip classification). */
Optional<ClassificationLabels> findByTeam(Long teamId);
/** Create or replace the team's labels. Returns the stored value. */
ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy);
/** Remove the team's labels (reset to default). Returns whether a set existed. */
boolean deleteByTeam(Long teamId);
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/store/InProcessClassificationLabelStore.java
+36
View File
@@ -0,0 +1,36 @@
package stirling.software.proprietary.classification.store;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import stirling.software.proprietary.classification.model.ClassificationLabels;
/**
* In-memory {@link ClassificationLabelStore} for tests and any future no-database mode. {@link
* JpaClassificationLabelStore} is the runtime bean.
*/
public class InProcessClassificationLabelStore implements ClassificationLabelStore {
private final Map<Long, ClassificationLabels> byTeam = new ConcurrentHashMap<>();
@Override
public Optional<ClassificationLabels> findByTeam(Long teamId) {
return Optional.ofNullable(byTeam.get(key(teamId)));
}
@Override
public ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy) {
byTeam.put(key(teamId), labels);
return labels;
}
@Override
public boolean deleteByTeam(Long teamId) {
return byTeam.remove(key(teamId)) != null;
}
private static long key(Long teamId) {
return teamId == null ? TeamLabelsEntity.NO_TEAM : teamId;
}
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/store/JpaClassificationLabelStore.java
+76
View File
@@ -0,0 +1,76 @@
package stirling.software.proprietary.classification.store;
import java.time.Instant;
import java.util.Optional;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.stereotype.Service;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.classification.model.ClassificationLabels;
import tools.jackson.core.JacksonException;
import tools.jackson.databind.ObjectMapper;
/**
* Durable {@link ClassificationLabelStore} backed by JPA; the runtime store. Gated on {@code
* policies.enabled} — stored labels only matter when the Classification policy can run — so it
* shares the policy subsystem's on/off switch. Each label set is persisted as JSON via {@link
* TeamLabelsEntity}.
*/
@Slf4j
@Service
@RequiredArgsConstructor
@ConditionalOnBooleanProperty(name = "policies.enabled")
public class JpaClassificationLabelStore implements ClassificationLabelStore {
private final TeamLabelsRepository teamRepository;
private final ObjectMapper objectMapper;
@Override
public Optional<ClassificationLabels> findByTeam(Long teamId) {
return teamRepository
.findById(key(teamId))
.flatMap(entity -> parse(entity.getLabelsJson(), "team " + teamId));
}
@Override
public ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy) {
TeamLabelsEntity entity = new TeamLabelsEntity();
entity.setTeamId(key(teamId));
entity.setLabelsJson(objectMapper.writeValueAsString(labels));
entity.setUpdatedAt(Instant.now());
entity.setUpdatedBy(updatedBy);
teamRepository.save(entity);
return labels;
}
@Override
public boolean deleteByTeam(Long teamId) {
long id = key(teamId);
if (!teamRepository.existsById(id)) {
return false;
}
teamRepository.deleteById(id);
return true;
}
private Optional<ClassificationLabels> parse(String json, String owner) {
try {
return Optional.of(objectMapper.readValue(json, ClassificationLabels.class));
} catch (JacksonException e) {
// A stored label set that no longer parses (corruption / manual DB edit) must not break
// classification: drop it so the caller treats the team as having no labels (and skips
// classification) rather than surfacing a 500 on every upload.
log.warn("Discarding unparseable stored labels for {}: {}", owner, e.getMessage());
return Optional.empty();
}
}
/** Map the nullable team id onto the entity's non-null key (sentinel for the unteamed case). */
private static long key(Long teamId) {
return teamId == null ? TeamLabelsEntity.NO_TEAM : teamId;
}
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/store/TeamLabelsEntity.java
+47
View File
@@ -0,0 +1,47 @@
package stirling.software.proprietary.classification.store;
import java.io.Serializable;
import java.time.Instant;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
/**
* JPA row for a team's classification labels — one row per team. The label set lives as JSON in
* {@code labelsJson} (authoritative on read). {@code teamId} is the natural key; the sentinel
* {@link #NO_TEAM} stands in for the unteamed (login-disabled / self-hosted single-team) case,
* since a primary key can't be null (policies store a nullable {@code team_id}, but this table is
* keyed one-per-team). Kept decoupled from the security entities — {@code teamId} is a plain value,
* not a foreign key — so classification can be enabled or disabled without touching them.
*/
@Entity
@Table(name = "classification_labels")
@NoArgsConstructor
@Getter
@Setter
public class TeamLabelsEntity implements Serializable {
private static final long serialVersionUID = 1L;
/** Sentinel key for the unteamed label set (login disabled / no resolvable team). */
public static final long NO_TEAM = 0L;
@Id
@Column(name = "team_id")
private long teamId;
@Column(name = "labels_json", columnDefinition = "text")
private String labelsJson;
@Column(name = "updated_at")
private Instant updatedAt;
@Column(name = "updated_by")
private String updatedBy;
}
app/proprietary/src/main/java/stirling/software/proprietary/classification/store/TeamLabelsRepository.java
+7
View File
@@ -0,0 +1,7 @@
package stirling.software.proprietary.classification.store;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface TeamLabelsRepository extends JpaRepository<TeamLabelsEntity, Long> {}
app/proprietary/src/main/java/stirling/software/proprietary/config/CustomAuditEventRepository.java
+3
View File
@@ -60,6 +60,8 @@ public class CustomAuditEventRepository implements AuditEventRepository {
clean.put("requestId", rid);
}
String source = MDC.get("auditSource");
String auditEventData = mapper.writeValueAsString(clean);
log.debug("AuditEvent data (JSON): {}", auditEventData);
@@ -67,6 +69,7 @@ public class CustomAuditEventRepository implements AuditEventRepository {
PersistentAuditEvent.builder()
.principal(safePrincipal(ev.getPrincipal()))
.type(ev.getType())
.source(source)
.data(auditEventData)
.timestamp(ev.getTimestamp())
.build();
app/proprietary/src/main/java/stirling/software/proprietary/controller/api/ClassifyLabelController.java
+218
View File
@@ -0,0 +1,218 @@
package stirling.software.proprietary.controller.api;
import java.io.IOException;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.io.Resource;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import io.github.pixee.security.Filenames;
import io.swagger.v3.oas.annotations.Hidden;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.service.CustomPDFDocumentFactory;
import stirling.software.common.service.PdfMetadataService;
import stirling.software.common.service.UserServiceInterface;
import stirling.software.common.util.TempFileManager;
import stirling.software.common.util.WebResponseUtils;
import stirling.software.proprietary.classification.model.ClassificationLabel;
import stirling.software.proprietary.classification.store.ClassificationLabelStore;
import stirling.software.proprietary.model.api.ai.AiPageText;
import stirling.software.proprietary.policy.config.PolicyManagementAuthority;
import stirling.software.proprietary.service.AiEngineClient;
import stirling.software.proprietary.service.PdfContentExtractor;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.ObjectMapper;
import tools.jackson.databind.node.ObjectNode;
/**
* Dispatchable tool that classifies a PDF and writes the result into its metadata.
*
* <p>Runs as a Classification-policy pipeline step: it reads a bounded page window, asks the AI
* engine to classify the document against the caller's team label set, and stores the engine's JSON
* answer — minus the transport-only {@code outcome} field — in the custom Info-dictionary key
* {@link PdfMetadataService#CLASSIFICATION_KEY}. Returns the labelled PDF. Not intended for direct
* client use.
*/
@Slf4j
@Hidden
@RestController
@RequestMapping("/api/v1/ai/tools")
@Tag(name = "AI Tools", description = "Dispatchable AI-backed tools.")
public class ClassifyLabelController {
/** Pages read from each end of the document — mirrors the engine's window. */
private static final int WINDOW_PAGES = 2;
private static final String CLASSIFY_ENDPOINT = "/api/v1/documents/classify";
private final CustomPDFDocumentFactory pdfDocumentFactory;
private final TempFileManager tempFileManager;
private final PdfContentExtractor pdfContentExtractor;
private final PdfMetadataService pdfMetadataService;
private final AiEngineClient aiEngineClient;
private final ObjectMapper objectMapper;
private final UserServiceInterface userService;
/**
* Present only when the policy subsystem is enabled ({@code policies.enabled}); the store and
* team authority are gated on it. Null otherwise, in which case there are no team labels to
* classify against and the document is passed through unlabelled.
*/
private final ClassificationLabelStore labelStore;
private final PolicyManagementAuthority policyManagementAuthority;
public ClassifyLabelController(
CustomPDFDocumentFactory pdfDocumentFactory,
TempFileManager tempFileManager,
PdfContentExtractor pdfContentExtractor,
PdfMetadataService pdfMetadataService,
AiEngineClient aiEngineClient,
ObjectMapper objectMapper,
@Autowired(required = false) UserServiceInterface userService,
@Autowired(required = false) ClassificationLabelStore labelStore,
@Autowired(required = false) PolicyManagementAuthority policyManagementAuthority) {
this.pdfDocumentFactory = pdfDocumentFactory;
this.tempFileManager = tempFileManager;
this.pdfContentExtractor = pdfContentExtractor;
this.pdfMetadataService = pdfMetadataService;
this.aiEngineClient = aiEngineClient;
this.objectMapper = objectMapper;
this.userService = userService;
this.labelStore = labelStore;
this.policyManagementAuthority = policyManagementAuthority;
}
@PostMapping(value = "/classify-and-label", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
@Operation(
summary = "Classify a PDF and label its metadata",
description =
"Reads the first two and last two pages, classifies the document via the AI"
+ " engine, and stores the result in the StirlingPDFClassification"
+ " metadata field. Dispatched by the Classification policy; not"
+ " intended for direct client use.")
public ResponseEntity<Resource> classifyAndLabel(
@RequestParam("fileInput") MultipartFile fileInput) throws IOException {
try (PDDocument document = pdfDocumentFactory.load(fileInput, true)) {
String fileName = safeFileName(fileInput.getOriginalFilename());
List<EngineLabel> allowed = resolveAllowedLabels();
if (allowed.isEmpty()) {
// No vocabulary to classify against (the team stored no labels): pass the file
// through unlabelled rather than ask the engine to classify against nothing.
log.debug("[classify-and-label] {} has no team labels; skipping", fileName);
return WebResponseUtils.pdfDocToWebResponse(document, fileName, tempFileManager);
}
List<AiPageText> pages = extractWindow(document);
String requestBody =
objectMapper.writeValueAsString(
new ClassifyEngineRequest(fileName, pages, allowed));
String userId = userService != null ? userService.getCurrentUsername() : null;
String responseJson = aiEngineClient.post(CLASSIFY_ENDPOINT, requestBody, userId);
pdfMetadataService.setClassificationMetadata(document, toMetadataValue(responseJson));
log.debug("[classify-and-label] labelled {} ({} window pages)", fileName, pages.size());
return WebResponseUtils.pdfDocToWebResponse(document, fileName, tempFileManager);
}
}
private List<AiPageText> extractWindow(PDDocument document) throws IOException {
List<AiPageText> pages = new ArrayList<>();
for (int pageNumber : windowPageNumbers(document.getNumberOfPages(), WINDOW_PAGES)) {
String text = pdfContentExtractor.extractPageTextRaw(document, pageNumber);
if (text != null && !text.isBlank()) {
pages.add(new AiPageText(pageNumber, text));
}
}
return pages;
}
/** First and last {@code window} page numbers (1-based), de-duplicated and in order. */
static List<Integer> windowPageNumbers(int pageCount, int window) {
Set<Integer> numbers = new LinkedHashSet<>();
for (int page = 1; page <= Math.min(window, pageCount); page++) {
numbers.add(page);
}
for (int page = Math.max(1, pageCount - window + 1); page <= pageCount; page++) {
numbers.add(page);
}
return new ArrayList<>(numbers);
}
/** Drop the transport-only {@code outcome} discriminator; keep the rest verbatim. */
private String toMetadataValue(String engineResponseJson) {
JsonNode node = objectMapper.readTree(engineResponseJson);
if (node instanceof ObjectNode object) {
object.remove("outcome");
}
return objectMapper.writeValueAsString(node);
}
private static String safeFileName(String originalFilename) {
String name = Filenames.toSimpleFileName(originalFilename);
return (name == null || name.isBlank()) ? "classified.pdf" : name;
}
/**
* The allowed labels for the caller's team as {@code {id, name}} pairs, de-duplicated by id.
* The engine shows the model the names and returns the ids (icons are presentational and never
* sent). Returns an empty list — the caller then skips classification — when the policy
* subsystem is disabled (no store) or the team has no stored labels. The engine holds no
* default vocabulary of its own, so a team's stored labels are the only source.
*/
private List<EngineLabel> resolveAllowedLabels() {
if (labelStore == null) {
return List.of();
}
Long teamId =
policyManagementAuthority == null
? null
: policyManagementAuthority.currentUserTeamId();
Map<String, EngineLabel> byId = new LinkedHashMap<>();
labelStore.findByTeam(teamId).ifPresent(labels -> collectLabels(labels.labels(), byId));
return List.copyOf(byId.values());
}
private static void collectLabels(
List<ClassificationLabel> labels, Map<String, EngineLabel> into) {
for (ClassificationLabel label : labels) {
if (label.id() == null
|| label.id().isBlank()
|| label.name() == null
|| label.name().isBlank()) {
continue;
}
into.putIfAbsent(label.id(), new EngineLabel(label.id(), label.name()));
}
}
/** One allowed label sent to the engine: stable id + the name the model reasons over. */
private record EngineLabel(String id, String name) {}
/** Request body for the engine's {@code /api/v1/documents/classify} endpoint. */
private record ClassifyEngineRequest(
String fileName, List<AiPageText> pages, List<EngineLabel> labels) {}
}
app/proprietary/src/main/java/stirling/software/proprietary/controller/api/FleetUsageController.java
+71
View File
@@ -0,0 +1,71 @@
package stirling.software.proprietary.controller.api;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.List;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.enumeration.Role;
import stirling.software.proprietary.audit.AuditLevel;
import stirling.software.proprietary.config.AuditConfigurationProperties;
import stirling.software.proprietary.model.api.usage.FleetUsageStats;
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
import stirling.software.proprietary.security.database.repository.UserRepository;
/**
* Admin endpoint exposing free-editor fleet usage for the portal Usage card. Audit-derived figures
* (active editors, PDFs processed) are null (rendered as "N/A") rather than a misleading 0 whenever
* the data can't exist: the events they count (PDF_PROCESS, FILE_OPERATION, HTTP_REQUEST) are all
* STANDARD level, so a gate on {@code isEnabled()} alone would still return 0 at level=OFF/BASIC —
* we gate on {@code isLevelEnabled(STANDARD)} instead.
*
* <p>Known limitation: on a login-disabled self-hosted instance every request is anonymous, so its
* audit origin is SYSTEM (not WEB) and it is excluded from these WEB-only counts — active/PDFs then
* read 0 despite real usage. Historical audit rows written before the {@code source} column existed
* carry {@code source=null}, so the cumulative "PDFs edited" figure effectively starts at deploy.
*/
@Slf4j
@RestController
@RequestMapping("/api/v1/usage")
@PreAuthorize("hasRole('ADMIN')")
@RequiredArgsConstructor
@EnterpriseEndpoint
public class FleetUsageController {
private final PersistentAuditEventRepository auditRepository;
private final UserRepository userRepository;
private final AuditConfigurationProperties auditConfig;
@GetMapping("/fleet-stats")
public FleetUsageStats fleetStats() {
// Exclude the reserved INTERNAL_API_USER row that InitialSecuritySetup creates on every
// install, so a fresh single-admin instance reads 1 editor, not 2.
Long deployed = userRepository.countByUsernameNot(Role.INTERNAL_API_USER.getRoleId());
// STANDARD is the level at which the counted events are recorded; below it the data
// can't exist, so report N/A instead of a 0 that would misrepresent an empty table.
boolean auditOn = auditConfig.isLevelEnabled(AuditLevel.STANDARD);
Instant since = Instant.now().minus(30, ChronoUnit.DAYS);
Long active =
auditOn
? auditRepository.countDistinctPrincipalsBySourceExcludingTypeAfter(
"WEB", "UI_DATA", since)
: null;
Long pdfs =
auditOn
? auditRepository.countByTypeInAndSourceAndTimestampAfter(
List.of("PDF_PROCESS", "FILE_OPERATION"), "WEB", Instant.EPOCH)
: null;
if (active != null && deployed != null && active > deployed) {
active = deployed; // active editors are a subset of those deployed
}
return new FleetUsageStats(deployed, active, pdfs);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/controller/api/PortalDocumentsController.java
+46
View File
@@ -0,0 +1,46 @@
package stirling.software.proprietary.controller.api;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import io.swagger.v3.oas.annotations.Operation;
import lombok.RequiredArgsConstructor;
import stirling.software.common.annotations.api.ProprietaryUiDataApi;
import stirling.software.proprietary.audit.PortalAuditScope;
import stirling.software.proprietary.audit.PortalAuditScopeResolver;
import stirling.software.proprietary.model.api.documents.PortalDocumentsResponseDto;
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
import stirling.software.proprietary.service.PortalDocumentsService;
/** Serves the portal Documents review queue, derived from real audit data and scoped per caller. */
@ProprietaryUiDataApi
@RequiredArgsConstructor
@EnterpriseEndpoint
public class PortalDocumentsController {
private final PortalDocumentsService portalDocumentsService;
private final PortalAuditScopeResolver auditScopeResolver;
// tier accepted for mock-seam symmetry; ignored (queue isn't tier-scoped).
@GetMapping("/documents")
@Operation(
summary = "Documents review queue",
description = "Files processed through the org, derived from the audit trail.")
public ResponseEntity<PortalDocumentsResponseDto> getDocuments(
@RequestParam(value = "tier", required = false) String tier) {
PortalAuditScope scope = auditScopeResolver.resolve();
if (!scope.allowed()) {
return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
}
PortalDocumentsResponseDto body =
scope.fullServer()
? portalDocumentsService.serverDocuments()
: portalDocumentsService.scopedDocuments(
scope.cacheKey(), scope.principals());
return ResponseEntity.ok(body);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/controller/api/PortalInfraAuditController.java
+47
View File
@@ -0,0 +1,47 @@
package stirling.software.proprietary.controller.api;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import io.swagger.v3.oas.annotations.Operation;
import lombok.RequiredArgsConstructor;
import stirling.software.common.annotations.api.ProprietaryUiDataApi;
import stirling.software.proprietary.audit.PortalAuditScope;
import stirling.software.proprietary.audit.PortalAuditScopeResolver;
import stirling.software.proprietary.model.api.audit.InfraAuditLogResponse;
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
import stirling.software.proprietary.service.PortalInfraAuditService;
/** Serves the Infrastructure → Audit tab from real audit data, scoped and cached per caller. */
@ProprietaryUiDataApi
@RequiredArgsConstructor
@EnterpriseEndpoint
public class PortalInfraAuditController {
private final PortalInfraAuditService portalInfraAuditService;
private final PortalAuditScopeResolver auditScopeResolver;
// tier accepted for endpoint symmetry; ignored (audit log isn't tier-scoped).
@GetMapping("/infrastructure/audit-log")
@Operation(
summary = "Infrastructure audit log",
description = "Recent audit events shaped for the portal Infrastructure → Audit tab.")
public ResponseEntity<InfraAuditLogResponse> getInfrastructureAuditLog(
@RequestParam(value = "tier", required = false) String tier) {
PortalAuditScope scope = auditScopeResolver.resolve();
if (!scope.allowed()) {
// Return 403 (not throw) so the tab shows its access message, not a generic 500.
return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
}
InfraAuditLogResponse body =
scope.fullServer()
? portalInfraAuditService.serverAuditLog()
: portalInfraAuditService.scopedAuditLog(
scope.cacheKey(), scope.principals());
return ResponseEntity.ok(body);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/controller/api/ProprietaryUIDataController.java
+50 -3
View File
@@ -5,6 +5,7 @@ import static stirling.software.common.util.ProviderUtils.validateProvider;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.*;
import java.util.stream.Collectors;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.http.ResponseEntity;
@@ -28,13 +29,16 @@ import stirling.software.common.model.ApplicationProperties.Security.OAUTH2.Clie
import stirling.software.common.model.ApplicationProperties.Security.SAML2;
import stirling.software.common.model.FileInfo;
import stirling.software.common.model.enumeration.Role;
import stirling.software.common.model.enumeration.TeamRole;
import stirling.software.common.model.oauth2.GitHubProvider;
import stirling.software.common.model.oauth2.GoogleProvider;
import stirling.software.common.model.oauth2.KeycloakProvider;
import stirling.software.proprietary.access.service.ResourceAccessService;
import stirling.software.proprietary.audit.AuditEventType;
import stirling.software.proprietary.audit.AuditLevel;
import stirling.software.proprietary.config.AuditConfigurationProperties;
import stirling.software.proprietary.model.Team;
import stirling.software.proprietary.model.TeamMembership;
import stirling.software.proprietary.model.dto.TeamWithUserCountDTO;
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
@@ -44,6 +48,7 @@ import stirling.software.proprietary.security.model.Authority;
import stirling.software.proprietary.security.model.SessionEntity;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.security.model.dto.AdminUserSummary;
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
import stirling.software.proprietary.security.repository.TeamRepository;
import stirling.software.proprietary.security.saml2.CustomSaml2AuthenticatedPrincipal;
import stirling.software.proprietary.security.service.DatabaseServiceInterface;
@@ -65,6 +70,7 @@ public class ProprietaryUIDataController {
private final SessionPersistentRegistry sessionPersistentRegistry;
private final UserRepository userRepository;
private final TeamRepository teamRepository;
private final TeamMembershipRepository teamMembershipRepository;
private final SessionRepository sessionRepository;
private final DatabaseServiceInterface databaseService;
private final boolean runningEE;
@@ -73,6 +79,7 @@ public class ProprietaryUIDataController {
private final PersistentAuditEventRepository auditRepository;
private final MfaService mfaService;
private final LoginAttemptService loginAttemptService;
private final ResourceAccessService resourceAccessService;
public ProprietaryUIDataController(
ApplicationProperties applicationProperties,
@@ -80,6 +87,7 @@ public class ProprietaryUIDataController {
SessionPersistentRegistry sessionPersistentRegistry,
UserRepository userRepository,
TeamRepository teamRepository,
TeamMembershipRepository teamMembershipRepository,
SessionRepository sessionRepository,
DatabaseServiceInterface databaseService,
ObjectMapper objectMapper,
@@ -87,12 +95,14 @@ public class ProprietaryUIDataController {
UserLicenseSettingsService licenseSettingsService,
PersistentAuditEventRepository auditRepository,
MfaService mfaService,
LoginAttemptService loginAttemptService) {
LoginAttemptService loginAttemptService,
ResourceAccessService resourceAccessService) {
this.applicationProperties = applicationProperties;
this.auditConfig = auditConfig;
this.sessionPersistentRegistry = sessionPersistentRegistry;
this.userRepository = userRepository;
this.teamRepository = teamRepository;
this.teamMembershipRepository = teamMembershipRepository;
this.sessionRepository = sessionRepository;
this.databaseService = databaseService;
this.objectMapper = objectMapper;
@@ -101,6 +111,7 @@ public class ProprietaryUIDataController {
this.auditRepository = auditRepository;
this.mfaService = mfaService;
this.loginAttemptService = loginAttemptService;
this.resourceAccessService = resourceAccessService;
}
/**
@@ -370,8 +381,11 @@ public class ProprietaryUIDataController {
boolean premiumEnabled = applicationProperties.getPremium().isEnabled();
// Convert User entities to AdminUserSummary DTOs to exclude sensitive fields
Set<Long> leaderUserIds = leaderUserIds();
List<AdminUserSummary> userSummaries =
sortedUsers.stream().map(this::convertUserToSummary).toList();
sortedUsers.stream()
.map(user -> convertUserToSummary(user, leaderUserIds))
.toList();
AdminSettingsData data = new AdminSettingsData();
data.setUsers(userSummaries);
@@ -390,6 +404,10 @@ public class ProprietaryUIDataController {
data.setLicenseMaxUsers(licenseMaxUsers);
data.setPremiumEnabled(premiumEnabled);
data.setMailEnabled(applicationProperties.getMail().isEnabled());
// Email invites need the invites toggle AND SMTP on; matches the inviteUsers precondition.
data.setEmailInvitesEnabled(
applicationProperties.getMail().isEnableInvites()
&& applicationProperties.getMail().isEnabled());
data.setUserSettings(userSettings);
data.setLockedUsers(loginAttemptService.getAllBlockedUsers());
@@ -468,9 +486,18 @@ public class ProprietaryUIDataController {
teamLastRequest.put(teamId, lastActivity);
}
Map<Long, List<String>> teamOwners = new HashMap<>();
for (TeamMembership row :
teamMembershipRepository.findByRoleFetchingUserAndTeam(TeamRole.LEADER)) {
teamOwners
.computeIfAbsent(row.getTeam().getId(), id -> new ArrayList<>())
.add(row.getUser().getUsername());
}
TeamsData data = new TeamsData();
data.setTeamsWithCounts(teamsWithCounts);
data.setTeamLastRequest(teamLastRequest);
data.setTeamOwners(teamOwners);
return ResponseEntity.ok(data);
}
@@ -510,11 +537,17 @@ public class ProprietaryUIDataController {
userLastRequest.put(username, lastRequest);
}
Set<Long> ownerUserIds =
teamMembershipRepository.findByTeamIdAndRole(id, TeamRole.LEADER).stream()
.map(row -> row.getUser().getId())
.collect(Collectors.toSet());
TeamDetailsData data = new TeamDetailsData();
data.setTeam(team);
data.setTeamUsers(teamUsers);
data.setAvailableUsers(availableUsers);
data.setUserLastRequest(userLastRequest);
data.setOwnerUserIds(ownerUserIds);
return ResponseEntity.ok(data);
}
@@ -535,13 +568,24 @@ public class ProprietaryUIDataController {
return ResponseEntity.ok(data);
}
/** User ids holding a LEADER membership on any team. */
private Set<Long> leaderUserIds() {
return teamMembershipRepository.findByRoleFetchingUserAndTeam(TeamRole.LEADER).stream()
.map(row -> row.getUser().getId())
.collect(Collectors.toSet());
}
/**
* Convert User entity to AdminUserSummary DTO, excluding sensitive fields like password and
* apiKey.
*/
private AdminUserSummary convertUserToSummary(User user) {
private AdminUserSummary convertUserToSummary(User user, Set<Long> leaderUserIds) {
AdminUserSummary summary = new AdminUserSummary();
summary.setId(user.getId());
summary.setTeamLead(leaderUserIds.contains(user.getId()));
// Authoritative portal access, same call /me uses, so the roster honors the configured
// policy instead of the frontend guessing from role/team-leadership.
summary.setPortalAccess(resourceAccessService.canAccessPortal(user));
summary.setUsername(user.getUsername());
summary.setEmail(user.getUsername()); // Use username as email for consistency
summary.setRoleName(user.getRoleName());
@@ -609,6 +653,7 @@ public class ProprietaryUIDataController {
private int licenseMaxUsers;
private boolean premiumEnabled;
private boolean mailEnabled;
private boolean emailInvitesEnabled;
private Map<String, Map<String, String>> userSettings;
private List<String> lockedUsers;
}
@@ -629,6 +674,7 @@ public class ProprietaryUIDataController {
public static class TeamsData {
private List<TeamWithUserCountDTO> teamsWithCounts;
private Map<Long, Date> teamLastRequest;
private Map<Long, List<String>> teamOwners;
}
@Data
@@ -637,6 +683,7 @@ public class ProprietaryUIDataController {
private List<User> teamUsers;
private List<User> availableUsers;
private Map<String, Date> userLastRequest;
private Set<Long> ownerUserIds;
}
@Data
app/proprietary/src/main/java/stirling/software/proprietary/integration/controller/IntegrationConfigController.java
+3 -1
View File
@@ -29,7 +29,9 @@ import stirling.software.proprietary.security.model.User;
@RestController
@RequestMapping("/api/v1/integrations")
@RequiredArgsConstructor
@PreAuthorize("isAuthenticated()")
// Portal-exclusive: server-side portal-access boundary, not just isAuthenticated. Per-config
// ownership is still enforced in the service layer.
@PreAuthorize("@resourceAccess.canUsePortal()")
@Tag(name = "Integrations", description = "Manage S3/MCP/API integration configurations")
public class IntegrationConfigController {
app/proprietary/src/main/java/stirling/software/proprietary/integration/crypto/CredentialEncryption.java
+24 -1
View File
@@ -1,12 +1,16 @@
package stirling.software.proprietary.integration.crypto;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.PosixFilePermissions;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.Base64;
import java.util.EnumSet;
import javax.crypto.Cipher;
import javax.crypto.KeyGenerator;
@@ -74,7 +78,7 @@ public class CredentialEncryption {
generator.init(256);
SecretKey generated = generator.generateKey();
Files.createDirectories(path.getParent());
Files.writeString(path, Base64.getEncoder().encodeToString(generated.getEncoded()));
writeOwnerOnly(path, Base64.getEncoder().encodeToString(generated.getEncoded()));
log.warn(
"Generated a new credential encryption key at {}. Back this file up: losing it"
+ " makes stored integration secrets unrecoverable.",
@@ -85,6 +89,25 @@ public class CredentialEncryption {
}
}
// The master key decrypts every stored integration secret, so create it 0600
// (owner-only) atomically. On non-POSIX filesystems (Windows) the config-dir
// ACL is the protection; we still create the file, just without POSIX perms.
private static void writeOwnerOnly(Path path, String content) throws IOException {
EnumSet<PosixFilePermission> ownerOnly =
EnumSet.of(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_WRITE);
try {
Files.createFile(path, PosixFilePermissions.asFileAttribute(ownerOnly));
} catch (UnsupportedOperationException e) {
Files.createFile(path);
}
Files.writeString(path, content);
try {
Files.setPosixFilePermissions(path, ownerOnly);
} catch (UnsupportedOperationException ignored) {
// Non-POSIX filesystem: nothing to tighten here.
}
}
public static String encrypt(String plaintext) {
if (plaintext == null) {
return null;
app/proprietary/src/main/java/stirling/software/proprietary/integration/repository/IntegrationConfigRepository.java
+9
View File
@@ -18,4 +18,13 @@ public interface IntegrationConfigRepository extends JpaRepository<IntegrationCo
List<IntegrationConfig> findByOwnerTeam(Team ownerTeam);
List<IntegrationConfig> findByScope(OwnerScope scope);
// Nested path: OwnedResource has a getOwnerTeamId() convenience getter but no such persistent
// attribute, so the plain "...OwnerTeamId" derivation resolves to a phantom property and throws
// UnknownPathException. The underscore forces the real ownerTeam.id association path.
boolean existsByOwnerTeam_Id(Long teamId);
void deleteByOwnerUser(User ownerUser);
void deleteByOwnerTeam_Id(Long teamId);
}
app/proprietary/src/main/java/stirling/software/proprietary/integration/service/IntegrationConfigService.java
+11
View File
@@ -16,6 +16,7 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.access.model.DefaultAccessPolicy;
import stirling.software.proprietary.access.model.OwnerScope;
import stirling.software.proprietary.access.model.ResourceType;
import stirling.software.proprietary.access.repository.ResourceGrantRepository;
import stirling.software.proprietary.access.service.OwnershipService;
import stirling.software.proprietary.access.service.SecretMasker;
import stirling.software.proprietary.integration.dto.IntegrationConfigRequest;
@@ -41,6 +42,7 @@ public class IntegrationConfigService {
private final IntegrationConfigRepository repository;
private final OwnershipService ownership;
private final SecretMasker secretMasker;
private final ResourceGrantRepository grantRepository;
// ---- commands ----
@@ -49,6 +51,13 @@ public class IntegrationConfigService {
OwnerScope scope = request.scope() == null ? OwnerScope.USER : request.scope();
IntegrationConfig cfg = new IntegrationConfig();
cfg.setIntegrationType(require(request.integrationType(), "integrationType"));
// S3 is infrastructure, not self-serve: no personal S3 for regular users. TEAM/SERVER
// scopes are already restricted to admins/team owners by assignOwnership.
if (cfg.getIntegrationType() == IntegrationType.S3
&& scope == OwnerScope.USER
&& !ownership.isAdmin(currentUser)) {
throw forbidden("S3 connections can only be created by administrators or team owners");
}
cfg.setName(require(request.name(), "name"));
cfg.setEnabled(request.enabled() == null || request.enabled());
cfg.setLocked(request.locked() != null && request.locked());
@@ -104,6 +113,8 @@ public class IntegrationConfigService {
if (!ownership.canManage(TYPE, cfg, currentUser)) {
throw forbidden("You cannot manage this integration");
}
// Drop grants sharing this config so they do not dangle as dead rows.
grantRepository.deleteByResourceTypeAndResourceId(TYPE, String.valueOf(cfg.getId()));
repository.delete(cfg);
}
app/saas/src/main/java/stirling/software/saas/model/TeamMembership.java → app/proprietary/src/main/java/stirling/software/proprietary/model/TeamMembership.java
+1 -2
View File
@@ -1,4 +1,4 @@
package stirling.software.saas.model;
package stirling.software.proprietary.model;
import java.io.Serializable;
import java.time.LocalDateTime;
@@ -15,7 +15,6 @@ import lombok.Setter;
import lombok.ToString;
import stirling.software.common.model.enumeration.TeamRole;
import stirling.software.proprietary.model.Team;
import stirling.software.proprietary.security.model.User;
/**
app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiWorkflowOutcome.java
+1 -2
View File
@@ -21,8 +21,7 @@ public enum AiWorkflowOutcome {
COMPLETED("completed"),
UNSUPPORTED_CAPABILITY("unsupported_capability"),
CANNOT_CONTINUE("cannot_continue"),
GENERATE_FILE("generate_file"),
CONVERT_MARKDOWN("convert_markdown");
GENERATE_FILE("generate_file");
private final String value;
app/proprietary/src/main/java/stirling/software/proprietary/model/api/ai/AiWorkflowResultFile.java
+7
View File
@@ -21,4 +21,11 @@ public class AiWorkflowResultFile {
@Schema(description = "MIME type of the file", example = "application/pdf")
private String contentType;
@Schema(
description =
"Index into the request's fileInputs that this output was derived from, or null"
+ " when it has no single source (e.g. a merge, or a generated file)."
+ " Lets the client replace that input in place as a new version.")
private Integer sourceIndex;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/audit/InfraAuditEventDto.java
+44
View File
@@ -0,0 +1,44 @@
package stirling.software.proprietary.model.api.audit;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* A single infrastructure audit-log row, shaped for the portal Infrastructure → Audit tab. Derived
* from a {@code audit_events} row: the real {@link
* stirling.software.proprietary.audit.AuditEventType} is mapped to a display category/action.
*/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class InfraAuditEventDto {
@Schema(description = "Audit event id", example = "8841")
private String id;
@Schema(description = "Display timestamp (UTC)", example = "2026-07-07 18:59:31")
private String timestamp;
@Schema(description = "Category: auth | config | elevation | processing | security")
private String category;
@Schema(description = "Human-readable action", example = "Compress PDF")
private String action;
@Schema(description = "Actor principal", example = "alice.chen@acme.com")
private String actor;
@Schema(description = "Affected target (file, endpoint, or session)")
private String target;
@Schema(description = "Status: success | warning | danger | info")
private String status;
@Schema(description = "Operation latency in milliseconds", example = "412")
private long latencyMs;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/audit/InfraAuditLogResponse.java
+30
View File
@@ -0,0 +1,30 @@
package stirling.software.proprietary.model.api.audit;
import java.util.List;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/** Response for the portal Infrastructure → Audit tab: summary strip + recent event rows. */
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class InfraAuditLogResponse {
@Schema(description = "Headline counts")
private InfraAuditSummary summary;
@Schema(description = "Most-recent audit events, newest first")
private List<InfraAuditEventDto> events;
@Schema(
description =
"True when this is the whole-server (admin) view. Team-scoped views are false; "
+ "drives whether the admin-only, whole-server CSV export is offered.")
private boolean fullServer;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/audit/InfraAuditSummary.java
+28
View File
@@ -0,0 +1,28 @@
package stirling.software.proprietary.model.api.audit;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/** Headline counts for the infrastructure audit-log tab, derived from the returned events. */
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class InfraAuditSummary {
@Schema(description = "Total events in the returned window", example = "40")
private int totalEvents;
@Schema(description = "Processing-category events", example = "24")
private int processing;
@Schema(description = "Elevation-category events", example = "0")
private int elevation;
@Schema(description = "Config-category events", example = "6")
private int config;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/documents/PortalDocAuditEventDto.java
+24
View File
@@ -0,0 +1,24 @@
package stirling.software.proprietary.model.api.documents;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/** One event in a document's lifecycle timeline. */
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PortalDocAuditEventDto {
private String id;
/** ingested | extracted | flagged | reviewed | approved | archived | elevation */
private String kind;
/** Relative-time string, e.g. "2m ago". */
private String time;
private String actor;
private String detail;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/documents/PortalDocumentsResponseDto.java
+18
View File
@@ -0,0 +1,18 @@
package stirling.software.proprietary.model.api.documents;
import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/** Response for the portal Documents review queue. */
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PortalDocumentsResponseDto {
private PortalDocumentsSummaryDto summary;
private List<PortalReviewDocumentDto> documents;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/documents/PortalDocumentsSummaryDto.java
+18
View File
@@ -0,0 +1,18 @@
package stirling.software.proprietary.model.api.documents;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/** KPI strip for the documents queue. */
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PortalDocumentsSummaryDto {
private int totalInQueue;
private int processed;
private int errors;
private int processedToday;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/documents/PortalExtractionDto.java
+17
View File
@@ -0,0 +1,17 @@
package stirling.software.proprietary.model.api.documents;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/** A single extracted field. Empty for audit-derived documents (no extraction data yet). */
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PortalExtractionDto {
private String field;
private String value;
private double confidence;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/documents/PortalReviewDocumentDto.java
+48
View File
@@ -0,0 +1,48 @@
package stirling.software.proprietary.model.api.documents;
import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* A document in the review queue, derived from the audit trail of a processed file. Extraction
* fields ({@code confidence}, {@code extractions}) are absent/empty - that data doesn't exist yet.
*/
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class PortalReviewDocumentDto {
private String id;
private String name;
private String type;
/** Where it was processed: "API" or "Editor". */
private String product;
/** The operation / pipeline, e.g. "Compress PDF" (or "Editor"). */
private String action;
/** The user who ran it. */
private String user;
/** processed | error */
private String status;
private String source;
/** Overall confidence 0..1, or null when there's no extraction data. */
private Double confidence;
private int fieldsExtracted;
/** Relative-time string, e.g. "4m ago". */
private String time;
private boolean sensitive;
private List<PortalExtractionDto> extractions;
private List<PortalDocAuditEventDto> audit;
}
app/proprietary/src/main/java/stirling/software/proprietary/model/api/usage/FleetUsageStats.java
+6
View File
@@ -0,0 +1,6 @@
package stirling.software.proprietary.model.api.usage;
/**
* Free-editor fleet usage for the portal Usage card. Null fields render as "N/A" (uncomputable).
*/
public record FleetUsageStats(Long editorsDeployed, Long activeThisMonth, Long pdfsProcessed) {}
app/proprietary/src/main/java/stirling/software/proprietary/model/security/PersistentAuditEvent.java
+10 -1
View File
@@ -18,7 +18,15 @@ import lombok.*;
columnList = "principal,type"),
@jakarta.persistence.Index(
name = "idx_audit_type_timestamp",
columnList = "type,timestamp")
columnList = "type,timestamp"),
@jakarta.persistence.Index(
name = "idx_audit_type_source_timestamp",
columnList = "type,source,timestamp"),
// Leads with source (equality) for the active-editors query, which filters on
// source then a timestamp range and counts distinct principal.
@jakarta.persistence.Index(
name = "idx_audit_source_timestamp_principal",
columnList = "source,timestamp,principal")
})
@Data
@Builder
@@ -32,6 +40,7 @@ public class PersistentAuditEvent {
private String principal;
private String type;
private String source;
@Column(columnDefinition = "text")
private String data; // JSON blob
app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java
+37
View File
@@ -18,6 +18,7 @@ import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestPart;
@@ -47,6 +48,7 @@ import stirling.software.proprietary.policy.engine.PolicyRunHandle;
import stirling.software.proprietary.policy.engine.PolicyRunRegistry;
import stirling.software.proprietary.policy.engine.PolicyRunner;
import stirling.software.proprietary.policy.engine.PolicyValidator;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
import stirling.software.proprietary.policy.model.PipelineDefinition;
import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.model.PolicyInputs;
@@ -86,6 +88,7 @@ public class PolicyController {
private final PolicyManagementAuthority policyManagementAuthority;
private final PolicyTriggerManager policyTriggerManager;
private final PolicyOverviewService policyOverviewService;
private final ProcessedLedger processedLedger;
private final List<PolicyTrigger> policyTriggers;
private final ApplicationProperties applicationProperties;
private final TempFileManager tempFileManager;
@@ -213,6 +216,20 @@ public class PolicyController {
return ResponseEntity.ok(saved);
}
@PutMapping("/order")
@Operation(
summary = "Set the team's policy run order",
description =
"Persists the team-wide order policies run in, from the given ordered list of"
+ " policy ids (position → order). The per-trigger order shown in the UI"
+ " is this one sequence filtered by trigger. Team-leader/admin only;"
+ " ids outside the caller's team are ignored.")
public ResponseEntity<Void> reorderPolicies(@RequestBody List<String> orderedPolicyIds) {
requirePolicyEditingAllowed();
policyStore.reorder(policyAccessGuard.teamForNewPolicy(), orderedPolicyIds);
return ResponseEntity.noContent().build();
}
/**
* Every {@code sourceId} a policy references must resolve to a source in the caller's team, so
* a client can neither reference a non-existent source nor reach across teams to use another
@@ -337,6 +354,7 @@ public class PolicyController {
boolean accessible =
policyStore.get(policyId).filter(policyAccessGuard::canAccess).isPresent();
if (accessible && policyStore.delete(policyId)) {
processedLedger.clearPolicy(policyId);
// Cancel any now-orphaned folder watch promptly rather than leaving the WatchKey open
// until the next reconcile sweep.
policyTriggerManager.notifyPoliciesChanged();
@@ -345,6 +363,25 @@ public class PolicyController {
return ResponseEntity.notFound().build();
}
@DeleteMapping("/{policyId}/processed-history")
@Operation(
summary = "Clear a policy's processed-file history",
description =
"Forgets which source files this policy has already processed, so its next"
+ " sweep reprocesses everything currently in its sources. Does not"
+ " touch the files themselves.")
public ResponseEntity<Void> clearProcessedHistory(@PathVariable String policyId) {
requirePolicyEditingAllowed();
// Scope to the caller's team: a policy in another team reads as not-found.
boolean accessible =
policyStore.get(policyId).filter(policyAccessGuard::canAccess).isPresent();
if (!accessible) {
return ResponseEntity.notFound().build();
}
processedLedger.clearPolicy(policyId);
return ResponseEntity.noContent().build();
}
@PostMapping(value = "/{policyId}/run", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
@Operation(
summary = "Run a stored policy",
app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyEngine.java
+7 -1
View File
@@ -35,6 +35,7 @@ import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.model.PolicyInputs;
import stirling.software.proprietary.policy.model.PolicyRun;
import stirling.software.proprietary.policy.model.WaitState;
import stirling.software.proprietary.policy.output.OutputDelivery;
import stirling.software.proprietary.policy.output.PolicyOutputSink;
import stirling.software.proprietary.policy.progress.PolicyProgressListener;
import stirling.software.proprietary.service.DownstreamEntitlementError;
@@ -203,7 +204,12 @@ public class PolicyEngine {
PolicyExecutionResult result =
stepExecutor.execute(run.getDefinition(), inputs, listener);
OutputSpec output = run.getDefinition().output();
List<ResultFile> outputs = sinkFor(output).deliver(runId, result.files(), output);
List<ResultFile> outputs =
sinkFor(output)
.deliver(
new OutputDelivery(runId, run.getPolicyId()),
result.files(),
output);
taskManager.setMultipleFileResults(runId, outputs);
taskManager.setComplete(runId);
run.complete(outputs);
app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyExecutionResult.java
+7 -3
View File
@@ -8,7 +8,11 @@ import tools.jackson.databind.JsonNode;
/**
* Result of a {@link PolicyExecutor} run. {@code files} are final temp files (not yet stored).
* {@code report}/{@code reportTool} carry the last step's structured report and its operation, or
* null if no step produced one.
* {@code origins} is parallel to {@code files}: each entry is the index into the original pipeline
* inputs that the output traces back to, or {@code null} when it has no single source (e.g. a merge
* combining several inputs, or a generated file). Callers use it to map an output back onto the
* file it came from. {@code report}/{@code reportTool} carry the last step's structured report and
* its operation, or null if no step produced one.
*/
public record PolicyExecutionResult(List<Resource> files, JsonNode report, String reportTool) {}
public record PolicyExecutionResult(
List<Resource> files, List<Integer> origins, JsonNode report, String reportTool) {}
app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyExecutor.java
+40 -9
View File
@@ -58,6 +58,11 @@ public class PolicyExecutor {
// payload the tool surfaced alongside or instead of a file.
private record ToolResult(List<Resource> files, JsonNode report) {}
// A step's output files paired with each file's origin (the index into the original pipeline
// inputs it traces back to, or null when it has no single source). Origins compose across steps
// so the final result can be mapped back onto the files that entered the pipeline.
private record StepOutput(List<Resource> files, List<Integer> origins, JsonNode report) {}
/**
* Run every step in order, feeding each step's output into the next. Supporting files in {@code
* inputs} bind to named file fields and never enter the document stream.
@@ -75,6 +80,12 @@ public class PolicyExecutor {
List<Resource> currentFiles = inputs.primary();
Map<String, List<Resource>> supportingFiles = inputs.supportingFiles();
// Seed each input with its own index as origin; steps carry these through so the final
// outputs can be traced back to the files that entered the pipeline.
List<Integer> currentOrigins = new ArrayList<>();
for (int k = 0; k < currentFiles.size(); k++) {
currentOrigins.add(k);
}
// Last non-null report wins: the terminal step defines the output.
JsonNode lastReport = null;
String lastReportTool = null;
@@ -87,8 +98,10 @@ public class PolicyExecutor {
"Pipeline step " + (i + 1) + " has no operation");
}
listener.onStepStart(i + 1, steps.size(), operation);
ToolResult stepResult = executeStep(step, currentFiles, supportingFiles);
StepOutput stepResult =
executeStep(step, currentFiles, currentOrigins, supportingFiles);
currentFiles = stepResult.files();
currentOrigins = stepResult.origins();
if (stepResult.report() != null) {
lastReport = stepResult.report();
lastReportTool = operation;
@@ -96,7 +109,7 @@ public class PolicyExecutor {
listener.onStepComplete(i + 1, steps.size(), operation);
}
return new PolicyExecutionResult(currentFiles, lastReport, lastReportTool);
return new PolicyExecutionResult(currentFiles, currentOrigins, lastReport, lastReportTool);
}
/**
@@ -104,32 +117,50 @@ public class PolicyExecutor {
* responses are unpacked so each inner file is its own result (e.g. split). For per-file
* dispatch the first non-null report wins.
*/
private ToolResult executeStep(
private StepOutput executeStep(
PipelineStep step,
List<Resource> inputFiles,
List<Integer> inputOrigins,
Map<String, List<Resource>> supportingFiles)
throws IOException {
requireAcceptedTypes(step.operation(), inputFiles);
List<Resource> files = new ArrayList<>();
List<Integer> origins = new ArrayList<>();
JsonNode report = null;
if (toolMetadataService.isMultiInput(step.operation())) {
// One call over all inputs. The outputs derive from a single input only when exactly
// one entered; otherwise (a genuine merge) there is no single source.
ToolResult r = callEndpoint(step, inputFiles, supportingFiles);
files.addAll(r.files());
Integer origin = inputOrigins.size() == 1 ? inputOrigins.get(0) : null;
for (Resource file : r.files()) {
files.add(file);
origins.add(origin);
}
report = r.report();
} else if (inputFiles.isEmpty()) {
ToolResult r = callEndpoint(step, List.of(), supportingFiles);
files.addAll(r.files());
for (Resource file : r.files()) {
files.add(file);
origins.add(null);
}
report = r.report();
} else {
for (Resource file : inputFiles) {
ToolResult r = callEndpoint(step, List.of(file), supportingFiles);
files.addAll(r.files());
// One call per file: every output of this call inherits that input's origin, so a 1:1
// op keeps its chain and a split (one input, many outputs) tags each output with the
// same source.
for (int k = 0; k < inputFiles.size(); k++) {
Integer origin = inputOrigins.get(k);
ToolResult r = callEndpoint(step, List.of(inputFiles.get(k)), supportingFiles);
for (Resource file : r.files()) {
files.add(file);
origins.add(origin);
}
if (report == null) {
report = r.report();
}
}
}
return new ToolResult(files, report);
return new StepOutput(files, origins, report);
}
/**
app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyRunner.java
+41 -8
View File
@@ -13,6 +13,7 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.policy.input.InputSource;
import stirling.software.proprietary.policy.input.ResolvedInput;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.model.PipelineDefinition;
import stirling.software.proprietary.policy.model.Policy;
@@ -27,7 +28,8 @@ import stirling.software.proprietary.policy.source.SourceStore;
/**
* Turns a policy's referenced sources into runs: each {@code sourceId} is resolved live to its
* persisted {@link Source}, then to an {@link InputSpec}. Triggers decide <em>when</em> and call
* {@link #run(Policy)}; the controller uses the supplied-input and ad-hoc entry points.
* {@link #run(Policy)}; the controller uses the supplied-input and ad-hoc entry points. A {@link
* SweepKind#FULL} sweep also reconciles the processed-file ledger against what is present.
*/
@Slf4j
@Service
@@ -39,6 +41,12 @@ public class PolicyRunner {
private final List<InputSource> inputSources;
private final SourceStore sourceStore;
private final SourceDocCounter docCounter;
private final ProcessedLedger processedLedger;
/** Full-listing sweep: resolve every source, then reconcile the ledger. */
public List<String> run(Policy policy) {
return run(policy, SweepKind.FULL);
}
/**
* Trigger entry point. Pulls every referenced source; each yielded unit becomes its own run so
@@ -47,15 +55,21 @@ public class PolicyRunner {
* rest. Returns the ids of the runs it started (empty when sources yielded no work), so a
* manual trigger can report back which runs to follow.
*/
public List<String> run(Policy policy) {
public List<String> run(Policy policy, SweepKind sweep) {
long sweepStart = System.currentTimeMillis();
PolicySweep context = new PolicySweep(policy.id(), sweep, processedLedger);
List<String> runIds = new ArrayList<>();
List<String> sourceIds = policy.sourceIds();
if (sourceIds.isEmpty()) {
return List.of(startRun(policy, PolicyInputs.of(List.of()), unused -> {}));
// Generator pipeline: one run with no input. Still fall through to the cleanup
// below so rows recorded for its folder outputs are pruned like anything else,
// instead of accumulating until the policy is deleted.
runIds.add(startRun(policy, PolicyInputs.of(List.of()), unused -> {}));
}
List<String> runIds = new ArrayList<>();
for (String sourceId : sourceIds) {
Source source = sourceStore.get(sourceId).orElse(null);
if (source == null) {
// No veto: a deleted source's rows should age out via the cleanup below.
log.warn("Policy {} references missing source {}; skipping", policy.id(), sourceId);
continue;
}
@@ -65,9 +79,21 @@ public class PolicyRunner {
sourceId,
source.name(),
policy.id());
// Veto: a paused source's files cannot be stamped, so they must not be pruned.
context.vetoCleanup();
continue;
}
runIds.addAll(pullAndRun(policy, sourceId, source.toInputSpec()));
runIds.addAll(pullAndRun(policy, sourceId, source.toInputSpec(), context));
}
if (context.cleanupAllowed()) {
processedLedger.markSeen(policy.id(), context.presentIdentities());
int removed = processedLedger.deleteUnseen(policy.id(), sweepStart);
if (removed > 0) {
log.debug(
"Pruned {} ledger row(s) for files no longer present (policy {})",
removed,
policy.id());
}
}
return runIds;
}
@@ -86,26 +112,33 @@ public class PolicyRunner {
/**
* Resolves the source and starts a run per unit; records how many documents the source fed and
* returns the ids of the runs started.
* returns the ids of the runs started. Any source that could not be listed completely vetoes
* this sweep's ledger cleanup.
*/
private List<String> pullAndRun(Policy policy, String sourceId, InputSpec spec) {
private List<String> pullAndRun(
Policy policy, String sourceId, InputSpec spec, PolicySweep context) {
InputSource source = sourceFor(spec);
if (source == null) {
log.warn(
"No input source for type '{}' (policy {}); skipping",
spec.type(),
policy.id());
context.vetoCleanup();
return List.of();
}
if (!source.listsExhaustively()) {
context.vetoCleanup();
}
List<ResolvedInput> work;
try {
work = source.resolve(spec);
work = source.resolve(spec, context);
} catch (IOException | RuntimeException e) {
log.warn(
"Failed to resolve source '{}' for policy {}: {}",
spec.type(),
policy.id(),
e.getMessage());
context.vetoCleanup();
return List.of();
}
List<String> runIds = new ArrayList<>();
app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicySweep.java
+89
View File
@@ -0,0 +1,89 @@
package stirling.software.proprietary.policy.engine;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.function.Supplier;
import stirling.software.proprietary.policy.input.ResolveContext;
import stirling.software.proprietary.policy.ledger.ClaimState;
import stirling.software.proprietary.policy.ledger.ProcessedFileStatus;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
/**
* The {@link ResolveContext} for one policy sweep: scopes ledger calls to the policy, gathers the
* present-identity union across sources, prefetches claim state in bulk so per-file claims skip
* their row lookup, and vetoes presence cleanup when any source could not be listed completely
* (pruning would wrongly forget its files).
*/
final class PolicySweep implements ResolveContext {
private final String policyId;
private final SweepKind kind;
private final ProcessedLedger ledger;
private final Set<String> present = new HashSet<>();
// Claim states loaded in bulk at reportPresent; a claim outside the prefetch falls back to a
// single lookup. A stale entry cannot double-claim (the ledger re-checks every transition),
// it can only defer a file to the next sweep.
private final Map<String, ClaimState> prefetched = new HashMap<>();
private final Set<String> prefetchedIdentities = new HashSet<>();
private boolean cleanupVetoed;
PolicySweep(String policyId, SweepKind kind, ProcessedLedger ledger) {
this.policyId = policyId;
this.kind = kind;
this.ledger = ledger;
}
@Override
public synchronized boolean claim(String identity, String gate, Supplier<String> contentHash) {
ClaimState observed =
prefetchedIdentities.contains(identity)
? prefetched.get(identity)
: ledger.statesFor(policyId, List.of(identity)).get(identity);
boolean claimed = ledger.claim(policyId, identity, gate, contentHash, observed);
if (claimed) {
// A nested source surfacing the same file later in this sweep sees it in flight
// without another lookup.
prefetchedIdentities.add(identity);
prefetched.put(identity, new ClaimState(ProcessedFileStatus.PROCESSING, gate, null));
}
return claimed;
}
@Override
public void settle(
String identity, String finalGate, String finalContentHash, boolean success) {
ledger.settle(policyId, identity, finalGate, finalContentHash, success);
}
@Override
public boolean allSettledDone(String identity) {
// Deliberately not policy-scoped: consume deletion needs every claimant's consensus.
return ledger.allSettledDone(identity);
}
@Override
public synchronized void reportPresent(Collection<String> identities) {
if (kind == SweepKind.FULL) {
present.addAll(identities);
}
prefetched.putAll(ledger.statesFor(policyId, identities));
prefetchedIdentities.addAll(identities);
}
synchronized void vetoCleanup() {
cleanupVetoed = true;
}
synchronized boolean cleanupAllowed() {
return kind == SweepKind.FULL && !cleanupVetoed;
}
synchronized Set<String> presentIdentities() {
return Set.copyOf(present);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/SweepKind.java
+10
View File
@@ -0,0 +1,10 @@
package stirling.software.proprietary.policy.engine;
/**
* How thorough a policy sweep is: {@link #FULL} (complete listing; also stamps presence and prunes
* the ledger) or {@link #LIGHT} (event-driven; claims only, cost proportional to what changed).
*/
public enum SweepKind {
FULL,
LIGHT
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/input/FolderInputSource.java
+200 -74
View File
@@ -1,12 +1,17 @@
package stirling.software.proprietary.policy.input;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.NoSuchFileException;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import java.util.stream.Stream;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
@@ -19,17 +24,20 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.common.util.FileReadinessChecker;
import stirling.software.proprietary.policy.config.FolderAccessGuard;
import stirling.software.proprietary.policy.ledger.FolderIdentities;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.model.PolicyInputs;
/**
* Reads input files from a directory; each ready file is its own unit of work so one failure does
* not affect the others.
*
* <p>Mode option: "consume" (default) claims each file by moving it into {@code
* .stirling/processing} then routes it to {@code .stirling/done} or {@code .stirling/error}, so
* each file runs once; "snapshot" reads without moving, so every run sees the full set. Readiness
* is checked first so files mid-write are skipped.
* Reads input files from a directory; each ready file is its own unit of work, claimed through the
* {@link ResolveContext} ledger rather than moved aside, so nothing accumulates in a work
* directory. Options: "mode" is "consume" (default: a processed file is removed once every policy
* that claimed it has settled successfully and it is still the version that ran; failures stay in
* place and are not retried until they change) or "snapshot" (stateless, every run sees the full
* set); "recursive" descends into subdirectories; "identity" is "stat" (default, any size/mtime
* change is a new version) or "hash" (content-verified, so a touch does not reprocess). Hidden
* files and directories, including the legacy {@code .stirling} work dir, are never picked up, and
* files mid-write are skipped by the readiness check.
*/
@Slf4j
@Service
@@ -38,11 +46,6 @@ import stirling.software.proprietary.policy.model.PolicyInputs;
public class FolderInputSource implements InputSource {
private static final String TYPE = FolderAccessGuard.FOLDER_TYPE;
// Bookkeeping lives under one hidden dir so the watched folder stays tidy.
private static final String WORK_SUBDIR = ".stirling";
private static final String PROCESSING_SUBDIR = "processing";
private static final String DONE_SUBDIR = "done";
private static final String ERROR_SUBDIR = "error";
private final FileReadinessChecker readinessChecker;
private final FolderAccessGuard accessGuard;
@@ -68,70 +71,195 @@ public class FolderInputSource implements InputSource {
}
@Override
public List<ResolvedInput> resolve(InputSpec spec) throws IOException {
public List<ResolvedInput> resolve(InputSpec spec, ResolveContext ctx) throws IOException {
FolderConfig config = FolderConfig.from(spec.options());
Path inputDir = accessGuard.requirePermitted(config.directory());
if (!Files.isDirectory(inputDir)) {
log.debug("Folder input dir does not exist: {}", inputDir);
return List.of();
// Fail rather than return empty: an unmounted drive must read as "could not list",
// which vetoes the sweep's presence cleanup, not as "verifiably no files", which
// would wipe the policy's history and reprocess everything on remount.
throw new NoSuchFileException(
inputDir.toString(), null, "input directory does not exist");
}
Path canonicalDir = FolderIdentities.canonicalDir(inputDir);
List<Path> present = listFiles(inputDir, config.recursive());
if (config.snapshot()) {
List<ResolvedInput> work = new ArrayList<>();
for (Path file : present) {
if (readinessChecker.isReady(file)) {
work.add(ResolvedInput.of(PolicyInputs.of(List.of(fileResource(file)))));
}
}
return work;
}
List<Path> ready = new ArrayList<>();
try (Stream<Path> entries = Files.list(inputDir)) {
entries.filter(Files::isRegularFile)
.filter(readinessChecker::isReady)
.forEach(ready::add);
}
ctx.reportPresent(
present.stream()
.map(file -> FolderIdentities.identity(canonicalDir, inputDir, file))
.toList());
List<ResolvedInput> work = new ArrayList<>();
for (Path file : ready) {
if (config.snapshot()) {
work.add(ResolvedInput.of(PolicyInputs.of(List.of(fileResource(file)))));
} else {
Path claimed = claim(inputDir, file);
if (claimed == null) {
continue; // another sweep/process grabbed it
}
work.add(
new ResolvedInput(
PolicyInputs.of(List.of(fileResource(claimed))),
success -> route(inputDir, claimed, success)));
for (Path file : present) {
if (!readinessChecker.isReady(file)) {
continue;
}
String identity = FolderIdentities.identity(canonicalDir, inputDir, file);
MemoizedContentHash contentHash =
config.hashIdentity() ? new MemoizedContentHash(file) : null;
String gate;
boolean claimed;
try {
gate = FolderIdentities.statGate(file);
claimed = ctx.claim(identity, gate, contentHash);
} catch (IOException | UncheckedIOException e) {
log.debug("Could not read {} for its version: {}", file, e.getMessage());
continue; // vanished or unreadable mid-sweep; the next sweep sees the truth
}
if (!claimed) {
continue;
}
work.add(
new ResolvedInput(
PolicyInputs.of(List.of(fileResource(file))),
success ->
completeConsumed(
ctx, identity, file, gate, contentHash, success)));
}
return work;
}
// Atomic move into processing/: only one sweep can win the claim, the rest see the file gone.
private Path claim(Path inputDir, Path file) {
/**
* Settle at the version this run claimed - never a re-read, so a file replaced mid-run reads as
* a new unclaimed version next sweep instead of being marked processed. Then remove the input
* only when it is still the processed version (a mid-run replacement must survive) and every
* policy that claimed it has settled DONE, so co-watching policies all read the original and
* one failure parks the file for everyone. A failed run settles ERROR and never deletes; the
* DONE row of a file that could not be deleted still stops reprocessing.
*/
private static void completeConsumed(
ResolveContext ctx,
String identity,
Path file,
String claimGate,
MemoizedContentHash contentHash,
boolean success) {
ctx.settle(identity, claimGate, claimedHash(file, claimGate, contentHash), success);
if (!success) {
return;
}
try {
Path processingDir = workDir(inputDir, PROCESSING_SUBDIR);
Files.createDirectories(processingDir);
Path claimed = uniqueTarget(processingDir, file.getFileName().toString());
Files.move(file, claimed, StandardCopyOption.ATOMIC_MOVE);
return claimed;
if (FolderIdentities.statGate(file).equals(claimGate) && ctx.allSettledDone(identity)) {
Files.deleteIfExists(file);
}
} catch (NoSuchFileException alreadyGone) {
// Removed by the user or a co-watching policy's own consensus delete: nothing to do.
} catch (IOException e) {
log.debug("Could not claim {}: {}", file, e.getMessage());
log.warn("Could not remove consumed input {}: {}", file, e.getMessage());
}
}
/**
* The claimed version's content hash: the value computed during the claim when the ledger
* consulted the verifier, else computed now while the file is still at the claimed gate (so the
* hash describes what actually ran), else null. Always null in stat mode.
*/
private static String claimedHash(Path file, String claimGate, MemoizedContentHash hash) {
if (hash == null) {
return null;
}
String computed = hash.valueIfComputed();
if (computed != null) {
return computed;
}
try {
if (FolderIdentities.statGate(file).equals(claimGate)) {
return hash.get();
}
} catch (IOException | UncheckedIOException e) {
log.debug("Could not hash {} at settle: {}", file, e.getMessage());
}
return null;
}
private void route(Path inputDir, Path claimed, boolean success) {
String subdir = success ? DONE_SUBDIR : ERROR_SUBDIR;
try {
Path destDir = workDir(inputDir, subdir);
Files.createDirectories(destDir);
Files.move(
claimed,
uniqueTarget(destDir, claimed.getFileName().toString()),
StandardCopyOption.ATOMIC_MOVE);
} catch (IOException e) {
log.warn(
"Could not move processed input {} to {}: {}", claimed, subdir, e.getMessage());
/** Lazy verification tier: invoked at most once by the ledger, retained for the settle. */
private static final class MemoizedContentHash implements Supplier<String> {
private final Path file;
private volatile String value;
private MemoizedContentHash(Path file) {
this.file = file;
}
@Override
public String get() {
if (value == null) {
try {
value = FolderIdentities.contentHash(file);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
return value;
}
String valueIfComputed() {
return value;
}
}
private static Path workDir(Path inputDir, String subdir) {
return inputDir.resolve(WORK_SUBDIR).resolve(subdir);
/** Every non-hidden regular file in the source, readable or not. */
private static List<Path> listFiles(Path inputDir, boolean recursive) throws IOException {
List<Path> files = new ArrayList<>();
if (!recursive) {
try (Stream<Path> entries = Files.list(inputDir)) {
entries.filter(Files::isRegularFile)
.filter(file -> !hidden(file))
.forEach(files::add);
}
return files;
}
// Hidden subtrees are pruned wholesale; symlinked directories are not followed.
Files.walkFileTree(
inputDir,
new SimpleFileVisitor<>() {
@Override
public FileVisitResult preVisitDirectory(
Path dir, BasicFileAttributes attributes) {
if (!dir.equals(inputDir) && hidden(dir)) {
return FileVisitResult.SKIP_SUBTREE;
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) {
if (attributes.isRegularFile() && !hidden(file)) {
files.add(file);
}
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult visitFileFailed(Path file, IOException e) {
log.debug("Skipping unreadable entry {}: {}", file, e.getMessage());
return FileVisitResult.CONTINUE;
}
});
return files;
}
private static boolean hidden(Path path) {
Path name = path.getFileName();
if (name != null && name.toString().startsWith(".")) {
return true;
}
try {
return Files.isHidden(path);
} catch (IOException e) {
return false;
}
}
private static Resource fileResource(Path path) {
@@ -144,27 +272,15 @@ public class FolderInputSource implements InputSource {
};
}
private static Path uniqueTarget(Path dir, String filename) {
Path candidate = dir.resolve(filename);
if (!Files.exists(candidate)) {
return candidate;
}
int dot = filename.lastIndexOf('.');
String base = dot < 0 ? filename : filename.substring(0, dot);
String ext = dot < 0 ? "" : filename.substring(dot);
for (int n = 1; ; n++) {
Path next = dir.resolve(base + " (" + n + ")" + ext);
if (!Files.exists(next)) {
return next;
}
}
}
record FolderConfig(Path directory, boolean snapshot) {
record FolderConfig(Path directory, boolean snapshot, boolean recursive, boolean hashIdentity) {
private static final String DIRECTORY_OPTION = "directory";
private static final String MODE_OPTION = "mode";
private static final String MODE_SNAPSHOT = "snapshot";
private static final String RECURSIVE_OPTION = "recursive";
private static final String IDENTITY_OPTION = "identity";
private static final String IDENTITY_STAT = "stat";
private static final String IDENTITY_HASH = "hash";
static FolderConfig from(Map<String, Object> options) {
Object directory = options.get(DIRECTORY_OPTION);
@@ -173,7 +289,17 @@ public class FolderInputSource implements InputSource {
}
Object mode = options.get(MODE_OPTION);
boolean snapshot = mode != null && MODE_SNAPSHOT.equals(mode.toString());
return new FolderConfig(Path.of(directory.toString()), snapshot);
Object recursive = options.get(RECURSIVE_OPTION);
boolean recurse = recursive != null && Boolean.parseBoolean(recursive.toString());
Object identity = options.get(IDENTITY_OPTION);
boolean hash = identity != null && IDENTITY_HASH.equals(identity.toString());
if (identity != null
&& !IDENTITY_STAT.equals(identity.toString())
&& !IDENTITY_HASH.equals(identity.toString())) {
throw new IllegalArgumentException(
"folder input 'identity' must be 'stat' or 'hash'");
}
return new FolderConfig(Path.of(directory.toString()), snapshot, recurse, hash);
}
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/input/InputSource.java
+14 -2
View File
@@ -24,9 +24,21 @@ public interface InputSource {
/**
* Resolve the spec into zero or more units of work, each carrying one run's files and a
* completion hook. Empty list means nothing to run right now.
* completion hook. Empty list means nothing to run right now. Discovery is read-only - files
* stay where the user put them; "already processed" is tracked through {@code ctx} (claim on
* pickup, settle on completion, report what is present so stale ledger rows can be pruned).
*/
List<ResolvedInput> resolve(InputSpec spec) throws IOException;
List<ResolvedInput> resolve(InputSpec spec, ResolveContext ctx) throws IOException;
/**
* Whether {@link #resolve} observes everything in the source (a complete listing) rather than
* e.g. only what events surfaced. Presence cleanup of the ledger is skipped for the whole
* policy unless every enabled source says true - wrongly pruning history would reprocess a
* whole folder, while keeping a few stale rows costs nothing.
*/
default boolean listsExhaustively() {
return true;
}
/**
* Filesystem dirs this source draws from, for the folder-watch trigger. Advisory: resolving is
app/proprietary/src/main/java/stirling/software/proprietary/policy/input/ResolveContext.java
+36
View File
@@ -0,0 +1,36 @@
package stirling.software.proprietary.policy.input;
import java.util.Collection;
import java.util.function.Supplier;
/**
* A source's policy-scoped window onto the processed-file ledger for one sweep. Thread-safe and
* valid for the lifetime of the work units the source issued ({@link #settle} fires from async run
* completions).
*/
public interface ResolveContext {
/**
* Atomically claim a file at its current version; true means this sweep runs it. A null {@code
* contentHash} makes any gate change a new version; a non-null supplier is invoked at most
* once, only on a gate mismatch, and a matching hash refreshes the stored gate instead of
* reprocessing. Supplier exceptions propagate.
*/
boolean claim(String identity, String gate, Supplier<String> contentHash);
/** Record a claimed file's outcome at its final version ({@code finalContentHash} nullable). */
void settle(String identity, String finalGate, String finalContentHash, boolean success);
/**
* Whether every policy holding a ledger row for this identity has settled it DONE. Cross-policy
* by design: consume-mode deletion is a consensus of all claimants, so a shared input is
* removed only once nobody still needs it (in-flight, failed, and interrupted rows all veto).
*/
boolean allSettledDone(String identity);
/**
* Report every identity present right now, readable or not; feeds presence cleanup of rows
* whose file is gone.
*/
void reportPresent(Collection<String> identities);
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/ClaimState.java
+9
View File
@@ -0,0 +1,9 @@
package stirling.software.proprietary.policy.ledger;
/**
* A row's claim-relevant state as read by {@link ProcessedLedger#statesFor}: what a sweep observed
* before deciding a claim. May be stale by the time the claim runs; every ledger transition
* re-checks the observed state in its WHERE clause, so staleness defers a claim to a later sweep
* rather than double-running one.
*/
public record ClaimState(ProcessedFileStatus status, String gate, String contentHash) {}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/FolderIdentities.java
+41
View File
@@ -0,0 +1,41 @@
package stirling.software.proprietary.policy.ledger;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.BasicFileAttributes;
import stirling.software.proprietary.billing.ContentHasher;
/**
* The folder backend's identity and version scheme, shared by {@code FolderInputSource} and {@code
* FolderOutputSink} so outputs are recorded under exactly the identity the next scan derives.
* Directories are canonicalised with {@code toRealPath()} so symlinked aliases agree.
*/
public final class FolderIdentities {
private FolderIdentities() {}
/** Canonical form of a configured directory; resolves symlinks, so the dir must exist. */
public static Path canonicalDir(Path dir) throws IOException {
return dir.toRealPath();
}
/** Identity of {@code file} under {@code dir}: its path re-rooted onto the canonical dir. */
public static String identity(Path canonicalDir, Path dir, Path file) {
return canonicalDir.resolve(dir.relativize(file)).normalize().toString();
}
/** The cheap version gate: a change to content length or mtime means "look closer". */
public static String statGate(Path file) throws IOException {
BasicFileAttributes attributes = Files.readAttributes(file, BasicFileAttributes.class);
return attributes.size() + ":" + attributes.lastModifiedTime().toMillis();
}
/**
* The strong version token: distinguishes a real change from a touch, at the cost of a read.
*/
public static String contentHash(Path file) throws IOException {
return ContentHasher.sha256(file);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/IdentityHasher.java
+19
View File
@@ -0,0 +1,19 @@
package stirling.software.proprietary.policy.ledger;
import java.nio.charset.StandardCharsets;
import stirling.software.proprietary.billing.ContentHasher;
/**
* Fixed-width key form of a source-owned identity, so any identity length fits the ledger's primary
* key. Backend-agnostic: every source type's identities are keyed through here, which is why this
* does not live with the folder backend's {@link FolderIdentities}.
*/
public final class IdentityHasher {
private IdentityHasher() {}
public static String identityHash(String identity) {
return ContentHasher.sha256(identity.getBytes(StandardCharsets.UTF_8));
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/InProcessProcessedLedger.java
+229
View File
@@ -0,0 +1,229 @@
package stirling.software.proprietary.policy.ledger;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.function.Supplier;
/**
* In-memory {@link ProcessedLedger} for tests and DB-less wiring; kept semantically identical to
* {@code JpaProcessedLedger} by the shared contract test.
*/
public class InProcessProcessedLedger implements ProcessedLedger {
private final Map<String, Map<String, Row>> rowsByPolicy = new HashMap<>();
private final Supplier<Long> nowMillis;
public InProcessProcessedLedger() {
this(System::currentTimeMillis);
}
public InProcessProcessedLedger(Supplier<Long> nowMillis) {
this.nowMillis = nowMillis;
}
@Override
public synchronized Map<String, ClaimState> statesFor(
String policyId, Collection<String> identities) {
Map<String, Row> rows = rowsByPolicy.getOrDefault(policyId, Map.of());
Map<String, ClaimState> states = new HashMap<>();
for (String identity : identities) {
Row row = rows.get(identity);
if (row != null) {
states.put(identity, new ClaimState(row.status, row.gate, row.contentHash));
}
}
return states;
}
// Single-lock store: the live row is never staler than any observed snapshot, so decide
// against it directly; the conditional updates of the JPA ledger yield the same outcomes.
@Override
public synchronized boolean claim(
String policyId,
String identity,
String gate,
Supplier<String> contentHash,
ClaimState observed) {
Map<String, Row> rows = rowsByPolicy.computeIfAbsent(policyId, key -> new HashMap<>());
long now = nowMillis.get();
Row row = rows.get(identity);
if (row == null) {
String hash = contentHash == null ? null : contentHash.get();
rows.put(identity, new Row(gate, hash, ProcessedFileStatus.PROCESSING, 1, now));
return true;
}
if (row.status == ProcessedFileStatus.PROCESSING) {
return false;
}
if (gate.equals(row.gate)) {
if (row.status == ProcessedFileStatus.INTERRUPTED && row.attempts < MAX_ATTEMPTS) {
row.status = ProcessedFileStatus.PROCESSING;
row.attempts++;
row.lastSeen = now;
return true;
}
return false;
}
if (contentHash == null) {
row.gate = gate;
row.contentHash = null;
row.status = ProcessedFileStatus.PROCESSING;
row.attempts = 1;
row.lastSeen = now;
return true;
}
String hash = contentHash.get();
if (Objects.equals(hash, row.contentHash)) {
if (row.status == ProcessedFileStatus.INTERRUPTED && row.attempts < MAX_ATTEMPTS) {
row.gate = gate;
row.status = ProcessedFileStatus.PROCESSING;
row.attempts++;
row.lastSeen = now;
return true;
}
if (row.status != ProcessedFileStatus.INTERRUPTED) {
row.gate = gate;
row.lastSeen = now;
}
return false;
}
row.gate = gate;
row.contentHash = hash;
row.status = ProcessedFileStatus.PROCESSING;
row.attempts = 1;
row.lastSeen = now;
return true;
}
@Override
public synchronized void settle(
String policyId,
String identity,
String finalGate,
String finalContentHash,
boolean success) {
upsertSettled(
policyId,
identity,
finalGate,
finalContentHash,
success ? ProcessedFileStatus.DONE : ProcessedFileStatus.ERROR);
}
@Override
public synchronized void recordOutput(
String policyId, String identity, String gate, String contentHash) {
upsertSettled(policyId, identity, gate, contentHash, ProcessedFileStatus.DONE);
}
@Override
public synchronized void forgetOutput(String policyId, String identity, String gate) {
Map<String, Row> rows = rowsByPolicy.get(policyId);
if (rows == null) {
return;
}
Row row = rows.get(identity);
if (row != null && row.status == ProcessedFileStatus.DONE && gate.equals(row.gate)) {
rows.remove(identity);
}
}
private void upsertSettled(
String policyId,
String identity,
String gate,
String contentHash,
ProcessedFileStatus status) {
Map<String, Row> rows = rowsByPolicy.computeIfAbsent(policyId, key -> new HashMap<>());
long now = nowMillis.get();
Row row = rows.get(identity);
if (row == null) {
rows.put(identity, new Row(gate, contentHash, status, 1, now));
return;
}
row.gate = gate;
row.contentHash = contentHash;
row.status = status;
row.lastSeen = now;
}
@Override
public synchronized boolean allSettledDone(String identity) {
for (Map<String, Row> rows : rowsByPolicy.values()) {
Row row = rows.get(identity);
if (row != null && row.status != ProcessedFileStatus.DONE) {
return false;
}
}
return true;
}
@Override
public synchronized void markSeen(String policyId, Collection<String> identities) {
Map<String, Row> rows = rowsByPolicy.get(policyId);
if (rows == null) {
return;
}
long now = nowMillis.get();
for (String identity : identities) {
Row row = rows.get(identity);
if (row != null) {
row.lastSeen = now;
}
}
}
@Override
public synchronized int deleteUnseen(String policyId, long seenSinceMillis) {
Map<String, Row> rows = rowsByPolicy.get(policyId);
if (rows == null) {
return 0;
}
int before = rows.size();
rows.values()
.removeIf(
row ->
row.lastSeen < seenSinceMillis
&& row.status != ProcessedFileStatus.PROCESSING);
return before - rows.size();
}
@Override
public synchronized void clearPolicy(String policyId) {
rowsByPolicy.remove(policyId);
}
@Override
public synchronized void recoverInterrupted() {
for (Map<String, Row> rows : rowsByPolicy.values()) {
for (Row row : rows.values()) {
if (row.status == ProcessedFileStatus.PROCESSING) {
row.status = ProcessedFileStatus.INTERRUPTED;
}
}
}
}
private static final class Row {
private String gate;
private String contentHash;
private ProcessedFileStatus status;
private int attempts;
private long lastSeen;
private Row(
String gate,
String contentHash,
ProcessedFileStatus status,
int attempts,
long lastSeen) {
this.gate = gate;
this.contentHash = contentHash;
this.status = status;
this.attempts = attempts;
this.lastSeen = lastSeen;
}
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/JpaProcessedLedger.java
+212
View File
@@ -0,0 +1,212 @@
package stirling.software.proprietary.policy.ledger;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.stereotype.Service;
import lombok.extern.slf4j.Slf4j;
/**
* Durable {@link ProcessedLedger}; the runtime bean. A fresh claim is a flushed insert so a
* concurrent winner surfaces as a constraint violation; every other transition is a conditional
* update that re-checks the observed state, so a lost race reports 0 rows and the caller skips.
* Boot recovery assumes the single node the folder-watch trigger assumes: runs live in memory, so
* after a restart every PROCESSING row is stale.
*/
@Slf4j
@Service
@ConditionalOnBooleanProperty(name = "policies.enabled")
public class JpaProcessedLedger implements ProcessedLedger {
private static final int STAMP_CHUNK = 500;
private final ProcessedFileRepository repository;
private final Supplier<Long> nowMillis;
@Autowired
public JpaProcessedLedger(ProcessedFileRepository repository) {
this(repository, System::currentTimeMillis);
}
// Clock seam so tests can pin "now"; the runtime bean uses the wall clock above.
JpaProcessedLedger(ProcessedFileRepository repository, Supplier<Long> nowMillis) {
this.repository = repository;
this.nowMillis = nowMillis;
}
@Override
public Map<String, ClaimState> statesFor(String policyId, Collection<String> identities) {
if (identities.isEmpty()) {
return Map.of();
}
Map<String, String> identityByHash = new HashMap<>();
for (String identity : identities) {
identityByHash.put(IdentityHasher.identityHash(identity), identity);
}
Map<String, ClaimState> states = new HashMap<>();
List<String> hashes = List.copyOf(identityByHash.keySet());
for (int from = 0; from < hashes.size(); from += STAMP_CHUNK) {
List<ProcessedFileEntity> rows =
repository.findByPolicyIdAndIdentityHashIn(
policyId,
hashes.subList(from, Math.min(from + STAMP_CHUNK, hashes.size())));
for (ProcessedFileEntity row : rows) {
states.put(
identityByHash.get(row.getIdentityHash()),
new ClaimState(row.getStatus(), row.getSignature(), row.getContentHash()));
}
}
return states;
}
@Override
public boolean claim(
String policyId,
String identity,
String gate,
Supplier<String> contentHash,
ClaimState observed) {
String identityHash = IdentityHasher.identityHash(identity);
long now = nowMillis.get();
if (observed == null) {
try {
repository.saveAndFlush(
new ProcessedFileEntity(
policyId,
identityHash,
identity,
gate,
contentHash == null ? null : contentHash.get(),
ProcessedFileStatus.PROCESSING,
now));
return true;
} catch (DataIntegrityViolationException concurrentClaim) {
return false;
}
}
if (observed.status() == ProcessedFileStatus.PROCESSING) {
return false;
}
if (gate.equals(observed.gate())) {
if (observed.status() == ProcessedFileStatus.INTERRUPTED) {
return repository.retryInterruptedAtGate(
policyId, identityHash, gate, MAX_ATTEMPTS, now)
> 0;
}
return false;
}
if (contentHash == null) {
return repository.reclaimAtNewGate(policyId, identityHash, gate, now) > 0;
}
String hash = contentHash.get();
if (hash.equals(observed.contentHash())) {
if (observed.status() == ProcessedFileStatus.INTERRUPTED) {
return repository.retryInterruptedSameContent(
policyId, identityHash, gate, hash, MAX_ATTEMPTS, now)
> 0;
}
repository.refreshGate(policyId, identityHash, gate, hash, now);
return false;
}
return repository.reclaimAtNewContent(policyId, identityHash, gate, hash, now) > 0;
}
@Override
public void settle(
String policyId,
String identity,
String finalGate,
String finalContentHash,
boolean success) {
upsertSettled(
policyId,
identity,
finalGate,
finalContentHash,
success ? ProcessedFileStatus.DONE : ProcessedFileStatus.ERROR);
}
@Override
public void recordOutput(String policyId, String identity, String gate, String contentHash) {
upsertSettled(policyId, identity, gate, contentHash, ProcessedFileStatus.DONE);
}
@Override
public void forgetOutput(String policyId, String identity, String gate) {
repository.deleteDoneAt(policyId, IdentityHasher.identityHash(identity), gate);
}
/**
* Settle-or-insert: the row may have been presence-cleaned mid-run, and an output row may be
* brand new.
*/
private void upsertSettled(
String policyId,
String identity,
String gate,
String contentHash,
ProcessedFileStatus status) {
String identityHash = IdentityHasher.identityHash(identity);
long now = nowMillis.get();
if (repository.settle(policyId, identityHash, gate, contentHash, status, now) > 0) {
return;
}
try {
ProcessedFileEntity row =
new ProcessedFileEntity(
policyId, identityHash, identity, gate, contentHash, status, now);
repository.saveAndFlush(row);
} catch (DataIntegrityViolationException concurrentInsert) {
repository.settle(policyId, identityHash, gate, contentHash, status, now);
}
}
@Override
public boolean allSettledDone(String identity) {
return !repository.existsByIdentityHashAndStatusNot(
IdentityHasher.identityHash(identity), ProcessedFileStatus.DONE);
}
@Override
public void markSeen(String policyId, Collection<String> identities) {
if (identities.isEmpty()) {
return;
}
List<String> hashes = identities.stream().map(IdentityHasher::identityHash).toList();
long now = nowMillis.get();
for (int from = 0; from < hashes.size(); from += STAMP_CHUNK) {
repository.stampSeen(
policyId,
hashes.subList(from, Math.min(from + STAMP_CHUNK, hashes.size())),
now);
}
}
@Override
public int deleteUnseen(String policyId, long seenSinceMillis) {
return repository.deleteUnseen(policyId, seenSinceMillis);
}
@Override
public void clearPolicy(String policyId) {
repository.deleteByPolicy(policyId);
}
@Override
@EventListener(ApplicationReadyEvent.class)
public void recoverInterrupted() {
int recovered = repository.markAllProcessingInterrupted(nowMillis.get());
if (recovered > 0) {
log.info("Recovered {} policy input file(s) interrupted by shutdown", recovered);
}
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/ProcessedFileEntity.java
+106
View File
@@ -0,0 +1,106 @@
package stirling.software.proprietary.policy.ledger;
import java.io.Serializable;
import org.springframework.data.domain.Persistable;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.EnumType;
import jakarta.persistence.Enumerated;
import jakarta.persistence.Id;
import jakarta.persistence.IdClass;
import jakarta.persistence.Index;
import jakarta.persistence.Table;
import jakarta.persistence.Transient;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
/**
* One processed-file ledger row: the version a policy last settled a file at, and where it is in
* the claim lifecycle. Keyed by SHA-256 of the source-owned identity so any identity length fits a
* fixed-width index. {@code isNew} is always true: the entity is only saved for fresh inserts
* (everything else is a conditional update), so a lost insert race surfaces as a constraint
* violation rather than a silent merge.
*/
@Entity
@Table(
name = "policy_processed_files",
indexes = {
// presence cleanup: delete this policy's rows unseen since the sweep began
@Index(name = "idx_processed_files_policy_seen", columnList = "policy_id, last_seen"),
// cross-policy deletion consensus: existsByIdentityHashAndStatusNot filters
// identity_hash on its own, so it cannot ride the (policy_id, identity_hash) PK
@Index(name = "idx_processed_files_identity", columnList = "identity_hash")
})
@IdClass(ProcessedFileId.class)
@NoArgsConstructor
@Getter
@Setter
public class ProcessedFileEntity implements Serializable, Persistable<ProcessedFileId> {
private static final long serialVersionUID = 1L;
@Id
@Column(name = "policy_id")
private String policyId;
@Id
@Column(name = "identity_hash", length = 64)
private String identityHash;
@Column(name = "identity", length = 4096)
private String identity;
@Column(name = "signature")
private String signature;
@Column(name = "content_hash", length = 64)
private String contentHash;
@Enumerated(EnumType.STRING)
@Column(name = "status", length = 16)
private ProcessedFileStatus status;
@Column(name = "attempts")
private int attempts;
@Column(name = "last_seen")
private long lastSeen;
@Column(name = "updated_at")
private long updatedAt;
public ProcessedFileEntity(
String policyId,
String identityHash,
String identity,
String signature,
String contentHash,
ProcessedFileStatus status,
long nowMillis) {
this.policyId = policyId;
this.identityHash = identityHash;
this.identity = identity;
this.signature = signature;
this.contentHash = contentHash;
this.status = status;
this.attempts = 1;
this.lastSeen = nowMillis;
this.updatedAt = nowMillis;
}
@Override
@Transient
public ProcessedFileId getId() {
return new ProcessedFileId(policyId, identityHash);
}
@Override
@Transient
public boolean isNew() {
return true;
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/ProcessedFileId.java
+37
View File
@@ -0,0 +1,37 @@
package stirling.software.proprietary.policy.ledger;
import java.io.Serializable;
import java.util.Objects;
/** Composite key for {@link ProcessedFileEntity}: one row per policy per file identity. */
public class ProcessedFileId implements Serializable {
private static final long serialVersionUID = 1L;
private String policyId;
private String identityHash;
public ProcessedFileId() {}
public ProcessedFileId(String policyId, String identityHash) {
this.policyId = policyId;
this.identityHash = identityHash;
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof ProcessedFileId other)) {
return false;
}
return Objects.equals(policyId, other.policyId)
&& Objects.equals(identityHash, other.identityHash);
}
@Override
public int hashCode() {
return Objects.hash(policyId, identityHash);
}
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/ProcessedFileRepository.java
+195
View File
@@ -0,0 +1,195 @@
package stirling.software.proprietary.policy.ledger;
import java.util.Collection;
import java.util.List;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
/**
* Conditional updates for the processed-file ledger: each claim variant re-checks in its WHERE
* clause the state it was decided against, so a racing claim loses cleanly with 0 rows updated.
* Transactional per call so the ledger can run them without an enclosing transaction.
*/
@Repository
public interface ProcessedFileRepository
extends JpaRepository<ProcessedFileEntity, ProcessedFileId> {
/**
* Re-claim a settled row at a new gate without content verification; clears the stored hash,
* which described content this claim never checked.
*/
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
+ " e.signature = :gate, e.contentHash = null, e.attempts = 1,"
+ " e.lastSeen = :now, e.updatedAt = :now"
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
+ " and e.status <>"
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING"
+ " and e.signature <> :gate")
int reclaimAtNewGate(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate,
@Param("now") long now);
/** Re-claim a settled row whose content verifiably changed (or was never hashed). */
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
+ " e.signature = :gate, e.contentHash = :contentHash, e.attempts = 1,"
+ " e.lastSeen = :now, e.updatedAt = :now"
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
+ " and e.status <>"
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING"
+ " and (e.contentHash is null or e.contentHash <> :contentHash)")
int reclaimAtNewContent(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate,
@Param("contentHash") String contentHash,
@Param("now") long now);
/** The gate moved but the content did not: track the new gate without changing status. */
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.signature = :gate, e.lastSeen = :now,"
+ " e.updatedAt = :now"
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
+ " and e.status <>"
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING"
+ " and e.contentHash = :contentHash and e.signature <> :gate")
int refreshGate(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate,
@Param("contentHash") String contentHash,
@Param("now") long now);
/** Bounded retry of an INTERRUPTED row at the same gate. */
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
+ " e.attempts = e.attempts + 1, e.lastSeen = :now, e.updatedAt = :now"
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
+ " and e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.INTERRUPTED"
+ " and e.signature = :gate and e.attempts < :maxAttempts")
int retryInterruptedAtGate(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate,
@Param("maxAttempts") int maxAttempts,
@Param("now") long now);
/** Bounded retry of an INTERRUPTED row whose gate moved but whose content is unchanged. */
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
+ " e.signature = :gate, e.attempts = e.attempts + 1, e.lastSeen = :now,"
+ " e.updatedAt = :now"
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
+ " and e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.INTERRUPTED"
+ " and e.contentHash = :contentHash and e.attempts < :maxAttempts")
int retryInterruptedSameContent(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate,
@Param("contentHash") String contentHash,
@Param("maxAttempts") int maxAttempts,
@Param("now") long now);
/**
* Unconditional settle (only the claiming run settles a row); returns 0 when the row was
* removed mid-run so the caller re-inserts.
*/
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.status = :status, e.signature = :gate,"
+ " e.contentHash = :contentHash, e.lastSeen = :now, e.updatedAt = :now"
+ " where e.policyId = :policyId and e.identityHash = :identityHash")
int settle(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate,
@Param("contentHash") String contentHash,
@Param("status") ProcessedFileStatus status,
@Param("now") long now);
/** Whether any policy's row at this identity is in a state other than {@code status}. */
boolean existsByIdentityHashAndStatusNot(String identityHash, ProcessedFileStatus status);
/** One policy's rows across a chunk of identity hashes, for a sweep's claim snapshot. */
List<ProcessedFileEntity> findByPolicyIdAndIdentityHashIn(
String policyId, Collection<String> identityHashes);
/**
* Remove an output record whose rename never landed, only while still settled exactly as
* recorded; a row a claim has since taken over is left alone.
*/
@Modifying
@Transactional
@Query(
"delete from ProcessedFileEntity e where e.policyId = :policyId"
+ " and e.identityHash = :identityHash and e.signature = :gate"
+ " and e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.DONE")
int deleteDoneAt(
@Param("policyId") String policyId,
@Param("identityHash") String identityHash,
@Param("gate") String gate);
/** Stamp presence for the given identities; chunked by the caller for very large folders. */
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.lastSeen = :now"
+ " where e.policyId = :policyId and e.identityHash in :identityHashes")
int stampSeen(
@Param("policyId") String policyId,
@Param("identityHashes") Collection<String> identityHashes,
@Param("now") long now);
/**
* Presence cleanup: remove rows not stamped since the sweep began, keeping in-flight claims.
*/
@Modifying
@Transactional
@Query(
"delete from ProcessedFileEntity e where e.policyId = :policyId"
+ " and e.lastSeen < :cutoff and e.status <>"
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING")
int deleteUnseen(@Param("policyId") String policyId, @Param("cutoff") long cutoff);
@Modifying
@Transactional
@Query("delete from ProcessedFileEntity e where e.policyId = :policyId")
int deleteByPolicy(@Param("policyId") String policyId);
/** Boot recovery: after a restart every PROCESSING row is stale (single node). */
@Modifying
@Transactional
@Query(
"update ProcessedFileEntity e set e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.INTERRUPTED,"
+ " e.updatedAt = :now"
+ " where e.status ="
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING")
int markAllProcessingInterrupted(@Param("now") long now);
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/ProcessedFileStatus.java
+17
View File
@@ -0,0 +1,17 @@
package stirling.software.proprietary.policy.ledger;
/** Lifecycle of one {@code (policy, file)} ledger row. */
public enum ProcessedFileStatus {
/** Claimed; a run is in flight. */
PROCESSING,
/** Run completed at this version. */
DONE,
/** Run failed; skipped until the file changes (clear-history is the manual retry). */
ERROR,
/** Was PROCESSING when the JVM died; retried a bounded number of times. */
INTERRUPTED
}
app/proprietary/src/main/java/stirling/software/proprietary/policy/ledger/ProcessedLedger.java
+100
View File
@@ -0,0 +1,100 @@
package stirling.software.proprietary.policy.ledger;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
/**
* Remembers which files a policy has processed, one row per {@code (policy, identity)}, so sources
* track files in place. Identities are opaque source-owned strings; versions are two-tier: a cheap
* gate compared every sweep plus an optional content hash consulted only when the gate moves.
* Presence reconciliation ({@link #markSeen} + {@link #deleteUnseen}) keeps the table bounded.
*/
public interface ProcessedLedger {
/**
* Claims of one version before an {@link ProcessedFileStatus#INTERRUPTED} row stops retrying.
*/
int MAX_ATTEMPTS = 3;
/**
* One-query snapshot of the rows for these identities, keyed by identity; identities with no
* row are absent. Feeds the {@code observed} parameter of {@link #claim(String, String, String,
* Supplier, ClaimState)} so a sweep decides its claims without a per-file lookup.
*/
Map<String, ClaimState> statesFor(String policyId, Collection<String> identities);
/**
* Atomically claim a file at its current version, deciding against {@code observed} (this row's
* entry from {@link #statesFor}; null means no row was seen); true means this caller runs it. A
* stale {@code observed} cannot double-claim - every transition re-checks the observed state,
* so a lost race skips until a later sweep. A null {@code contentHash} makes any gate change a
* new version; a non-null supplier is invoked at most once, only on a gate mismatch, and a
* matching hash refreshes the stored gate instead of reprocessing. Supplier exceptions
* propagate.
*/
boolean claim(
String policyId,
String identity,
String gate,
Supplier<String> contentHash,
ClaimState observed);
/** Snapshot-then-claim convenience for a single file; sweeps batch via {@link #statesFor}. */
default boolean claim(
String policyId, String identity, String gate, Supplier<String> contentHash) {
return claim(
policyId,
identity,
gate,
contentHash,
statesFor(policyId, List.of(identity)).get(identity));
}
/** Record a claimed file's outcome at its final version ({@code finalContentHash} nullable). */
void settle(
String policyId,
String identity,
String finalGate,
String finalContentHash,
boolean success);
/**
* Record a produced file as {@link ProcessedFileStatus#DONE} so the policy skips its own
* outputs. Must be called before the file is visible at this identity; other policies have no
* row and still process it.
*/
void recordOutput(String policyId, String identity, String gate, String contentHash);
/**
* Remove an output record whose file never became visible (its rename lost the name race to a
* concurrent writer), so whatever file actually owns that identity is claimable at any version.
* A no-op unless the row is still settled exactly as recorded, so a claim that took the row
* over in the meantime is left alone.
*/
void forgetOutput(String policyId, String identity, String gate);
/**
* Whether every row at this identity - across all policies, by design - is {@link
* ProcessedFileStatus#DONE}. Consume-mode deletion gates on this so a shared input is removed
* only once every claimant has processed it; in-flight, failed, and interrupted rows all veto,
* parking the file. Vacuously true when no rows exist.
*/
boolean allSettledDone(String identity);
/** Stamp presence for every identity a full-listing sweep observed. */
void markSeen(String policyId, Collection<String> identities);
/**
* Remove rows not seen since {@code seenSinceMillis}, keeping in-flight claims. Only call after
* every enabled source listed completely; returns the number of rows removed.
*/
int deleteUnseen(String policyId, long seenSinceMillis);
/** Forget everything for a policy. */
void clearPolicy(String policyId);
/** Boot recovery: flip stale in-flight claims to {@link ProcessedFileStatus#INTERRUPTED}. */
void recoverInterrupted();
}

Some files were not shown because too many files have changed in this diff Show More