mirror of
https://github.com/Stirling-Tools/Stirling-PDF.git
synced 2026-09-03 21:30:14 +03:00
Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e2deab2a81 | ||
|
|
3d69006d9d | ||
|
|
35cb66bd5e | ||
|
|
3a57da928d | ||
|
|
596f5fd42c | ||
|
|
68b1c86d4b | ||
|
|
4a1773cd88 | ||
|
|
c73dd3d8de |
@@ -1,97 +0,0 @@
|
||||
---
|
||||
name: feature-walkthrough
|
||||
description: >-
|
||||
Explain the full logic and process of the current branch end-to-end so someone
|
||||
with no prior knowledge of the task can understand, review, and reproduce it.
|
||||
Scopes the change from the branch diff, traces the flow across every layer it
|
||||
touches (frontend tool/hook/component, Java controller/service/endpoint, Python
|
||||
engine, config, i18n, tests), and produces a self-contained walkthrough document
|
||||
with Mermaid diagrams (sequence/flow/architecture), annotated file map with
|
||||
clickable references, before/after behavior, screenshots where a UI is involved,
|
||||
a "try it locally" section, and edge cases/risks. Use when asked for a feature or
|
||||
branch walkthrough, "explain what this branch does", a design/logic writeup, PR
|
||||
reviewer onboarding, or a hand-off doc. Pass --html to also emit a rendered HTML
|
||||
version; --no-screens to skip screenshots.
|
||||
argument-hint: "[branch-or-area] [--html] [--no-screens]"
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# Feature / Branch Walkthrough
|
||||
|
||||
Turn the current branch into a walkthrough a newcomer can follow. Audience:
|
||||
**someone who has never seen this task**. Explain the *why*, the *flow*, and *how to
|
||||
try it* - not just a diff summary.
|
||||
|
||||
`$ARGUMENTS` may name a branch or area to focus on; default is the current branch
|
||||
vs `main`. Flags: `--html` (also emit a rendered HTML twin), `--no-screens`.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Scope the change
|
||||
- `git log --oneline main..HEAD` and `git diff --stat main...HEAD` for the shape.
|
||||
- Read the PR description / commit messages for stated intent. Do **not** invent
|
||||
history or motivation that isn't evidenced (state current behavior in present tense).
|
||||
- Classify touched files by layer:
|
||||
- **Frontend**: tools (`frontend/editor/src/core/components/tools/*` or `.../core/tools/*`),
|
||||
hooks (`core/hooks/tools/*`, `useToolOperation`), contexts, routes, i18n
|
||||
(`public/locales/en-US`).
|
||||
- **Java backend**: controllers (`.../controller/api/...`), services, models, config.
|
||||
- **Engine**: `engine/src/stirling/{agents,contracts,api,services}`.
|
||||
- **Config / build / docker / tests.**
|
||||
|
||||
### 2. Trace the flow end-to-end
|
||||
Follow one real path from user action to result. For a typical PDF tool that's:
|
||||
UI control → `useToolOperation` hook → `POST /api/v1/...` → Spring controller →
|
||||
service (PDFBox / LibreOffice / engine call) → response → review panel → download.
|
||||
Read the actual files so the narrative is true to the code, and collect the exact
|
||||
file:line anchors you'll cite.
|
||||
|
||||
### 3. Draw the diagrams (Mermaid)
|
||||
Pick what fits; usually 2-3 of:
|
||||
- **Sequence diagram** - request/response across frontend → backend → engine.
|
||||
- **Flowchart** - the core decision/branching logic of the feature.
|
||||
- **Architecture/component** - new pieces and how they wire to existing ones.
|
||||
- **State** - if the feature has modes/steps.
|
||||
Keep nodes labeled in plain language. Validate the Mermaid parses before shipping.
|
||||
|
||||
### 4. Screenshots (unless --no-screens)
|
||||
If a UI is involved, capture key states with the stubbed Playwright harness
|
||||
(see the **ui-walkthrough** skill and `files-page-screenshots.spec.ts` for the
|
||||
pattern) or, for before/after, capture `main` then the branch. Drop PNGs in
|
||||
`walkthrough/<feature>/` and reference them from the doc. For backend-only
|
||||
changes, show request/response examples (curl + JSON) instead.
|
||||
|
||||
### 5. Write the walkthrough
|
||||
Create `walkthrough/<feature>/FEATURE-WALKTHROUGH.md` with:
|
||||
1. **TL;DR** - what the branch does and who it's for, in 3-4 sentences.
|
||||
2. **Problem & approach** - what wasn't possible before; the chosen solution.
|
||||
3. **Architecture diagram** + 1-paragraph orientation.
|
||||
4. **End-to-end flow** - the sequence diagram + a numbered walk of each step,
|
||||
each citing the real file (clickable `path:line`).
|
||||
5. **Key files** - annotated map (path → one line on its role).
|
||||
6. **Logic deep-dive** - the flowchart + prose for the non-obvious decisions.
|
||||
7. **Behavior** - before vs after; screenshots or request/response examples.
|
||||
8. **Try it locally** - exact steps (`task dev` / `task dev:all`, the route to
|
||||
open or the curl to run, any env like `DOCKER_ENABLE_SECURITY` or a test
|
||||
license key). Make it copy-pasteable.
|
||||
9. **Edge cases, risks, follow-ups** - what's untested, known limits, gotchas.
|
||||
|
||||
Markdown is the primary deliverable - it renders with diagrams in GitHub PRs and
|
||||
IDEs, no build step, ideal for review.
|
||||
|
||||
### 6. If `--html`
|
||||
Also emit `walkthrough/<feature>/walkthrough.html`: the same content with Mermaid
|
||||
rendered via `mermaid.initialize({startOnLoad:true})` (script from CDN; note in
|
||||
the file that rendering diagrams needs network, the `.md` is the offline copy) and
|
||||
screenshots inline. Keep it self-contained otherwise.
|
||||
|
||||
### 7. Deliver
|
||||
Give the doc path and a short chat summary. Offer to `SendUserFile` it.
|
||||
|
||||
## Principles
|
||||
- **True to the code.** Every claim traces to a file you read; cite `path:line`.
|
||||
No fabricated migration/version history.
|
||||
- **Newcomer-first.** Define repo-specific terms (FileContext, `useToolOperation`,
|
||||
the `@app/*` layer cascade, stubbed vs live tests) on first use.
|
||||
- **Show, don't assert.** Prefer a diagram + a real example over adjectives.
|
||||
- Don't commit the `walkthrough/` output unless asked.
|
||||
@@ -1,122 +0,0 @@
|
||||
---
|
||||
name: ui-before-after
|
||||
description: >-
|
||||
Analyse a branch or PR and automatically capture before/after screenshots of
|
||||
every UI surface its changes touch, then pixel-diff the pairs to surface what
|
||||
actually changed and assemble PR-ready before/after montage images. Generic and
|
||||
diff-driven: it derives the capture targets from the diff (changed tools/routes →
|
||||
URLs) instead of hand-listing screens, captures "before" from the base branch and
|
||||
"after" from the head, then keeps only the views that visually differ. Each
|
||||
comparison is auto-cropped to the region that actually changed (the bounding box of
|
||||
differing pixels), falling back to the full page only when the change spans most of
|
||||
it. Use for before/after shots, a visual diff of a branch/PR, "screenshots for the
|
||||
PR description", "show what changed in the UI", or a side-by-side of UI changes.
|
||||
Takes a PR number/URL (resolved via gh) or a branch; defaults to the current branch
|
||||
vs its base. Flags: --scope <selector>, --base <ref|merge-base>, --theme
|
||||
light|dark|both, --all (capture every route, not just changed), --no-autocrop,
|
||||
--pagewide <n>, --threshold <n>.
|
||||
argument-hint: "[PR# | PR-url | branch] [--scope <sel>] [--base <ref>] [--theme both] [--all] [--no-autocrop]"
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# UI Before / After (generic visual diff)
|
||||
|
||||
Point it at a branch or PR; it figures out which UI changed, screenshots every
|
||||
affected surface **before** (base) and **after** (head), pixel-diffs the pairs, and
|
||||
montages the ones that actually changed into images for the PR description.
|
||||
|
||||
`$ARGUMENTS`: a PR number/URL, a branch, or nothing (current branch vs base).
|
||||
By default it captures the full viewport and auto-crops each comparison to the region
|
||||
that changed. Flags: `--scope <css>` (narrow the *capture* to a container, e.g.
|
||||
`[data-sidebar="tool-panel"]`, when you already know where the change is),
|
||||
`--no-autocrop` (keep full frames), `--pagewide <fraction>` (above this share of the
|
||||
page, skip cropping; default 0.6), `--base <ref|merge-base>`,
|
||||
`--theme light|dark|both`, `--all` (walk every route, not just changed),
|
||||
`--threshold <fraction>` (diff sensitivity, default 0.001).
|
||||
|
||||
Shares the capture harness with **ui-walkthrough** - read its SKILL.md for the
|
||||
stubbed-Playwright setup, worktree node_modules + `generate-icons`, the
|
||||
stale-`:5173` gotcha, and the dark-mode init-script. Bundled helpers:
|
||||
[capture-spec.template.ts](capture-spec.template.ts), [diff-shots.mjs](diff-shots.mjs),
|
||||
[montage-template.html](montage-template.html), [shoot-sections.mjs](shoot-sections.mjs).
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Resolve target + base
|
||||
```
|
||||
gh pr view <pr> --json number,title,headRefName,baseRefName,url,files # PR
|
||||
# or branch: base = merge-base(main, HEAD); head = HEAD
|
||||
gh pr diff <pr> --name-only # or: git diff --name-only <base>...HEAD
|
||||
```
|
||||
|
||||
### 2. Derive capture targets from the diff (the "analyse" step - no hand-listing)
|
||||
Map changed frontend files to URLs generically:
|
||||
- **Tools**: a changed `components/tools/<toolDir>/…` or `hooks/tools/<tool>/…` →
|
||||
toolId → URL via the repo's own rule `getToolUrlPath` in
|
||||
[toolsTaxonomy.ts:200](frontend/editor/src/core/data/toolsTaxonomy.ts): `/` + the
|
||||
id kebab-cased (`addPageNumbers` → `/add-page-numbers`).
|
||||
- **Pages/routes**: changed `filesPage/*` → `/files`, etc.
|
||||
- `--all`: enumerate every tool in the registry instead of just changed ones.
|
||||
Write `frontend/editor/screenshots/ui-diff/targets.json` =
|
||||
`[{ "id":"compress", "url":"/compress", "name":"Compress" }]`. This is what makes
|
||||
it generic - the spec never names a tool.
|
||||
|
||||
### 3. Capture AFTER (head) then BEFORE (base)
|
||||
Copy [capture-spec.template.ts](capture-spec.template.ts) →
|
||||
`src/core/tests/stubbed/ui-before-after.spec.ts` (it loops `targets.json`, seeds a
|
||||
sample PDF so file-dependent panels render, navigates to each URL, and screenshots
|
||||
the full viewport - or the `--scope` container if given). Ensure the harness is ready
|
||||
(node_modules + icons).
|
||||
```
|
||||
# after = current head
|
||||
cd frontend/editor && PR_SHOT_SIDE=after PR_SHOT_THEME=light \
|
||||
npx playwright test --project=stubbed ui-before-after.spec.ts
|
||||
# before = base, in an isolated worktree (copy the spec + targets.json in)
|
||||
git worktree add ../ba-base origin/<baseRefName> # or the merge-base
|
||||
# set up its frontend, copy spec + screenshots/ui-diff/targets.json across, then:
|
||||
cd ../ba-base/frontend/editor && PR_SHOT_SIDE=before PR_SHOT_THEME=light \
|
||||
npx playwright test --project=stubbed ui-before-after.spec.ts
|
||||
# copy its screenshots/ui-diff/before/ back next to after/. Repeat with
|
||||
# PR_SHOT_THEME=dark if --theme includes dark. Remove worktree when done.
|
||||
```
|
||||
|
||||
### 4. Auto-diff (surface what changed)
|
||||
```
|
||||
cd frontend/editor && node <skill>/diff-shots.mjs \
|
||||
screenshots/ui-diff/before screenshots/ui-diff/after screenshots/ui-diff
|
||||
```
|
||||
Produces `diff-report.json` classifying each view `unchanged | changed | added |
|
||||
removed`. For each changed view it computes the bounding box of differing pixels and
|
||||
writes cropped `__before_crop.png` / `__after_crop.png` / `__diff.png` to that region
|
||||
(+ padding) - **unless** the change covers more than `--pagewide` of the frame, where
|
||||
it keeps the full frame (`pageWide:true`). Drop `unchanged` - that's the noise the
|
||||
user doesn't want.
|
||||
|
||||
### 5. Montage the changes
|
||||
Build the manifest from the non-unchanged entries (group by tab/tool; each becomes a
|
||||
state row with before/after). For changed views use the cropped `cropBefore` /
|
||||
`cropAfter` from `diff-report.json` (tight on the affected region; full frame when
|
||||
`pageWide`); `added`/`removed` render the "not present" placeholder. Fill
|
||||
[montage-template.html](montage-template.html) (replace the `window.__BA__` data
|
||||
block; base64-inline the PNGs for portability), then render one PNG per section with
|
||||
[shoot-sections.mjs](shoot-sections.mjs). Optionally include the `__diff.png` overlay
|
||||
as a third column.
|
||||
|
||||
### 6. Deliver
|
||||
Output the `montage_<tab>.png` files + a short summary (N changed / added / removed,
|
||||
M unchanged skipped) and a paste-ready Markdown block. GitHub has no PR-body image
|
||||
API, so tell the user to drag the PNGs into the description. Do **not** post to the
|
||||
PR.
|
||||
|
||||
## Gotchas
|
||||
- Two installs (base worktree + head); junction main's node_modules only if its deps
|
||||
match that ref, else `npm ci` (see ui-walkthrough's stale-dep note).
|
||||
- A view that errors on one side (refactored/removed) → that side is missing; the
|
||||
diff marks it added/removed rather than failing the run.
|
||||
- Pixel diff needs equal dimensions, so capture at a fixed viewport (the template
|
||||
does); a view whose size changed is reported as "changed (dimensions differ)",
|
||||
uncropped.
|
||||
- Auto-crop uses a single bounding box, so two far-apart changes give one large crop
|
||||
(or trip `--pagewide`); narrow with `--scope` if that happens.
|
||||
- `getToolUrlPath` is the source of truth for tool URLs - use it, don't guess slugs.
|
||||
- Don't commit `screenshots/`, the throwaway spec, or the base worktree.
|
||||
@@ -1,67 +0,0 @@
|
||||
// Generic before/after capturer. NOT app-specific: it walks a targets.json that
|
||||
// the ui-before-after skill generates from the branch/PR diff, so nothing here is
|
||||
// hand-listed. Copy to src/core/tests/stubbed/ui-before-after.spec.ts, then run
|
||||
// once per (side, theme):
|
||||
// PR_SHOT_SIDE=after PR_SHOT_THEME=light \
|
||||
// npx playwright test --project=stubbed ui-before-after.spec.ts
|
||||
//
|
||||
// targets.json shape: [{ "id":"compress", "url":"/compress", "name":"Compress",
|
||||
// "needsFile": true }]
|
||||
import { test } from "@app/tests/helpers/stub-test-base";
|
||||
import type { Page } from "@playwright/test";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const SIDE = process.env.PR_SHOT_SIDE ?? "after";
|
||||
const THEME = process.env.PR_SHOT_THEME ?? "light";
|
||||
// Capture the full viewport by default so the affected region is in frame
|
||||
// wherever it is; diff-shots.mjs crops each comparison to what actually changed.
|
||||
// Set PR_SHOT_SCOPE to a selector to narrow the capture to one container.
|
||||
const SCOPE = process.env.PR_SHOT_SCOPE ?? "";
|
||||
const ROOT = path.resolve(process.cwd(), "screenshots", "ui-diff");
|
||||
const OUT = path.join(ROOT, SIDE);
|
||||
// A tiny sample PDF so file-dependent tool panels render. Point at a real fixture.
|
||||
const SAMPLE_PDF = process.env.PR_SHOT_SAMPLE ?? "src/core/tests/test-fixtures/sample.pdf";
|
||||
|
||||
type Target = { id: string; url: string; name?: string; needsFile?: boolean };
|
||||
const targets: Target[] = JSON.parse(fs.readFileSync(path.join(ROOT, "targets.json"), "utf-8"));
|
||||
|
||||
test.use({ autoGoto: false, viewport: { width: 1600, height: 900 }, seedJwt: true });
|
||||
|
||||
async function applyTheme(page: Page): Promise<void> {
|
||||
if (THEME !== "dark") return;
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem("mantine-color-scheme", "dark");
|
||||
localStorage.setItem("mantine-color-scheme-value", "dark");
|
||||
});
|
||||
await page.emulateMedia({ colorScheme: "dark" });
|
||||
}
|
||||
|
||||
async function seedFile(page: Page): Promise<void> {
|
||||
if (!fs.existsSync(SAMPLE_PDF)) return;
|
||||
await page.goto("/", { waitUntil: "domcontentloaded" });
|
||||
await page.getByTestId("files-button").click().catch(() => {});
|
||||
await page.locator('[data-testid="file-input"]').setInputFiles(SAMPLE_PDF).catch(() => {});
|
||||
await page.locator(".file-sidebar-file-item").first().isVisible({ timeout: 8_000 }).catch(() => {});
|
||||
}
|
||||
|
||||
for (const t of targets) {
|
||||
// One test per target so a single failure doesn't drop the rest.
|
||||
test(`${SIDE}/${THEME} ${t.id}`, async ({ page }) => {
|
||||
fs.mkdirSync(OUT, { recursive: true });
|
||||
await applyTheme(page);
|
||||
if (t.needsFile !== false) await seedFile(page);
|
||||
await page.goto(t.url, { waitUntil: "domcontentloaded" });
|
||||
await page.waitForTimeout(400); // settle Mantine portals/transitions
|
||||
const shot = path.join(OUT, `${t.id}__${THEME}.png`);
|
||||
if (SCOPE) {
|
||||
const scope = page.locator(SCOPE).first();
|
||||
if (await scope.isVisible({ timeout: 8_000 }).catch(() => false)) {
|
||||
await scope.screenshot({ path: shot });
|
||||
return;
|
||||
}
|
||||
}
|
||||
// Full viewport (fixed size → stable dimensions for pixel diffing).
|
||||
await page.screenshot({ path: shot });
|
||||
});
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// Auto-diff before/ vs after/ screenshots, classify each as
|
||||
// unchanged | changed | added | removed, and CROP each changed pair to the
|
||||
// affected region (bounding box of differing pixels + padding) - unless the
|
||||
// change spans most of the page, in which case the full frame is kept.
|
||||
// Run from frontend/editor (so deps resolve):
|
||||
// node <skill>/diff-shots.mjs <beforeDir> <afterDir> [outDir]
|
||||
// Env:
|
||||
// DIFF_THRESHOLD min fraction of differing pixels to count as changed (default 0.001)
|
||||
// DIFF_PAD padding px around the affected region (default 24)
|
||||
// DIFF_PAGEWIDE if affected bbox area / image area exceeds this, keep full frame (default 0.6)
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
const require = createRequire(path.join(process.cwd(), "noop.js"));
|
||||
const pm = require("pixelmatch");
|
||||
const pixelmatch = pm.default || pm;
|
||||
const { PNG } = require("pngjs");
|
||||
|
||||
const beforeDir = path.resolve(process.argv[2]);
|
||||
const afterDir = path.resolve(process.argv[3]);
|
||||
const outDir = path.resolve(process.argv[4] || afterDir);
|
||||
const THRESHOLD = Number(process.env.DIFF_THRESHOLD ?? "0.001");
|
||||
const PAD = Number(process.env.DIFF_PAD ?? "24");
|
||||
const PAGEWIDE = Number(process.env.DIFF_PAGEWIDE ?? "0.6");
|
||||
|
||||
const read = (p) => PNG.sync.read(fs.readFileSync(p));
|
||||
const isShot = (f) => f.endsWith(".png") && !/__(diff|before_crop|after_crop)\.png$/.test(f);
|
||||
const list = (d) => (fs.existsSync(d) ? fs.readdirSync(d).filter(isShot) : []);
|
||||
const names = [...new Set([...list(beforeDir), ...list(afterDir)])].sort();
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
|
||||
function cropPNG(src, x, y, w, h) {
|
||||
const out = new PNG({ width: w, height: h });
|
||||
PNG.bitblt(src, out, x, y, w, h, 0, 0);
|
||||
return out;
|
||||
}
|
||||
const writePNG = (p, png) => fs.writeFileSync(p, PNG.sync.write(png));
|
||||
|
||||
// Bounding box of differing pixels using a diff mask (alpha>0 where changed).
|
||||
function changedBBox(before, after, w, h) {
|
||||
const mask = new PNG({ width: w, height: h });
|
||||
pixelmatch(before.data, after.data, mask.data, w, h, { threshold: 0.1, diffMask: true });
|
||||
let minX = w, minY = h, maxX = -1, maxY = -1, count = 0;
|
||||
for (let y = 0; y < h; y++) {
|
||||
for (let x = 0; x < w; x++) {
|
||||
if (mask.data[(y * w + x) * 4 + 3] > 0) {
|
||||
count++;
|
||||
if (x < minX) minX = x; if (x > maxX) maxX = x;
|
||||
if (y < minY) minY = y; if (y > maxY) maxY = y;
|
||||
}
|
||||
}
|
||||
}
|
||||
return maxX < 0 ? null : { minX, minY, maxX, maxY, count };
|
||||
}
|
||||
|
||||
const report = [];
|
||||
for (const name of names) {
|
||||
const id = name.replace(/\.png$/, "");
|
||||
const bp = path.join(beforeDir, name), ap = path.join(afterDir, name);
|
||||
const hasB = fs.existsSync(bp), hasA = fs.existsSync(ap);
|
||||
if (hasB && !hasA) { report.push({ id, status: "removed", before: bp }); continue; }
|
||||
if (!hasB && hasA) { report.push({ id, status: "added", after: ap }); continue; }
|
||||
|
||||
const before = read(bp), after = read(ap);
|
||||
if (before.width !== after.width || before.height !== after.height) {
|
||||
report.push({ id, status: "changed", note: "dimensions differ", before: bp, after: ap });
|
||||
continue;
|
||||
}
|
||||
const w = after.width, h = after.height;
|
||||
const overlay = new PNG({ width: w, height: h });
|
||||
const px = pixelmatch(before.data, after.data, overlay.data, w, h, { threshold: 0.1 });
|
||||
const ratio = px / (w * h);
|
||||
if (ratio <= THRESHOLD) { report.push({ id, status: "unchanged", ratio: Number(ratio.toFixed(5)), before: bp, after: ap }); continue; }
|
||||
|
||||
const box = changedBBox(before, after, w, h);
|
||||
// Pad + clamp the affected region.
|
||||
const x = Math.max(0, box.minX - PAD), y = Math.max(0, box.minY - PAD);
|
||||
const x2 = Math.min(w, box.maxX + 1 + PAD), y2 = Math.min(h, box.maxY + 1 + PAD);
|
||||
const bw = x2 - x, bh = y2 - y;
|
||||
const pageWide = (bw * bh) / (w * h) > PAGEWIDE;
|
||||
|
||||
const entry = { id, status: "changed", ratio: Number(ratio.toFixed(5)), before: bp, after: ap, pageWide };
|
||||
if (pageWide) {
|
||||
// Change spans most of the page - keep the full frame, full overlay.
|
||||
const dp = path.join(outDir, `${id}__diff.png`); writePNG(dp, overlay);
|
||||
entry.diff = dp;
|
||||
} else {
|
||||
entry.bbox = { x, y, w: bw, h: bh };
|
||||
const cb = path.join(outDir, `${id}__before_crop.png`); writePNG(cb, cropPNG(before, x, y, bw, bh));
|
||||
const ca = path.join(outDir, `${id}__after_crop.png`); writePNG(ca, cropPNG(after, x, y, bw, bh));
|
||||
const dp = path.join(outDir, `${id}__diff.png`); writePNG(dp, cropPNG(overlay, x, y, bw, bh));
|
||||
entry.cropBefore = cb; entry.cropAfter = ca; entry.diff = dp;
|
||||
}
|
||||
report.push(entry);
|
||||
}
|
||||
|
||||
fs.writeFileSync(path.join(outDir, "diff-report.json"), JSON.stringify(report, null, 2));
|
||||
const changed = report.filter((r) => r.status !== "unchanged");
|
||||
console.log(`diffed ${report.length} view(s): ${changed.length} changed/added/removed, ${report.length - changed.length} unchanged`);
|
||||
for (const r of changed) {
|
||||
const tail = r.status !== "changed" ? ""
|
||||
: r.pageWide ? " (page-wide → full frame)"
|
||||
: ` (${(r.ratio * 100).toFixed(2)}%, cropped to ${r.bbox.w}×${r.bbox.h})`;
|
||||
console.log(` ${r.status.padEnd(9)} ${r.id}${tail}${r.note ? " - " + r.note : ""}`);
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
"""Build EXAMPLE.html from montage-template.html using REAL files-page shots as
|
||||
stand-in before/after pairs (layout demo, not an actual PR diff). Inlines PNGs as
|
||||
data URIs so the HTML is portable. Run: python make_example.py"""
|
||||
import base64
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
HERE = pathlib.Path(__file__).parent
|
||||
SHOTS = pathlib.Path(
|
||||
r"C:\Users\systo\git\Stirling-PDFNew\.claude\worktrees\kind-faraday-522a30"
|
||||
r"\frontend\editor\screenshots\files-page"
|
||||
)
|
||||
|
||||
|
||||
def uri(fname):
|
||||
p = SHOTS / fname
|
||||
return "data:image/png;base64," + base64.b64encode(p.read_bytes()).decode() if p.exists() else None
|
||||
|
||||
|
||||
data = {
|
||||
"pr": "DEMO",
|
||||
"title": "EXAMPLE — before/after montage (layout demo, real Files-page shots; not a real PR diff)",
|
||||
"base": "main", "head": "demo-branch",
|
||||
"cropSelector": "[data-sidebar=\"tool-panel\"] (real runs crop to the side; these demo shots are full-page)",
|
||||
"tabs": [
|
||||
{"id": "files", "title": "Files page", "ctx": "Each row = one flow state; left = base branch, right = this PR.",
|
||||
"states": [
|
||||
{"name": "Empty folder", "before": uri("01_empty_state_ctas.png"), "after": uri("02_empty_state_storage_off.png")},
|
||||
{"name": "Files + details panel", "before": uri("03_subtoolbar_with_files.png"), "after": uri("06_details_panel_save_to_server.png")},
|
||||
{"name": "Delete folder confirm", "before": None, "after": uri("19_delete_folder_dialog.png"), "note": "New in this PR"},
|
||||
]},
|
||||
{"id": "move", "title": "Move-to-folder dialog",
|
||||
"states": [
|
||||
{"name": "Dialog opened", "before": uri("07_move_dialog_collapsed.png"), "after": uri("08_move_dialog_create_folder_expanded.png")},
|
||||
{"name": "After folder created", "before": None, "after": uri("08b_move_dialog_after_create_folder.png"), "note": "New flow"},
|
||||
]},
|
||||
],
|
||||
}
|
||||
|
||||
tpl = (HERE / "montage-template.html").read_text(encoding="utf-8")
|
||||
out = re.sub(
|
||||
r"/\*__DATA__\*/.*?/\*__END__\*/",
|
||||
lambda _m: "/*__DATA__*/" + json.dumps(data) + "/*__END__*/",
|
||||
tpl, count=1, flags=re.S,
|
||||
)
|
||||
(HERE / "EXAMPLE.html").write_text(out, encoding="utf-8")
|
||||
print("wrote", HERE / "EXAMPLE.html", "(", (HERE / "EXAMPLE.html").stat().st_size // 1024, "KB )")
|
||||
@@ -1,106 +0,0 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
Before/After montage for a PR description. The ui-before-after skill replaces
|
||||
the JSON in the window.__BA__ data block below with the captured manifest, then
|
||||
screenshots each .tab-section (id="section-<tabId>") into a PNG to drag into the
|
||||
PR description. Self-contained; images may be relative paths or data URIs.
|
||||
|
||||
Data shape:
|
||||
{
|
||||
"pr":"6552","title":"...","base":"main","head":"feat/x",
|
||||
"cropSelector":"[data-sidebar=\"tool-panel\"]",
|
||||
"tabs":[
|
||||
{ "id":"sign","title":"Sign tool","states":[
|
||||
{"name":"Initial","before":"before/sign__initial.png","after":"after/sign__initial.png"},
|
||||
{"name":"Cert selected","before":null,"after":"after/sign__cert.png","note":"New in this PR"}
|
||||
]}
|
||||
]
|
||||
}
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>Before / After</title>
|
||||
<style>
|
||||
:root { --bg:#ffffff; --ink:#0b0c0e; --muted:#6b7280; --line:#e5e7eb;
|
||||
--before:#6b7280; --after:#1f883d; --frame:#f3f4f6; --note:#b45309; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin:0; background:var(--bg); color:var(--ink);
|
||||
font:14px/1.5 -apple-system,"Segoe UI",Roboto,system-ui,sans-serif; }
|
||||
.wrap { max-width:1100px; margin:0 auto; padding:24px; }
|
||||
.doc-head { margin-bottom:8px; }
|
||||
.doc-head h1 { font-size:18px; margin:0 0 2px; }
|
||||
.doc-head .sub { color:var(--muted); font-size:12.5px; }
|
||||
.legend { display:flex; gap:14px; align-items:center; margin:10px 0 4px; font-size:12px; color:var(--muted); }
|
||||
.chip { font-size:10px; font-weight:700; letter-spacing:.04em; text-transform:uppercase;
|
||||
padding:2px 8px; border-radius:999px; color:#fff; }
|
||||
.chip.before { background:var(--before); } .chip.after { background:var(--after); }
|
||||
|
||||
.tab-section { border:1px solid var(--line); border-radius:14px; padding:18px 18px 8px;
|
||||
margin:18px 0; background:var(--bg); }
|
||||
.tab-section > h2 { font-size:16px; margin:0 0 2px; }
|
||||
.tab-section > .ctx { color:var(--muted); font-size:12px; margin-bottom:14px; }
|
||||
.state { margin-bottom:18px; }
|
||||
.state .name { font-weight:600; font-size:13.5px; margin-bottom:8px; display:flex; gap:8px; align-items:center; }
|
||||
.state .name .note { font-weight:500; color:var(--note); font-size:12px; }
|
||||
.pair { display:grid; grid-template-columns:1fr 1fr; gap:14px; align-items:start; }
|
||||
.cell { border:1px solid var(--line); border-radius:10px; overflow:hidden; background:var(--frame); }
|
||||
.cell .cap { display:flex; align-items:center; gap:8px; padding:7px 10px; border-bottom:1px solid var(--line);
|
||||
background:var(--bg); }
|
||||
.cell .cap .meta { color:var(--muted); font-size:11px; }
|
||||
.cell img { display:block; width:100%; height:auto; background:#fff; }
|
||||
.cell.empty .ph { display:flex; align-items:center; justify-content:center; height:160px; color:var(--muted);
|
||||
font-size:12.5px; text-align:center; padding:0 16px; }
|
||||
.single .pair { grid-template-columns:1fr; }
|
||||
.empty-doc { color:var(--muted); padding:40px; text-align:center; }
|
||||
@media (max-width:760px){ .pair{ grid-template-columns:1fr; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap" id="root"></div>
|
||||
|
||||
<script id="data">
|
||||
window.__BA__ = /*__DATA__*/{"pr":"","title":"No data","base":"","head":"","cropSelector":"","tabs":[]}/*__END__*/;
|
||||
</script>
|
||||
<script>
|
||||
(function(){
|
||||
var D = window.__BA__ || { tabs: [] };
|
||||
var root = document.getElementById("root");
|
||||
function el(html){ var t=document.createElement("template"); t.innerHTML=html.trim(); return t.content.firstChild; }
|
||||
function esc(s){ return (s==null?"":String(s)).replace(/[&<>]/g, function(c){return {"&":"&","<":"<",">":">"}[c];}); }
|
||||
|
||||
function cell(kind, src){
|
||||
if (src) {
|
||||
return '<div class="cell"><div class="cap"><span class="chip '+kind+'">'+kind+'</span></div>'+
|
||||
'<img src="'+esc(src)+'" alt="'+kind+'"/></div>';
|
||||
}
|
||||
return '<div class="cell empty"><div class="cap"><span class="chip '+kind+'">'+kind+'</span>'+
|
||||
'<span class="meta">not present</span></div><div class="ph">No '+kind+' screenshot for this state</div></div>';
|
||||
}
|
||||
|
||||
var head = '<div class="doc-head"><h1>'+esc(D.title || ("PR #"+D.pr))+'</h1>'+
|
||||
'<div class="sub">Before / after · base <code>'+esc(D.base)+'</code> → head <code>'+esc(D.head)+'</code>'+
|
||||
(D.cropSelector ? ' · cropped to <code>'+esc(D.cropSelector)+'</code>' : '')+'</div></div>'+
|
||||
'<div class="legend"><span class="chip before">Before</span> base branch'+
|
||||
'<span class="chip after">After</span> this PR</div>';
|
||||
root.appendChild(el('<div>'+head+'</div>'));
|
||||
|
||||
if (!D.tabs || !D.tabs.length){ root.appendChild(el('<div class="empty-doc">No tabs captured yet.</div>')); return; }
|
||||
|
||||
D.tabs.forEach(function(tab){
|
||||
var states = (tab.states||[]).map(function(s){
|
||||
var onlyOne = (!s.before || !s.after);
|
||||
return '<div class="state'+(onlyOne?' ':'')+'">'+
|
||||
'<div class="name">'+esc(s.name)+(s.note?'<span class="note">'+esc(s.note)+'</span>':'')+'</div>'+
|
||||
'<div class="pair">'+cell("before", s.before)+cell("after", s.after)+'</div></div>';
|
||||
}).join("");
|
||||
var sec = '<section class="tab-section" id="section-'+esc(tab.id)+'">'+
|
||||
'<h2>'+esc(tab.title)+'</h2>'+
|
||||
(tab.ctx?'<div class="ctx">'+esc(tab.ctx)+'</div>':'')+
|
||||
states+'</section>';
|
||||
root.appendChild(el(sec));
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,25 +0,0 @@
|
||||
// Render each .tab-section of a montage HTML into its own PNG (the PR-ready image).
|
||||
// Run from frontend/editor (so @playwright/test resolves):
|
||||
// node <skill>/shoot-sections.mjs <montage.html> <outDir>
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
const require = createRequire(path.join(process.cwd(), "noop.js"));
|
||||
const { chromium } = require("@playwright/test");
|
||||
|
||||
const htmlPath = path.resolve(process.argv[2]);
|
||||
const outDir = path.resolve(process.argv[3] || path.dirname(htmlPath));
|
||||
|
||||
const browser = await chromium.launch();
|
||||
const page = await browser.newPage({ viewport: { width: 1200, height: 1200 }, deviceScaleFactor: 2 });
|
||||
await page.goto(pathToFileURL(htmlPath).href, { waitUntil: "load" });
|
||||
await page.waitForTimeout(250); // let images/fonts paint
|
||||
const ids = await page.$$eval(".tab-section", (els) => els.map((e) => e.id));
|
||||
if (!ids.length) { console.error("no .tab-section found"); process.exit(1); }
|
||||
for (const id of ids) {
|
||||
const name = id.replace(/^section-/, "");
|
||||
await page.locator("#" + id).screenshot({ path: path.join(outDir, `montage_${name}.png`) });
|
||||
console.log("wrote montage_" + name + ".png");
|
||||
}
|
||||
await browser.close();
|
||||
@@ -1,120 +0,0 @@
|
||||
---
|
||||
name: ui-walkthrough
|
||||
description: >-
|
||||
Full UI investigation of the current branch's feature. Enumerates every view
|
||||
and state (empty, populated, loading, error, each dialog/menu/panel, responsive
|
||||
breakpoints, light + dark + RTL), captures them with the stubbed Playwright
|
||||
harness, assembles a single-image HTML walkthrough with a global light/dark
|
||||
toggle slider, then runs two review passes: visual/consistency (alignment,
|
||||
spacing, professionalism, dark/light parity, contrast, truncation) and
|
||||
UX/ease-of-use (flow, discoverability, affordances, empty/error states,
|
||||
expectations). Use when asked for a UI walkthrough, screenshot review, design
|
||||
or QA pass, "find anywhere to make it easier/better for users", or before
|
||||
merging frontend work. Pass --fix to auto-apply safe frontend fixes and
|
||||
re-capture; --theme to limit themes; --no-rtl to skip RTL.
|
||||
argument-hint: "[feature/area] [--fix] [--theme light|dark|both] [--no-rtl] [--breakpoints]"
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# UI Walkthrough
|
||||
|
||||
Produce a reviewable HTML walkthrough of a feature's UI in every state and theme,
|
||||
then critique it. Optionally auto-fix and re-capture.
|
||||
|
||||
`$ARGUMENTS` may name the feature/area to focus on. If empty, scope from the
|
||||
current branch diff. Flags: `--fix`, `--theme light|dark|both` (default both),
|
||||
`--no-rtl`, `--breakpoints` (also capture phone/narrow widths).
|
||||
|
||||
## What this repo gives you (use it, don't reinvent)
|
||||
|
||||
- **Stubbed Playwright project** = backend-free screenshots via `page.route()` mocks.
|
||||
Reference implementation: `frontend/editor/src/core/tests/stubbed/files-page-screenshots.spec.ts`.
|
||||
It already shows the light / **dark** / **RTL** passes, JWT seeding, IndexedDB
|
||||
seeding, and dumping PNGs to a `screenshots/<area>/` folder. Copy its shape.
|
||||
- Helpers: `frontend/editor/src/core/tests/helpers/ui-helpers.ts`
|
||||
(`uploadFiles`, `openSettings`, `waitForModalOpen`, `dismissTourTooltip`, …)
|
||||
and the `stub-test-base` fixtures (`autoGoto`, `seedJwt`, `viewport`).
|
||||
- Config: `frontend/editor/playwright.config.ts` (run from `frontend/editor/`).
|
||||
- Report template: [report-template.html](report-template.html) - self-contained,
|
||||
one big image at a time, a global light/dark slider that flips every shot,
|
||||
thumbnail rail, prev/next + arrow keys, and a Findings tab.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Scope the feature
|
||||
- If `$ARGUMENTS` is empty: `git diff --name-only main...HEAD` and read the PR/commits.
|
||||
Identify changed pages, tools (`core/components/tools/<tool>` or `core/tools/<tool>`),
|
||||
dialogs, panels, and routes.
|
||||
- Enumerate **every view and state** to capture, e.g.:
|
||||
empty / populated / loading / error / disabled; each dialog, menu, popover, tooltip;
|
||||
each tab or step; selection + multi-select; success/result panel; and (if relevant)
|
||||
permission/role variants. Write the list down before capturing - it's the report's spine.
|
||||
|
||||
### 2. Prepare the harness (worktree-safe)
|
||||
Worktrees have no `node_modules` and no generated icons. From repo root:
|
||||
```
|
||||
cd frontend && npm ci # or junction main's node_modules (see memory)
|
||||
cd frontend/editor && node scripts/generate-icons.js
|
||||
```
|
||||
Kill any stale dev server first (it serves old modules):
|
||||
`Get-NetTCPConnection -LocalPort 5173 -State Listen | %{ Stop-Process -Id $_.OwningProcess -Force }`
|
||||
|
||||
### 3. Write the capture spec
|
||||
Create `frontend/editor/src/core/tests/stubbed/<feature>-walkthrough.spec.ts`,
|
||||
modeled on `files-page-screenshots.spec.ts`. For each enumerated view:
|
||||
- stub the APIs it needs, drive the UI to that state, wait on a real locator
|
||||
(not a fixed sleep), `await settle(page)` for Mantine portals, then
|
||||
`page.screenshot({ path: shotPath("NN_name_<theme>") })`.
|
||||
- Capture each view in **light and dark** (and RTL unless `--no-rtl`). Reuse the
|
||||
`enableDarkMode` / `enableRtl` init-script pattern from the reference spec
|
||||
(`localStorage["mantine-color-scheme"]="dark"` + `emulateMedia({colorScheme:"dark"})`).
|
||||
- Name shots `NN_<view>_<theme>.png` so light/dark pair up by suffix.
|
||||
- Prefer **stable test-ids** over translated accessible names (RTL/i18n breaks text locators).
|
||||
|
||||
Run it: `cd frontend/editor && npx playwright test --project=stubbed <feature>-walkthrough.spec.ts`.
|
||||
Add `--project=stubbed-firefox`/`-webkit` only if cross-browser layout matters.
|
||||
|
||||
### 4. Build the report
|
||||
- Copy `report-template.html` to `screenshots/<feature>/walkthrough.html` (so the
|
||||
relative `screenshots/...` image paths resolve, or rewrite paths to sit beside it).
|
||||
- Build the manifest and inject it: replace the JSON between the
|
||||
`/*__DATA__*/` … `/*__END__*/` markers with one `views[]` entry per view
|
||||
(`{id,title,light,dark,viewport,notes}`) and an empty `findings` object you'll
|
||||
fill in step 5. Keep `light`/`dark` as relative paths.
|
||||
- The toggle slider answers the "one big image + flip light/dark for all" request:
|
||||
it shows a single large screenshot, and switching the slider re-themes every view.
|
||||
|
||||
### 5. Review pass 1 - visual & consistency
|
||||
Open each screenshot (Read the PNG) and judge against the others:
|
||||
alignment & spacing rhythm, control placement, button hierarchy, typography,
|
||||
**light/dark parity** (contrast, invisible borders, washed-out text, wrong tokens),
|
||||
truncation/overflow, RTL mirroring, focus states, icon consistency, professional polish.
|
||||
Record each issue as a finding `{severity:high|med|low, view, title, detail, fix}`.
|
||||
|
||||
### 6. Review pass 2 - UX & ease of use
|
||||
Walk the flow as a first-time user: discoverability, number of steps, affordance
|
||||
clarity, empty-state guidance, error recovery, destructive-action confirmation,
|
||||
defaults, loading feedback, mobile reachability, accessible names, and whether the
|
||||
UI matches user expectations for this kind of tool. Record findings the same way.
|
||||
|
||||
Write both finding lists into the report's `findings.visual` / `findings.ux`,
|
||||
and add short per-view `notes`. Re-inject the manifest.
|
||||
|
||||
### 7. If `--fix`
|
||||
Only safe, self-contained frontend fixes (spacing, alignment, tokens, missing
|
||||
dark-mode colors, labels, aria, obvious copy). For each: edit the component/CSS,
|
||||
mark the finding `fixed:true` with what changed, then **re-run the spec** to
|
||||
re-capture the affected shots and regenerate the report. Run `task frontend:check`.
|
||||
Leave anything risky or ambiguous as a finding, not a change.
|
||||
|
||||
### 8. Deliver
|
||||
Tell the user the report path and give a tight chat summary: N views ×
|
||||
themes captured, top findings by severity, and (if `--fix`) what changed.
|
||||
Optionally `SendUserFile` the `walkthrough.html`.
|
||||
|
||||
## Gotchas
|
||||
- Stale `:5173` server serves old bundles - kill it before capturing (see step 2).
|
||||
- Missing `material-symbols-icons.json` → blank app → every shot times out. Run
|
||||
`generate-icons.js` first.
|
||||
- `await settle(page)` before shots or portals/transitions tear mid-capture.
|
||||
- Don't commit the generated `screenshots/` or the throwaway spec unless asked.
|
||||
@@ -1,116 +0,0 @@
|
||||
"""Build a self-contained EXAMPLE.html from report-template.html with mock
|
||||
light/dark screenshots, so the viewer + global theme slider can be demoed
|
||||
without a real capture run. Run: python make_example.py"""
|
||||
import base64
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
HERE = pathlib.Path(__file__).parent
|
||||
|
||||
|
||||
def svg(bg, fg, panel, accent, muted, label, kind):
|
||||
"""A simple fake 'screen' SVG: title bar, sidebar, content varies by kind."""
|
||||
parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" width="1600" height="900" viewBox="0 0 1600 900">',
|
||||
f'<rect width="1600" height="900" fill="{bg}"/>',
|
||||
# top bar
|
||||
f'<rect width="1600" height="64" fill="{panel}"/>',
|
||||
f'<circle cx="40" cy="32" r="12" fill="{accent}"/>',
|
||||
f'<rect x="64" y="24" width="160" height="16" rx="6" fill="{muted}"/>',
|
||||
f'<rect x="1430" y="20" width="130" height="24" rx="12" fill="{accent}"/>',
|
||||
# left sidebar
|
||||
f'<rect x="0" y="64" width="220" height="836" fill="{panel}"/>',
|
||||
]
|
||||
for i in range(6):
|
||||
y = 100 + i * 56
|
||||
parts.append(f'<rect x="24" y="{y}" width="172" height="32" rx="8" fill="{bg}"/>')
|
||||
if kind == "empty":
|
||||
parts += [
|
||||
f'<rect x="700" y="360" width="200" height="120" rx="16" fill="none" stroke="{muted}" stroke-width="3" stroke-dasharray="10 8"/>',
|
||||
f'<rect x="690" y="510" width="220" height="44" rx="10" fill="{accent}"/>',
|
||||
f'<text x="800" y="600" fill="{muted}" font-family="sans-serif" font-size="26" text-anchor="middle">{label}</text>',
|
||||
]
|
||||
elif kind == "form":
|
||||
for i in range(4):
|
||||
y = 140 + i * 90
|
||||
parts.append(f'<rect x="280" y="{y}" width="160" height="16" rx="6" fill="{muted}"/>')
|
||||
parts.append(f'<rect x="280" y="{y+26}" width="900" height="44" rx="8" fill="{panel}" stroke="{muted}" stroke-width="1"/>')
|
||||
parts.append(f'<rect x="280" y="560" width="200" height="50" rx="10" fill="{accent}"/>')
|
||||
parts.append(f'<text x="800" y="850" fill="{muted}" font-family="sans-serif" font-size="24" text-anchor="middle">{label}</text>')
|
||||
else: # dialog
|
||||
parts += [
|
||||
f'<rect width="1600" height="900" fill="{fg}" opacity="0.45"/>',
|
||||
f'<rect x="520" y="280" width="560" height="360" rx="18" fill="{panel}"/>',
|
||||
f'<rect x="556" y="320" width="280" height="22" rx="8" fill="{fg}"/>',
|
||||
f'<rect x="556" y="372" width="488" height="14" rx="6" fill="{muted}"/>',
|
||||
f'<rect x="556" y="398" width="420" height="14" rx="6" fill="{muted}"/>',
|
||||
f'<rect x="820" y="560" width="110" height="44" rx="9" fill="{bg}" stroke="{muted}"/>',
|
||||
f'<rect x="946" y="560" width="98" height="44" rx="9" fill="{accent}"/>',
|
||||
f'<text x="800" y="700" fill="#fff" font-family="sans-serif" font-size="24" text-anchor="middle">{label}</text>',
|
||||
]
|
||||
parts.append("</svg>")
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def data_uri(s):
|
||||
return "data:image/svg+xml;base64," + base64.b64encode(s.encode()).decode()
|
||||
|
||||
|
||||
LIGHT = dict(bg="#ffffff", fg="#111418", panel="#f1f3f6", accent="#2f6fed", muted="#c2c8d0")
|
||||
DARK = dict(bg="#16181c", fg="#000000", panel="#1f232a", accent="#5b8cff", muted="#3a414b")
|
||||
|
||||
|
||||
def pair(kind, label):
|
||||
return (
|
||||
data_uri(svg(LIGHT["bg"], LIGHT["fg"], LIGHT["panel"], LIGHT["accent"], LIGHT["muted"], label, kind)),
|
||||
data_uri(svg(DARK["bg"], DARK["fg"], DARK["panel"], DARK["accent"], DARK["muted"], label, kind)),
|
||||
)
|
||||
|
||||
|
||||
views = []
|
||||
for idx, (kind, title, label) in enumerate([
|
||||
("empty", "Empty state", "Drop a PDF to start"),
|
||||
("form", "Tool options panel", "Compress options"),
|
||||
("dialog", "Confirm dialog", "Replace original file?"),
|
||||
], start=1):
|
||||
light, dark = pair(kind, label)
|
||||
views.append({
|
||||
"id": f"{idx:02d}_{kind}",
|
||||
"title": title,
|
||||
"light": light,
|
||||
"dark": dark,
|
||||
"viewport": "1600x900",
|
||||
"notes": ["This is mock data to demo the viewer."],
|
||||
})
|
||||
|
||||
data = {
|
||||
"feature": "EXAMPLE - Compress PDF (mock data)",
|
||||
"branch": "demo",
|
||||
"generated": "example",
|
||||
"views": views,
|
||||
"findings": {
|
||||
"visual": [
|
||||
{"severity": "high", "view": "03_dialog", "title": "Dialog buttons too close",
|
||||
"detail": "Cancel/Confirm have only 8px gap; easy to misclick.",
|
||||
"fix": "Increase gap to var(--mantine-spacing-md)."},
|
||||
{"severity": "low", "view": "02_form", "title": "Field labels low contrast in dark mode",
|
||||
"detail": "Muted token fails WCAG AA on the dark panel.",
|
||||
"fix": "Use --mantine-color-dimmed instead of a hard-coded grey."},
|
||||
],
|
||||
"ux": [
|
||||
{"severity": "med", "view": "01_empty", "title": "Primary CTA below the dropzone",
|
||||
"detail": "Users expect the action button adjacent to the dropzone.",
|
||||
"fix": "Move the button directly under the dashed zone."},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
tpl = (HERE / "report-template.html").read_text(encoding="utf-8")
|
||||
out = re.sub(
|
||||
r"/\*__DATA__\*/.*?/\*__END__\*/",
|
||||
lambda _m: "/*__DATA__*/" + json.dumps(data) + "/*__END__*/",
|
||||
tpl, count=1, flags=re.S,
|
||||
)
|
||||
(HERE / "EXAMPLE.html").write_text(out, encoding="utf-8")
|
||||
print("wrote", (HERE / "EXAMPLE.html"))
|
||||
@@ -1,298 +0,0 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
UI Walkthrough report template (self-contained, works from file://).
|
||||
The ui-walkthrough skill replaces the JSON in the window.__WALKTHROUGH__ data
|
||||
block below with the captured manifest. Do not add external CDN deps - it must open offline.
|
||||
|
||||
Data shape:
|
||||
{
|
||||
"feature": "Compress PDF tool",
|
||||
"branch": "claude/...",
|
||||
"generated": "2026-06-21",
|
||||
"views": [
|
||||
{ "id": "01_empty", "title": "Empty state",
|
||||
"light": "screenshots/compress/01_empty_light.png",
|
||||
"dark": "screenshots/compress/01_empty_dark.png",
|
||||
"viewport": "1600x900",
|
||||
"notes": ["Heading is centered", "Primary CTA below the fold on mobile"] }
|
||||
],
|
||||
"findings": {
|
||||
"visual": [ { "severity":"high", "view":"01_empty", "title":"...", "detail":"...", "fix":"..." } ],
|
||||
"ux": [ { "severity":"med", "view":"03_dialog", "title":"...", "detail":"...", "fix":"..." } ]
|
||||
}
|
||||
}
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>UI Walkthrough</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f6f7f9; --panel: #ffffff; --panel-2: #f0f2f5; --text: #1a1b1e;
|
||||
--muted: #6b7280; --border: #e2e5ea; --accent: #2f6fed; --accent-weak: #e8f0fe;
|
||||
--shadow: 0 1px 3px rgba(0,0,0,.08), 0 8px 24px rgba(0,0,0,.06);
|
||||
--hi: #d92d20; --med: #d98e00; --low: #2f6fed; --stage: #0b0c0e;
|
||||
}
|
||||
html[data-theme="dark"] {
|
||||
--bg: #0d0e10; --panel: #16181c; --panel-2: #1d2024; --text: #e6e8eb;
|
||||
--muted: #9aa3ad; --border: #2a2e35; --accent: #5b8cff; --accent-weak: #1a2336;
|
||||
--shadow: 0 1px 3px rgba(0,0,0,.5), 0 8px 24px rgba(0,0,0,.4); --stage: #000;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; font: 14px/1.5 -apple-system, "Segoe UI", Roboto, system-ui, sans-serif;
|
||||
background: var(--bg); color: var(--text); }
|
||||
header { display: flex; align-items: center; gap: 16px; padding: 12px 20px;
|
||||
background: var(--panel); border-bottom: 1px solid var(--border); position: sticky; top: 0; z-index: 5; }
|
||||
header h1 { font-size: 15px; margin: 0; font-weight: 650; }
|
||||
header .sub { color: var(--muted); font-size: 12px; }
|
||||
.spacer { flex: 1; }
|
||||
.counter { color: var(--muted); font-variant-numeric: tabular-nums; font-size: 13px; }
|
||||
.tabs { display: flex; gap: 4px; }
|
||||
.tab { border: 1px solid var(--border); background: var(--panel-2); color: var(--text);
|
||||
padding: 6px 12px; border-radius: 8px; cursor: pointer; font-size: 13px; }
|
||||
.tab.active { background: var(--accent); color: #fff; border-color: var(--accent); }
|
||||
|
||||
/* Light/Dark slider */
|
||||
.theme-toggle { display: flex; align-items: center; gap: 9px; user-select: none; }
|
||||
.theme-toggle .lbl { font-size: 12px; color: var(--muted); }
|
||||
.theme-toggle .lbl.on { color: var(--text); font-weight: 600; }
|
||||
.switch { position: relative; width: 52px; height: 28px; }
|
||||
.switch input { opacity: 0; width: 0; height: 0; }
|
||||
.slider { position: absolute; inset: 0; cursor: pointer; background: var(--panel-2);
|
||||
border: 1px solid var(--border); border-radius: 999px; transition: .2s; }
|
||||
.slider:before { content: ""; position: absolute; height: 20px; width: 20px; left: 3px; top: 3px;
|
||||
background: #fbbf24; border-radius: 50%; transition: .2s; box-shadow: 0 1px 2px rgba(0,0,0,.3); }
|
||||
.switch input:checked + .slider { background: var(--accent); }
|
||||
.switch input:checked + .slider:before { transform: translateX(24px); background: #c7d2fe; }
|
||||
|
||||
main { display: grid; grid-template-columns: 240px 1fr; height: calc(100vh - 53px); }
|
||||
.rail { border-right: 1px solid var(--border); overflow-y: auto; background: var(--panel); padding: 8px; }
|
||||
.rail .group-label { font-size: 11px; text-transform: uppercase; letter-spacing: .05em;
|
||||
color: var(--muted); padding: 10px 8px 4px; }
|
||||
.thumb { display: flex; gap: 9px; align-items: center; padding: 7px; border-radius: 8px;
|
||||
cursor: pointer; border: 1px solid transparent; }
|
||||
.thumb:hover { background: var(--panel-2); }
|
||||
.thumb.active { background: var(--accent-weak); border-color: var(--accent); }
|
||||
.thumb img { width: 64px; height: 40px; object-fit: cover; border-radius: 4px; border: 1px solid var(--border); background: var(--stage); }
|
||||
.thumb .t { font-size: 12.5px; line-height: 1.3; }
|
||||
.thumb .badge { font-size: 10px; color: var(--muted); }
|
||||
.thumb .dot { width: 7px; height: 7px; border-radius: 50%; margin-left: auto; flex: none; }
|
||||
|
||||
.stagewrap { display: flex; flex-direction: column; min-width: 0; }
|
||||
.stage { flex: 1; display: flex; align-items: center; justify-content: center; padding: 22px;
|
||||
background: var(--stage); position: relative; min-height: 0; }
|
||||
.stage img { max-width: 100%; max-height: 100%; object-fit: contain; border-radius: 8px;
|
||||
box-shadow: 0 4px 30px rgba(0,0,0,.4); background: #fff; }
|
||||
html[data-theme="dark"] .stage img { background: #16181c; }
|
||||
.nav-btn { position: absolute; top: 50%; transform: translateY(-50%); width: 42px; height: 42px;
|
||||
border-radius: 50%; border: 1px solid var(--border); background: var(--panel);
|
||||
color: var(--text); cursor: pointer; font-size: 18px; opacity: .85; }
|
||||
.nav-btn:hover { opacity: 1; } .nav-btn.prev { left: 16px; } .nav-btn.next { right: 16px; }
|
||||
.nav-btn:disabled { opacity: .25; cursor: default; }
|
||||
.missing { color: var(--muted); font-size: 13px; text-align: center; }
|
||||
|
||||
.detail { border-top: 1px solid var(--border); background: var(--panel); padding: 14px 20px;
|
||||
max-height: 38vh; overflow-y: auto; }
|
||||
.detail h2 { margin: 0 0 4px; font-size: 15px; }
|
||||
.detail .meta { color: var(--muted); font-size: 12px; margin-bottom: 10px; }
|
||||
.notes { list-style: none; padding: 0; margin: 0; display: grid; gap: 6px; }
|
||||
.notes li { display: flex; gap: 8px; align-items: flex-start; }
|
||||
.sev { font-size: 10px; font-weight: 700; text-transform: uppercase; padding: 2px 7px; border-radius: 999px;
|
||||
color: #fff; flex: none; margin-top: 1px; }
|
||||
.sev.high { background: var(--hi); } .sev.med { background: var(--med); } .sev.low { background: var(--low); }
|
||||
.finding .fix { color: var(--muted); font-size: 12.5px; }
|
||||
.finding .fix b { color: var(--text); font-weight: 600; }
|
||||
|
||||
/* Summary tab */
|
||||
.summary { padding: 20px 28px; overflow-y: auto; }
|
||||
.summary h2 { font-size: 16px; margin: 22px 0 8px; }
|
||||
.summary .empty { color: var(--muted); }
|
||||
.card { background: var(--panel); border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 12px 14px; margin-bottom: 8px; box-shadow: var(--shadow); }
|
||||
.card .head { display: flex; gap: 8px; align-items: center; }
|
||||
.card a { color: var(--accent); text-decoration: none; cursor: pointer; }
|
||||
.hide { display: none !important; }
|
||||
kbd { font: 11px ui-monospace, monospace; background: var(--panel-2); border: 1px solid var(--border);
|
||||
border-radius: 4px; padding: 1px 5px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div>
|
||||
<h1 id="feature-title">UI Walkthrough</h1>
|
||||
<div class="sub" id="feature-sub"></div>
|
||||
</div>
|
||||
<div class="spacer"></div>
|
||||
<div class="tabs">
|
||||
<button class="tab active" data-tab="viewer">Walkthrough</button>
|
||||
<button class="tab" data-tab="summary">Findings</button>
|
||||
</div>
|
||||
<div class="counter" id="counter"></div>
|
||||
<label class="theme-toggle" title="Toggle light / dark for every screenshot">
|
||||
<span class="lbl" id="lbl-light">Light</span>
|
||||
<span class="switch"><input type="checkbox" id="theme-switch" /><span class="slider"></span></span>
|
||||
<span class="lbl" id="lbl-dark">Dark</span>
|
||||
</label>
|
||||
</header>
|
||||
|
||||
<main id="viewer-pane">
|
||||
<aside class="rail" id="rail"></aside>
|
||||
<section class="stagewrap">
|
||||
<div class="stage">
|
||||
<button class="nav-btn prev" id="prev" aria-label="Previous">‹</button>
|
||||
<img id="stage-img" alt="" />
|
||||
<div class="missing hide" id="missing"></div>
|
||||
<button class="nav-btn next" id="next" aria-label="Next">›</button>
|
||||
</div>
|
||||
<div class="detail">
|
||||
<h2 id="view-title"></h2>
|
||||
<div class="meta" id="view-meta"></div>
|
||||
<ul class="notes" id="view-notes"></ul>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<section class="summary hide" id="summary-pane"></section>
|
||||
|
||||
<script id="data">
|
||||
window.__WALKTHROUGH__ = /*__DATA__*/{"feature":"No data","branch":"","generated":"","views":[],"findings":{"visual":[],"ux":[]}}/*__END__*/;
|
||||
</script>
|
||||
<script>
|
||||
(function () {
|
||||
var D = window.__WALKTHROUGH__ || { views: [], findings: { visual: [], ux: [] } };
|
||||
var views = D.views || [];
|
||||
var state = { i: 0, theme: localStorage.getItem("ui-wt-theme") || "light", tab: "viewer" };
|
||||
|
||||
var $ = function (id) { return document.getElementById(id); };
|
||||
function sevClass(s) { return s === "high" ? "high" : s === "med" || s === "medium" ? "med" : "low"; }
|
||||
|
||||
function applyChrome() {
|
||||
document.documentElement.setAttribute("data-theme", state.theme);
|
||||
$("theme-switch").checked = state.theme === "dark";
|
||||
$("lbl-light").classList.toggle("on", state.theme === "light");
|
||||
$("lbl-dark").classList.toggle("on", state.theme === "dark");
|
||||
}
|
||||
|
||||
function srcFor(v) { return state.theme === "dark" ? (v.dark || v.light) : (v.light || v.dark); }
|
||||
|
||||
function findingsForView(id) {
|
||||
var all = (D.findings && D.findings.visual || []).concat(D.findings && D.findings.ux || []);
|
||||
return all.filter(function (f) { return f.view === id; });
|
||||
}
|
||||
|
||||
function renderRail() {
|
||||
var rail = $("rail");
|
||||
rail.innerHTML = "";
|
||||
if (!views.length) { rail.innerHTML = '<div class="group-label">No views captured</div>'; return; }
|
||||
views.forEach(function (v, idx) {
|
||||
var fs = findingsForView(v.id);
|
||||
var worst = fs.some(function (f){return sevClass(f.severity)==="high";}) ? "var(--hi)"
|
||||
: fs.some(function (f){return sevClass(f.severity)==="med";}) ? "var(--med)"
|
||||
: fs.length ? "var(--low)" : "transparent";
|
||||
var el = document.createElement("div");
|
||||
el.className = "thumb" + (idx === state.i ? " active" : "");
|
||||
el.innerHTML = '<img src="' + srcFor(v) + '" alt="" />' +
|
||||
'<div><div class="t">' + (v.title || v.id) + '</div>' +
|
||||
'<div class="badge">' + (v.viewport || "") + '</div></div>' +
|
||||
'<span class="dot" style="background:' + worst + '"></span>';
|
||||
el.onclick = function () { state.i = idx; render(); };
|
||||
rail.appendChild(el);
|
||||
});
|
||||
}
|
||||
|
||||
function render() {
|
||||
applyChrome();
|
||||
if (!views.length) {
|
||||
$("missing").classList.remove("hide"); $("stage-img").classList.add("hide");
|
||||
$("missing").textContent = "No screenshots in this report yet.";
|
||||
$("counter").textContent = ""; return;
|
||||
}
|
||||
var v = views[state.i];
|
||||
var src = srcFor(v);
|
||||
var img = $("stage-img");
|
||||
if (src) {
|
||||
img.classList.remove("hide"); $("missing").classList.add("hide");
|
||||
img.src = src; img.alt = v.title || v.id;
|
||||
} else {
|
||||
img.classList.add("hide"); $("missing").classList.remove("hide");
|
||||
$("missing").textContent = "No " + state.theme + " screenshot for this view.";
|
||||
}
|
||||
$("counter").textContent = (state.i + 1) + " / " + views.length;
|
||||
$("view-title").textContent = v.title || v.id;
|
||||
$("view-meta").textContent = [v.viewport, state.theme + " mode"].filter(Boolean).join(" · ");
|
||||
var notes = $("view-notes"); notes.innerHTML = "";
|
||||
var fs = findingsForView(v.id);
|
||||
(v.notes || []).forEach(function (n) {
|
||||
var li = document.createElement("li"); li.textContent = "· " + n; notes.appendChild(li);
|
||||
});
|
||||
fs.forEach(function (f) {
|
||||
var li = document.createElement("li"); li.className = "finding";
|
||||
li.innerHTML = '<span class="sev ' + sevClass(f.severity) + '">' + (f.severity || "note") + '</span>' +
|
||||
'<span><b>' + (f.title || "") + '</b> — ' + (f.detail || "") +
|
||||
(f.fix ? ' <span class="fix"><b>Fix:</b> ' + f.fix + '</span>' : '') + '</span>';
|
||||
notes.appendChild(li);
|
||||
});
|
||||
$("prev").disabled = state.i === 0;
|
||||
$("next").disabled = state.i === views.length - 1;
|
||||
renderRail();
|
||||
}
|
||||
|
||||
function renderSummary() {
|
||||
var pane = $("summary-pane");
|
||||
function block(title, arr) {
|
||||
var h = '<h2>' + title + ' (' + arr.length + ')</h2>';
|
||||
if (!arr.length) return h + '<div class="empty">None found.</div>';
|
||||
return h + arr.map(function (f) {
|
||||
return '<div class="card"><div class="head">' +
|
||||
'<span class="sev ' + sevClass(f.severity) + '">' + (f.severity || "note") + '</span>' +
|
||||
'<b>' + (f.title || "") + '</b>' +
|
||||
(f.view ? ' <a data-jump="' + f.view + '">' + f.view + '</a>' : '') + '</div>' +
|
||||
'<div style="margin-top:6px">' + (f.detail || "") + '</div>' +
|
||||
(f.fix ? '<div class="finding" style="margin-top:6px"><span class="fix"><b>Fix:</b> ' + f.fix + '</span></div>' : '') +
|
||||
'</div>';
|
||||
}).join("");
|
||||
}
|
||||
pane.innerHTML = block("Visual & consistency", (D.findings && D.findings.visual) || []) +
|
||||
block("UX & ease of use", (D.findings && D.findings.ux) || []);
|
||||
pane.querySelectorAll("[data-jump]").forEach(function (a) {
|
||||
a.onclick = function () {
|
||||
var id = a.getAttribute("data-jump");
|
||||
var idx = views.findIndex(function (v) { return v.id === id; });
|
||||
if (idx >= 0) { state.i = idx; setTab("viewer"); }
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function setTab(t) {
|
||||
state.tab = t;
|
||||
document.querySelectorAll(".tab").forEach(function (b) { b.classList.toggle("active", b.dataset.tab === t); });
|
||||
$("viewer-pane").classList.toggle("hide", t !== "viewer");
|
||||
$("summary-pane").classList.toggle("hide", t !== "summary");
|
||||
if (t === "viewer") $("viewer-pane").style.display = "grid";
|
||||
if (t === "summary") renderSummary();
|
||||
}
|
||||
|
||||
// wiring
|
||||
$("feature-title").textContent = D.feature || "UI Walkthrough";
|
||||
$("feature-sub").textContent = [D.branch, D.generated].filter(Boolean).join(" · ");
|
||||
$("theme-switch").onchange = function () {
|
||||
state.theme = this.checked ? "dark" : "light";
|
||||
localStorage.setItem("ui-wt-theme", state.theme);
|
||||
render();
|
||||
};
|
||||
$("prev").onclick = function () { if (state.i > 0) { state.i--; render(); } };
|
||||
$("next").onclick = function () { if (state.i < views.length - 1) { state.i++; render(); } };
|
||||
document.addEventListener("keydown", function (e) {
|
||||
if (state.tab !== "viewer") return;
|
||||
if (e.key === "ArrowLeft") $("prev").click();
|
||||
if (e.key === "ArrowRight") $("next").click();
|
||||
if (e.key.toLowerCase() === "t") $("theme-switch").click();
|
||||
});
|
||||
document.querySelectorAll(".tab").forEach(function (b) { b.onclick = function () { setTab(b.dataset.tab); }; });
|
||||
|
||||
render();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -116,9 +116,6 @@ jobs:
|
||||
env:
|
||||
USE_DEPOT: ${{ needs.pick.outputs.is_fork != 'true' }}
|
||||
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
|
||||
# Single source of truth for whether this preview embeds the admin portal:
|
||||
# drives the image build-arg and the deployment comment.
|
||||
BUILD_PORTAL: "true"
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
@@ -249,9 +246,7 @@ jobs:
|
||||
file: ./docker/embedded/Dockerfile
|
||||
push: true
|
||||
tags: ${{ secrets.DOCKER_HUB_USERNAME }}/test:v2-${{ steps.commit-hash.outputs.app_short }}
|
||||
build-args: |
|
||||
VERSION_TAG=v2-alpha
|
||||
BUILD_PORTAL=${{ env.BUILD_PORTAL }}
|
||||
build-args: VERSION_TAG=v2-alpha
|
||||
platforms: linux/amd64
|
||||
|
||||
- name: Build and push V2 image (Docker fork fallback)
|
||||
@@ -264,9 +259,7 @@ jobs:
|
||||
cache-from: type=gha,scope=stirling-pdf-latest
|
||||
cache-to: type=gha,mode=max,scope=stirling-pdf-latest
|
||||
tags: ${{ secrets.DOCKER_HUB_USERNAME }}/test:v2-${{ steps.commit-hash.outputs.app_short }}
|
||||
build-args: |
|
||||
VERSION_TAG=v2-alpha
|
||||
BUILD_PORTAL=${{ env.BUILD_PORTAL }}
|
||||
build-args: VERSION_TAG=v2-alpha
|
||||
platforms: linux/amd64
|
||||
|
||||
- name: Set up SSH
|
||||
@@ -297,8 +290,6 @@ jobs:
|
||||
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/storage:/storage:rw
|
||||
environment:
|
||||
DISABLE_ADDITIONAL_FEATURES: "false"
|
||||
POLICIES_ENABLED: "true"
|
||||
STIRLING_BILLING_ACCOUNT_LINK_ENABLED: "true"
|
||||
SECURITY_ENABLELOGIN: "true"
|
||||
SECURITY_INITIALLOGIN_USERNAME: "${{ secrets.TEST_LOGIN_USERNAME }}"
|
||||
SECURITY_INITIALLOGIN_PASSWORD: "${{ secrets.TEST_LOGIN_PASSWORD }}"
|
||||
@@ -342,70 +333,9 @@ jobs:
|
||||
# Set port for output
|
||||
echo "v2_port=${V2_PORT}" >> $GITHUB_OUTPUT
|
||||
|
||||
# ---- Storybook preview (only when this PR touches stories/.storybook) ----
|
||||
# Runs inside the same approved-contributor-gated deploy job, so it deploys
|
||||
# under the exact same access rules as the app preview.
|
||||
- name: Detect Storybook changes
|
||||
id: sb-changes
|
||||
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
|
||||
with:
|
||||
list-files: json
|
||||
filters: |
|
||||
storybook:
|
||||
- 'frontend/**/*.stories.@(ts|tsx|mdx)'
|
||||
- 'frontend/**/*.mdx'
|
||||
- 'frontend/.storybook/**'
|
||||
|
||||
- name: Set up Node.js for Storybook
|
||||
if: steps.sb-changes.outputs.storybook == 'true'
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: "npm"
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install Task for Storybook
|
||||
if: steps.sb-changes.outputs.storybook == 'true'
|
||||
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
|
||||
|
||||
- name: Build and deploy Storybook
|
||||
id: storybook
|
||||
if: steps.sb-changes.outputs.storybook == 'true'
|
||||
env:
|
||||
VPS_HOST: ${{ secrets.NEW_VPS_HOST }}
|
||||
VPS_USER: ${{ secrets.NEW_VPS_USERNAME }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `prepare` generates the icon set stories import (not committed).
|
||||
task frontend:prepare
|
||||
task frontend:storybook:build
|
||||
PR=${{ needs.check-pr.outputs.pr_number }}
|
||||
# Served at the ROOT of its own port so Storybook's global MSW worker
|
||||
# (/mockServiceWorker.js) resolves. Port = PR + 20000 (bijective, offset
|
||||
# from the app preview's bare-PR-number port).
|
||||
SB_PORT=$((PR + 20000))
|
||||
DIR=/stirling/SB-PR-$PR
|
||||
tar czf storybook.tgz -C frontend/storybook-static .
|
||||
scp -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
storybook.tgz "$VPS_USER@$VPS_HOST:/tmp/storybook-$PR.tgz"
|
||||
ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -T \
|
||||
"$VPS_USER@$VPS_HOST" << ENDSSH
|
||||
set -e
|
||||
rm -rf "$DIR" && mkdir -p "$DIR"
|
||||
tar xzf /tmp/storybook-$PR.tgz -C "$DIR"
|
||||
rm -f /tmp/storybook-$PR.tgz
|
||||
docker rm -f storybook-pr-$PR 2>/dev/null || true
|
||||
docker run -d --name storybook-pr-$PR --restart unless-stopped \
|
||||
-p $SB_PORT:80 -v "$DIR":/usr/share/nginx/html:ro nginx:alpine
|
||||
ENDSSH
|
||||
echo "url=http://$VPS_HOST:$SB_PORT/" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Post V2 deployment URL to PR
|
||||
if: success()
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
SB_URL: ${{ steps.storybook.outputs.url }}
|
||||
SB_FILES: ${{ steps.sb-changes.outputs.storybook_files }}
|
||||
with:
|
||||
github-token: ${{ steps.setup-bot.outputs.token }}
|
||||
script: |
|
||||
@@ -429,40 +359,12 @@ jobs:
|
||||
}
|
||||
|
||||
const deploymentUrl = `http://${{ secrets.NEW_VPS_HOST }}:${v2Port}`;
|
||||
|
||||
// Only mention the portal when this image actually embeds it.
|
||||
// Use the direct IP URL - the SSL hostname isn't supported yet.
|
||||
const withPortal = "${{ env.BUILD_PORTAL }}" === "true";
|
||||
const portalNote = withPortal
|
||||
? `🧩 **Admin portal** included - try it at [${deploymentUrl}/portal](${deploymentUrl}/portal).\n\n`
|
||||
: ``;
|
||||
|
||||
// Storybook preview: only present when this PR changed stories/config.
|
||||
const sbUrl = process.env.SB_URL;
|
||||
let storybookNote = "";
|
||||
if (sbUrl) {
|
||||
const files = JSON.parse(process.env.SB_FILES || "[]");
|
||||
const stories = files.filter((f) => /\.stories\.(ts|tsx|mdx)$/.test(f));
|
||||
const config = files.filter((f) => f.startsWith("frontend/.storybook/"));
|
||||
const shorten = (f) =>
|
||||
f.replace(/^frontend\/editor\/src\//, "").replace(/^frontend\//, "");
|
||||
const storyList = stories.map((f) => `- \`${shorten(f)}\``).join("\n");
|
||||
const configList = config.map((f) => `- \`${shorten(f)}\``).join("\n");
|
||||
const summary =
|
||||
`${stories.length} stor${stories.length === 1 ? "y" : "ies"} changed` +
|
||||
(config.length ? ` (+${config.length} config file${config.length === 1 ? "" : "s"})` : "");
|
||||
storybookNote =
|
||||
`📚 **Storybook:** [${sbUrl}](${sbUrl})\n\n` +
|
||||
`<details>\n<summary>${summary}</summary>\n\n` +
|
||||
(storyList ? `**Stories**\n${storyList}\n\n` : "") +
|
||||
(configList ? `**Config**\n${configList}\n` : "") +
|
||||
`</details>\n\n`;
|
||||
}
|
||||
const httpsUrl = `https://${v2Port}.ssl.stirlingpdf.cloud`;
|
||||
|
||||
const commentBody = `## 🚀 V2 Auto-Deployment Complete!\n\n` +
|
||||
`Your V2 PR with embedded architecture has been deployed!\n\n` +
|
||||
`🔗 **Direct Test URL (non-SSL)** [${deploymentUrl}](${deploymentUrl})\n\n` +
|
||||
portalNote +
|
||||
storybookNote +
|
||||
`🔐 **Secure HTTPS URL**: [${httpsUrl}](${httpsUrl})\n\n` +
|
||||
`_This deployment will be automatically cleaned up when the PR is closed._\n\n` +
|
||||
`🔄 **Auto-deployed** for approved V2 contributors.`;
|
||||
|
||||
@@ -558,11 +460,7 @@ jobs:
|
||||
else
|
||||
echo "V2 PR directory not found, nothing to clean up"
|
||||
fi
|
||||
|
||||
# Remove this PR's Storybook preview (container + files), if any.
|
||||
docker rm -f storybook-pr-${{ github.event.pull_request.number }} 2>/dev/null || true
|
||||
rm -rf /stirling/SB-PR-${{ github.event.pull_request.number }}
|
||||
|
||||
|
||||
# Clean up old unused images (older than 2 weeks) but keep recent ones for reuse
|
||||
docker image prune -af --filter "until=336h" --filter "label!=keep=true" || true
|
||||
|
||||
|
||||
+10
-25
@@ -80,12 +80,6 @@ tasks:
|
||||
OPEN: '{{.OPEN | default ""}}'
|
||||
env:
|
||||
BACKEND_URL: '{{.BACKEND_URL}}'
|
||||
# Dev-only browser-tab label so concurrent worktrees are distinguishable.
|
||||
# Only the worktree folder basename (e.g. "wt1") is exposed — never the
|
||||
# full path, hostname, or user. Consumed at dev-serve time by vite.config
|
||||
# and dropped from production builds.
|
||||
STIRLING_DEV_LABEL:
|
||||
sh: basename "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cmds:
|
||||
- npx vite editor --mode {{.MODE}} --port {{.PORT}}{{if .OPEN}} --open{{end}}
|
||||
|
||||
@@ -201,40 +195,25 @@ tasks:
|
||||
desc: "Run linting"
|
||||
deps: [install]
|
||||
cmds:
|
||||
- task: lint:eslint
|
||||
- task: lint:dpdm
|
||||
|
||||
lint:eslint:
|
||||
desc: "Run ESLint linting"
|
||||
deps: [install]
|
||||
cmds:
|
||||
- npx eslint --max-warnings=0
|
||||
|
||||
lint:dpdm:
|
||||
desc: "Run circular import linting"
|
||||
deps: [install]
|
||||
cmds:
|
||||
# Globs so dpdm walks the whole tree. dpdm expands the braces itself, so this is
|
||||
# shell-agnostic. Covers the whole editor tree, including the portal layer.
|
||||
- npx dpdm "editor/src/**/*.{ts,tsx}" --circular --no-warning --no-tree --exit-code circular:1
|
||||
- task: lint:oxlint
|
||||
|
||||
lint:fix:
|
||||
desc: "Auto-fix lint issues"
|
||||
deps: [install]
|
||||
cmds:
|
||||
- npx eslint --fix
|
||||
- npx oxlint --fix
|
||||
|
||||
format:
|
||||
desc: "Auto-fix code formatting"
|
||||
deps: [install]
|
||||
cmds:
|
||||
- npx prettier --write .
|
||||
- npx oxfmt --write .
|
||||
|
||||
format:check:
|
||||
desc: "Check code formatting"
|
||||
deps: [install]
|
||||
cmds:
|
||||
- npx prettier --check .
|
||||
- npx oxfmt --check .
|
||||
|
||||
fix:
|
||||
desc: "Auto-fix lint and format"
|
||||
@@ -242,6 +221,12 @@ tasks:
|
||||
- task: format
|
||||
- task: lint:fix
|
||||
|
||||
lint:oxlint:
|
||||
desc: "Run oxlint linting"
|
||||
deps: [install]
|
||||
cmds:
|
||||
- npx oxlint --max-warnings=0
|
||||
|
||||
typecheck:
|
||||
desc: "Typecheck default build of the app"
|
||||
cmds:
|
||||
|
||||
Vendored
+1
-1
@@ -19,6 +19,6 @@
|
||||
"yzhang.markdown-all-in-one", // Markdown All-in-One extension for enhanced Markdown editing
|
||||
"stylelint.vscode-stylelint", // Stylelint extension for CSS and SCSS linting
|
||||
"redhat.vscode-yaml", // YAML extension for Visual Studio Code
|
||||
"dbaeumer.vscode-eslint", // ESLint extension for TypeScript linting
|
||||
"oxc.oxc-vscode", // Oxc (oxlint) extension for TypeScript/JavaScript linting
|
||||
]
|
||||
}
|
||||
|
||||
@@ -453,7 +453,6 @@ The frontend is organized with a clear separation of concerns:
|
||||
|
||||
- **CRITICAL**: Always update translations in `en-US` only - all other languages (including `en-GB`) are handled separately
|
||||
- Translation files are located in `frontend/editor/public/locales/`
|
||||
- After changing any translation file, run `task pre-commit:fix`
|
||||
|
||||
## Important Notes
|
||||
|
||||
|
||||
+1
-1
@@ -158,7 +158,7 @@ Stirling-PDF/
|
||||
│ │ │ └── locales/ # Internationalization files (JSON)
|
||||
│ │ └── vite.config.ts # Vite configuration
|
||||
│ ├── package.json # Shared workspace dependencies
|
||||
│ └── eslint.config.mjs # Shared lint config
|
||||
│ └── oxlint.config.ts # Shared lint config (oxlint)
|
||||
├── customFiles/ # Custom static files and templates (generated at runtime used to replace existing files)
|
||||
├── docs/ # Documentation files
|
||||
├── exampleYmlFiles/ # Example YAML configuration files
|
||||
|
||||
@@ -7,7 +7,6 @@ import java.time.format.DateTimeFormatter;
|
||||
import java.util.Calendar;
|
||||
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.apache.pdfbox.pdmodel.PDDocumentInformation;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.stereotype.Service;
|
||||
@@ -18,9 +17,6 @@ import stirling.software.common.model.PdfMetadata;
|
||||
@Service
|
||||
public class PdfMetadataService {
|
||||
|
||||
/** ({@code {labels}}). Written by the classify-and-label tool. */
|
||||
public static final String CLASSIFICATION_KEY = "StirlingPDFClassification";
|
||||
|
||||
private final ApplicationProperties applicationProperties;
|
||||
private final String stirlingPDFLabel;
|
||||
private final UserServiceInterface userService;
|
||||
@@ -181,14 +177,4 @@ public class PdfMetadataService {
|
||||
}
|
||||
pdf.getDocumentInformation().setAuthor(author);
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the document classifier's JSON result into the custom Info-dictionary field {@link
|
||||
* #CLASSIFICATION_KEY}, leaving all other metadata untouched.
|
||||
*/
|
||||
public void setClassificationMetadata(PDDocument pdf, String classificationJson) {
|
||||
PDDocumentInformation info = pdf.getDocumentInformation();
|
||||
info.setCustomMetadataValue(CLASSIFICATION_KEY, classificationJson);
|
||||
pdf.setDocumentInformation(info);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,16 +57,6 @@ public class RequestUriUtils {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Admin portal SPA shell (mounted at /processor — must match the frontend
|
||||
// PORTAL_BASENAME). Served publicly like the editor root so a direct nav /
|
||||
// refresh to /processor loads the app (the JWT lives in localStorage, not a
|
||||
// cookie, so the server can't authenticate the navigation itself). The
|
||||
// portal gates access via its own auth gate + RequirePortalAccess, and its
|
||||
// data APIs stay protected, so serving the shell pre-auth is safe.
|
||||
if (normalizedUri.equals("/processor") || normalizedUri.startsWith("/processor/")) {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Treat common static file extensions as static resources
|
||||
return normalizedUri.endsWith(".svg")
|
||||
|| normalizedUri.endsWith(".png")
|
||||
|
||||
@@ -73,14 +73,6 @@ class RequestUriUtilsTest {
|
||||
assertTrue(RequestUriUtils.isStaticResource("/mobile-scanner"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void testIsStaticResource_portalShell() {
|
||||
// The admin portal SPA shell (/processor) is served pre-auth so it's directly navigable.
|
||||
assertTrue(RequestUriUtils.isStaticResource("/processor"));
|
||||
assertTrue(RequestUriUtils.isStaticResource("/processor/users"));
|
||||
assertTrue(RequestUriUtils.isStaticResource("/app", "/app/processor"));
|
||||
}
|
||||
|
||||
// --- isFrontendRoute tests ---
|
||||
|
||||
@Test
|
||||
|
||||
+1
-11
@@ -175,14 +175,6 @@ springBoot {
|
||||
// Frontend build tasks - only enabled with -PbuildWithFrontend=true
|
||||
def buildWithFrontend = project.hasProperty('buildWithFrontend') && project.property('buildWithFrontend') == 'true'
|
||||
def buildPrototypes = project.hasProperty('prototypesMode') && project.property('prototypesMode') == 'true'
|
||||
// The admin portal ships as a lazy route inside the editor bundle (see
|
||||
// proprietary/routes/adminRouteExtensions). -PbuildWithPortal=true includes that
|
||||
// chunk via VITE_INCLUDE_PORTAL on the editor build; the deploy GHA sets it when
|
||||
// the portal or AI layers change. Building the portal implies building the editor.
|
||||
def buildWithPortal = project.hasProperty('buildWithPortal') && project.property('buildWithPortal') == 'true'
|
||||
if (buildWithPortal) {
|
||||
buildWithFrontend = true
|
||||
}
|
||||
// Workspace root holds package.json and node_modules (shared across editor /
|
||||
// future portal). Editor-specific paths (src, public, dist, tauri) live one
|
||||
// level deeper under frontend/editor/.
|
||||
@@ -305,11 +297,9 @@ tasks.register('npmBuild', Exec) {
|
||||
// Override VITE_API_BASE_URL to use relative paths for production builds
|
||||
// This ensures JARs work regardless of how they're deployed (direct, proxied, etc.)
|
||||
environment 'VITE_API_BASE_URL', '/'
|
||||
// Include the admin portal's lazy route/chunk in the editor build when requested.
|
||||
environment 'VITE_INCLUDE_PORTAL', (buildWithPortal ? 'true' : 'false')
|
||||
|
||||
doFirst {
|
||||
println "Building editor frontend application for production (mode=${frontendMode}, VITE_API_BASE_URL=/, portal=${buildWithPortal})"
|
||||
println "Building editor frontend application for production (mode=${frontendMode}, VITE_API_BASE_URL=/)"
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
-29
@@ -305,23 +305,6 @@ public class GetInfoOnPDF {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Info-dictionary keys exposed above via typed getters; any other key in the dictionary is
|
||||
* surfaced as custom metadata (e.g. the classification policy's StirlingPDFClassification
|
||||
* entry).
|
||||
*/
|
||||
private static final java.util.Set<String> STANDARD_INFO_KEYS =
|
||||
java.util.Set.of(
|
||||
"Title",
|
||||
"Author",
|
||||
"Subject",
|
||||
"Keywords",
|
||||
"Producer",
|
||||
"Creator",
|
||||
"CreationDate",
|
||||
"ModDate",
|
||||
"Trapped");
|
||||
|
||||
private static ObjectNode extractMetadata(PDDocument document) {
|
||||
ObjectNode metadata = objectMapper.createObjectNode();
|
||||
|
||||
@@ -352,18 +335,6 @@ public class GetInfoOnPDF {
|
||||
if (modificationDate != null) {
|
||||
metadata.put("ModificationDate", modificationDate);
|
||||
}
|
||||
|
||||
// Surface custom Info-dictionary entries (anything beyond the
|
||||
// standard fields above) — e.g. StirlingPDFClassification
|
||||
for (String key : info.getMetadataKeys()) {
|
||||
if (STANDARD_INFO_KEYS.contains(key)) {
|
||||
continue;
|
||||
}
|
||||
String value = info.getCustomMetadataValue(key);
|
||||
if (value != null && !value.isBlank()) {
|
||||
metadata.put(key, value);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.error("Error extracting metadata: {}", e.getMessage());
|
||||
|
||||
-9
@@ -4,9 +4,7 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
import stirling.software.proprietary.access.service.DefaultPrincipalResolver;
|
||||
import stirling.software.proprietary.access.service.DefaultTeamLeadLookup;
|
||||
import stirling.software.proprietary.access.service.PrincipalResolver;
|
||||
import stirling.software.proprietary.access.service.TeamLeadLookup;
|
||||
|
||||
/** Access-layer bean wiring. */
|
||||
@@ -19,11 +17,4 @@ public class AccessConfig {
|
||||
TeamLeadLookup defaultTeamLeadLookup() {
|
||||
return new DefaultTeamLeadLookup();
|
||||
}
|
||||
|
||||
/** USER/TEAM projection unless another bean is defined (e.g. the saas resolver). */
|
||||
@Bean
|
||||
@ConditionalOnMissingBean(PrincipalResolver.class)
|
||||
PrincipalResolver defaultPrincipalResolver() {
|
||||
return new DefaultPrincipalResolver();
|
||||
}
|
||||
}
|
||||
|
||||
+4
-33
@@ -25,9 +25,7 @@ import stirling.software.proprietary.access.model.PrincipalType;
|
||||
import stirling.software.proprietary.access.model.ResourceGrant;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
import stirling.software.proprietary.access.service.ResourceAccessService;
|
||||
import stirling.software.proprietary.security.database.repository.UserRepository;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.repository.TeamRepository;
|
||||
|
||||
/** Admin endpoints to grant/revoke access to gated resources (the portal, integration configs). */
|
||||
@RestController
|
||||
@@ -38,8 +36,6 @@ import stirling.software.proprietary.security.repository.TeamRepository;
|
||||
public class ResourceGrantController {
|
||||
|
||||
private final ResourceAccessService accessService;
|
||||
private final UserRepository userRepository;
|
||||
private final TeamRepository teamRepository;
|
||||
|
||||
@GetMapping("/grants")
|
||||
public ResponseEntity<?> list(
|
||||
@@ -49,14 +45,6 @@ public class ResourceGrantController {
|
||||
return ResponseEntity.ok(grants.stream().map(this::toDto).toList());
|
||||
}
|
||||
|
||||
@GetMapping("/grants/by-principal")
|
||||
public ResponseEntity<?> listByPrincipal(
|
||||
@RequestParam PrincipalType principalType, @RequestParam Long principalId) {
|
||||
List<ResourceGrant> grants =
|
||||
accessService.listGrantsForPrincipal(principalType, principalId);
|
||||
return ResponseEntity.ok(grants.stream().map(this::toDto).toList());
|
||||
}
|
||||
|
||||
@PostMapping("/grants")
|
||||
public ResponseEntity<?> create(
|
||||
@RequestBody GrantRequest request, @AuthenticationPrincipal User admin) {
|
||||
@@ -69,26 +57,17 @@ public class ResourceGrantController {
|
||||
"error",
|
||||
"resourceType, principalType and principalId are required"));
|
||||
}
|
||||
// PORTAL is a singleton (empty resourceId); every other type must name a resource.
|
||||
boolean portal = request.resourceType() == ResourceType.PORTAL;
|
||||
if (!portal && (request.resourceId() == null || request.resourceId().isBlank())) {
|
||||
return ResponseEntity.badRequest()
|
||||
.body(Map.of("error", "resourceId is required for " + request.resourceType()));
|
||||
}
|
||||
Long principalId = request.principalId();
|
||||
String principalError = validatePrincipalExists(request.principalType(), principalId);
|
||||
if (principalError != null) {
|
||||
return ResponseEntity.badRequest().body(Map.of("error", principalError));
|
||||
}
|
||||
AccessPermission permission =
|
||||
request.permission() == null ? AccessPermission.USE : request.permission();
|
||||
String resourceId = portal ? "" : request.resourceId();
|
||||
// PORTAL is a singleton resource; its grants always target the whole type.
|
||||
String resourceId =
|
||||
request.resourceType() == ResourceType.PORTAL ? "" : request.resourceId();
|
||||
ResourceGrant grant =
|
||||
accessService.grant(
|
||||
request.resourceType(),
|
||||
resourceId,
|
||||
request.principalType(),
|
||||
principalId,
|
||||
request.principalId(),
|
||||
permission,
|
||||
admin);
|
||||
return ResponseEntity.ok(toDto(grant));
|
||||
@@ -100,14 +79,6 @@ public class ResourceGrantController {
|
||||
return ResponseEntity.ok(Map.of("message", "Grant revoked"));
|
||||
}
|
||||
|
||||
// Rejects grants to nonexistent principals (dead rows otherwise).
|
||||
private String validatePrincipalExists(PrincipalType type, Long id) {
|
||||
return switch (type) {
|
||||
case USER -> userRepository.existsById(id) ? null : "User " + id + " does not exist";
|
||||
case TEAM -> teamRepository.existsById(id) ? null : "Team " + id + " does not exist";
|
||||
};
|
||||
}
|
||||
|
||||
private Map<String, Object> toDto(ResourceGrant g) {
|
||||
Map<String, Object> m = new HashMap<>();
|
||||
m.put("id", g.getId());
|
||||
|
||||
-11
@@ -54,15 +54,4 @@ public abstract class OwnedResource {
|
||||
public Long getOwnerTeamId() {
|
||||
return ownerTeam != null ? ownerTeam.getId() : null;
|
||||
}
|
||||
|
||||
/** Owner as a principal ref; null when server-owned (admin-only ownership). */
|
||||
public PrincipalRef getOwnerRef() {
|
||||
if (getOwnerUserId() != null) {
|
||||
return PrincipalRef.user(getOwnerUserId());
|
||||
}
|
||||
if (getOwnerTeamId() != null) {
|
||||
return PrincipalRef.team(getOwnerTeamId());
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
package stirling.software.proprietary.access.model;
|
||||
|
||||
import java.util.Locale;
|
||||
|
||||
/** A (type, id) principal pair; the atom grants and ownership are expressed in. */
|
||||
public record PrincipalRef(PrincipalType type, Long id) {
|
||||
|
||||
public static PrincipalRef user(Long id) {
|
||||
return new PrincipalRef(PrincipalType.USER, id);
|
||||
}
|
||||
|
||||
public static PrincipalRef team(Long id) {
|
||||
return new PrincipalRef(PrincipalType.TEAM, id);
|
||||
}
|
||||
|
||||
/** Canonical engine wire form, e.g. "user:12". */
|
||||
public String token() {
|
||||
return type.name().toLowerCase(Locale.ROOT) + ":" + id;
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
package stirling.software.proprietary.access.model;
|
||||
|
||||
/** Who a {@link ResourceGrant} is granted to. */
|
||||
/** Who a {@link ResourceGrant} is granted to. Org-wide access is expressed via default policy. */
|
||||
public enum PrincipalType {
|
||||
USER,
|
||||
TEAM
|
||||
|
||||
-16
@@ -3,15 +3,11 @@ package stirling.software.proprietary.access.repository;
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Modifying;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import stirling.software.proprietary.access.model.PrincipalType;
|
||||
import stirling.software.proprietary.access.model.ResourceGrant;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
|
||||
@Repository
|
||||
public interface ResourceGrantRepository extends JpaRepository<ResourceGrant, Long> {
|
||||
@@ -22,20 +18,8 @@ public interface ResourceGrantRepository extends JpaRepository<ResourceGrant, Lo
|
||||
List<ResourceGrant> findByResourceTypeAndPrincipalTypeAndPrincipalId(
|
||||
ResourceType resourceType, PrincipalType principalType, Long principalId);
|
||||
|
||||
/** All grants held by a principal, across resource types (for the manage-access view). */
|
||||
List<ResourceGrant> findByPrincipalTypeAndPrincipalId(
|
||||
PrincipalType principalType, Long principalId);
|
||||
|
||||
void deleteByResourceTypeAndResourceId(ResourceType resourceType, String resourceId);
|
||||
|
||||
/** Removes every grant held by a principal; used when the user/team behind it is deleted. */
|
||||
void deleteByPrincipalTypeAndPrincipalId(PrincipalType principalType, Long principalId);
|
||||
|
||||
// Detach issued grants so deleting the granting user does not hit the FK.
|
||||
@Modifying
|
||||
@Query("update ResourceGrant g set g.grantedBy = null where g.grantedBy = :user")
|
||||
void clearGrantedBy(@Param("user") User user);
|
||||
|
||||
boolean existsByResourceTypeAndResourceIdAndPrincipalTypeAndPrincipalId(
|
||||
ResourceType resourceType,
|
||||
String resourceId,
|
||||
|
||||
+1
-6
@@ -11,12 +11,7 @@ import stirling.software.proprietary.access.service.ResourceAccessService;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.service.UserService;
|
||||
|
||||
/**
|
||||
* {@code @PreAuthorize} bean for portal-access checks. Active in self-hosted and saas. Convention:
|
||||
* every portal-exclusive endpoint is gated with
|
||||
* {@code @PreAuthorize("@resourceAccess.canUsePortal()")}; endpoints shared with the editor (e.g.
|
||||
* the policies API) must NOT be.
|
||||
*/
|
||||
/** {@code @PreAuthorize} bean for portal-access checks. Active in self-hosted and saas. */
|
||||
@Component("resourceAccess")
|
||||
@RequiredArgsConstructor
|
||||
public class ResourceAccessSecurity {
|
||||
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
package stirling.software.proprietary.access.service;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
|
||||
import stirling.software.proprietary.access.model.PrincipalRef;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
|
||||
/**
|
||||
* Self-hosted projection: the user and their team. One deployment = one org, so ORG_ALL is open.
|
||||
*/
|
||||
public class DefaultPrincipalResolver implements PrincipalResolver {
|
||||
|
||||
@Override
|
||||
public Set<PrincipalRef> principalsOf(User user) {
|
||||
if (user == null) {
|
||||
return Set.of();
|
||||
}
|
||||
Set<PrincipalRef> principals = new HashSet<>();
|
||||
principals.add(PrincipalRef.user(user.getId()));
|
||||
if (user.getTeam() != null) {
|
||||
principals.add(PrincipalRef.team(user.getTeam().getId()));
|
||||
}
|
||||
return principals;
|
||||
}
|
||||
|
||||
// Self-hosted is a single deployment-wide org, so ORG_ALL admits every authenticated user.
|
||||
@Override
|
||||
public boolean allowsDeploymentWideAccess() {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
-33
@@ -1,33 +0,0 @@
|
||||
package stirling.software.proprietary.access.service;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.model.enumeration.TeamRole;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
|
||||
|
||||
/** Real lookup backed by team_memberships LEADER rows; wins over the no-op default bean. */
|
||||
@Component
|
||||
@RequiredArgsConstructor
|
||||
public class MembershipTeamLeadLookup implements TeamLeadLookup {
|
||||
|
||||
private final TeamMembershipRepository memberships;
|
||||
|
||||
@Override
|
||||
public boolean isAnyTeamLeader(User user) {
|
||||
return user != null
|
||||
&& user.getId() != null
|
||||
&& memberships.existsByUserIdAndRole(user.getId(), TeamRole.LEADER);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isLeaderOfTeam(User user, Long teamId) {
|
||||
return user != null
|
||||
&& user.getId() != null
|
||||
&& teamId != null
|
||||
&& memberships.existsByTeamIdAndUserIdAndRole(
|
||||
teamId, user.getId(), TeamRole.LEADER);
|
||||
}
|
||||
}
|
||||
+2
-6
@@ -36,19 +36,15 @@ public class OwnershipService {
|
||||
return accessService.canUseResource(
|
||||
type,
|
||||
String.valueOf(resource.getId()),
|
||||
resource.getOwnerRef(),
|
||||
resource.getOwnerUserId(),
|
||||
resource.getDefaultAccess(),
|
||||
user);
|
||||
}
|
||||
|
||||
/** Whether the user may manage the resource. */
|
||||
public boolean canManage(ResourceType type, OwnedResource resource, User user) {
|
||||
// Disabled resources bypass grants for MANAGE too: admin/owner only.
|
||||
if (!resource.isEnabled()) {
|
||||
return isAdmin(user) || isOwner(resource, user);
|
||||
}
|
||||
return accessService.canManageResource(
|
||||
type, String.valueOf(resource.getId()), resource.getOwnerRef(), user);
|
||||
type, String.valueOf(resource.getId()), resource.getOwnerUserId(), user);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
-28
@@ -1,28 +0,0 @@
|
||||
package stirling.software.proprietary.access.service;
|
||||
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import stirling.software.proprietary.access.model.PrincipalRef;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
|
||||
/** Projects a user onto the set of principals they act as. */
|
||||
public interface PrincipalResolver {
|
||||
|
||||
/** Every principal the user acts as; empty for a null user. */
|
||||
Set<PrincipalRef> principalsOf(User user);
|
||||
|
||||
/**
|
||||
* Whether this deployment treats every authenticated user as one org, so the {@code ORG_ALL}
|
||||
* default policy admits anyone. Self-hosted: true. Multi-tenant saas: false, so an {@code
|
||||
* ORG_ALL} resource can't leak across tenants. Defaults to false (deny) for safety.
|
||||
*/
|
||||
default boolean allowsDeploymentWideAccess() {
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Canonical wire tokens for the engine, e.g. "user:12". */
|
||||
default Set<String> principalTokens(User user) {
|
||||
return principalsOf(user).stream().map(PrincipalRef::token).collect(Collectors.toSet());
|
||||
}
|
||||
}
|
||||
+27
-44
@@ -14,7 +14,6 @@ import lombok.extern.slf4j.Slf4j;
|
||||
import stirling.software.common.model.enumeration.Role;
|
||||
import stirling.software.proprietary.access.model.AccessPermission;
|
||||
import stirling.software.proprietary.access.model.DefaultAccessPolicy;
|
||||
import stirling.software.proprietary.access.model.PrincipalRef;
|
||||
import stirling.software.proprietary.access.model.PrincipalType;
|
||||
import stirling.software.proprietary.access.model.ResourceGrant;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
@@ -30,7 +29,6 @@ public class ResourceAccessService {
|
||||
|
||||
private final ResourceGrantRepository grantRepository;
|
||||
private final TeamLeadLookup teamLeadLookup;
|
||||
private final PrincipalResolver principalResolver;
|
||||
|
||||
@Value("${security.portal.defaultAccess:ADMINS_AND_TEAM_LEADS}")
|
||||
private DefaultAccessPolicy portalDefaultPolicy;
|
||||
@@ -46,28 +44,28 @@ public class ResourceAccessService {
|
||||
public boolean canUseResource(
|
||||
ResourceType type,
|
||||
String resourceId,
|
||||
PrincipalRef owner,
|
||||
Long ownerUserId,
|
||||
DefaultAccessPolicy defaultPolicy,
|
||||
User user) {
|
||||
if (user == null) {
|
||||
return false;
|
||||
}
|
||||
if (isOwner(owner, user) || isAdmin(user)) {
|
||||
if (isOwner(ownerUserId, user) || isAdmin(user)) {
|
||||
return true;
|
||||
}
|
||||
if (hasGrant(type, normalize(resourceId), user, AccessPermission.USE)) {
|
||||
return true;
|
||||
}
|
||||
return matchesDefault(defaultPolicy, owner, user);
|
||||
return matchesDefault(defaultPolicy, user);
|
||||
}
|
||||
|
||||
/** Whether the user may manage (edit/delete/share) a resource. No default-policy fallback. */
|
||||
public boolean canManageResource(
|
||||
ResourceType type, String resourceId, PrincipalRef owner, User user) {
|
||||
ResourceType type, String resourceId, Long ownerUserId, User user) {
|
||||
if (user == null) {
|
||||
return false;
|
||||
}
|
||||
if (isOwner(owner, user) || isAdmin(user)) {
|
||||
if (isOwner(ownerUserId, user) || isAdmin(user)) {
|
||||
return true;
|
||||
}
|
||||
return hasGrant(type, normalize(resourceId), user, AccessPermission.MANAGE);
|
||||
@@ -112,22 +110,21 @@ public class ResourceAccessService {
|
||||
return grantRepository.findByResourceTypeAndResourceId(type, normalize(resourceId));
|
||||
}
|
||||
|
||||
/** Every grant a principal holds, for the per-user/per-team manage-access view. */
|
||||
public List<ResourceGrant> listGrantsForPrincipal(
|
||||
PrincipalType principalType, Long principalId) {
|
||||
return grantRepository.findByPrincipalTypeAndPrincipalId(principalType, principalId);
|
||||
}
|
||||
|
||||
/** Resource ids of the given type that any of the user's principals holds a grant on. */
|
||||
/** Resource ids of the given type that this user (or their team) holds any grant on. */
|
||||
public Set<String> grantedResourceIds(ResourceType type, User user) {
|
||||
if (user == null) {
|
||||
return Set.of();
|
||||
}
|
||||
Set<String> ids = new HashSet<>();
|
||||
for (PrincipalRef principal : principalResolver.principalsOf(user)) {
|
||||
for (ResourceGrant g :
|
||||
grantRepository.findByResourceTypeAndPrincipalTypeAndPrincipalId(
|
||||
type, PrincipalType.USER, user.getId())) {
|
||||
ids.add(g.getResourceId());
|
||||
}
|
||||
if (user.getTeam() != null) {
|
||||
for (ResourceGrant g :
|
||||
grantRepository.findByResourceTypeAndPrincipalTypeAndPrincipalId(
|
||||
type, principal.type(), principal.id())) {
|
||||
type, PrincipalType.TEAM, user.getTeam().getId())) {
|
||||
ids.add(g.getResourceId());
|
||||
}
|
||||
}
|
||||
@@ -138,12 +135,18 @@ public class ResourceAccessService {
|
||||
|
||||
private boolean hasGrant(
|
||||
ResourceType type, String resourceId, User user, AccessPermission required) {
|
||||
Set<PrincipalRef> principals = principalResolver.principalsOf(user);
|
||||
Long teamId = user.getTeam() != null ? user.getTeam().getId() : null;
|
||||
for (ResourceGrant g : grantRepository.findByResourceTypeAndResourceId(type, resourceId)) {
|
||||
if (!permissionSatisfies(g.getPermission(), required)) {
|
||||
continue;
|
||||
}
|
||||
if (principals.contains(new PrincipalRef(g.getPrincipalType(), g.getPrincipalId()))) {
|
||||
if (g.getPrincipalType() == PrincipalType.USER
|
||||
&& g.getPrincipalId().equals(user.getId())) {
|
||||
return true;
|
||||
}
|
||||
if (g.getPrincipalType() == PrincipalType.TEAM
|
||||
&& teamId != null
|
||||
&& g.getPrincipalId().equals(teamId)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -158,40 +161,20 @@ public class ResourceAccessService {
|
||||
return held == AccessPermission.MANAGE;
|
||||
}
|
||||
|
||||
private boolean matchesDefault(DefaultAccessPolicy policy, PrincipalRef owner, User user) {
|
||||
private boolean matchesDefault(DefaultAccessPolicy policy, User user) {
|
||||
if (policy == null) {
|
||||
return false;
|
||||
}
|
||||
return switch (policy) {
|
||||
// Deployment-wide only where the resolver treats everyone as one org; saas resolvers
|
||||
// return false, so ORG_ALL cannot leak a tenant's resource to another tenant's users.
|
||||
case ORG_ALL -> principalResolver.allowsDeploymentWideAccess();
|
||||
// Admins already pass above; only team leads here, scoped to the owning team.
|
||||
case ADMINS_AND_TEAM_LEADS -> matchesTeamLeadDefault(owner, user);
|
||||
case ORG_ALL -> true;
|
||||
// Admins already pass above; only team leads here.
|
||||
case ADMINS_AND_TEAM_LEADS -> teamLeadLookup.isAnyTeamLeader(user);
|
||||
case EXPLICIT_ONLY -> false;
|
||||
};
|
||||
}
|
||||
|
||||
// Portal (no owner) admits any team lead; a team-owned resource admits only that team's
|
||||
// leads; a user-owned resource admits no extra leads.
|
||||
private boolean matchesTeamLeadDefault(PrincipalRef owner, User user) {
|
||||
if (owner == null) {
|
||||
return teamLeadLookup.isAnyTeamLeader(user);
|
||||
}
|
||||
return owner.type() == PrincipalType.TEAM
|
||||
&& owner.id() != null
|
||||
&& teamLeadLookup.isLeaderOfTeam(user, owner.id());
|
||||
}
|
||||
|
||||
// Team owners are the owning team's leaders; plain members are not.
|
||||
private boolean isOwner(PrincipalRef owner, User user) {
|
||||
if (owner == null || owner.id() == null) {
|
||||
return false;
|
||||
}
|
||||
return switch (owner.type()) {
|
||||
case USER -> owner.id().equals(user.getId());
|
||||
case TEAM -> teamLeadLookup.isLeaderOfTeam(user, owner.id());
|
||||
};
|
||||
private boolean isOwner(Long ownerUserId, User user) {
|
||||
return ownerUserId != null && ownerUserId.equals(user.getId());
|
||||
}
|
||||
|
||||
private boolean isAdmin(User user) {
|
||||
|
||||
+5
-22
@@ -18,26 +18,15 @@ public class SecretMasker {
|
||||
// Cap recursion so a pathologically nested payload cannot overflow the stack.
|
||||
private static final int MAX_DEPTH = 32;
|
||||
|
||||
// Key-name substrings that mark a value sensitive. Over-masking a non-secret is
|
||||
// safe; leaking a secret is not, so this errs broad - but a per-type schema
|
||||
// whitelist would be a stronger boundary for free-form config (follow-up).
|
||||
private static final Set<String> SENSITIVE_HINTS =
|
||||
Set.of(
|
||||
"secret",
|
||||
"password",
|
||||
"passphrase",
|
||||
"pwd",
|
||||
"token",
|
||||
"apikey",
|
||||
"accesskey",
|
||||
"credential",
|
||||
"privatekey",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"session",
|
||||
"connectionstring",
|
||||
"bearer",
|
||||
"signature");
|
||||
"privatekey");
|
||||
|
||||
/** Replace sensitive values with the mask (recursively) for safe display. */
|
||||
public Map<String, Object> mask(Map<String, Object> config) {
|
||||
@@ -84,24 +73,18 @@ public class SecretMasker {
|
||||
|
||||
private Map<String, Object> merge(
|
||||
Map<String, Object> stored, Map<String, Object> incoming, int depth) {
|
||||
// Replace semantics (PUT): the result is the incoming document, except a redacted secret
|
||||
// keeps its stored value. Keys absent from incoming are dropped, so edits can remove them.
|
||||
Map<String, Object> out = new LinkedHashMap<>();
|
||||
Map<String, Object> out = new LinkedHashMap<>(stored);
|
||||
for (Map.Entry<String, Object> e : incoming.entrySet()) {
|
||||
String key = e.getKey();
|
||||
Object value = e.getValue();
|
||||
if (isSensitive(key)) {
|
||||
if (isRedacted(value, depth)) {
|
||||
if (stored.containsKey(key)) {
|
||||
out.put(key, stored.get(key)); // keep the stored secret
|
||||
}
|
||||
} else {
|
||||
if (!isRedacted(value, depth)) {
|
||||
out.put(key, value); // a real new secret replaces the stored one
|
||||
}
|
||||
continue;
|
||||
continue; // redacted (blank / mask) -> keep stored
|
||||
}
|
||||
if (depth < MAX_DEPTH
|
||||
&& stored.get(key) instanceof Map<?, ?> s
|
||||
&& out.get(key) instanceof Map<?, ?> s
|
||||
&& value instanceof Map<?, ?> i) {
|
||||
out.put(key, merge(castMap(s), castMap(i), depth + 1));
|
||||
} else {
|
||||
|
||||
-10
@@ -130,7 +130,6 @@ public class ControllerAuditAspect {
|
||||
|
||||
String previousPrincipal = MDC.get("auditPrincipal");
|
||||
String previousOrigin = MDC.get("auditOrigin");
|
||||
String previousSource = MDC.get("auditSource");
|
||||
String previousIp = MDC.get("auditIp");
|
||||
|
||||
// EARLY CAPTURE: Capture from SecurityContext on request thread, store in MDC for async
|
||||
@@ -162,14 +161,6 @@ public class ControllerAuditAspect {
|
||||
return joinPoint.proceed();
|
||||
}
|
||||
|
||||
// Stamp the free-UI source only for non-@Audited controller traffic — an actual
|
||||
// tool / UI action. @Audited events (login, settings) return above without a source,
|
||||
// so they never count as an "active editor" or a free UI run. The finally block
|
||||
// restores auditSource, so a pooled thread can't leak a stale "WEB" into them.
|
||||
if (previousSource == null) {
|
||||
MDC.put("auditSource", auditService.captureCurrentSource());
|
||||
}
|
||||
|
||||
long start = System.currentTimeMillis();
|
||||
|
||||
// Use auditService to create the base audit data
|
||||
@@ -256,7 +247,6 @@ public class ControllerAuditAspect {
|
||||
} finally {
|
||||
restoreMdcValue("auditPrincipal", previousPrincipal);
|
||||
restoreMdcValue("auditOrigin", previousOrigin);
|
||||
restoreMdcValue("auditSource", previousSource);
|
||||
restoreMdcValue("auditIp", previousIp);
|
||||
}
|
||||
}
|
||||
|
||||
-15
@@ -1,15 +0,0 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/** Self-hosted default: admins see the whole-server audit log, everyone else is denied. */
|
||||
@Component
|
||||
public class DefaultPortalAuditScopeResolver implements PortalAuditScopeResolver {
|
||||
|
||||
@Override
|
||||
public PortalAuditScope resolve() {
|
||||
return PortalAuditScopeResolver.hasAdminAuthority()
|
||||
? PortalAuditScope.server()
|
||||
: PortalAuditScope.denied();
|
||||
}
|
||||
}
|
||||
-7
@@ -1,7 +0,0 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import java.time.Instant;
|
||||
|
||||
/** Immutable, cacheable projection of an {@code audit_events} row, shared across portal views. */
|
||||
public record PortalAuditEventRow(
|
||||
long id, String principal, String type, String data, Instant timestamp) {}
|
||||
-21
@@ -1,21 +0,0 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/** Resolved audit visibility: fullServer (admin), principals-scoped (team lead), or !allowed. */
|
||||
public record PortalAuditScope(
|
||||
boolean allowed, boolean fullServer, List<String> principals, String cacheKey) {
|
||||
|
||||
public static PortalAuditScope denied() {
|
||||
return new PortalAuditScope(false, false, List.of(), "denied");
|
||||
}
|
||||
|
||||
// Named server()/team() to avoid colliding with the record's fullServer() accessor.
|
||||
public static PortalAuditScope server() {
|
||||
return new PortalAuditScope(true, true, List.of(), "server");
|
||||
}
|
||||
|
||||
public static PortalAuditScope team(String cacheKey, List<String> principals) {
|
||||
return new PortalAuditScope(true, false, List.copyOf(principals), cacheKey);
|
||||
}
|
||||
}
|
||||
-18
@@ -1,18 +0,0 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import org.springframework.security.core.Authentication;
|
||||
import org.springframework.security.core.context.SecurityContextHolder;
|
||||
|
||||
/** Resolves which slice of the audit log the caller may see. */
|
||||
public interface PortalAuditScopeResolver {
|
||||
|
||||
PortalAuditScope resolve();
|
||||
|
||||
/** True when the current authentication carries {@code ROLE_ADMIN}. */
|
||||
static boolean hasAdminAuthority() {
|
||||
Authentication auth = SecurityContextHolder.getContext().getAuthentication();
|
||||
return auth != null
|
||||
&& auth.getAuthorities().stream()
|
||||
.anyMatch(a -> "ROLE_ADMIN".equals(a.getAuthority()));
|
||||
}
|
||||
}
|
||||
-136
@@ -1,136 +0,0 @@
|
||||
package stirling.software.proprietary.classification;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.DeleteMapping;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PutMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.server.ResponseStatusException;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Hidden;
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.model.ApplicationProperties;
|
||||
import stirling.software.common.service.UserServiceInterface;
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
import stirling.software.proprietary.classification.model.LabelsValidator;
|
||||
import stirling.software.proprietary.classification.store.ClassificationLabelStore;
|
||||
import stirling.software.proprietary.classification.store.TeamLabelsEntity;
|
||||
import stirling.software.proprietary.policy.config.PolicyManagementAuthority;
|
||||
|
||||
/**
|
||||
* Read/write the team's classification label set — the flat vocabulary the document classifier runs
|
||||
* against. Shared and team-scoped exactly like policies: every user reads their own team's labels,
|
||||
* and only a user who may edit policies (a team leader on SaaS, the global admin self-hosted; see
|
||||
* {@link PolicyManagementAuthority}) may change it — gated only when login is enabled, since
|
||||
* single-user deployments trust the local operator. A team with no stored labels reads as {@code
|
||||
* 204}; that team has no vocabulary, so its documents are not classified (there is no built-in
|
||||
* default on the backend or the engine — the label data lives only in the frontend).
|
||||
*/
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/classification/labels")
|
||||
@Hidden
|
||||
@RequiredArgsConstructor
|
||||
@Tag(name = "Classification", description = "Team-scoped document-classification labels")
|
||||
@ConditionalOnBooleanProperty(name = "policies.enabled")
|
||||
public class ClassificationLabelsController {
|
||||
|
||||
private final ClassificationLabelStore labelStore;
|
||||
private final PolicyManagementAuthority policyManagementAuthority;
|
||||
private final ApplicationProperties applicationProperties;
|
||||
private final UserServiceInterface userService;
|
||||
|
||||
@GetMapping
|
||||
@Operation(
|
||||
summary = "Get the team's classification labels",
|
||||
description =
|
||||
"Returns the caller's team label set, or 204 when the team has none (its"
|
||||
+ " documents are then not classified).")
|
||||
public ResponseEntity<ClassificationLabels> getTeamLabels() {
|
||||
return labelStore
|
||||
.findByTeam(currentTeamId())
|
||||
.map(ResponseEntity::ok)
|
||||
.orElseGet(() -> ResponseEntity.noContent().build());
|
||||
}
|
||||
|
||||
@PutMapping(consumes = MediaType.APPLICATION_JSON_VALUE)
|
||||
@Operation(
|
||||
summary = "Save the team's classification labels",
|
||||
description =
|
||||
"Validates and stores the label set for the caller's team, shared by everyone"
|
||||
+ " on the team. Requires the policy-editor role for the team.")
|
||||
public ResponseEntity<ClassificationLabels> saveTeamLabels(
|
||||
@RequestBody ClassificationLabels labels) {
|
||||
requireEditingAllowed();
|
||||
validate(labels);
|
||||
ClassificationLabels saved = labelStore.save(currentTeamId(), labels, currentUsername());
|
||||
return ResponseEntity.ok(saved);
|
||||
}
|
||||
|
||||
@DeleteMapping
|
||||
@Operation(
|
||||
summary = "Reset the team's classification labels",
|
||||
description =
|
||||
"Removes the team's stored label set; its documents are then not classified"
|
||||
+ " until labels are saved again. Requires the policy-editor role for the"
|
||||
+ " team.")
|
||||
public ResponseEntity<Void> resetTeamLabels() {
|
||||
requireEditingAllowed();
|
||||
labelStore.deleteByTeam(currentTeamId());
|
||||
return ResponseEntity.noContent().build();
|
||||
}
|
||||
|
||||
private static void validate(ClassificationLabels labels) {
|
||||
try {
|
||||
LabelsValidator.validate(labels);
|
||||
} catch (IllegalArgumentException e) {
|
||||
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Editing the team labels requires the editor role for the caller's team — the same gate
|
||||
* policies use (team leader on SaaS, global admin self-hosted). Single-user deployments (login
|
||||
* disabled) have no such role, so they trust the local operator.
|
||||
*/
|
||||
private void requireEditingAllowed() {
|
||||
if (!applicationProperties.getSecurity().isEnableLogin()) {
|
||||
return;
|
||||
}
|
||||
if (!policyManagementAuthority.canEditPolicies()) {
|
||||
throw new ResponseStatusException(
|
||||
HttpStatus.FORBIDDEN,
|
||||
"The team classification labels may only be changed by a team leader");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The caller's team key. With login disabled the single operator owns the {@link
|
||||
* TeamLabelsEntity#NO_TEAM} sentinel row; with login enabled a caller with no resolvable team
|
||||
* is an error rather than being dropped into the shared sentinel bucket (which would let
|
||||
* unteamed users read and overwrite each other's "team" labels).
|
||||
*/
|
||||
private Long currentTeamId() {
|
||||
Long teamId = policyManagementAuthority.currentUserTeamId();
|
||||
if (teamId != null) {
|
||||
return teamId;
|
||||
}
|
||||
if (!applicationProperties.getSecurity().isEnableLogin()) {
|
||||
return TeamLabelsEntity.NO_TEAM;
|
||||
}
|
||||
throw new ResponseStatusException(
|
||||
HttpStatus.UNAUTHORIZED, "Could not resolve the current user's team");
|
||||
}
|
||||
|
||||
private String currentUsername() {
|
||||
return userService == null ? null : userService.getCurrentUsername();
|
||||
}
|
||||
}
|
||||
-9
@@ -1,9 +0,0 @@
|
||||
package stirling.software.proprietary.classification.model;
|
||||
|
||||
/**
|
||||
* One entry in the classification vocabulary. {@code id} is the label's stable identity (a slug,
|
||||
* unique within a set): it is what the engine returns and what is stored on the document. {@code
|
||||
* name} is the human display text the classifier model reasons over. {@code icon} is an optional
|
||||
* presentational key (a Material Symbols name shown in the file sidebar); the engine never sees it.
|
||||
*/
|
||||
public record ClassificationLabel(String id, String name, String icon) {}
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
package stirling.software.proprietary.classification.model;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* A flat multi-label classification vocabulary — the set of labels a document may be assigned.
|
||||
* Stored per team (admin-edited, shared by everyone on the team); the classifier runs against these
|
||||
* label names. A team with no stored set has no vocabulary, so its documents are not classified —
|
||||
* neither the backend nor the engine holds a default of its own.
|
||||
*/
|
||||
public record ClassificationLabels(List<ClassificationLabel> labels) {
|
||||
|
||||
public ClassificationLabels {
|
||||
labels = labels == null ? List.of() : List.copyOf(labels);
|
||||
}
|
||||
}
|
||||
-71
@@ -1,71 +0,0 @@
|
||||
package stirling.software.proprietary.classification.model;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Locale;
|
||||
import java.util.Set;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Structural validation for a user- or admin-supplied label set, run before it is stored so a
|
||||
* malformed vocabulary can never reach the classifier. Mirrors the invariants the engine relies on:
|
||||
* non-blank ids and names, each unique within the set (ids exactly, names case-insensitively).
|
||||
*/
|
||||
public final class LabelsValidator {
|
||||
|
||||
private LabelsValidator() {}
|
||||
|
||||
// Generous upper bounds so a legitimate label set is never blocked, but a single team or user
|
||||
// can't store an unbounded blob that would bloat the row, balloon the classifier prompt, or
|
||||
// exhaust memory on deserialize.
|
||||
static final int MAX_LABELS = 500;
|
||||
static final int MAX_TEXT_LENGTH = 128;
|
||||
|
||||
// Icon is a Material Symbols key (lowercase, digits, hyphens). Enforce the SHAPE server-side —
|
||||
// the exact allowlist lives in the frontend — so a client bypassing the UI can't store
|
||||
// arbitrary
|
||||
// text that would render as garbage (or worse) in every teammate's sidebar.
|
||||
private static final Pattern ICON_KEY = Pattern.compile("^[a-z0-9-]+$");
|
||||
|
||||
/**
|
||||
* @throws IllegalArgumentException with a human-readable message when the label set is invalid.
|
||||
*/
|
||||
public static void validate(ClassificationLabels labels) {
|
||||
if (labels == null || labels.labels() == null) {
|
||||
throw new IllegalArgumentException("Labels are required");
|
||||
}
|
||||
if (labels.labels().size() > MAX_LABELS) {
|
||||
throw new IllegalArgumentException("Too many labels (max " + MAX_LABELS + ")");
|
||||
}
|
||||
Set<String> ids = new HashSet<>();
|
||||
Set<String> names = new HashSet<>();
|
||||
for (ClassificationLabel label : labels.labels()) {
|
||||
requireText(label.id(), "Label id");
|
||||
requireText(label.name(), "Label name");
|
||||
if (label.icon() != null && !label.icon().isEmpty()) {
|
||||
if (label.icon().length() > MAX_TEXT_LENGTH) {
|
||||
throw new IllegalArgumentException(
|
||||
"Label icon is too long (max " + MAX_TEXT_LENGTH + " characters)");
|
||||
}
|
||||
if (!ICON_KEY.matcher(label.icon()).matches()) {
|
||||
throw new IllegalArgumentException("Invalid label icon: " + label.icon());
|
||||
}
|
||||
}
|
||||
if (!ids.add(label.id().trim())) {
|
||||
throw new IllegalArgumentException("Duplicate label id: " + label.id());
|
||||
}
|
||||
if (!names.add(label.name().trim().toLowerCase(Locale.ROOT))) {
|
||||
throw new IllegalArgumentException("Duplicate label name: " + label.name());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void requireText(String value, String field) {
|
||||
if (value == null || value.isBlank()) {
|
||||
throw new IllegalArgumentException(field + " must not be blank");
|
||||
}
|
||||
if (value.trim().length() > MAX_TEXT_LENGTH) {
|
||||
throw new IllegalArgumentException(
|
||||
field + " is too long (max " + MAX_TEXT_LENGTH + " characters)");
|
||||
}
|
||||
}
|
||||
}
|
||||
-22
@@ -1,22 +0,0 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.util.Optional;
|
||||
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
|
||||
/**
|
||||
* Stores one {@link ClassificationLabels} set per team. A {@code null} teamId addresses the
|
||||
* unteamed set (login disabled / no resolvable team), mirroring how the policy store treats a null
|
||||
* team.
|
||||
*/
|
||||
public interface ClassificationLabelStore {
|
||||
|
||||
/** The team's stored labels, or empty when it has none (callers then skip classification). */
|
||||
Optional<ClassificationLabels> findByTeam(Long teamId);
|
||||
|
||||
/** Create or replace the team's labels. Returns the stored value. */
|
||||
ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy);
|
||||
|
||||
/** Remove the team's labels (reset to default). Returns whether a set existed. */
|
||||
boolean deleteByTeam(Long teamId);
|
||||
}
|
||||
-36
@@ -1,36 +0,0 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
|
||||
/**
|
||||
* In-memory {@link ClassificationLabelStore} for tests and any future no-database mode. {@link
|
||||
* JpaClassificationLabelStore} is the runtime bean.
|
||||
*/
|
||||
public class InProcessClassificationLabelStore implements ClassificationLabelStore {
|
||||
|
||||
private final Map<Long, ClassificationLabels> byTeam = new ConcurrentHashMap<>();
|
||||
|
||||
@Override
|
||||
public Optional<ClassificationLabels> findByTeam(Long teamId) {
|
||||
return Optional.ofNullable(byTeam.get(key(teamId)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy) {
|
||||
byTeam.put(key(teamId), labels);
|
||||
return labels;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean deleteByTeam(Long teamId) {
|
||||
return byTeam.remove(key(teamId)) != null;
|
||||
}
|
||||
|
||||
private static long key(Long teamId) {
|
||||
return teamId == null ? TeamLabelsEntity.NO_TEAM : teamId;
|
||||
}
|
||||
}
|
||||
-76
@@ -1,76 +0,0 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Optional;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
|
||||
import tools.jackson.core.JacksonException;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
|
||||
/**
|
||||
* Durable {@link ClassificationLabelStore} backed by JPA; the runtime store. Gated on {@code
|
||||
* policies.enabled} — stored labels only matter when the Classification policy can run — so it
|
||||
* shares the policy subsystem's on/off switch. Each label set is persisted as JSON via {@link
|
||||
* TeamLabelsEntity}.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@RequiredArgsConstructor
|
||||
@ConditionalOnBooleanProperty(name = "policies.enabled")
|
||||
public class JpaClassificationLabelStore implements ClassificationLabelStore {
|
||||
|
||||
private final TeamLabelsRepository teamRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
@Override
|
||||
public Optional<ClassificationLabels> findByTeam(Long teamId) {
|
||||
return teamRepository
|
||||
.findById(key(teamId))
|
||||
.flatMap(entity -> parse(entity.getLabelsJson(), "team " + teamId));
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy) {
|
||||
TeamLabelsEntity entity = new TeamLabelsEntity();
|
||||
entity.setTeamId(key(teamId));
|
||||
entity.setLabelsJson(objectMapper.writeValueAsString(labels));
|
||||
entity.setUpdatedAt(Instant.now());
|
||||
entity.setUpdatedBy(updatedBy);
|
||||
teamRepository.save(entity);
|
||||
return labels;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean deleteByTeam(Long teamId) {
|
||||
long id = key(teamId);
|
||||
if (!teamRepository.existsById(id)) {
|
||||
return false;
|
||||
}
|
||||
teamRepository.deleteById(id);
|
||||
return true;
|
||||
}
|
||||
|
||||
private Optional<ClassificationLabels> parse(String json, String owner) {
|
||||
try {
|
||||
return Optional.of(objectMapper.readValue(json, ClassificationLabels.class));
|
||||
} catch (JacksonException e) {
|
||||
// A stored label set that no longer parses (corruption / manual DB edit) must not break
|
||||
// classification: drop it so the caller treats the team as having no labels (and skips
|
||||
// classification) rather than surfacing a 500 on every upload.
|
||||
log.warn("Discarding unparseable stored labels for {}: {}", owner, e.getMessage());
|
||||
return Optional.empty();
|
||||
}
|
||||
}
|
||||
|
||||
/** Map the nullable team id onto the entity's non-null key (sentinel for the unteamed case). */
|
||||
private static long key(Long teamId) {
|
||||
return teamId == null ? TeamLabelsEntity.NO_TEAM : teamId;
|
||||
}
|
||||
}
|
||||
-47
@@ -1,47 +0,0 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.time.Instant;
|
||||
|
||||
import jakarta.persistence.Column;
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.Table;
|
||||
|
||||
import lombok.Getter;
|
||||
import lombok.NoArgsConstructor;
|
||||
import lombok.Setter;
|
||||
|
||||
/**
|
||||
* JPA row for a team's classification labels — one row per team. The label set lives as JSON in
|
||||
* {@code labelsJson} (authoritative on read). {@code teamId} is the natural key; the sentinel
|
||||
* {@link #NO_TEAM} stands in for the unteamed (login-disabled / self-hosted single-team) case,
|
||||
* since a primary key can't be null (policies store a nullable {@code team_id}, but this table is
|
||||
* keyed one-per-team). Kept decoupled from the security entities — {@code teamId} is a plain value,
|
||||
* not a foreign key — so classification can be enabled or disabled without touching them.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "classification_labels")
|
||||
@NoArgsConstructor
|
||||
@Getter
|
||||
@Setter
|
||||
public class TeamLabelsEntity implements Serializable {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
/** Sentinel key for the unteamed label set (login disabled / no resolvable team). */
|
||||
public static final long NO_TEAM = 0L;
|
||||
|
||||
@Id
|
||||
@Column(name = "team_id")
|
||||
private long teamId;
|
||||
|
||||
@Column(name = "labels_json", columnDefinition = "text")
|
||||
private String labelsJson;
|
||||
|
||||
@Column(name = "updated_at")
|
||||
private Instant updatedAt;
|
||||
|
||||
@Column(name = "updated_by")
|
||||
private String updatedBy;
|
||||
}
|
||||
-7
@@ -1,7 +0,0 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
@Repository
|
||||
public interface TeamLabelsRepository extends JpaRepository<TeamLabelsEntity, Long> {}
|
||||
-3
@@ -60,8 +60,6 @@ public class CustomAuditEventRepository implements AuditEventRepository {
|
||||
clean.put("requestId", rid);
|
||||
}
|
||||
|
||||
String source = MDC.get("auditSource");
|
||||
|
||||
String auditEventData = mapper.writeValueAsString(clean);
|
||||
log.debug("AuditEvent data (JSON): {}", auditEventData);
|
||||
|
||||
@@ -69,7 +67,6 @@ public class CustomAuditEventRepository implements AuditEventRepository {
|
||||
PersistentAuditEvent.builder()
|
||||
.principal(safePrincipal(ev.getPrincipal()))
|
||||
.type(ev.getType())
|
||||
.source(source)
|
||||
.data(auditEventData)
|
||||
.timestamp(ev.getTimestamp())
|
||||
.build();
|
||||
|
||||
-218
@@ -1,218 +0,0 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
|
||||
import io.github.pixee.security.Filenames;
|
||||
import io.swagger.v3.oas.annotations.Hidden;
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.service.CustomPDFDocumentFactory;
|
||||
import stirling.software.common.service.PdfMetadataService;
|
||||
import stirling.software.common.service.UserServiceInterface;
|
||||
import stirling.software.common.util.TempFileManager;
|
||||
import stirling.software.common.util.WebResponseUtils;
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabel;
|
||||
import stirling.software.proprietary.classification.store.ClassificationLabelStore;
|
||||
import stirling.software.proprietary.model.api.ai.AiPageText;
|
||||
import stirling.software.proprietary.policy.config.PolicyManagementAuthority;
|
||||
import stirling.software.proprietary.service.AiEngineClient;
|
||||
import stirling.software.proprietary.service.PdfContentExtractor;
|
||||
|
||||
import tools.jackson.databind.JsonNode;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
import tools.jackson.databind.node.ObjectNode;
|
||||
|
||||
/**
|
||||
* Dispatchable tool that classifies a PDF and writes the result into its metadata.
|
||||
*
|
||||
* <p>Runs as a Classification-policy pipeline step: it reads a bounded page window, asks the AI
|
||||
* engine to classify the document against the caller's team label set, and stores the engine's JSON
|
||||
* answer — minus the transport-only {@code outcome} field — in the custom Info-dictionary key
|
||||
* {@link PdfMetadataService#CLASSIFICATION_KEY}. Returns the labelled PDF. Not intended for direct
|
||||
* client use.
|
||||
*/
|
||||
@Slf4j
|
||||
@Hidden
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/ai/tools")
|
||||
@Tag(name = "AI Tools", description = "Dispatchable AI-backed tools.")
|
||||
public class ClassifyLabelController {
|
||||
|
||||
/** Pages read from each end of the document — mirrors the engine's window. */
|
||||
private static final int WINDOW_PAGES = 2;
|
||||
|
||||
private static final String CLASSIFY_ENDPOINT = "/api/v1/documents/classify";
|
||||
|
||||
private final CustomPDFDocumentFactory pdfDocumentFactory;
|
||||
private final TempFileManager tempFileManager;
|
||||
private final PdfContentExtractor pdfContentExtractor;
|
||||
private final PdfMetadataService pdfMetadataService;
|
||||
private final AiEngineClient aiEngineClient;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final UserServiceInterface userService;
|
||||
|
||||
/**
|
||||
* Present only when the policy subsystem is enabled ({@code policies.enabled}); the store and
|
||||
* team authority are gated on it. Null otherwise, in which case there are no team labels to
|
||||
* classify against and the document is passed through unlabelled.
|
||||
*/
|
||||
private final ClassificationLabelStore labelStore;
|
||||
|
||||
private final PolicyManagementAuthority policyManagementAuthority;
|
||||
|
||||
public ClassifyLabelController(
|
||||
CustomPDFDocumentFactory pdfDocumentFactory,
|
||||
TempFileManager tempFileManager,
|
||||
PdfContentExtractor pdfContentExtractor,
|
||||
PdfMetadataService pdfMetadataService,
|
||||
AiEngineClient aiEngineClient,
|
||||
ObjectMapper objectMapper,
|
||||
@Autowired(required = false) UserServiceInterface userService,
|
||||
@Autowired(required = false) ClassificationLabelStore labelStore,
|
||||
@Autowired(required = false) PolicyManagementAuthority policyManagementAuthority) {
|
||||
this.pdfDocumentFactory = pdfDocumentFactory;
|
||||
this.tempFileManager = tempFileManager;
|
||||
this.pdfContentExtractor = pdfContentExtractor;
|
||||
this.pdfMetadataService = pdfMetadataService;
|
||||
this.aiEngineClient = aiEngineClient;
|
||||
this.objectMapper = objectMapper;
|
||||
this.userService = userService;
|
||||
this.labelStore = labelStore;
|
||||
this.policyManagementAuthority = policyManagementAuthority;
|
||||
}
|
||||
|
||||
@PostMapping(value = "/classify-and-label", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
|
||||
@Operation(
|
||||
summary = "Classify a PDF and label its metadata",
|
||||
description =
|
||||
"Reads the first two and last two pages, classifies the document via the AI"
|
||||
+ " engine, and stores the result in the StirlingPDFClassification"
|
||||
+ " metadata field. Dispatched by the Classification policy; not"
|
||||
+ " intended for direct client use.")
|
||||
public ResponseEntity<Resource> classifyAndLabel(
|
||||
@RequestParam("fileInput") MultipartFile fileInput) throws IOException {
|
||||
try (PDDocument document = pdfDocumentFactory.load(fileInput, true)) {
|
||||
String fileName = safeFileName(fileInput.getOriginalFilename());
|
||||
|
||||
List<EngineLabel> allowed = resolveAllowedLabels();
|
||||
if (allowed.isEmpty()) {
|
||||
// No vocabulary to classify against (the team stored no labels): pass the file
|
||||
// through unlabelled rather than ask the engine to classify against nothing.
|
||||
log.debug("[classify-and-label] {} has no team labels; skipping", fileName);
|
||||
return WebResponseUtils.pdfDocToWebResponse(document, fileName, tempFileManager);
|
||||
}
|
||||
|
||||
List<AiPageText> pages = extractWindow(document);
|
||||
String requestBody =
|
||||
objectMapper.writeValueAsString(
|
||||
new ClassifyEngineRequest(fileName, pages, allowed));
|
||||
|
||||
String userId = userService != null ? userService.getCurrentUsername() : null;
|
||||
String responseJson = aiEngineClient.post(CLASSIFY_ENDPOINT, requestBody, userId);
|
||||
|
||||
pdfMetadataService.setClassificationMetadata(document, toMetadataValue(responseJson));
|
||||
log.debug("[classify-and-label] labelled {} ({} window pages)", fileName, pages.size());
|
||||
|
||||
return WebResponseUtils.pdfDocToWebResponse(document, fileName, tempFileManager);
|
||||
}
|
||||
}
|
||||
|
||||
private List<AiPageText> extractWindow(PDDocument document) throws IOException {
|
||||
List<AiPageText> pages = new ArrayList<>();
|
||||
for (int pageNumber : windowPageNumbers(document.getNumberOfPages(), WINDOW_PAGES)) {
|
||||
String text = pdfContentExtractor.extractPageTextRaw(document, pageNumber);
|
||||
if (text != null && !text.isBlank()) {
|
||||
pages.add(new AiPageText(pageNumber, text));
|
||||
}
|
||||
}
|
||||
return pages;
|
||||
}
|
||||
|
||||
/** First and last {@code window} page numbers (1-based), de-duplicated and in order. */
|
||||
static List<Integer> windowPageNumbers(int pageCount, int window) {
|
||||
Set<Integer> numbers = new LinkedHashSet<>();
|
||||
for (int page = 1; page <= Math.min(window, pageCount); page++) {
|
||||
numbers.add(page);
|
||||
}
|
||||
for (int page = Math.max(1, pageCount - window + 1); page <= pageCount; page++) {
|
||||
numbers.add(page);
|
||||
}
|
||||
return new ArrayList<>(numbers);
|
||||
}
|
||||
|
||||
/** Drop the transport-only {@code outcome} discriminator; keep the rest verbatim. */
|
||||
private String toMetadataValue(String engineResponseJson) {
|
||||
JsonNode node = objectMapper.readTree(engineResponseJson);
|
||||
if (node instanceof ObjectNode object) {
|
||||
object.remove("outcome");
|
||||
}
|
||||
return objectMapper.writeValueAsString(node);
|
||||
}
|
||||
|
||||
private static String safeFileName(String originalFilename) {
|
||||
String name = Filenames.toSimpleFileName(originalFilename);
|
||||
return (name == null || name.isBlank()) ? "classified.pdf" : name;
|
||||
}
|
||||
|
||||
/**
|
||||
* The allowed labels for the caller's team as {@code {id, name}} pairs, de-duplicated by id.
|
||||
* The engine shows the model the names and returns the ids (icons are presentational and never
|
||||
* sent). Returns an empty list — the caller then skips classification — when the policy
|
||||
* subsystem is disabled (no store) or the team has no stored labels. The engine holds no
|
||||
* default vocabulary of its own, so a team's stored labels are the only source.
|
||||
*/
|
||||
private List<EngineLabel> resolveAllowedLabels() {
|
||||
if (labelStore == null) {
|
||||
return List.of();
|
||||
}
|
||||
Long teamId =
|
||||
policyManagementAuthority == null
|
||||
? null
|
||||
: policyManagementAuthority.currentUserTeamId();
|
||||
|
||||
Map<String, EngineLabel> byId = new LinkedHashMap<>();
|
||||
labelStore.findByTeam(teamId).ifPresent(labels -> collectLabels(labels.labels(), byId));
|
||||
|
||||
return List.copyOf(byId.values());
|
||||
}
|
||||
|
||||
private static void collectLabels(
|
||||
List<ClassificationLabel> labels, Map<String, EngineLabel> into) {
|
||||
for (ClassificationLabel label : labels) {
|
||||
if (label.id() == null
|
||||
|| label.id().isBlank()
|
||||
|| label.name() == null
|
||||
|| label.name().isBlank()) {
|
||||
continue;
|
||||
}
|
||||
into.putIfAbsent(label.id(), new EngineLabel(label.id(), label.name()));
|
||||
}
|
||||
}
|
||||
|
||||
/** One allowed label sent to the engine: stable id + the name the model reasons over. */
|
||||
private record EngineLabel(String id, String name) {}
|
||||
|
||||
/** Request body for the engine's {@code /api/v1/documents/classify} endpoint. */
|
||||
private record ClassifyEngineRequest(
|
||||
String fileName, List<AiPageText> pages, List<EngineLabel> labels) {}
|
||||
}
|
||||
-71
@@ -1,71 +0,0 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.security.access.prepost.PreAuthorize;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.model.enumeration.Role;
|
||||
import stirling.software.proprietary.audit.AuditLevel;
|
||||
import stirling.software.proprietary.config.AuditConfigurationProperties;
|
||||
import stirling.software.proprietary.model.api.usage.FleetUsageStats;
|
||||
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
import stirling.software.proprietary.security.database.repository.UserRepository;
|
||||
|
||||
/**
|
||||
* Admin endpoint exposing free-editor fleet usage for the portal Usage card. Audit-derived figures
|
||||
* (active editors, PDFs processed) are null (rendered as "N/A") rather than a misleading 0 whenever
|
||||
* the data can't exist: the events they count (PDF_PROCESS, FILE_OPERATION, HTTP_REQUEST) are all
|
||||
* STANDARD level, so a gate on {@code isEnabled()} alone would still return 0 at level=OFF/BASIC —
|
||||
* we gate on {@code isLevelEnabled(STANDARD)} instead.
|
||||
*
|
||||
* <p>Known limitation: on a login-disabled self-hosted instance every request is anonymous, so its
|
||||
* audit origin is SYSTEM (not WEB) and it is excluded from these WEB-only counts — active/PDFs then
|
||||
* read 0 despite real usage. Historical audit rows written before the {@code source} column existed
|
||||
* carry {@code source=null}, so the cumulative "PDFs edited" figure effectively starts at deploy.
|
||||
*/
|
||||
@Slf4j
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/usage")
|
||||
@PreAuthorize("hasRole('ADMIN')")
|
||||
@RequiredArgsConstructor
|
||||
@EnterpriseEndpoint
|
||||
public class FleetUsageController {
|
||||
|
||||
private final PersistentAuditEventRepository auditRepository;
|
||||
private final UserRepository userRepository;
|
||||
private final AuditConfigurationProperties auditConfig;
|
||||
|
||||
@GetMapping("/fleet-stats")
|
||||
public FleetUsageStats fleetStats() {
|
||||
// Exclude the reserved INTERNAL_API_USER row that InitialSecuritySetup creates on every
|
||||
// install, so a fresh single-admin instance reads 1 editor, not 2.
|
||||
Long deployed = userRepository.countByUsernameNot(Role.INTERNAL_API_USER.getRoleId());
|
||||
// STANDARD is the level at which the counted events are recorded; below it the data
|
||||
// can't exist, so report N/A instead of a 0 that would misrepresent an empty table.
|
||||
boolean auditOn = auditConfig.isLevelEnabled(AuditLevel.STANDARD);
|
||||
Instant since = Instant.now().minus(30, ChronoUnit.DAYS);
|
||||
Long active =
|
||||
auditOn
|
||||
? auditRepository.countDistinctPrincipalsBySourceExcludingTypeAfter(
|
||||
"WEB", "UI_DATA", since)
|
||||
: null;
|
||||
Long pdfs =
|
||||
auditOn
|
||||
? auditRepository.countByTypeInAndSourceAndTimestampAfter(
|
||||
List.of("PDF_PROCESS", "FILE_OPERATION"), "WEB", Instant.EPOCH)
|
||||
: null;
|
||||
if (active != null && deployed != null && active > deployed) {
|
||||
active = deployed; // active editors are a subset of those deployed
|
||||
}
|
||||
return new FleetUsageStats(deployed, active, pdfs);
|
||||
}
|
||||
}
|
||||
-46
@@ -1,46 +0,0 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.annotations.api.ProprietaryUiDataApi;
|
||||
import stirling.software.proprietary.audit.PortalAuditScope;
|
||||
import stirling.software.proprietary.audit.PortalAuditScopeResolver;
|
||||
import stirling.software.proprietary.model.api.documents.PortalDocumentsResponseDto;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
import stirling.software.proprietary.service.PortalDocumentsService;
|
||||
|
||||
/** Serves the portal Documents review queue, derived from real audit data and scoped per caller. */
|
||||
@ProprietaryUiDataApi
|
||||
@RequiredArgsConstructor
|
||||
@EnterpriseEndpoint
|
||||
public class PortalDocumentsController {
|
||||
|
||||
private final PortalDocumentsService portalDocumentsService;
|
||||
private final PortalAuditScopeResolver auditScopeResolver;
|
||||
|
||||
// tier accepted for mock-seam symmetry; ignored (queue isn't tier-scoped).
|
||||
@GetMapping("/documents")
|
||||
@Operation(
|
||||
summary = "Documents review queue",
|
||||
description = "Files processed through the org, derived from the audit trail.")
|
||||
public ResponseEntity<PortalDocumentsResponseDto> getDocuments(
|
||||
@RequestParam(value = "tier", required = false) String tier) {
|
||||
PortalAuditScope scope = auditScopeResolver.resolve();
|
||||
if (!scope.allowed()) {
|
||||
return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
|
||||
}
|
||||
PortalDocumentsResponseDto body =
|
||||
scope.fullServer()
|
||||
? portalDocumentsService.serverDocuments()
|
||||
: portalDocumentsService.scopedDocuments(
|
||||
scope.cacheKey(), scope.principals());
|
||||
return ResponseEntity.ok(body);
|
||||
}
|
||||
}
|
||||
-47
@@ -1,47 +0,0 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.annotations.api.ProprietaryUiDataApi;
|
||||
import stirling.software.proprietary.audit.PortalAuditScope;
|
||||
import stirling.software.proprietary.audit.PortalAuditScopeResolver;
|
||||
import stirling.software.proprietary.model.api.audit.InfraAuditLogResponse;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
import stirling.software.proprietary.service.PortalInfraAuditService;
|
||||
|
||||
/** Serves the Infrastructure → Audit tab from real audit data, scoped and cached per caller. */
|
||||
@ProprietaryUiDataApi
|
||||
@RequiredArgsConstructor
|
||||
@EnterpriseEndpoint
|
||||
public class PortalInfraAuditController {
|
||||
|
||||
private final PortalInfraAuditService portalInfraAuditService;
|
||||
private final PortalAuditScopeResolver auditScopeResolver;
|
||||
|
||||
// tier accepted for endpoint symmetry; ignored (audit log isn't tier-scoped).
|
||||
@GetMapping("/infrastructure/audit-log")
|
||||
@Operation(
|
||||
summary = "Infrastructure audit log",
|
||||
description = "Recent audit events shaped for the portal Infrastructure → Audit tab.")
|
||||
public ResponseEntity<InfraAuditLogResponse> getInfrastructureAuditLog(
|
||||
@RequestParam(value = "tier", required = false) String tier) {
|
||||
PortalAuditScope scope = auditScopeResolver.resolve();
|
||||
if (!scope.allowed()) {
|
||||
// Return 403 (not throw) so the tab shows its access message, not a generic 500.
|
||||
return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
|
||||
}
|
||||
InfraAuditLogResponse body =
|
||||
scope.fullServer()
|
||||
? portalInfraAuditService.serverAuditLog()
|
||||
: portalInfraAuditService.scopedAuditLog(
|
||||
scope.cacheKey(), scope.principals());
|
||||
return ResponseEntity.ok(body);
|
||||
}
|
||||
}
|
||||
+3
-50
@@ -5,7 +5,6 @@ import static stirling.software.common.util.ProviderUtils.validateProvider;
|
||||
import java.time.Instant;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
@@ -29,16 +28,13 @@ import stirling.software.common.model.ApplicationProperties.Security.OAUTH2.Clie
|
||||
import stirling.software.common.model.ApplicationProperties.Security.SAML2;
|
||||
import stirling.software.common.model.FileInfo;
|
||||
import stirling.software.common.model.enumeration.Role;
|
||||
import stirling.software.common.model.enumeration.TeamRole;
|
||||
import stirling.software.common.model.oauth2.GitHubProvider;
|
||||
import stirling.software.common.model.oauth2.GoogleProvider;
|
||||
import stirling.software.common.model.oauth2.KeycloakProvider;
|
||||
import stirling.software.proprietary.access.service.ResourceAccessService;
|
||||
import stirling.software.proprietary.audit.AuditEventType;
|
||||
import stirling.software.proprietary.audit.AuditLevel;
|
||||
import stirling.software.proprietary.config.AuditConfigurationProperties;
|
||||
import stirling.software.proprietary.model.Team;
|
||||
import stirling.software.proprietary.model.TeamMembership;
|
||||
import stirling.software.proprietary.model.dto.TeamWithUserCountDTO;
|
||||
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
@@ -48,7 +44,6 @@ import stirling.software.proprietary.security.model.Authority;
|
||||
import stirling.software.proprietary.security.model.SessionEntity;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.model.dto.AdminUserSummary;
|
||||
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
|
||||
import stirling.software.proprietary.security.repository.TeamRepository;
|
||||
import stirling.software.proprietary.security.saml2.CustomSaml2AuthenticatedPrincipal;
|
||||
import stirling.software.proprietary.security.service.DatabaseServiceInterface;
|
||||
@@ -70,7 +65,6 @@ public class ProprietaryUIDataController {
|
||||
private final SessionPersistentRegistry sessionPersistentRegistry;
|
||||
private final UserRepository userRepository;
|
||||
private final TeamRepository teamRepository;
|
||||
private final TeamMembershipRepository teamMembershipRepository;
|
||||
private final SessionRepository sessionRepository;
|
||||
private final DatabaseServiceInterface databaseService;
|
||||
private final boolean runningEE;
|
||||
@@ -79,7 +73,6 @@ public class ProprietaryUIDataController {
|
||||
private final PersistentAuditEventRepository auditRepository;
|
||||
private final MfaService mfaService;
|
||||
private final LoginAttemptService loginAttemptService;
|
||||
private final ResourceAccessService resourceAccessService;
|
||||
|
||||
public ProprietaryUIDataController(
|
||||
ApplicationProperties applicationProperties,
|
||||
@@ -87,7 +80,6 @@ public class ProprietaryUIDataController {
|
||||
SessionPersistentRegistry sessionPersistentRegistry,
|
||||
UserRepository userRepository,
|
||||
TeamRepository teamRepository,
|
||||
TeamMembershipRepository teamMembershipRepository,
|
||||
SessionRepository sessionRepository,
|
||||
DatabaseServiceInterface databaseService,
|
||||
ObjectMapper objectMapper,
|
||||
@@ -95,14 +87,12 @@ public class ProprietaryUIDataController {
|
||||
UserLicenseSettingsService licenseSettingsService,
|
||||
PersistentAuditEventRepository auditRepository,
|
||||
MfaService mfaService,
|
||||
LoginAttemptService loginAttemptService,
|
||||
ResourceAccessService resourceAccessService) {
|
||||
LoginAttemptService loginAttemptService) {
|
||||
this.applicationProperties = applicationProperties;
|
||||
this.auditConfig = auditConfig;
|
||||
this.sessionPersistentRegistry = sessionPersistentRegistry;
|
||||
this.userRepository = userRepository;
|
||||
this.teamRepository = teamRepository;
|
||||
this.teamMembershipRepository = teamMembershipRepository;
|
||||
this.sessionRepository = sessionRepository;
|
||||
this.databaseService = databaseService;
|
||||
this.objectMapper = objectMapper;
|
||||
@@ -111,7 +101,6 @@ public class ProprietaryUIDataController {
|
||||
this.auditRepository = auditRepository;
|
||||
this.mfaService = mfaService;
|
||||
this.loginAttemptService = loginAttemptService;
|
||||
this.resourceAccessService = resourceAccessService;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -381,11 +370,8 @@ public class ProprietaryUIDataController {
|
||||
boolean premiumEnabled = applicationProperties.getPremium().isEnabled();
|
||||
|
||||
// Convert User entities to AdminUserSummary DTOs to exclude sensitive fields
|
||||
Set<Long> leaderUserIds = leaderUserIds();
|
||||
List<AdminUserSummary> userSummaries =
|
||||
sortedUsers.stream()
|
||||
.map(user -> convertUserToSummary(user, leaderUserIds))
|
||||
.toList();
|
||||
sortedUsers.stream().map(this::convertUserToSummary).toList();
|
||||
|
||||
AdminSettingsData data = new AdminSettingsData();
|
||||
data.setUsers(userSummaries);
|
||||
@@ -404,10 +390,6 @@ public class ProprietaryUIDataController {
|
||||
data.setLicenseMaxUsers(licenseMaxUsers);
|
||||
data.setPremiumEnabled(premiumEnabled);
|
||||
data.setMailEnabled(applicationProperties.getMail().isEnabled());
|
||||
// Email invites need the invites toggle AND SMTP on; matches the inviteUsers precondition.
|
||||
data.setEmailInvitesEnabled(
|
||||
applicationProperties.getMail().isEnableInvites()
|
||||
&& applicationProperties.getMail().isEnabled());
|
||||
data.setUserSettings(userSettings);
|
||||
data.setLockedUsers(loginAttemptService.getAllBlockedUsers());
|
||||
|
||||
@@ -486,18 +468,9 @@ public class ProprietaryUIDataController {
|
||||
teamLastRequest.put(teamId, lastActivity);
|
||||
}
|
||||
|
||||
Map<Long, List<String>> teamOwners = new HashMap<>();
|
||||
for (TeamMembership row :
|
||||
teamMembershipRepository.findByRoleFetchingUserAndTeam(TeamRole.LEADER)) {
|
||||
teamOwners
|
||||
.computeIfAbsent(row.getTeam().getId(), id -> new ArrayList<>())
|
||||
.add(row.getUser().getUsername());
|
||||
}
|
||||
|
||||
TeamsData data = new TeamsData();
|
||||
data.setTeamsWithCounts(teamsWithCounts);
|
||||
data.setTeamLastRequest(teamLastRequest);
|
||||
data.setTeamOwners(teamOwners);
|
||||
|
||||
return ResponseEntity.ok(data);
|
||||
}
|
||||
@@ -537,17 +510,11 @@ public class ProprietaryUIDataController {
|
||||
userLastRequest.put(username, lastRequest);
|
||||
}
|
||||
|
||||
Set<Long> ownerUserIds =
|
||||
teamMembershipRepository.findByTeamIdAndRole(id, TeamRole.LEADER).stream()
|
||||
.map(row -> row.getUser().getId())
|
||||
.collect(Collectors.toSet());
|
||||
|
||||
TeamDetailsData data = new TeamDetailsData();
|
||||
data.setTeam(team);
|
||||
data.setTeamUsers(teamUsers);
|
||||
data.setAvailableUsers(availableUsers);
|
||||
data.setUserLastRequest(userLastRequest);
|
||||
data.setOwnerUserIds(ownerUserIds);
|
||||
|
||||
return ResponseEntity.ok(data);
|
||||
}
|
||||
@@ -568,24 +535,13 @@ public class ProprietaryUIDataController {
|
||||
return ResponseEntity.ok(data);
|
||||
}
|
||||
|
||||
/** User ids holding a LEADER membership on any team. */
|
||||
private Set<Long> leaderUserIds() {
|
||||
return teamMembershipRepository.findByRoleFetchingUserAndTeam(TeamRole.LEADER).stream()
|
||||
.map(row -> row.getUser().getId())
|
||||
.collect(Collectors.toSet());
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert User entity to AdminUserSummary DTO, excluding sensitive fields like password and
|
||||
* apiKey.
|
||||
*/
|
||||
private AdminUserSummary convertUserToSummary(User user, Set<Long> leaderUserIds) {
|
||||
private AdminUserSummary convertUserToSummary(User user) {
|
||||
AdminUserSummary summary = new AdminUserSummary();
|
||||
summary.setId(user.getId());
|
||||
summary.setTeamLead(leaderUserIds.contains(user.getId()));
|
||||
// Authoritative portal access, same call /me uses, so the roster honors the configured
|
||||
// policy instead of the frontend guessing from role/team-leadership.
|
||||
summary.setPortalAccess(resourceAccessService.canAccessPortal(user));
|
||||
summary.setUsername(user.getUsername());
|
||||
summary.setEmail(user.getUsername()); // Use username as email for consistency
|
||||
summary.setRoleName(user.getRoleName());
|
||||
@@ -653,7 +609,6 @@ public class ProprietaryUIDataController {
|
||||
private int licenseMaxUsers;
|
||||
private boolean premiumEnabled;
|
||||
private boolean mailEnabled;
|
||||
private boolean emailInvitesEnabled;
|
||||
private Map<String, Map<String, String>> userSettings;
|
||||
private List<String> lockedUsers;
|
||||
}
|
||||
@@ -674,7 +629,6 @@ public class ProprietaryUIDataController {
|
||||
public static class TeamsData {
|
||||
private List<TeamWithUserCountDTO> teamsWithCounts;
|
||||
private Map<Long, Date> teamLastRequest;
|
||||
private Map<Long, List<String>> teamOwners;
|
||||
}
|
||||
|
||||
@Data
|
||||
@@ -683,7 +637,6 @@ public class ProprietaryUIDataController {
|
||||
private List<User> teamUsers;
|
||||
private List<User> availableUsers;
|
||||
private Map<String, Date> userLastRequest;
|
||||
private Set<Long> ownerUserIds;
|
||||
}
|
||||
|
||||
@Data
|
||||
|
||||
+1
-3
@@ -29,9 +29,7 @@ import stirling.software.proprietary.security.model.User;
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/integrations")
|
||||
@RequiredArgsConstructor
|
||||
// Portal-exclusive: server-side portal-access boundary, not just isAuthenticated. Per-config
|
||||
// ownership is still enforced in the service layer.
|
||||
@PreAuthorize("@resourceAccess.canUsePortal()")
|
||||
@PreAuthorize("isAuthenticated()")
|
||||
@Tag(name = "Integrations", description = "Manage S3/MCP/API integration configurations")
|
||||
public class IntegrationConfigController {
|
||||
|
||||
|
||||
+1
-24
@@ -1,16 +1,12 @@
|
||||
package stirling.software.proprietary.integration.crypto;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.attribute.PosixFilePermission;
|
||||
import java.nio.file.attribute.PosixFilePermissions;
|
||||
import java.security.GeneralSecurityException;
|
||||
import java.security.SecureRandom;
|
||||
import java.util.Arrays;
|
||||
import java.util.Base64;
|
||||
import java.util.EnumSet;
|
||||
|
||||
import javax.crypto.Cipher;
|
||||
import javax.crypto.KeyGenerator;
|
||||
@@ -78,7 +74,7 @@ public class CredentialEncryption {
|
||||
generator.init(256);
|
||||
SecretKey generated = generator.generateKey();
|
||||
Files.createDirectories(path.getParent());
|
||||
writeOwnerOnly(path, Base64.getEncoder().encodeToString(generated.getEncoded()));
|
||||
Files.writeString(path, Base64.getEncoder().encodeToString(generated.getEncoded()));
|
||||
log.warn(
|
||||
"Generated a new credential encryption key at {}. Back this file up: losing it"
|
||||
+ " makes stored integration secrets unrecoverable.",
|
||||
@@ -89,25 +85,6 @@ public class CredentialEncryption {
|
||||
}
|
||||
}
|
||||
|
||||
// The master key decrypts every stored integration secret, so create it 0600
|
||||
// (owner-only) atomically. On non-POSIX filesystems (Windows) the config-dir
|
||||
// ACL is the protection; we still create the file, just without POSIX perms.
|
||||
private static void writeOwnerOnly(Path path, String content) throws IOException {
|
||||
EnumSet<PosixFilePermission> ownerOnly =
|
||||
EnumSet.of(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_WRITE);
|
||||
try {
|
||||
Files.createFile(path, PosixFilePermissions.asFileAttribute(ownerOnly));
|
||||
} catch (UnsupportedOperationException e) {
|
||||
Files.createFile(path);
|
||||
}
|
||||
Files.writeString(path, content);
|
||||
try {
|
||||
Files.setPosixFilePermissions(path, ownerOnly);
|
||||
} catch (UnsupportedOperationException ignored) {
|
||||
// Non-POSIX filesystem: nothing to tighten here.
|
||||
}
|
||||
}
|
||||
|
||||
public static String encrypt(String plaintext) {
|
||||
if (plaintext == null) {
|
||||
return null;
|
||||
|
||||
-9
@@ -18,13 +18,4 @@ public interface IntegrationConfigRepository extends JpaRepository<IntegrationCo
|
||||
List<IntegrationConfig> findByOwnerTeam(Team ownerTeam);
|
||||
|
||||
List<IntegrationConfig> findByScope(OwnerScope scope);
|
||||
|
||||
// Nested path: OwnedResource has a getOwnerTeamId() convenience getter but no such persistent
|
||||
// attribute, so the plain "...OwnerTeamId" derivation resolves to a phantom property and throws
|
||||
// UnknownPathException. The underscore forces the real ownerTeam.id association path.
|
||||
boolean existsByOwnerTeam_Id(Long teamId);
|
||||
|
||||
void deleteByOwnerUser(User ownerUser);
|
||||
|
||||
void deleteByOwnerTeam_Id(Long teamId);
|
||||
}
|
||||
|
||||
-11
@@ -16,7 +16,6 @@ import lombok.extern.slf4j.Slf4j;
|
||||
import stirling.software.proprietary.access.model.DefaultAccessPolicy;
|
||||
import stirling.software.proprietary.access.model.OwnerScope;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
import stirling.software.proprietary.access.repository.ResourceGrantRepository;
|
||||
import stirling.software.proprietary.access.service.OwnershipService;
|
||||
import stirling.software.proprietary.access.service.SecretMasker;
|
||||
import stirling.software.proprietary.integration.dto.IntegrationConfigRequest;
|
||||
@@ -42,7 +41,6 @@ public class IntegrationConfigService {
|
||||
private final IntegrationConfigRepository repository;
|
||||
private final OwnershipService ownership;
|
||||
private final SecretMasker secretMasker;
|
||||
private final ResourceGrantRepository grantRepository;
|
||||
|
||||
// ---- commands ----
|
||||
|
||||
@@ -51,13 +49,6 @@ public class IntegrationConfigService {
|
||||
OwnerScope scope = request.scope() == null ? OwnerScope.USER : request.scope();
|
||||
IntegrationConfig cfg = new IntegrationConfig();
|
||||
cfg.setIntegrationType(require(request.integrationType(), "integrationType"));
|
||||
// S3 is infrastructure, not self-serve: no personal S3 for regular users. TEAM/SERVER
|
||||
// scopes are already restricted to admins/team owners by assignOwnership.
|
||||
if (cfg.getIntegrationType() == IntegrationType.S3
|
||||
&& scope == OwnerScope.USER
|
||||
&& !ownership.isAdmin(currentUser)) {
|
||||
throw forbidden("S3 connections can only be created by administrators or team owners");
|
||||
}
|
||||
cfg.setName(require(request.name(), "name"));
|
||||
cfg.setEnabled(request.enabled() == null || request.enabled());
|
||||
cfg.setLocked(request.locked() != null && request.locked());
|
||||
@@ -113,8 +104,6 @@ public class IntegrationConfigService {
|
||||
if (!ownership.canManage(TYPE, cfg, currentUser)) {
|
||||
throw forbidden("You cannot manage this integration");
|
||||
}
|
||||
// Drop grants sharing this config so they do not dangle as dead rows.
|
||||
grantRepository.deleteByResourceTypeAndResourceId(TYPE, String.valueOf(cfg.getId()));
|
||||
repository.delete(cfg);
|
||||
}
|
||||
|
||||
|
||||
-7
@@ -21,11 +21,4 @@ public class AiWorkflowResultFile {
|
||||
|
||||
@Schema(description = "MIME type of the file", example = "application/pdf")
|
||||
private String contentType;
|
||||
|
||||
@Schema(
|
||||
description =
|
||||
"Index into the request's fileInputs that this output was derived from, or null"
|
||||
+ " when it has no single source (e.g. a merge, or a generated file)."
|
||||
+ " Lets the client replace that input in place as a new version.")
|
||||
private Integer sourceIndex;
|
||||
}
|
||||
|
||||
-44
@@ -1,44 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.audit;
|
||||
|
||||
import io.swagger.v3.oas.annotations.media.Schema;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/**
|
||||
* A single infrastructure audit-log row, shaped for the portal Infrastructure → Audit tab. Derived
|
||||
* from a {@code audit_events} row: the real {@link
|
||||
* stirling.software.proprietary.audit.AuditEventType} is mapped to a display category/action.
|
||||
*/
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class InfraAuditEventDto {
|
||||
|
||||
@Schema(description = "Audit event id", example = "8841")
|
||||
private String id;
|
||||
|
||||
@Schema(description = "Display timestamp (UTC)", example = "2026-07-07 18:59:31")
|
||||
private String timestamp;
|
||||
|
||||
@Schema(description = "Category: auth | config | elevation | processing | security")
|
||||
private String category;
|
||||
|
||||
@Schema(description = "Human-readable action", example = "Compress PDF")
|
||||
private String action;
|
||||
|
||||
@Schema(description = "Actor principal", example = "alice.chen@acme.com")
|
||||
private String actor;
|
||||
|
||||
@Schema(description = "Affected target (file, endpoint, or session)")
|
||||
private String target;
|
||||
|
||||
@Schema(description = "Status: success | warning | danger | info")
|
||||
private String status;
|
||||
|
||||
@Schema(description = "Operation latency in milliseconds", example = "412")
|
||||
private long latencyMs;
|
||||
}
|
||||
-30
@@ -1,30 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.audit;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import io.swagger.v3.oas.annotations.media.Schema;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/** Response for the portal Infrastructure → Audit tab: summary strip + recent event rows. */
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class InfraAuditLogResponse {
|
||||
|
||||
@Schema(description = "Headline counts")
|
||||
private InfraAuditSummary summary;
|
||||
|
||||
@Schema(description = "Most-recent audit events, newest first")
|
||||
private List<InfraAuditEventDto> events;
|
||||
|
||||
@Schema(
|
||||
description =
|
||||
"True when this is the whole-server (admin) view. Team-scoped views are false; "
|
||||
+ "drives whether the admin-only, whole-server CSV export is offered.")
|
||||
private boolean fullServer;
|
||||
}
|
||||
-28
@@ -1,28 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.audit;
|
||||
|
||||
import io.swagger.v3.oas.annotations.media.Schema;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/** Headline counts for the infrastructure audit-log tab, derived from the returned events. */
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class InfraAuditSummary {
|
||||
|
||||
@Schema(description = "Total events in the returned window", example = "40")
|
||||
private int totalEvents;
|
||||
|
||||
@Schema(description = "Processing-category events", example = "24")
|
||||
private int processing;
|
||||
|
||||
@Schema(description = "Elevation-category events", example = "0")
|
||||
private int elevation;
|
||||
|
||||
@Schema(description = "Config-category events", example = "6")
|
||||
private int config;
|
||||
}
|
||||
-24
@@ -1,24 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.documents;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/** One event in a document's lifecycle timeline. */
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class PortalDocAuditEventDto {
|
||||
private String id;
|
||||
|
||||
/** ingested | extracted | flagged | reviewed | approved | archived | elevation */
|
||||
private String kind;
|
||||
|
||||
/** Relative-time string, e.g. "2m ago". */
|
||||
private String time;
|
||||
|
||||
private String actor;
|
||||
private String detail;
|
||||
}
|
||||
-18
@@ -1,18 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.documents;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/** Response for the portal Documents review queue. */
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class PortalDocumentsResponseDto {
|
||||
private PortalDocumentsSummaryDto summary;
|
||||
private List<PortalReviewDocumentDto> documents;
|
||||
}
|
||||
-18
@@ -1,18 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.documents;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/** KPI strip for the documents queue. */
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class PortalDocumentsSummaryDto {
|
||||
private int totalInQueue;
|
||||
private int processed;
|
||||
private int errors;
|
||||
private int processedToday;
|
||||
}
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.documents;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/** A single extracted field. Empty for audit-derived documents (no extraction data yet). */
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class PortalExtractionDto {
|
||||
private String field;
|
||||
private String value;
|
||||
private double confidence;
|
||||
}
|
||||
-48
@@ -1,48 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.documents;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Builder;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/**
|
||||
* A document in the review queue, derived from the audit trail of a processed file. Extraction
|
||||
* fields ({@code confidence}, {@code extractions}) are absent/empty - that data doesn't exist yet.
|
||||
*/
|
||||
@Data
|
||||
@Builder
|
||||
@NoArgsConstructor
|
||||
@AllArgsConstructor
|
||||
public class PortalReviewDocumentDto {
|
||||
private String id;
|
||||
private String name;
|
||||
private String type;
|
||||
|
||||
/** Where it was processed: "API" or "Editor". */
|
||||
private String product;
|
||||
|
||||
/** The operation / pipeline, e.g. "Compress PDF" (or "Editor"). */
|
||||
private String action;
|
||||
|
||||
/** The user who ran it. */
|
||||
private String user;
|
||||
|
||||
/** processed | error */
|
||||
private String status;
|
||||
|
||||
private String source;
|
||||
|
||||
/** Overall confidence 0..1, or null when there's no extraction data. */
|
||||
private Double confidence;
|
||||
|
||||
private int fieldsExtracted;
|
||||
|
||||
/** Relative-time string, e.g. "4m ago". */
|
||||
private String time;
|
||||
|
||||
private boolean sensitive;
|
||||
private List<PortalExtractionDto> extractions;
|
||||
private List<PortalDocAuditEventDto> audit;
|
||||
}
|
||||
-6
@@ -1,6 +0,0 @@
|
||||
package stirling.software.proprietary.model.api.usage;
|
||||
|
||||
/**
|
||||
* Free-editor fleet usage for the portal Usage card. Null fields render as "N/A" (uncomputable).
|
||||
*/
|
||||
public record FleetUsageStats(Long editorsDeployed, Long activeThisMonth, Long pdfsProcessed) {}
|
||||
+1
-10
@@ -18,15 +18,7 @@ import lombok.*;
|
||||
columnList = "principal,type"),
|
||||
@jakarta.persistence.Index(
|
||||
name = "idx_audit_type_timestamp",
|
||||
columnList = "type,timestamp"),
|
||||
@jakarta.persistence.Index(
|
||||
name = "idx_audit_type_source_timestamp",
|
||||
columnList = "type,source,timestamp"),
|
||||
// Leads with source (equality) for the active-editors query, which filters on
|
||||
// source then a timestamp range and counts distinct principal.
|
||||
@jakarta.persistence.Index(
|
||||
name = "idx_audit_source_timestamp_principal",
|
||||
columnList = "source,timestamp,principal")
|
||||
columnList = "type,timestamp")
|
||||
})
|
||||
@Data
|
||||
@Builder
|
||||
@@ -40,7 +32,6 @@ public class PersistentAuditEvent {
|
||||
|
||||
private String principal;
|
||||
private String type;
|
||||
private String source;
|
||||
|
||||
@Column(columnDefinition = "text")
|
||||
private String data; // JSON blob
|
||||
|
||||
-37
@@ -18,7 +18,6 @@ import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.ModelAttribute;
|
||||
import org.springframework.web.bind.annotation.PathVariable;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.PutMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RequestPart;
|
||||
@@ -48,7 +47,6 @@ import stirling.software.proprietary.policy.engine.PolicyRunHandle;
|
||||
import stirling.software.proprietary.policy.engine.PolicyRunRegistry;
|
||||
import stirling.software.proprietary.policy.engine.PolicyRunner;
|
||||
import stirling.software.proprietary.policy.engine.PolicyValidator;
|
||||
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
|
||||
import stirling.software.proprietary.policy.model.PipelineDefinition;
|
||||
import stirling.software.proprietary.policy.model.Policy;
|
||||
import stirling.software.proprietary.policy.model.PolicyInputs;
|
||||
@@ -88,7 +86,6 @@ public class PolicyController {
|
||||
private final PolicyManagementAuthority policyManagementAuthority;
|
||||
private final PolicyTriggerManager policyTriggerManager;
|
||||
private final PolicyOverviewService policyOverviewService;
|
||||
private final ProcessedLedger processedLedger;
|
||||
private final List<PolicyTrigger> policyTriggers;
|
||||
private final ApplicationProperties applicationProperties;
|
||||
private final TempFileManager tempFileManager;
|
||||
@@ -216,20 +213,6 @@ public class PolicyController {
|
||||
return ResponseEntity.ok(saved);
|
||||
}
|
||||
|
||||
@PutMapping("/order")
|
||||
@Operation(
|
||||
summary = "Set the team's policy run order",
|
||||
description =
|
||||
"Persists the team-wide order policies run in, from the given ordered list of"
|
||||
+ " policy ids (position → order). The per-trigger order shown in the UI"
|
||||
+ " is this one sequence filtered by trigger. Team-leader/admin only;"
|
||||
+ " ids outside the caller's team are ignored.")
|
||||
public ResponseEntity<Void> reorderPolicies(@RequestBody List<String> orderedPolicyIds) {
|
||||
requirePolicyEditingAllowed();
|
||||
policyStore.reorder(policyAccessGuard.teamForNewPolicy(), orderedPolicyIds);
|
||||
return ResponseEntity.noContent().build();
|
||||
}
|
||||
|
||||
/**
|
||||
* Every {@code sourceId} a policy references must resolve to a source in the caller's team, so
|
||||
* a client can neither reference a non-existent source nor reach across teams to use another
|
||||
@@ -354,7 +337,6 @@ public class PolicyController {
|
||||
boolean accessible =
|
||||
policyStore.get(policyId).filter(policyAccessGuard::canAccess).isPresent();
|
||||
if (accessible && policyStore.delete(policyId)) {
|
||||
processedLedger.clearPolicy(policyId);
|
||||
// Cancel any now-orphaned folder watch promptly rather than leaving the WatchKey open
|
||||
// until the next reconcile sweep.
|
||||
policyTriggerManager.notifyPoliciesChanged();
|
||||
@@ -363,25 +345,6 @@ public class PolicyController {
|
||||
return ResponseEntity.notFound().build();
|
||||
}
|
||||
|
||||
@DeleteMapping("/{policyId}/processed-history")
|
||||
@Operation(
|
||||
summary = "Clear a policy's processed-file history",
|
||||
description =
|
||||
"Forgets which source files this policy has already processed, so its next"
|
||||
+ " sweep reprocesses everything currently in its sources. Does not"
|
||||
+ " touch the files themselves.")
|
||||
public ResponseEntity<Void> clearProcessedHistory(@PathVariable String policyId) {
|
||||
requirePolicyEditingAllowed();
|
||||
// Scope to the caller's team: a policy in another team reads as not-found.
|
||||
boolean accessible =
|
||||
policyStore.get(policyId).filter(policyAccessGuard::canAccess).isPresent();
|
||||
if (!accessible) {
|
||||
return ResponseEntity.notFound().build();
|
||||
}
|
||||
processedLedger.clearPolicy(policyId);
|
||||
return ResponseEntity.noContent().build();
|
||||
}
|
||||
|
||||
@PostMapping(value = "/{policyId}/run", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
|
||||
@Operation(
|
||||
summary = "Run a stored policy",
|
||||
|
||||
+1
-7
@@ -35,7 +35,6 @@ import stirling.software.proprietary.policy.model.Policy;
|
||||
import stirling.software.proprietary.policy.model.PolicyInputs;
|
||||
import stirling.software.proprietary.policy.model.PolicyRun;
|
||||
import stirling.software.proprietary.policy.model.WaitState;
|
||||
import stirling.software.proprietary.policy.output.OutputDelivery;
|
||||
import stirling.software.proprietary.policy.output.PolicyOutputSink;
|
||||
import stirling.software.proprietary.policy.progress.PolicyProgressListener;
|
||||
import stirling.software.proprietary.service.DownstreamEntitlementError;
|
||||
@@ -204,12 +203,7 @@ public class PolicyEngine {
|
||||
PolicyExecutionResult result =
|
||||
stepExecutor.execute(run.getDefinition(), inputs, listener);
|
||||
OutputSpec output = run.getDefinition().output();
|
||||
List<ResultFile> outputs =
|
||||
sinkFor(output)
|
||||
.deliver(
|
||||
new OutputDelivery(runId, run.getPolicyId()),
|
||||
result.files(),
|
||||
output);
|
||||
List<ResultFile> outputs = sinkFor(output).deliver(runId, result.files(), output);
|
||||
taskManager.setMultipleFileResults(runId, outputs);
|
||||
taskManager.setComplete(runId);
|
||||
run.complete(outputs);
|
||||
|
||||
+3
-7
@@ -8,11 +8,7 @@ import tools.jackson.databind.JsonNode;
|
||||
|
||||
/**
|
||||
* Result of a {@link PolicyExecutor} run. {@code files} are final temp files (not yet stored).
|
||||
* {@code origins} is parallel to {@code files}: each entry is the index into the original pipeline
|
||||
* inputs that the output traces back to, or {@code null} when it has no single source (e.g. a merge
|
||||
* combining several inputs, or a generated file). Callers use it to map an output back onto the
|
||||
* file it came from. {@code report}/{@code reportTool} carry the last step's structured report and
|
||||
* its operation, or null if no step produced one.
|
||||
* {@code report}/{@code reportTool} carry the last step's structured report and its operation, or
|
||||
* null if no step produced one.
|
||||
*/
|
||||
public record PolicyExecutionResult(
|
||||
List<Resource> files, List<Integer> origins, JsonNode report, String reportTool) {}
|
||||
public record PolicyExecutionResult(List<Resource> files, JsonNode report, String reportTool) {}
|
||||
|
||||
+9
-40
@@ -58,11 +58,6 @@ public class PolicyExecutor {
|
||||
// payload the tool surfaced alongside or instead of a file.
|
||||
private record ToolResult(List<Resource> files, JsonNode report) {}
|
||||
|
||||
// A step's output files paired with each file's origin (the index into the original pipeline
|
||||
// inputs it traces back to, or null when it has no single source). Origins compose across steps
|
||||
// so the final result can be mapped back onto the files that entered the pipeline.
|
||||
private record StepOutput(List<Resource> files, List<Integer> origins, JsonNode report) {}
|
||||
|
||||
/**
|
||||
* Run every step in order, feeding each step's output into the next. Supporting files in {@code
|
||||
* inputs} bind to named file fields and never enter the document stream.
|
||||
@@ -80,12 +75,6 @@ public class PolicyExecutor {
|
||||
|
||||
List<Resource> currentFiles = inputs.primary();
|
||||
Map<String, List<Resource>> supportingFiles = inputs.supportingFiles();
|
||||
// Seed each input with its own index as origin; steps carry these through so the final
|
||||
// outputs can be traced back to the files that entered the pipeline.
|
||||
List<Integer> currentOrigins = new ArrayList<>();
|
||||
for (int k = 0; k < currentFiles.size(); k++) {
|
||||
currentOrigins.add(k);
|
||||
}
|
||||
// Last non-null report wins: the terminal step defines the output.
|
||||
JsonNode lastReport = null;
|
||||
String lastReportTool = null;
|
||||
@@ -98,10 +87,8 @@ public class PolicyExecutor {
|
||||
"Pipeline step " + (i + 1) + " has no operation");
|
||||
}
|
||||
listener.onStepStart(i + 1, steps.size(), operation);
|
||||
StepOutput stepResult =
|
||||
executeStep(step, currentFiles, currentOrigins, supportingFiles);
|
||||
ToolResult stepResult = executeStep(step, currentFiles, supportingFiles);
|
||||
currentFiles = stepResult.files();
|
||||
currentOrigins = stepResult.origins();
|
||||
if (stepResult.report() != null) {
|
||||
lastReport = stepResult.report();
|
||||
lastReportTool = operation;
|
||||
@@ -109,7 +96,7 @@ public class PolicyExecutor {
|
||||
listener.onStepComplete(i + 1, steps.size(), operation);
|
||||
}
|
||||
|
||||
return new PolicyExecutionResult(currentFiles, currentOrigins, lastReport, lastReportTool);
|
||||
return new PolicyExecutionResult(currentFiles, lastReport, lastReportTool);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -117,50 +104,32 @@ public class PolicyExecutor {
|
||||
* responses are unpacked so each inner file is its own result (e.g. split). For per-file
|
||||
* dispatch the first non-null report wins.
|
||||
*/
|
||||
private StepOutput executeStep(
|
||||
private ToolResult executeStep(
|
||||
PipelineStep step,
|
||||
List<Resource> inputFiles,
|
||||
List<Integer> inputOrigins,
|
||||
Map<String, List<Resource>> supportingFiles)
|
||||
throws IOException {
|
||||
requireAcceptedTypes(step.operation(), inputFiles);
|
||||
List<Resource> files = new ArrayList<>();
|
||||
List<Integer> origins = new ArrayList<>();
|
||||
JsonNode report = null;
|
||||
if (toolMetadataService.isMultiInput(step.operation())) {
|
||||
// One call over all inputs. The outputs derive from a single input only when exactly
|
||||
// one entered; otherwise (a genuine merge) there is no single source.
|
||||
ToolResult r = callEndpoint(step, inputFiles, supportingFiles);
|
||||
Integer origin = inputOrigins.size() == 1 ? inputOrigins.get(0) : null;
|
||||
for (Resource file : r.files()) {
|
||||
files.add(file);
|
||||
origins.add(origin);
|
||||
}
|
||||
files.addAll(r.files());
|
||||
report = r.report();
|
||||
} else if (inputFiles.isEmpty()) {
|
||||
ToolResult r = callEndpoint(step, List.of(), supportingFiles);
|
||||
for (Resource file : r.files()) {
|
||||
files.add(file);
|
||||
origins.add(null);
|
||||
}
|
||||
files.addAll(r.files());
|
||||
report = r.report();
|
||||
} else {
|
||||
// One call per file: every output of this call inherits that input's origin, so a 1:1
|
||||
// op keeps its chain and a split (one input, many outputs) tags each output with the
|
||||
// same source.
|
||||
for (int k = 0; k < inputFiles.size(); k++) {
|
||||
Integer origin = inputOrigins.get(k);
|
||||
ToolResult r = callEndpoint(step, List.of(inputFiles.get(k)), supportingFiles);
|
||||
for (Resource file : r.files()) {
|
||||
files.add(file);
|
||||
origins.add(origin);
|
||||
}
|
||||
for (Resource file : inputFiles) {
|
||||
ToolResult r = callEndpoint(step, List.of(file), supportingFiles);
|
||||
files.addAll(r.files());
|
||||
if (report == null) {
|
||||
report = r.report();
|
||||
}
|
||||
}
|
||||
}
|
||||
return new StepOutput(files, origins, report);
|
||||
return new ToolResult(files, report);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+8
-41
@@ -13,7 +13,6 @@ import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.policy.input.InputSource;
|
||||
import stirling.software.proprietary.policy.input.ResolvedInput;
|
||||
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
|
||||
import stirling.software.proprietary.policy.model.InputSpec;
|
||||
import stirling.software.proprietary.policy.model.PipelineDefinition;
|
||||
import stirling.software.proprietary.policy.model.Policy;
|
||||
@@ -28,8 +27,7 @@ import stirling.software.proprietary.policy.source.SourceStore;
|
||||
/**
|
||||
* Turns a policy's referenced sources into runs: each {@code sourceId} is resolved live to its
|
||||
* persisted {@link Source}, then to an {@link InputSpec}. Triggers decide <em>when</em> and call
|
||||
* {@link #run(Policy)}; the controller uses the supplied-input and ad-hoc entry points. A {@link
|
||||
* SweepKind#FULL} sweep also reconciles the processed-file ledger against what is present.
|
||||
* {@link #run(Policy)}; the controller uses the supplied-input and ad-hoc entry points.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@@ -41,12 +39,6 @@ public class PolicyRunner {
|
||||
private final List<InputSource> inputSources;
|
||||
private final SourceStore sourceStore;
|
||||
private final SourceDocCounter docCounter;
|
||||
private final ProcessedLedger processedLedger;
|
||||
|
||||
/** Full-listing sweep: resolve every source, then reconcile the ledger. */
|
||||
public List<String> run(Policy policy) {
|
||||
return run(policy, SweepKind.FULL);
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger entry point. Pulls every referenced source; each yielded unit becomes its own run so
|
||||
@@ -55,21 +47,15 @@ public class PolicyRunner {
|
||||
* rest. Returns the ids of the runs it started (empty when sources yielded no work), so a
|
||||
* manual trigger can report back which runs to follow.
|
||||
*/
|
||||
public List<String> run(Policy policy, SweepKind sweep) {
|
||||
long sweepStart = System.currentTimeMillis();
|
||||
PolicySweep context = new PolicySweep(policy.id(), sweep, processedLedger);
|
||||
List<String> runIds = new ArrayList<>();
|
||||
public List<String> run(Policy policy) {
|
||||
List<String> sourceIds = policy.sourceIds();
|
||||
if (sourceIds.isEmpty()) {
|
||||
// Generator pipeline: one run with no input. Still fall through to the cleanup
|
||||
// below so rows recorded for its folder outputs are pruned like anything else,
|
||||
// instead of accumulating until the policy is deleted.
|
||||
runIds.add(startRun(policy, PolicyInputs.of(List.of()), unused -> {}));
|
||||
return List.of(startRun(policy, PolicyInputs.of(List.of()), unused -> {}));
|
||||
}
|
||||
List<String> runIds = new ArrayList<>();
|
||||
for (String sourceId : sourceIds) {
|
||||
Source source = sourceStore.get(sourceId).orElse(null);
|
||||
if (source == null) {
|
||||
// No veto: a deleted source's rows should age out via the cleanup below.
|
||||
log.warn("Policy {} references missing source {}; skipping", policy.id(), sourceId);
|
||||
continue;
|
||||
}
|
||||
@@ -79,21 +65,9 @@ public class PolicyRunner {
|
||||
sourceId,
|
||||
source.name(),
|
||||
policy.id());
|
||||
// Veto: a paused source's files cannot be stamped, so they must not be pruned.
|
||||
context.vetoCleanup();
|
||||
continue;
|
||||
}
|
||||
runIds.addAll(pullAndRun(policy, sourceId, source.toInputSpec(), context));
|
||||
}
|
||||
if (context.cleanupAllowed()) {
|
||||
processedLedger.markSeen(policy.id(), context.presentIdentities());
|
||||
int removed = processedLedger.deleteUnseen(policy.id(), sweepStart);
|
||||
if (removed > 0) {
|
||||
log.debug(
|
||||
"Pruned {} ledger row(s) for files no longer present (policy {})",
|
||||
removed,
|
||||
policy.id());
|
||||
}
|
||||
runIds.addAll(pullAndRun(policy, sourceId, source.toInputSpec()));
|
||||
}
|
||||
return runIds;
|
||||
}
|
||||
@@ -112,33 +86,26 @@ public class PolicyRunner {
|
||||
|
||||
/**
|
||||
* Resolves the source and starts a run per unit; records how many documents the source fed and
|
||||
* returns the ids of the runs started. Any source that could not be listed completely vetoes
|
||||
* this sweep's ledger cleanup.
|
||||
* returns the ids of the runs started.
|
||||
*/
|
||||
private List<String> pullAndRun(
|
||||
Policy policy, String sourceId, InputSpec spec, PolicySweep context) {
|
||||
private List<String> pullAndRun(Policy policy, String sourceId, InputSpec spec) {
|
||||
InputSource source = sourceFor(spec);
|
||||
if (source == null) {
|
||||
log.warn(
|
||||
"No input source for type '{}' (policy {}); skipping",
|
||||
spec.type(),
|
||||
policy.id());
|
||||
context.vetoCleanup();
|
||||
return List.of();
|
||||
}
|
||||
if (!source.listsExhaustively()) {
|
||||
context.vetoCleanup();
|
||||
}
|
||||
List<ResolvedInput> work;
|
||||
try {
|
||||
work = source.resolve(spec, context);
|
||||
work = source.resolve(spec);
|
||||
} catch (IOException | RuntimeException e) {
|
||||
log.warn(
|
||||
"Failed to resolve source '{}' for policy {}: {}",
|
||||
spec.type(),
|
||||
policy.id(),
|
||||
e.getMessage());
|
||||
context.vetoCleanup();
|
||||
return List.of();
|
||||
}
|
||||
List<String> runIds = new ArrayList<>();
|
||||
|
||||
-89
@@ -1,89 +0,0 @@
|
||||
package stirling.software.proprietary.policy.engine;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import stirling.software.proprietary.policy.input.ResolveContext;
|
||||
import stirling.software.proprietary.policy.ledger.ClaimState;
|
||||
import stirling.software.proprietary.policy.ledger.ProcessedFileStatus;
|
||||
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
|
||||
|
||||
/**
|
||||
* The {@link ResolveContext} for one policy sweep: scopes ledger calls to the policy, gathers the
|
||||
* present-identity union across sources, prefetches claim state in bulk so per-file claims skip
|
||||
* their row lookup, and vetoes presence cleanup when any source could not be listed completely
|
||||
* (pruning would wrongly forget its files).
|
||||
*/
|
||||
final class PolicySweep implements ResolveContext {
|
||||
|
||||
private final String policyId;
|
||||
private final SweepKind kind;
|
||||
private final ProcessedLedger ledger;
|
||||
private final Set<String> present = new HashSet<>();
|
||||
// Claim states loaded in bulk at reportPresent; a claim outside the prefetch falls back to a
|
||||
// single lookup. A stale entry cannot double-claim (the ledger re-checks every transition),
|
||||
// it can only defer a file to the next sweep.
|
||||
private final Map<String, ClaimState> prefetched = new HashMap<>();
|
||||
private final Set<String> prefetchedIdentities = new HashSet<>();
|
||||
private boolean cleanupVetoed;
|
||||
|
||||
PolicySweep(String policyId, SweepKind kind, ProcessedLedger ledger) {
|
||||
this.policyId = policyId;
|
||||
this.kind = kind;
|
||||
this.ledger = ledger;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized boolean claim(String identity, String gate, Supplier<String> contentHash) {
|
||||
ClaimState observed =
|
||||
prefetchedIdentities.contains(identity)
|
||||
? prefetched.get(identity)
|
||||
: ledger.statesFor(policyId, List.of(identity)).get(identity);
|
||||
boolean claimed = ledger.claim(policyId, identity, gate, contentHash, observed);
|
||||
if (claimed) {
|
||||
// A nested source surfacing the same file later in this sweep sees it in flight
|
||||
// without another lookup.
|
||||
prefetchedIdentities.add(identity);
|
||||
prefetched.put(identity, new ClaimState(ProcessedFileStatus.PROCESSING, gate, null));
|
||||
}
|
||||
return claimed;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void settle(
|
||||
String identity, String finalGate, String finalContentHash, boolean success) {
|
||||
ledger.settle(policyId, identity, finalGate, finalContentHash, success);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean allSettledDone(String identity) {
|
||||
// Deliberately not policy-scoped: consume deletion needs every claimant's consensus.
|
||||
return ledger.allSettledDone(identity);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void reportPresent(Collection<String> identities) {
|
||||
if (kind == SweepKind.FULL) {
|
||||
present.addAll(identities);
|
||||
}
|
||||
prefetched.putAll(ledger.statesFor(policyId, identities));
|
||||
prefetchedIdentities.addAll(identities);
|
||||
}
|
||||
|
||||
synchronized void vetoCleanup() {
|
||||
cleanupVetoed = true;
|
||||
}
|
||||
|
||||
synchronized boolean cleanupAllowed() {
|
||||
return kind == SweepKind.FULL && !cleanupVetoed;
|
||||
}
|
||||
|
||||
synchronized Set<String> presentIdentities() {
|
||||
return Set.copyOf(present);
|
||||
}
|
||||
}
|
||||
-10
@@ -1,10 +0,0 @@
|
||||
package stirling.software.proprietary.policy.engine;
|
||||
|
||||
/**
|
||||
* How thorough a policy sweep is: {@link #FULL} (complete listing; also stamps presence and prunes
|
||||
* the ledger) or {@link #LIGHT} (event-driven; claims only, cost proportional to what changed).
|
||||
*/
|
||||
public enum SweepKind {
|
||||
FULL,
|
||||
LIGHT
|
||||
}
|
||||
+76
-202
@@ -1,17 +1,12 @@
|
||||
package stirling.software.proprietary.policy.input;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.UncheckedIOException;
|
||||
import java.nio.file.FileVisitResult;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.NoSuchFileException;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.SimpleFileVisitor;
|
||||
import java.nio.file.attribute.BasicFileAttributes;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
@@ -24,20 +19,17 @@ import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.util.FileReadinessChecker;
|
||||
import stirling.software.proprietary.policy.config.FolderAccessGuard;
|
||||
import stirling.software.proprietary.policy.ledger.FolderIdentities;
|
||||
import stirling.software.proprietary.policy.model.InputSpec;
|
||||
import stirling.software.proprietary.policy.model.PolicyInputs;
|
||||
|
||||
/**
|
||||
* Reads input files from a directory; each ready file is its own unit of work, claimed through the
|
||||
* {@link ResolveContext} ledger rather than moved aside, so nothing accumulates in a work
|
||||
* directory. Options: "mode" is "consume" (default: a processed file is removed once every policy
|
||||
* that claimed it has settled successfully and it is still the version that ran; failures stay in
|
||||
* place and are not retried until they change) or "snapshot" (stateless, every run sees the full
|
||||
* set); "recursive" descends into subdirectories; "identity" is "stat" (default, any size/mtime
|
||||
* change is a new version) or "hash" (content-verified, so a touch does not reprocess). Hidden
|
||||
* files and directories, including the legacy {@code .stirling} work dir, are never picked up, and
|
||||
* files mid-write are skipped by the readiness check.
|
||||
* Reads input files from a directory; each ready file is its own unit of work so one failure does
|
||||
* not affect the others.
|
||||
*
|
||||
* <p>Mode option: "consume" (default) claims each file by moving it into {@code
|
||||
* .stirling/processing} then routes it to {@code .stirling/done} or {@code .stirling/error}, so
|
||||
* each file runs once; "snapshot" reads without moving, so every run sees the full set. Readiness
|
||||
* is checked first so files mid-write are skipped.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@@ -46,6 +38,11 @@ import stirling.software.proprietary.policy.model.PolicyInputs;
|
||||
public class FolderInputSource implements InputSource {
|
||||
|
||||
private static final String TYPE = FolderAccessGuard.FOLDER_TYPE;
|
||||
// Bookkeeping lives under one hidden dir so the watched folder stays tidy.
|
||||
private static final String WORK_SUBDIR = ".stirling";
|
||||
private static final String PROCESSING_SUBDIR = "processing";
|
||||
private static final String DONE_SUBDIR = "done";
|
||||
private static final String ERROR_SUBDIR = "error";
|
||||
|
||||
private final FileReadinessChecker readinessChecker;
|
||||
private final FolderAccessGuard accessGuard;
|
||||
@@ -71,197 +68,72 @@ public class FolderInputSource implements InputSource {
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ResolvedInput> resolve(InputSpec spec, ResolveContext ctx) throws IOException {
|
||||
public List<ResolvedInput> resolve(InputSpec spec) throws IOException {
|
||||
FolderConfig config = FolderConfig.from(spec.options());
|
||||
Path inputDir = accessGuard.requirePermitted(config.directory());
|
||||
if (!Files.isDirectory(inputDir)) {
|
||||
// Fail rather than return empty: an unmounted drive must read as "could not list",
|
||||
// which vetoes the sweep's presence cleanup, not as "verifiably no files", which
|
||||
// would wipe the policy's history and reprocess everything on remount.
|
||||
throw new NoSuchFileException(
|
||||
inputDir.toString(), null, "input directory does not exist");
|
||||
}
|
||||
Path canonicalDir = FolderIdentities.canonicalDir(inputDir);
|
||||
List<Path> present = listFiles(inputDir, config.recursive());
|
||||
|
||||
if (config.snapshot()) {
|
||||
List<ResolvedInput> work = new ArrayList<>();
|
||||
for (Path file : present) {
|
||||
if (readinessChecker.isReady(file)) {
|
||||
work.add(ResolvedInput.of(PolicyInputs.of(List.of(fileResource(file)))));
|
||||
}
|
||||
}
|
||||
return work;
|
||||
log.debug("Folder input dir does not exist: {}", inputDir);
|
||||
return List.of();
|
||||
}
|
||||
|
||||
ctx.reportPresent(
|
||||
present.stream()
|
||||
.map(file -> FolderIdentities.identity(canonicalDir, inputDir, file))
|
||||
.toList());
|
||||
List<Path> ready = new ArrayList<>();
|
||||
try (Stream<Path> entries = Files.list(inputDir)) {
|
||||
entries.filter(Files::isRegularFile)
|
||||
.filter(readinessChecker::isReady)
|
||||
.forEach(ready::add);
|
||||
}
|
||||
|
||||
List<ResolvedInput> work = new ArrayList<>();
|
||||
for (Path file : present) {
|
||||
if (!readinessChecker.isReady(file)) {
|
||||
continue;
|
||||
for (Path file : ready) {
|
||||
if (config.snapshot()) {
|
||||
work.add(ResolvedInput.of(PolicyInputs.of(List.of(fileResource(file)))));
|
||||
} else {
|
||||
Path claimed = claim(inputDir, file);
|
||||
if (claimed == null) {
|
||||
continue; // another sweep/process grabbed it
|
||||
}
|
||||
work.add(
|
||||
new ResolvedInput(
|
||||
PolicyInputs.of(List.of(fileResource(claimed))),
|
||||
success -> route(inputDir, claimed, success)));
|
||||
}
|
||||
String identity = FolderIdentities.identity(canonicalDir, inputDir, file);
|
||||
MemoizedContentHash contentHash =
|
||||
config.hashIdentity() ? new MemoizedContentHash(file) : null;
|
||||
String gate;
|
||||
boolean claimed;
|
||||
try {
|
||||
gate = FolderIdentities.statGate(file);
|
||||
claimed = ctx.claim(identity, gate, contentHash);
|
||||
} catch (IOException | UncheckedIOException e) {
|
||||
log.debug("Could not read {} for its version: {}", file, e.getMessage());
|
||||
continue; // vanished or unreadable mid-sweep; the next sweep sees the truth
|
||||
}
|
||||
if (!claimed) {
|
||||
continue;
|
||||
}
|
||||
work.add(
|
||||
new ResolvedInput(
|
||||
PolicyInputs.of(List.of(fileResource(file))),
|
||||
success ->
|
||||
completeConsumed(
|
||||
ctx, identity, file, gate, contentHash, success)));
|
||||
}
|
||||
return work;
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle at the version this run claimed - never a re-read, so a file replaced mid-run reads as
|
||||
* a new unclaimed version next sweep instead of being marked processed. Then remove the input
|
||||
* only when it is still the processed version (a mid-run replacement must survive) and every
|
||||
* policy that claimed it has settled DONE, so co-watching policies all read the original and
|
||||
* one failure parks the file for everyone. A failed run settles ERROR and never deletes; the
|
||||
* DONE row of a file that could not be deleted still stops reprocessing.
|
||||
*/
|
||||
private static void completeConsumed(
|
||||
ResolveContext ctx,
|
||||
String identity,
|
||||
Path file,
|
||||
String claimGate,
|
||||
MemoizedContentHash contentHash,
|
||||
boolean success) {
|
||||
ctx.settle(identity, claimGate, claimedHash(file, claimGate, contentHash), success);
|
||||
if (!success) {
|
||||
return;
|
||||
}
|
||||
// Atomic move into processing/: only one sweep can win the claim, the rest see the file gone.
|
||||
private Path claim(Path inputDir, Path file) {
|
||||
try {
|
||||
if (FolderIdentities.statGate(file).equals(claimGate) && ctx.allSettledDone(identity)) {
|
||||
Files.deleteIfExists(file);
|
||||
}
|
||||
} catch (NoSuchFileException alreadyGone) {
|
||||
// Removed by the user or a co-watching policy's own consensus delete: nothing to do.
|
||||
Path processingDir = workDir(inputDir, PROCESSING_SUBDIR);
|
||||
Files.createDirectories(processingDir);
|
||||
Path claimed = uniqueTarget(processingDir, file.getFileName().toString());
|
||||
Files.move(file, claimed, StandardCopyOption.ATOMIC_MOVE);
|
||||
return claimed;
|
||||
} catch (IOException e) {
|
||||
log.warn("Could not remove consumed input {}: {}", file, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The claimed version's content hash: the value computed during the claim when the ledger
|
||||
* consulted the verifier, else computed now while the file is still at the claimed gate (so the
|
||||
* hash describes what actually ran), else null. Always null in stat mode.
|
||||
*/
|
||||
private static String claimedHash(Path file, String claimGate, MemoizedContentHash hash) {
|
||||
if (hash == null) {
|
||||
log.debug("Could not claim {}: {}", file, e.getMessage());
|
||||
return null;
|
||||
}
|
||||
String computed = hash.valueIfComputed();
|
||||
if (computed != null) {
|
||||
return computed;
|
||||
}
|
||||
}
|
||||
|
||||
private void route(Path inputDir, Path claimed, boolean success) {
|
||||
String subdir = success ? DONE_SUBDIR : ERROR_SUBDIR;
|
||||
try {
|
||||
if (FolderIdentities.statGate(file).equals(claimGate)) {
|
||||
return hash.get();
|
||||
}
|
||||
} catch (IOException | UncheckedIOException e) {
|
||||
log.debug("Could not hash {} at settle: {}", file, e.getMessage());
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Lazy verification tier: invoked at most once by the ledger, retained for the settle. */
|
||||
private static final class MemoizedContentHash implements Supplier<String> {
|
||||
|
||||
private final Path file;
|
||||
private volatile String value;
|
||||
|
||||
private MemoizedContentHash(Path file) {
|
||||
this.file = file;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String get() {
|
||||
if (value == null) {
|
||||
try {
|
||||
value = FolderIdentities.contentHash(file);
|
||||
} catch (IOException e) {
|
||||
throw new UncheckedIOException(e);
|
||||
}
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
String valueIfComputed() {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
|
||||
/** Every non-hidden regular file in the source, readable or not. */
|
||||
private static List<Path> listFiles(Path inputDir, boolean recursive) throws IOException {
|
||||
List<Path> files = new ArrayList<>();
|
||||
if (!recursive) {
|
||||
try (Stream<Path> entries = Files.list(inputDir)) {
|
||||
entries.filter(Files::isRegularFile)
|
||||
.filter(file -> !hidden(file))
|
||||
.forEach(files::add);
|
||||
}
|
||||
return files;
|
||||
}
|
||||
// Hidden subtrees are pruned wholesale; symlinked directories are not followed.
|
||||
Files.walkFileTree(
|
||||
inputDir,
|
||||
new SimpleFileVisitor<>() {
|
||||
@Override
|
||||
public FileVisitResult preVisitDirectory(
|
||||
Path dir, BasicFileAttributes attributes) {
|
||||
if (!dir.equals(inputDir) && hidden(dir)) {
|
||||
return FileVisitResult.SKIP_SUBTREE;
|
||||
}
|
||||
return FileVisitResult.CONTINUE;
|
||||
}
|
||||
|
||||
@Override
|
||||
public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) {
|
||||
if (attributes.isRegularFile() && !hidden(file)) {
|
||||
files.add(file);
|
||||
}
|
||||
return FileVisitResult.CONTINUE;
|
||||
}
|
||||
|
||||
@Override
|
||||
public FileVisitResult visitFileFailed(Path file, IOException e) {
|
||||
log.debug("Skipping unreadable entry {}: {}", file, e.getMessage());
|
||||
return FileVisitResult.CONTINUE;
|
||||
}
|
||||
});
|
||||
return files;
|
||||
}
|
||||
|
||||
private static boolean hidden(Path path) {
|
||||
Path name = path.getFileName();
|
||||
if (name != null && name.toString().startsWith(".")) {
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
return Files.isHidden(path);
|
||||
Path destDir = workDir(inputDir, subdir);
|
||||
Files.createDirectories(destDir);
|
||||
Files.move(
|
||||
claimed,
|
||||
uniqueTarget(destDir, claimed.getFileName().toString()),
|
||||
StandardCopyOption.ATOMIC_MOVE);
|
||||
} catch (IOException e) {
|
||||
return false;
|
||||
log.warn(
|
||||
"Could not move processed input {} to {}: {}", claimed, subdir, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static Path workDir(Path inputDir, String subdir) {
|
||||
return inputDir.resolve(WORK_SUBDIR).resolve(subdir);
|
||||
}
|
||||
|
||||
private static Resource fileResource(Path path) {
|
||||
String name = path.getFileName().toString();
|
||||
return new FileSystemResource(path.toFile()) {
|
||||
@@ -272,15 +144,27 @@ public class FolderInputSource implements InputSource {
|
||||
};
|
||||
}
|
||||
|
||||
record FolderConfig(Path directory, boolean snapshot, boolean recursive, boolean hashIdentity) {
|
||||
private static Path uniqueTarget(Path dir, String filename) {
|
||||
Path candidate = dir.resolve(filename);
|
||||
if (!Files.exists(candidate)) {
|
||||
return candidate;
|
||||
}
|
||||
int dot = filename.lastIndexOf('.');
|
||||
String base = dot < 0 ? filename : filename.substring(0, dot);
|
||||
String ext = dot < 0 ? "" : filename.substring(dot);
|
||||
for (int n = 1; ; n++) {
|
||||
Path next = dir.resolve(base + " (" + n + ")" + ext);
|
||||
if (!Files.exists(next)) {
|
||||
return next;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
record FolderConfig(Path directory, boolean snapshot) {
|
||||
|
||||
private static final String DIRECTORY_OPTION = "directory";
|
||||
private static final String MODE_OPTION = "mode";
|
||||
private static final String MODE_SNAPSHOT = "snapshot";
|
||||
private static final String RECURSIVE_OPTION = "recursive";
|
||||
private static final String IDENTITY_OPTION = "identity";
|
||||
private static final String IDENTITY_STAT = "stat";
|
||||
private static final String IDENTITY_HASH = "hash";
|
||||
|
||||
static FolderConfig from(Map<String, Object> options) {
|
||||
Object directory = options.get(DIRECTORY_OPTION);
|
||||
@@ -289,17 +173,7 @@ public class FolderInputSource implements InputSource {
|
||||
}
|
||||
Object mode = options.get(MODE_OPTION);
|
||||
boolean snapshot = mode != null && MODE_SNAPSHOT.equals(mode.toString());
|
||||
Object recursive = options.get(RECURSIVE_OPTION);
|
||||
boolean recurse = recursive != null && Boolean.parseBoolean(recursive.toString());
|
||||
Object identity = options.get(IDENTITY_OPTION);
|
||||
boolean hash = identity != null && IDENTITY_HASH.equals(identity.toString());
|
||||
if (identity != null
|
||||
&& !IDENTITY_STAT.equals(identity.toString())
|
||||
&& !IDENTITY_HASH.equals(identity.toString())) {
|
||||
throw new IllegalArgumentException(
|
||||
"folder input 'identity' must be 'stat' or 'hash'");
|
||||
}
|
||||
return new FolderConfig(Path.of(directory.toString()), snapshot, recurse, hash);
|
||||
return new FolderConfig(Path.of(directory.toString()), snapshot);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+2
-14
@@ -24,21 +24,9 @@ public interface InputSource {
|
||||
|
||||
/**
|
||||
* Resolve the spec into zero or more units of work, each carrying one run's files and a
|
||||
* completion hook. Empty list means nothing to run right now. Discovery is read-only - files
|
||||
* stay where the user put them; "already processed" is tracked through {@code ctx} (claim on
|
||||
* pickup, settle on completion, report what is present so stale ledger rows can be pruned).
|
||||
* completion hook. Empty list means nothing to run right now.
|
||||
*/
|
||||
List<ResolvedInput> resolve(InputSpec spec, ResolveContext ctx) throws IOException;
|
||||
|
||||
/**
|
||||
* Whether {@link #resolve} observes everything in the source (a complete listing) rather than
|
||||
* e.g. only what events surfaced. Presence cleanup of the ledger is skipped for the whole
|
||||
* policy unless every enabled source says true - wrongly pruning history would reprocess a
|
||||
* whole folder, while keeping a few stale rows costs nothing.
|
||||
*/
|
||||
default boolean listsExhaustively() {
|
||||
return true;
|
||||
}
|
||||
List<ResolvedInput> resolve(InputSpec spec) throws IOException;
|
||||
|
||||
/**
|
||||
* Filesystem dirs this source draws from, for the folder-watch trigger. Advisory: resolving is
|
||||
|
||||
-36
@@ -1,36 +0,0 @@
|
||||
package stirling.software.proprietary.policy.input;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
/**
|
||||
* A source's policy-scoped window onto the processed-file ledger for one sweep. Thread-safe and
|
||||
* valid for the lifetime of the work units the source issued ({@link #settle} fires from async run
|
||||
* completions).
|
||||
*/
|
||||
public interface ResolveContext {
|
||||
|
||||
/**
|
||||
* Atomically claim a file at its current version; true means this sweep runs it. A null {@code
|
||||
* contentHash} makes any gate change a new version; a non-null supplier is invoked at most
|
||||
* once, only on a gate mismatch, and a matching hash refreshes the stored gate instead of
|
||||
* reprocessing. Supplier exceptions propagate.
|
||||
*/
|
||||
boolean claim(String identity, String gate, Supplier<String> contentHash);
|
||||
|
||||
/** Record a claimed file's outcome at its final version ({@code finalContentHash} nullable). */
|
||||
void settle(String identity, String finalGate, String finalContentHash, boolean success);
|
||||
|
||||
/**
|
||||
* Whether every policy holding a ledger row for this identity has settled it DONE. Cross-policy
|
||||
* by design: consume-mode deletion is a consensus of all claimants, so a shared input is
|
||||
* removed only once nobody still needs it (in-flight, failed, and interrupted rows all veto).
|
||||
*/
|
||||
boolean allSettledDone(String identity);
|
||||
|
||||
/**
|
||||
* Report every identity present right now, readable or not; feeds presence cleanup of rows
|
||||
* whose file is gone.
|
||||
*/
|
||||
void reportPresent(Collection<String> identities);
|
||||
}
|
||||
-9
@@ -1,9 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
/**
|
||||
* A row's claim-relevant state as read by {@link ProcessedLedger#statesFor}: what a sweep observed
|
||||
* before deciding a claim. May be stale by the time the claim runs; every ledger transition
|
||||
* re-checks the observed state in its WHERE clause, so staleness defers a claim to a later sweep
|
||||
* rather than double-running one.
|
||||
*/
|
||||
public record ClaimState(ProcessedFileStatus status, String gate, String contentHash) {}
|
||||
-41
@@ -1,41 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.attribute.BasicFileAttributes;
|
||||
|
||||
import stirling.software.proprietary.billing.ContentHasher;
|
||||
|
||||
/**
|
||||
* The folder backend's identity and version scheme, shared by {@code FolderInputSource} and {@code
|
||||
* FolderOutputSink} so outputs are recorded under exactly the identity the next scan derives.
|
||||
* Directories are canonicalised with {@code toRealPath()} so symlinked aliases agree.
|
||||
*/
|
||||
public final class FolderIdentities {
|
||||
|
||||
private FolderIdentities() {}
|
||||
|
||||
/** Canonical form of a configured directory; resolves symlinks, so the dir must exist. */
|
||||
public static Path canonicalDir(Path dir) throws IOException {
|
||||
return dir.toRealPath();
|
||||
}
|
||||
|
||||
/** Identity of {@code file} under {@code dir}: its path re-rooted onto the canonical dir. */
|
||||
public static String identity(Path canonicalDir, Path dir, Path file) {
|
||||
return canonicalDir.resolve(dir.relativize(file)).normalize().toString();
|
||||
}
|
||||
|
||||
/** The cheap version gate: a change to content length or mtime means "look closer". */
|
||||
public static String statGate(Path file) throws IOException {
|
||||
BasicFileAttributes attributes = Files.readAttributes(file, BasicFileAttributes.class);
|
||||
return attributes.size() + ":" + attributes.lastModifiedTime().toMillis();
|
||||
}
|
||||
|
||||
/**
|
||||
* The strong version token: distinguishes a real change from a touch, at the cost of a read.
|
||||
*/
|
||||
public static String contentHash(Path file) throws IOException {
|
||||
return ContentHasher.sha256(file);
|
||||
}
|
||||
}
|
||||
-19
@@ -1,19 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
|
||||
import stirling.software.proprietary.billing.ContentHasher;
|
||||
|
||||
/**
|
||||
* Fixed-width key form of a source-owned identity, so any identity length fits the ledger's primary
|
||||
* key. Backend-agnostic: every source type's identities are keyed through here, which is why this
|
||||
* does not live with the folder backend's {@link FolderIdentities}.
|
||||
*/
|
||||
public final class IdentityHasher {
|
||||
|
||||
private IdentityHasher() {}
|
||||
|
||||
public static String identityHash(String identity) {
|
||||
return ContentHasher.sha256(identity.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
-229
@@ -1,229 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
/**
|
||||
* In-memory {@link ProcessedLedger} for tests and DB-less wiring; kept semantically identical to
|
||||
* {@code JpaProcessedLedger} by the shared contract test.
|
||||
*/
|
||||
public class InProcessProcessedLedger implements ProcessedLedger {
|
||||
|
||||
private final Map<String, Map<String, Row>> rowsByPolicy = new HashMap<>();
|
||||
private final Supplier<Long> nowMillis;
|
||||
|
||||
public InProcessProcessedLedger() {
|
||||
this(System::currentTimeMillis);
|
||||
}
|
||||
|
||||
public InProcessProcessedLedger(Supplier<Long> nowMillis) {
|
||||
this.nowMillis = nowMillis;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized Map<String, ClaimState> statesFor(
|
||||
String policyId, Collection<String> identities) {
|
||||
Map<String, Row> rows = rowsByPolicy.getOrDefault(policyId, Map.of());
|
||||
Map<String, ClaimState> states = new HashMap<>();
|
||||
for (String identity : identities) {
|
||||
Row row = rows.get(identity);
|
||||
if (row != null) {
|
||||
states.put(identity, new ClaimState(row.status, row.gate, row.contentHash));
|
||||
}
|
||||
}
|
||||
return states;
|
||||
}
|
||||
|
||||
// Single-lock store: the live row is never staler than any observed snapshot, so decide
|
||||
// against it directly; the conditional updates of the JPA ledger yield the same outcomes.
|
||||
@Override
|
||||
public synchronized boolean claim(
|
||||
String policyId,
|
||||
String identity,
|
||||
String gate,
|
||||
Supplier<String> contentHash,
|
||||
ClaimState observed) {
|
||||
Map<String, Row> rows = rowsByPolicy.computeIfAbsent(policyId, key -> new HashMap<>());
|
||||
long now = nowMillis.get();
|
||||
Row row = rows.get(identity);
|
||||
if (row == null) {
|
||||
String hash = contentHash == null ? null : contentHash.get();
|
||||
rows.put(identity, new Row(gate, hash, ProcessedFileStatus.PROCESSING, 1, now));
|
||||
return true;
|
||||
}
|
||||
if (row.status == ProcessedFileStatus.PROCESSING) {
|
||||
return false;
|
||||
}
|
||||
if (gate.equals(row.gate)) {
|
||||
if (row.status == ProcessedFileStatus.INTERRUPTED && row.attempts < MAX_ATTEMPTS) {
|
||||
row.status = ProcessedFileStatus.PROCESSING;
|
||||
row.attempts++;
|
||||
row.lastSeen = now;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (contentHash == null) {
|
||||
row.gate = gate;
|
||||
row.contentHash = null;
|
||||
row.status = ProcessedFileStatus.PROCESSING;
|
||||
row.attempts = 1;
|
||||
row.lastSeen = now;
|
||||
return true;
|
||||
}
|
||||
String hash = contentHash.get();
|
||||
if (Objects.equals(hash, row.contentHash)) {
|
||||
if (row.status == ProcessedFileStatus.INTERRUPTED && row.attempts < MAX_ATTEMPTS) {
|
||||
row.gate = gate;
|
||||
row.status = ProcessedFileStatus.PROCESSING;
|
||||
row.attempts++;
|
||||
row.lastSeen = now;
|
||||
return true;
|
||||
}
|
||||
if (row.status != ProcessedFileStatus.INTERRUPTED) {
|
||||
row.gate = gate;
|
||||
row.lastSeen = now;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
row.gate = gate;
|
||||
row.contentHash = hash;
|
||||
row.status = ProcessedFileStatus.PROCESSING;
|
||||
row.attempts = 1;
|
||||
row.lastSeen = now;
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void settle(
|
||||
String policyId,
|
||||
String identity,
|
||||
String finalGate,
|
||||
String finalContentHash,
|
||||
boolean success) {
|
||||
upsertSettled(
|
||||
policyId,
|
||||
identity,
|
||||
finalGate,
|
||||
finalContentHash,
|
||||
success ? ProcessedFileStatus.DONE : ProcessedFileStatus.ERROR);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void recordOutput(
|
||||
String policyId, String identity, String gate, String contentHash) {
|
||||
upsertSettled(policyId, identity, gate, contentHash, ProcessedFileStatus.DONE);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void forgetOutput(String policyId, String identity, String gate) {
|
||||
Map<String, Row> rows = rowsByPolicy.get(policyId);
|
||||
if (rows == null) {
|
||||
return;
|
||||
}
|
||||
Row row = rows.get(identity);
|
||||
if (row != null && row.status == ProcessedFileStatus.DONE && gate.equals(row.gate)) {
|
||||
rows.remove(identity);
|
||||
}
|
||||
}
|
||||
|
||||
private void upsertSettled(
|
||||
String policyId,
|
||||
String identity,
|
||||
String gate,
|
||||
String contentHash,
|
||||
ProcessedFileStatus status) {
|
||||
Map<String, Row> rows = rowsByPolicy.computeIfAbsent(policyId, key -> new HashMap<>());
|
||||
long now = nowMillis.get();
|
||||
Row row = rows.get(identity);
|
||||
if (row == null) {
|
||||
rows.put(identity, new Row(gate, contentHash, status, 1, now));
|
||||
return;
|
||||
}
|
||||
row.gate = gate;
|
||||
row.contentHash = contentHash;
|
||||
row.status = status;
|
||||
row.lastSeen = now;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized boolean allSettledDone(String identity) {
|
||||
for (Map<String, Row> rows : rowsByPolicy.values()) {
|
||||
Row row = rows.get(identity);
|
||||
if (row != null && row.status != ProcessedFileStatus.DONE) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void markSeen(String policyId, Collection<String> identities) {
|
||||
Map<String, Row> rows = rowsByPolicy.get(policyId);
|
||||
if (rows == null) {
|
||||
return;
|
||||
}
|
||||
long now = nowMillis.get();
|
||||
for (String identity : identities) {
|
||||
Row row = rows.get(identity);
|
||||
if (row != null) {
|
||||
row.lastSeen = now;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized int deleteUnseen(String policyId, long seenSinceMillis) {
|
||||
Map<String, Row> rows = rowsByPolicy.get(policyId);
|
||||
if (rows == null) {
|
||||
return 0;
|
||||
}
|
||||
int before = rows.size();
|
||||
rows.values()
|
||||
.removeIf(
|
||||
row ->
|
||||
row.lastSeen < seenSinceMillis
|
||||
&& row.status != ProcessedFileStatus.PROCESSING);
|
||||
return before - rows.size();
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void clearPolicy(String policyId) {
|
||||
rowsByPolicy.remove(policyId);
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void recoverInterrupted() {
|
||||
for (Map<String, Row> rows : rowsByPolicy.values()) {
|
||||
for (Row row : rows.values()) {
|
||||
if (row.status == ProcessedFileStatus.PROCESSING) {
|
||||
row.status = ProcessedFileStatus.INTERRUPTED;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static final class Row {
|
||||
private String gate;
|
||||
private String contentHash;
|
||||
private ProcessedFileStatus status;
|
||||
private int attempts;
|
||||
private long lastSeen;
|
||||
|
||||
private Row(
|
||||
String gate,
|
||||
String contentHash,
|
||||
ProcessedFileStatus status,
|
||||
int attempts,
|
||||
long lastSeen) {
|
||||
this.gate = gate;
|
||||
this.contentHash = contentHash;
|
||||
this.status = status;
|
||||
this.attempts = attempts;
|
||||
this.lastSeen = lastSeen;
|
||||
}
|
||||
}
|
||||
}
|
||||
-212
@@ -1,212 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
import org.springframework.boot.context.event.ApplicationReadyEvent;
|
||||
import org.springframework.context.event.EventListener;
|
||||
import org.springframework.dao.DataIntegrityViolationException;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
/**
|
||||
* Durable {@link ProcessedLedger}; the runtime bean. A fresh claim is a flushed insert so a
|
||||
* concurrent winner surfaces as a constraint violation; every other transition is a conditional
|
||||
* update that re-checks the observed state, so a lost race reports 0 rows and the caller skips.
|
||||
* Boot recovery assumes the single node the folder-watch trigger assumes: runs live in memory, so
|
||||
* after a restart every PROCESSING row is stale.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@ConditionalOnBooleanProperty(name = "policies.enabled")
|
||||
public class JpaProcessedLedger implements ProcessedLedger {
|
||||
|
||||
private static final int STAMP_CHUNK = 500;
|
||||
|
||||
private final ProcessedFileRepository repository;
|
||||
private final Supplier<Long> nowMillis;
|
||||
|
||||
@Autowired
|
||||
public JpaProcessedLedger(ProcessedFileRepository repository) {
|
||||
this(repository, System::currentTimeMillis);
|
||||
}
|
||||
|
||||
// Clock seam so tests can pin "now"; the runtime bean uses the wall clock above.
|
||||
JpaProcessedLedger(ProcessedFileRepository repository, Supplier<Long> nowMillis) {
|
||||
this.repository = repository;
|
||||
this.nowMillis = nowMillis;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, ClaimState> statesFor(String policyId, Collection<String> identities) {
|
||||
if (identities.isEmpty()) {
|
||||
return Map.of();
|
||||
}
|
||||
Map<String, String> identityByHash = new HashMap<>();
|
||||
for (String identity : identities) {
|
||||
identityByHash.put(IdentityHasher.identityHash(identity), identity);
|
||||
}
|
||||
Map<String, ClaimState> states = new HashMap<>();
|
||||
List<String> hashes = List.copyOf(identityByHash.keySet());
|
||||
for (int from = 0; from < hashes.size(); from += STAMP_CHUNK) {
|
||||
List<ProcessedFileEntity> rows =
|
||||
repository.findByPolicyIdAndIdentityHashIn(
|
||||
policyId,
|
||||
hashes.subList(from, Math.min(from + STAMP_CHUNK, hashes.size())));
|
||||
for (ProcessedFileEntity row : rows) {
|
||||
states.put(
|
||||
identityByHash.get(row.getIdentityHash()),
|
||||
new ClaimState(row.getStatus(), row.getSignature(), row.getContentHash()));
|
||||
}
|
||||
}
|
||||
return states;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean claim(
|
||||
String policyId,
|
||||
String identity,
|
||||
String gate,
|
||||
Supplier<String> contentHash,
|
||||
ClaimState observed) {
|
||||
String identityHash = IdentityHasher.identityHash(identity);
|
||||
long now = nowMillis.get();
|
||||
if (observed == null) {
|
||||
try {
|
||||
repository.saveAndFlush(
|
||||
new ProcessedFileEntity(
|
||||
policyId,
|
||||
identityHash,
|
||||
identity,
|
||||
gate,
|
||||
contentHash == null ? null : contentHash.get(),
|
||||
ProcessedFileStatus.PROCESSING,
|
||||
now));
|
||||
return true;
|
||||
} catch (DataIntegrityViolationException concurrentClaim) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if (observed.status() == ProcessedFileStatus.PROCESSING) {
|
||||
return false;
|
||||
}
|
||||
if (gate.equals(observed.gate())) {
|
||||
if (observed.status() == ProcessedFileStatus.INTERRUPTED) {
|
||||
return repository.retryInterruptedAtGate(
|
||||
policyId, identityHash, gate, MAX_ATTEMPTS, now)
|
||||
> 0;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (contentHash == null) {
|
||||
return repository.reclaimAtNewGate(policyId, identityHash, gate, now) > 0;
|
||||
}
|
||||
String hash = contentHash.get();
|
||||
if (hash.equals(observed.contentHash())) {
|
||||
if (observed.status() == ProcessedFileStatus.INTERRUPTED) {
|
||||
return repository.retryInterruptedSameContent(
|
||||
policyId, identityHash, gate, hash, MAX_ATTEMPTS, now)
|
||||
> 0;
|
||||
}
|
||||
repository.refreshGate(policyId, identityHash, gate, hash, now);
|
||||
return false;
|
||||
}
|
||||
return repository.reclaimAtNewContent(policyId, identityHash, gate, hash, now) > 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void settle(
|
||||
String policyId,
|
||||
String identity,
|
||||
String finalGate,
|
||||
String finalContentHash,
|
||||
boolean success) {
|
||||
upsertSettled(
|
||||
policyId,
|
||||
identity,
|
||||
finalGate,
|
||||
finalContentHash,
|
||||
success ? ProcessedFileStatus.DONE : ProcessedFileStatus.ERROR);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void recordOutput(String policyId, String identity, String gate, String contentHash) {
|
||||
upsertSettled(policyId, identity, gate, contentHash, ProcessedFileStatus.DONE);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void forgetOutput(String policyId, String identity, String gate) {
|
||||
repository.deleteDoneAt(policyId, IdentityHasher.identityHash(identity), gate);
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle-or-insert: the row may have been presence-cleaned mid-run, and an output row may be
|
||||
* brand new.
|
||||
*/
|
||||
private void upsertSettled(
|
||||
String policyId,
|
||||
String identity,
|
||||
String gate,
|
||||
String contentHash,
|
||||
ProcessedFileStatus status) {
|
||||
String identityHash = IdentityHasher.identityHash(identity);
|
||||
long now = nowMillis.get();
|
||||
if (repository.settle(policyId, identityHash, gate, contentHash, status, now) > 0) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
ProcessedFileEntity row =
|
||||
new ProcessedFileEntity(
|
||||
policyId, identityHash, identity, gate, contentHash, status, now);
|
||||
repository.saveAndFlush(row);
|
||||
} catch (DataIntegrityViolationException concurrentInsert) {
|
||||
repository.settle(policyId, identityHash, gate, contentHash, status, now);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean allSettledDone(String identity) {
|
||||
return !repository.existsByIdentityHashAndStatusNot(
|
||||
IdentityHasher.identityHash(identity), ProcessedFileStatus.DONE);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void markSeen(String policyId, Collection<String> identities) {
|
||||
if (identities.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
List<String> hashes = identities.stream().map(IdentityHasher::identityHash).toList();
|
||||
long now = nowMillis.get();
|
||||
for (int from = 0; from < hashes.size(); from += STAMP_CHUNK) {
|
||||
repository.stampSeen(
|
||||
policyId,
|
||||
hashes.subList(from, Math.min(from + STAMP_CHUNK, hashes.size())),
|
||||
now);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int deleteUnseen(String policyId, long seenSinceMillis) {
|
||||
return repository.deleteUnseen(policyId, seenSinceMillis);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void clearPolicy(String policyId) {
|
||||
repository.deleteByPolicy(policyId);
|
||||
}
|
||||
|
||||
@Override
|
||||
@EventListener(ApplicationReadyEvent.class)
|
||||
public void recoverInterrupted() {
|
||||
int recovered = repository.markAllProcessingInterrupted(nowMillis.get());
|
||||
if (recovered > 0) {
|
||||
log.info("Recovered {} policy input file(s) interrupted by shutdown", recovered);
|
||||
}
|
||||
}
|
||||
}
|
||||
-106
@@ -1,106 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.io.Serializable;
|
||||
|
||||
import org.springframework.data.domain.Persistable;
|
||||
|
||||
import jakarta.persistence.Column;
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.EnumType;
|
||||
import jakarta.persistence.Enumerated;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.IdClass;
|
||||
import jakarta.persistence.Index;
|
||||
import jakarta.persistence.Table;
|
||||
import jakarta.persistence.Transient;
|
||||
|
||||
import lombok.Getter;
|
||||
import lombok.NoArgsConstructor;
|
||||
import lombok.Setter;
|
||||
|
||||
/**
|
||||
* One processed-file ledger row: the version a policy last settled a file at, and where it is in
|
||||
* the claim lifecycle. Keyed by SHA-256 of the source-owned identity so any identity length fits a
|
||||
* fixed-width index. {@code isNew} is always true: the entity is only saved for fresh inserts
|
||||
* (everything else is a conditional update), so a lost insert race surfaces as a constraint
|
||||
* violation rather than a silent merge.
|
||||
*/
|
||||
@Entity
|
||||
@Table(
|
||||
name = "policy_processed_files",
|
||||
indexes = {
|
||||
// presence cleanup: delete this policy's rows unseen since the sweep began
|
||||
@Index(name = "idx_processed_files_policy_seen", columnList = "policy_id, last_seen"),
|
||||
// cross-policy deletion consensus: existsByIdentityHashAndStatusNot filters
|
||||
// identity_hash on its own, so it cannot ride the (policy_id, identity_hash) PK
|
||||
@Index(name = "idx_processed_files_identity", columnList = "identity_hash")
|
||||
})
|
||||
@IdClass(ProcessedFileId.class)
|
||||
@NoArgsConstructor
|
||||
@Getter
|
||||
@Setter
|
||||
public class ProcessedFileEntity implements Serializable, Persistable<ProcessedFileId> {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
@Id
|
||||
@Column(name = "policy_id")
|
||||
private String policyId;
|
||||
|
||||
@Id
|
||||
@Column(name = "identity_hash", length = 64)
|
||||
private String identityHash;
|
||||
|
||||
@Column(name = "identity", length = 4096)
|
||||
private String identity;
|
||||
|
||||
@Column(name = "signature")
|
||||
private String signature;
|
||||
|
||||
@Column(name = "content_hash", length = 64)
|
||||
private String contentHash;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "status", length = 16)
|
||||
private ProcessedFileStatus status;
|
||||
|
||||
@Column(name = "attempts")
|
||||
private int attempts;
|
||||
|
||||
@Column(name = "last_seen")
|
||||
private long lastSeen;
|
||||
|
||||
@Column(name = "updated_at")
|
||||
private long updatedAt;
|
||||
|
||||
public ProcessedFileEntity(
|
||||
String policyId,
|
||||
String identityHash,
|
||||
String identity,
|
||||
String signature,
|
||||
String contentHash,
|
||||
ProcessedFileStatus status,
|
||||
long nowMillis) {
|
||||
this.policyId = policyId;
|
||||
this.identityHash = identityHash;
|
||||
this.identity = identity;
|
||||
this.signature = signature;
|
||||
this.contentHash = contentHash;
|
||||
this.status = status;
|
||||
this.attempts = 1;
|
||||
this.lastSeen = nowMillis;
|
||||
this.updatedAt = nowMillis;
|
||||
}
|
||||
|
||||
@Override
|
||||
@Transient
|
||||
public ProcessedFileId getId() {
|
||||
return new ProcessedFileId(policyId, identityHash);
|
||||
}
|
||||
|
||||
@Override
|
||||
@Transient
|
||||
public boolean isNew() {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
-37
@@ -1,37 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.Objects;
|
||||
|
||||
/** Composite key for {@link ProcessedFileEntity}: one row per policy per file identity. */
|
||||
public class ProcessedFileId implements Serializable {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private String policyId;
|
||||
private String identityHash;
|
||||
|
||||
public ProcessedFileId() {}
|
||||
|
||||
public ProcessedFileId(String policyId, String identityHash) {
|
||||
this.policyId = policyId;
|
||||
this.identityHash = identityHash;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(Object o) {
|
||||
if (this == o) {
|
||||
return true;
|
||||
}
|
||||
if (!(o instanceof ProcessedFileId other)) {
|
||||
return false;
|
||||
}
|
||||
return Objects.equals(policyId, other.policyId)
|
||||
&& Objects.equals(identityHash, other.identityHash);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode() {
|
||||
return Objects.hash(policyId, identityHash);
|
||||
}
|
||||
}
|
||||
-195
@@ -1,195 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Modifying;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
import org.springframework.stereotype.Repository;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
/**
|
||||
* Conditional updates for the processed-file ledger: each claim variant re-checks in its WHERE
|
||||
* clause the state it was decided against, so a racing claim loses cleanly with 0 rows updated.
|
||||
* Transactional per call so the ledger can run them without an enclosing transaction.
|
||||
*/
|
||||
@Repository
|
||||
public interface ProcessedFileRepository
|
||||
extends JpaRepository<ProcessedFileEntity, ProcessedFileId> {
|
||||
|
||||
/**
|
||||
* Re-claim a settled row at a new gate without content verification; clears the stored hash,
|
||||
* which described content this claim never checked.
|
||||
*/
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
|
||||
+ " e.signature = :gate, e.contentHash = null, e.attempts = 1,"
|
||||
+ " e.lastSeen = :now, e.updatedAt = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
|
||||
+ " and e.status <>"
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING"
|
||||
+ " and e.signature <> :gate")
|
||||
int reclaimAtNewGate(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate,
|
||||
@Param("now") long now);
|
||||
|
||||
/** Re-claim a settled row whose content verifiably changed (or was never hashed). */
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
|
||||
+ " e.signature = :gate, e.contentHash = :contentHash, e.attempts = 1,"
|
||||
+ " e.lastSeen = :now, e.updatedAt = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
|
||||
+ " and e.status <>"
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING"
|
||||
+ " and (e.contentHash is null or e.contentHash <> :contentHash)")
|
||||
int reclaimAtNewContent(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate,
|
||||
@Param("contentHash") String contentHash,
|
||||
@Param("now") long now);
|
||||
|
||||
/** The gate moved but the content did not: track the new gate without changing status. */
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.signature = :gate, e.lastSeen = :now,"
|
||||
+ " e.updatedAt = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
|
||||
+ " and e.status <>"
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING"
|
||||
+ " and e.contentHash = :contentHash and e.signature <> :gate")
|
||||
int refreshGate(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate,
|
||||
@Param("contentHash") String contentHash,
|
||||
@Param("now") long now);
|
||||
|
||||
/** Bounded retry of an INTERRUPTED row at the same gate. */
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
|
||||
+ " e.attempts = e.attempts + 1, e.lastSeen = :now, e.updatedAt = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
|
||||
+ " and e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.INTERRUPTED"
|
||||
+ " and e.signature = :gate and e.attempts < :maxAttempts")
|
||||
int retryInterruptedAtGate(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate,
|
||||
@Param("maxAttempts") int maxAttempts,
|
||||
@Param("now") long now);
|
||||
|
||||
/** Bounded retry of an INTERRUPTED row whose gate moved but whose content is unchanged. */
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING,"
|
||||
+ " e.signature = :gate, e.attempts = e.attempts + 1, e.lastSeen = :now,"
|
||||
+ " e.updatedAt = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash = :identityHash"
|
||||
+ " and e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.INTERRUPTED"
|
||||
+ " and e.contentHash = :contentHash and e.attempts < :maxAttempts")
|
||||
int retryInterruptedSameContent(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate,
|
||||
@Param("contentHash") String contentHash,
|
||||
@Param("maxAttempts") int maxAttempts,
|
||||
@Param("now") long now);
|
||||
|
||||
/**
|
||||
* Unconditional settle (only the claiming run settles a row); returns 0 when the row was
|
||||
* removed mid-run so the caller re-inserts.
|
||||
*/
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.status = :status, e.signature = :gate,"
|
||||
+ " e.contentHash = :contentHash, e.lastSeen = :now, e.updatedAt = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash = :identityHash")
|
||||
int settle(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate,
|
||||
@Param("contentHash") String contentHash,
|
||||
@Param("status") ProcessedFileStatus status,
|
||||
@Param("now") long now);
|
||||
|
||||
/** Whether any policy's row at this identity is in a state other than {@code status}. */
|
||||
boolean existsByIdentityHashAndStatusNot(String identityHash, ProcessedFileStatus status);
|
||||
|
||||
/** One policy's rows across a chunk of identity hashes, for a sweep's claim snapshot. */
|
||||
List<ProcessedFileEntity> findByPolicyIdAndIdentityHashIn(
|
||||
String policyId, Collection<String> identityHashes);
|
||||
|
||||
/**
|
||||
* Remove an output record whose rename never landed, only while still settled exactly as
|
||||
* recorded; a row a claim has since taken over is left alone.
|
||||
*/
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"delete from ProcessedFileEntity e where e.policyId = :policyId"
|
||||
+ " and e.identityHash = :identityHash and e.signature = :gate"
|
||||
+ " and e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.DONE")
|
||||
int deleteDoneAt(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHash") String identityHash,
|
||||
@Param("gate") String gate);
|
||||
|
||||
/** Stamp presence for the given identities; chunked by the caller for very large folders. */
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.lastSeen = :now"
|
||||
+ " where e.policyId = :policyId and e.identityHash in :identityHashes")
|
||||
int stampSeen(
|
||||
@Param("policyId") String policyId,
|
||||
@Param("identityHashes") Collection<String> identityHashes,
|
||||
@Param("now") long now);
|
||||
|
||||
/**
|
||||
* Presence cleanup: remove rows not stamped since the sweep began, keeping in-flight claims.
|
||||
*/
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"delete from ProcessedFileEntity e where e.policyId = :policyId"
|
||||
+ " and e.lastSeen < :cutoff and e.status <>"
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING")
|
||||
int deleteUnseen(@Param("policyId") String policyId, @Param("cutoff") long cutoff);
|
||||
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query("delete from ProcessedFileEntity e where e.policyId = :policyId")
|
||||
int deleteByPolicy(@Param("policyId") String policyId);
|
||||
|
||||
/** Boot recovery: after a restart every PROCESSING row is stale (single node). */
|
||||
@Modifying
|
||||
@Transactional
|
||||
@Query(
|
||||
"update ProcessedFileEntity e set e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.INTERRUPTED,"
|
||||
+ " e.updatedAt = :now"
|
||||
+ " where e.status ="
|
||||
+ " stirling.software.proprietary.policy.ledger.ProcessedFileStatus.PROCESSING")
|
||||
int markAllProcessingInterrupted(@Param("now") long now);
|
||||
}
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
/** Lifecycle of one {@code (policy, file)} ledger row. */
|
||||
public enum ProcessedFileStatus {
|
||||
|
||||
/** Claimed; a run is in flight. */
|
||||
PROCESSING,
|
||||
|
||||
/** Run completed at this version. */
|
||||
DONE,
|
||||
|
||||
/** Run failed; skipped until the file changes (clear-history is the manual retry). */
|
||||
ERROR,
|
||||
|
||||
/** Was PROCESSING when the JVM died; retried a bounded number of times. */
|
||||
INTERRUPTED
|
||||
}
|
||||
-100
@@ -1,100 +0,0 @@
|
||||
package stirling.software.proprietary.policy.ledger;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
/**
|
||||
* Remembers which files a policy has processed, one row per {@code (policy, identity)}, so sources
|
||||
* track files in place. Identities are opaque source-owned strings; versions are two-tier: a cheap
|
||||
* gate compared every sweep plus an optional content hash consulted only when the gate moves.
|
||||
* Presence reconciliation ({@link #markSeen} + {@link #deleteUnseen}) keeps the table bounded.
|
||||
*/
|
||||
public interface ProcessedLedger {
|
||||
|
||||
/**
|
||||
* Claims of one version before an {@link ProcessedFileStatus#INTERRUPTED} row stops retrying.
|
||||
*/
|
||||
int MAX_ATTEMPTS = 3;
|
||||
|
||||
/**
|
||||
* One-query snapshot of the rows for these identities, keyed by identity; identities with no
|
||||
* row are absent. Feeds the {@code observed} parameter of {@link #claim(String, String, String,
|
||||
* Supplier, ClaimState)} so a sweep decides its claims without a per-file lookup.
|
||||
*/
|
||||
Map<String, ClaimState> statesFor(String policyId, Collection<String> identities);
|
||||
|
||||
/**
|
||||
* Atomically claim a file at its current version, deciding against {@code observed} (this row's
|
||||
* entry from {@link #statesFor}; null means no row was seen); true means this caller runs it. A
|
||||
* stale {@code observed} cannot double-claim - every transition re-checks the observed state,
|
||||
* so a lost race skips until a later sweep. A null {@code contentHash} makes any gate change a
|
||||
* new version; a non-null supplier is invoked at most once, only on a gate mismatch, and a
|
||||
* matching hash refreshes the stored gate instead of reprocessing. Supplier exceptions
|
||||
* propagate.
|
||||
*/
|
||||
boolean claim(
|
||||
String policyId,
|
||||
String identity,
|
||||
String gate,
|
||||
Supplier<String> contentHash,
|
||||
ClaimState observed);
|
||||
|
||||
/** Snapshot-then-claim convenience for a single file; sweeps batch via {@link #statesFor}. */
|
||||
default boolean claim(
|
||||
String policyId, String identity, String gate, Supplier<String> contentHash) {
|
||||
return claim(
|
||||
policyId,
|
||||
identity,
|
||||
gate,
|
||||
contentHash,
|
||||
statesFor(policyId, List.of(identity)).get(identity));
|
||||
}
|
||||
|
||||
/** Record a claimed file's outcome at its final version ({@code finalContentHash} nullable). */
|
||||
void settle(
|
||||
String policyId,
|
||||
String identity,
|
||||
String finalGate,
|
||||
String finalContentHash,
|
||||
boolean success);
|
||||
|
||||
/**
|
||||
* Record a produced file as {@link ProcessedFileStatus#DONE} so the policy skips its own
|
||||
* outputs. Must be called before the file is visible at this identity; other policies have no
|
||||
* row and still process it.
|
||||
*/
|
||||
void recordOutput(String policyId, String identity, String gate, String contentHash);
|
||||
|
||||
/**
|
||||
* Remove an output record whose file never became visible (its rename lost the name race to a
|
||||
* concurrent writer), so whatever file actually owns that identity is claimable at any version.
|
||||
* A no-op unless the row is still settled exactly as recorded, so a claim that took the row
|
||||
* over in the meantime is left alone.
|
||||
*/
|
||||
void forgetOutput(String policyId, String identity, String gate);
|
||||
|
||||
/**
|
||||
* Whether every row at this identity - across all policies, by design - is {@link
|
||||
* ProcessedFileStatus#DONE}. Consume-mode deletion gates on this so a shared input is removed
|
||||
* only once every claimant has processed it; in-flight, failed, and interrupted rows all veto,
|
||||
* parking the file. Vacuously true when no rows exist.
|
||||
*/
|
||||
boolean allSettledDone(String identity);
|
||||
|
||||
/** Stamp presence for every identity a full-listing sweep observed. */
|
||||
void markSeen(String policyId, Collection<String> identities);
|
||||
|
||||
/**
|
||||
* Remove rows not seen since {@code seenSinceMillis}, keeping in-flight claims. Only call after
|
||||
* every enabled source listed completely; returns the number of rows removed.
|
||||
*/
|
||||
int deleteUnseen(String policyId, long seenSinceMillis);
|
||||
|
||||
/** Forget everything for a policy. */
|
||||
void clearPolicy(String policyId);
|
||||
|
||||
/** Boot recovery: flip stale in-flight claims to {@link ProcessedFileStatus#INTERRUPTED}. */
|
||||
void recoverInterrupted();
|
||||
}
|
||||
+12
-116
@@ -2,18 +2,11 @@ package stirling.software.proprietary.policy.output;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.file.FileAlreadyExistsException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.security.DigestOutputStream;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
import org.apache.commons.io.FilenameUtils;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
@@ -26,18 +19,14 @@ import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.model.job.ResultFile;
|
||||
import stirling.software.proprietary.billing.ContentHasher;
|
||||
import stirling.software.proprietary.policy.config.FolderAccessGuard;
|
||||
import stirling.software.proprietary.policy.ledger.FolderIdentities;
|
||||
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
|
||||
import stirling.software.proprietary.policy.model.OutputSpec;
|
||||
|
||||
/**
|
||||
* Writes a run's outputs to the {@code directory} given in the {@link OutputSpec}. Each output is
|
||||
* staged under a hidden {@code .stirling/tmp} dir, recorded in the processed-file ledger, then
|
||||
* atomically renamed into place, so the producing policy's row exists before the file is
|
||||
* discoverable and half-written outputs are never visible. Returned {@link ResultFile}s carry a
|
||||
* synthetic id since the deliverable is the file on disk, not a {@code FileStorage} entry.
|
||||
* Writes a run's outputs to the {@code directory} given in the {@link OutputSpec}. Files are
|
||||
* streamed (not buffered) and uniquely named to avoid clobbering. Returned {@link ResultFile}s
|
||||
* carry a synthetic id since the deliverable is the file on disk, not a {@code FileStorage} entry,
|
||||
* so folder outputs are not downloadable via {@code /files/{id}}.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@@ -48,11 +37,7 @@ public class FolderOutputSink implements PolicyOutputSink {
|
||||
static final String TYPE = FolderAccessGuard.FOLDER_TYPE;
|
||||
static final String DIRECTORY_OPTION = "directory";
|
||||
|
||||
// Staging entries are renamed away within one delivery; anything older is a crash leftover.
|
||||
private static final Duration STALE_TMP_AGE = Duration.ofDays(1);
|
||||
|
||||
private final FolderAccessGuard accessGuard;
|
||||
private final ProcessedLedger processedLedger;
|
||||
|
||||
@Override
|
||||
public String type() {
|
||||
@@ -70,25 +55,20 @@ public class FolderOutputSink implements PolicyOutputSink {
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ResultFile> deliver(
|
||||
OutputDelivery delivery, List<Resource> outputs, OutputSpec spec) throws IOException {
|
||||
public List<ResultFile> deliver(String runId, List<Resource> outputs, OutputSpec spec)
|
||||
throws IOException {
|
||||
Path targetDir = accessGuard.requirePermitted(directoryOf(spec));
|
||||
Files.createDirectories(targetDir);
|
||||
Path canonicalDir = FolderIdentities.canonicalDir(targetDir);
|
||||
Path tmpDir = canonicalDir.resolve(".stirling").resolve("tmp");
|
||||
Files.createDirectories(tmpDir);
|
||||
sweepStaleTmp(tmpDir);
|
||||
|
||||
List<ResultFile> results = new ArrayList<>();
|
||||
for (int i = 0; i < outputs.size(); i++) {
|
||||
Resource resource = outputs.get(i);
|
||||
String name = safeName(resource.getFilename(), i);
|
||||
Path staged = tmpDir.resolve(UUID.randomUUID().toString());
|
||||
String contentHash = stage(resource, staged, delivery.policyId() != null);
|
||||
long size = Files.size(staged);
|
||||
// Size and mtime survive the rename.
|
||||
String gate = FolderIdentities.statGate(staged);
|
||||
Path target = moveIntoPlace(delivery, canonicalDir, name, staged, gate, contentHash);
|
||||
Path target = uniqueTarget(targetDir, name);
|
||||
try (InputStream is = resource.getInputStream()) {
|
||||
Files.copy(is, target);
|
||||
}
|
||||
long size = Files.size(target);
|
||||
String contentType =
|
||||
MediaTypeFactory.getMediaType(name)
|
||||
.orElse(MediaType.APPLICATION_OCTET_STREAM)
|
||||
@@ -100,95 +80,11 @@ public class FolderOutputSink implements PolicyOutputSink {
|
||||
.contentType(contentType)
|
||||
.fileSize(size)
|
||||
.build());
|
||||
log.debug("Wrote policy run {} output to {}", delivery.runId(), target);
|
||||
log.debug("Wrote policy run {} output to {}", runId, target);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream the output to its staging path. For a recorded delivery (stored policy) the content
|
||||
* hash is digested in the same pass, so the ledger gets both version tiers without re-reading a
|
||||
* possibly huge output; ad-hoc runs record nothing and skip the digest entirely.
|
||||
*/
|
||||
private static String stage(Resource resource, Path staged, boolean hashed) throws IOException {
|
||||
if (!hashed) {
|
||||
try (InputStream is = resource.getInputStream()) {
|
||||
Files.copy(is, staged);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
MessageDigest digest = ContentHasher.newSha256();
|
||||
try (InputStream is = resource.getInputStream();
|
||||
DigestOutputStream out =
|
||||
new DigestOutputStream(Files.newOutputStream(staged), digest)) {
|
||||
is.transferTo(out);
|
||||
}
|
||||
return ContentHasher.toHex(digest.digest());
|
||||
}
|
||||
|
||||
/**
|
||||
* The ledger row must exist before the file is visible at its final path, or a sweep could
|
||||
* claim the producing policy's own output in the gap. Losing the chosen name to a concurrent
|
||||
* writer forgets the just-recorded row - whatever file actually owns that name must stay
|
||||
* claimable at any version - then re-picks.
|
||||
*/
|
||||
private Path moveIntoPlace(
|
||||
OutputDelivery delivery,
|
||||
Path dir,
|
||||
String name,
|
||||
Path staged,
|
||||
String gate,
|
||||
String contentHash)
|
||||
throws IOException {
|
||||
while (true) {
|
||||
Path target = uniqueTarget(dir, name);
|
||||
if (delivery.policyId() != null) {
|
||||
processedLedger.recordOutput(
|
||||
delivery.policyId(), target.toString(), gate, contentHash);
|
||||
}
|
||||
try {
|
||||
Files.move(staged, target, StandardCopyOption.ATOMIC_MOVE);
|
||||
return target;
|
||||
} catch (FileAlreadyExistsException raced) {
|
||||
if (delivery.policyId() != null) {
|
||||
processedLedger.forgetOutput(delivery.policyId(), target.toString(), gate);
|
||||
}
|
||||
log.debug("Output name {} taken concurrently; re-picking", target);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Best-effort removal of staging leftovers from crashed deliveries. */
|
||||
private static void sweepStaleTmp(Path tmpDir) {
|
||||
Instant cutoff = Instant.now().minus(STALE_TMP_AGE);
|
||||
try (Stream<Path> entries = Files.list(tmpDir)) {
|
||||
entries.filter(Files::isRegularFile)
|
||||
.filter(
|
||||
entry -> {
|
||||
try {
|
||||
return Files.getLastModifiedTime(entry)
|
||||
.toInstant()
|
||||
.isBefore(cutoff);
|
||||
} catch (IOException e) {
|
||||
return false;
|
||||
}
|
||||
})
|
||||
.forEach(
|
||||
entry -> {
|
||||
try {
|
||||
Files.deleteIfExists(entry);
|
||||
} catch (IOException e) {
|
||||
log.debug(
|
||||
"Could not remove stale staging file {}: {}",
|
||||
entry,
|
||||
e.getMessage());
|
||||
}
|
||||
});
|
||||
} catch (IOException e) {
|
||||
log.debug("Could not sweep staging dir {}: {}", tmpDir, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static Path directoryOf(OutputSpec spec) {
|
||||
Object directory = spec.options().get(DIRECTORY_OPTION);
|
||||
if (directory == null || directory.toString().isBlank()) {
|
||||
|
||||
+2
-2
@@ -41,8 +41,8 @@ public class InlineOutputSink implements PolicyOutputSink {
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<ResultFile> deliver(
|
||||
OutputDelivery delivery, List<Resource> outputs, OutputSpec spec) throws IOException {
|
||||
public List<ResultFile> deliver(String runId, List<Resource> outputs, OutputSpec spec)
|
||||
throws IOException {
|
||||
List<ResultFile> results = new ArrayList<>();
|
||||
for (int i = 0; i < outputs.size(); i++) {
|
||||
Resource resource = outputs.get(i);
|
||||
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
package stirling.software.proprietary.policy.output;
|
||||
|
||||
/**
|
||||
* Context for one run's output delivery. {@code policyId} is null for ad-hoc pipelines; when
|
||||
* present, sinks record outputs in the processed-file ledger so the producing policy does not
|
||||
* re-ingest them.
|
||||
*/
|
||||
public record OutputDelivery(String runId, String policyId) {}
|
||||
+1
-1
@@ -25,6 +25,6 @@ public interface PolicyOutputSink {
|
||||
default void validate(OutputSpec spec) {}
|
||||
|
||||
/** Persist/deliver the output files and return their descriptors. */
|
||||
List<ResultFile> deliver(OutputDelivery delivery, List<Resource> outputs, OutputSpec spec)
|
||||
List<ResultFile> deliver(String runId, List<Resource> outputs, OutputSpec spec)
|
||||
throws IOException;
|
||||
}
|
||||
|
||||
+1
-34
@@ -1,6 +1,5 @@
|
||||
package stirling.software.proprietary.policy.store;
|
||||
|
||||
import java.util.Comparator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
@@ -17,8 +16,6 @@ import stirling.software.proprietary.policy.model.Policy;
|
||||
public class InProcessPolicyStore implements PolicyStore {
|
||||
|
||||
private final Map<String, Policy> policies = new ConcurrentHashMap<>();
|
||||
// Run-order position per policy id, mirroring JpaPolicyStore's sort_order column.
|
||||
private final Map<String, Integer> sortOrders = new ConcurrentHashMap<>();
|
||||
|
||||
@Override
|
||||
public Policy save(Policy policy) {
|
||||
@@ -38,25 +35,9 @@ public class InProcessPolicyStore implements PolicyStore {
|
||||
policy.output(),
|
||||
policy.teamId());
|
||||
policies.put(id, stored);
|
||||
// Existing policy keeps its position; a new one appends to the end of its team's queue.
|
||||
sortOrders.computeIfAbsent(id, key -> nextSortOrder(stored.teamId()));
|
||||
return stored;
|
||||
}
|
||||
|
||||
private int nextSortOrder(Long teamId) {
|
||||
return policies.values().stream()
|
||||
.filter(policy -> Objects.equals(policy.teamId(), teamId))
|
||||
.map(policy -> sortOrders.getOrDefault(policy.id(), 0))
|
||||
.max(Comparator.naturalOrder())
|
||||
.orElse(-1)
|
||||
+ 1;
|
||||
}
|
||||
|
||||
private Comparator<Policy> byRunOrder() {
|
||||
return Comparator.<Policy>comparingInt(policy -> sortOrders.getOrDefault(policy.id(), 0))
|
||||
.thenComparing(Policy::id);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<Policy> get(String id) {
|
||||
return Optional.ofNullable(policies.get(id));
|
||||
@@ -64,14 +45,13 @@ public class InProcessPolicyStore implements PolicyStore {
|
||||
|
||||
@Override
|
||||
public List<Policy> all() {
|
||||
return policies.values().stream().sorted(byRunOrder()).toList();
|
||||
return List.copyOf(policies.values());
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Policy> findByTeam(Long teamId) {
|
||||
return policies.values().stream()
|
||||
.filter(policy -> Objects.equals(policy.teamId(), teamId))
|
||||
.sorted(byRunOrder())
|
||||
.toList();
|
||||
}
|
||||
|
||||
@@ -84,21 +64,8 @@ public class InProcessPolicyStore implements PolicyStore {
|
||||
.toList();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void reorder(Long teamId, List<String> orderedIds) {
|
||||
int position = 0;
|
||||
for (String id : orderedIds) {
|
||||
Policy policy = policies.get(id);
|
||||
if (policy == null || !Objects.equals(policy.teamId(), teamId)) {
|
||||
continue;
|
||||
}
|
||||
sortOrders.put(id, position++);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean delete(String id) {
|
||||
sortOrders.remove(id);
|
||||
return policies.remove(id) != null;
|
||||
}
|
||||
}
|
||||
|
||||
+1
-42
@@ -1,13 +1,11 @@
|
||||
package stirling.software.proprietary.policy.store;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
@@ -28,7 +26,6 @@ public class JpaPolicyStore implements PolicyStore {
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
@Override
|
||||
@Transactional
|
||||
public Policy save(Policy policy) {
|
||||
String id =
|
||||
policy.id() == null || policy.id().isBlank()
|
||||
@@ -53,49 +50,11 @@ public class JpaPolicyStore implements PolicyStore {
|
||||
entity.setEnabled(stored.enabled());
|
||||
entity.setTriggerType(stored.trigger() == null ? null : stored.trigger().type());
|
||||
entity.setTeamId(stored.teamId());
|
||||
// Preserve an existing policy's run-order position; append a new one to the end of its
|
||||
// team's queue (max + 1), so setting up a policy adds it last by default.
|
||||
entity.setSortOrder(
|
||||
repository
|
||||
.findById(id)
|
||||
.map(PolicyEntity::getSortOrder)
|
||||
.orElseGet(() -> nextSortOrder(stored.teamId())));
|
||||
entity.setPolicyJson(objectMapper.writeValueAsString(stored));
|
||||
repository.save(entity);
|
||||
return stored;
|
||||
}
|
||||
|
||||
/**
|
||||
* Append position for a new policy: max(existing) + 1, computed under a pessimistic lock on the
|
||||
* team's rows (see {@link PolicyRepository#findByTeamForUpdate}) so two concurrent creates
|
||||
* can't both read the same max and assign a duplicate order. (A brand-new team has no rows to
|
||||
* lock; a rare simultaneous first-create there ties at 0 — harmless, since the ordering query
|
||||
* breaks ties by id and any later reorder normalises it.)
|
||||
*/
|
||||
private int nextSortOrder(Long teamId) {
|
||||
return repository.findByTeamForUpdate(teamId).stream()
|
||||
.map(entity -> entity.getSortOrder() == null ? 0 : entity.getSortOrder())
|
||||
.max(Integer::compareTo)
|
||||
.orElse(-1)
|
||||
+ 1;
|
||||
}
|
||||
|
||||
@Override
|
||||
@Transactional
|
||||
public void reorder(Long teamId, List<String> orderedIds) {
|
||||
int position = 0;
|
||||
for (String id : orderedIds) {
|
||||
PolicyEntity entity = repository.findById(id).orElse(null);
|
||||
// Ignore unknown ids and any policy outside the caller's team — a reorder can't reach
|
||||
// across teams.
|
||||
if (entity == null || !Objects.equals(entity.getTeamId(), teamId)) {
|
||||
continue;
|
||||
}
|
||||
entity.setSortOrder(position++);
|
||||
repository.save(entity);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<Policy> get(String id) {
|
||||
return repository.findById(id).map(this::toPolicy);
|
||||
@@ -103,7 +62,7 @@ public class JpaPolicyStore implements PolicyStore {
|
||||
|
||||
@Override
|
||||
public List<Policy> all() {
|
||||
return repository.findAllOrdered().stream().map(this::toPolicy).toList();
|
||||
return repository.findAll().stream().map(this::toPolicy).toList();
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
-8
@@ -47,14 +47,6 @@ public class PolicyEntity implements Serializable {
|
||||
@Column(name = "team_id")
|
||||
private Long teamId;
|
||||
|
||||
/**
|
||||
* Position in the team's run order (ascending). Team-wide and admin-editable; the per-trigger
|
||||
* order the UI shows is this single sequence filtered by trigger. New policies are appended
|
||||
* (max + 1). Nullable for pre-existing rows; treated as 0 when sorting.
|
||||
*/
|
||||
@Column(name = "sort_order")
|
||||
private Integer sortOrder;
|
||||
|
||||
@Column(name = "policy_json", columnDefinition = "text")
|
||||
private String policyJson;
|
||||
}
|
||||
|
||||
+4
-24
@@ -3,13 +3,10 @@ package stirling.software.proprietary.policy.store;
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Lock;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import jakarta.persistence.LockModeType;
|
||||
|
||||
@Repository
|
||||
public interface PolicyRepository extends JpaRepository<PolicyEntity, String> {
|
||||
|
||||
@@ -17,29 +14,12 @@ public interface PolicyRepository extends JpaRepository<PolicyEntity, String> {
|
||||
List<PolicyEntity> findByTriggerTypeAndEnabledTrue(String triggerType);
|
||||
|
||||
/**
|
||||
* Policies belonging to a team, in run order (ascending {@code sortOrder}; a null order sorts
|
||||
* first, id breaks ties for stability). A {@code null} teamId matches the rows with no team
|
||||
* (login-disabled / pre-team data), mirroring the in-memory team filter rather than the empty
|
||||
* result a plain {@code = null} would give.
|
||||
* Policies belonging to a team, loaded without scanning every team's rows. A {@code null}
|
||||
* teamId matches the rows with no team (login-disabled / pre-team data), mirroring the
|
||||
* in-memory team filter rather than the empty result a plain {@code = null} would give.
|
||||
*/
|
||||
@Query(
|
||||
"select p from PolicyEntity p where ((:teamId is null and p.teamId is null) or"
|
||||
+ " p.teamId = :teamId) order by coalesce(p.sortOrder, 0) asc, p.id asc")
|
||||
List<PolicyEntity> findByTeam(@Param("teamId") Long teamId);
|
||||
|
||||
/** All policies in run order — used when team scoping is off (login-disabled). */
|
||||
@Query("select p from PolicyEntity p order by coalesce(p.sortOrder, 0) asc, p.id asc")
|
||||
List<PolicyEntity> findAllOrdered();
|
||||
|
||||
/**
|
||||
* The team's policy rows, locked for the transaction (SELECT … FOR UPDATE). Appending a new
|
||||
* policy reads the max {@code sortOrder} from these under the lock, so two concurrent creates
|
||||
* serialize instead of both reading a stale max and assigning the same position. Must be called
|
||||
* inside a transaction.
|
||||
*/
|
||||
@Lock(LockModeType.PESSIMISTIC_WRITE)
|
||||
@Query(
|
||||
"select p from PolicyEntity p where (:teamId is null and p.teamId is null) or"
|
||||
+ " p.teamId = :teamId")
|
||||
List<PolicyEntity> findByTeamForUpdate(@Param("teamId") Long teamId);
|
||||
List<PolicyEntity> findByTeam(@Param("teamId") Long teamId);
|
||||
}
|
||||
|
||||
-7
@@ -21,13 +21,6 @@ public interface PolicyStore {
|
||||
/** Enabled policies with the given trigger type, for background triggers. */
|
||||
List<Policy> findByTriggerType(String triggerType);
|
||||
|
||||
/**
|
||||
* Set the team's run order from {@code orderedIds} (position → sortOrder). Only policies that
|
||||
* belong to {@code teamId} are touched; unknown/other-team ids are ignored, so a caller can't
|
||||
* reorder across teams. Ids omitted from the list keep their existing order.
|
||||
*/
|
||||
void reorder(Long teamId, List<String> orderedIds);
|
||||
|
||||
/** Returns whether the policy existed. */
|
||||
boolean delete(String id);
|
||||
}
|
||||
|
||||
+1
-3
@@ -29,7 +29,6 @@ import lombok.extern.slf4j.Slf4j;
|
||||
import stirling.software.common.model.ApplicationProperties;
|
||||
import stirling.software.proprietary.policy.config.FolderAccessGuard;
|
||||
import stirling.software.proprietary.policy.engine.PolicyRunner;
|
||||
import stirling.software.proprietary.policy.engine.SweepKind;
|
||||
import stirling.software.proprietary.policy.input.InputSource;
|
||||
import stirling.software.proprietary.policy.model.InputSpec;
|
||||
import stirling.software.proprietary.policy.model.Policy;
|
||||
@@ -203,8 +202,7 @@ public class FolderWatchTrigger implements PolicyTrigger {
|
||||
}
|
||||
if (dirs.stream().anyMatch(changedDirs::contains)) {
|
||||
log.debug("Folder-watch policy {} ({}) saw activity", policy.id(), policy.name());
|
||||
// Light: the periodic reconcile does the full sweep.
|
||||
policyRunner.run(policy, SweepKind.LIGHT);
|
||||
policyRunner.run(policy);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
-17
@@ -259,21 +259,4 @@ public interface PersistentAuditEventRepository extends JpaRepository<Persistent
|
||||
"SELECT e FROM PersistentAuditEvent e WHERE e.type != :excludeType AND e.timestamp > :startDate")
|
||||
List<PersistentAuditEvent> findAllExceptTypeAndTimestampAfterForExport(
|
||||
@Param("excludeType") String excludeType, @Param("startDate") Instant startDate);
|
||||
|
||||
// Free-editor fleet usage: count genuine free-UI operations (source = "WEB") by type.
|
||||
@Query(
|
||||
"SELECT COUNT(e) FROM PersistentAuditEvent e "
|
||||
+ "WHERE e.type IN :types AND e.source = :source AND e.timestamp > :since")
|
||||
long countByTypeInAndSourceAndTimestampAfter(
|
||||
@Param("types") List<String> types,
|
||||
@Param("source") String source,
|
||||
@Param("since") Instant since);
|
||||
|
||||
@Query(
|
||||
"SELECT COUNT(DISTINCT e.principal) FROM PersistentAuditEvent e "
|
||||
+ "WHERE e.source = :source AND e.type <> :excludeType AND e.timestamp > :since")
|
||||
long countDistinctPrincipalsBySourceExcludingTypeAfter(
|
||||
@Param("source") String source,
|
||||
@Param("excludeType") String excludeType,
|
||||
@Param("since") Instant since);
|
||||
}
|
||||
|
||||
-3
@@ -22,7 +22,6 @@ import stirling.software.proprietary.model.Team;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.service.DatabaseServiceInterface;
|
||||
import stirling.software.proprietary.security.service.SaveUserRequest;
|
||||
import stirling.software.proprietary.security.service.TeamMembershipService;
|
||||
import stirling.software.proprietary.security.service.TeamService;
|
||||
import stirling.software.proprietary.security.service.UserService;
|
||||
import stirling.software.proprietary.service.UserLicenseSettingsService;
|
||||
@@ -41,7 +40,6 @@ public class InitialSecuritySetup {
|
||||
private final DatabaseServiceInterface databaseService;
|
||||
private final UserLicenseSettingsService licenseSettingsService;
|
||||
private final Environment environment;
|
||||
private final TeamMembershipService teamMembershipService;
|
||||
|
||||
/**
|
||||
* SaaS manages identity in Supabase and billing via PAYG, so the self-host bootstrap steps that
|
||||
@@ -116,7 +114,6 @@ public class InitialSecuritySetup {
|
||||
}
|
||||
|
||||
userService.saveAll(usersWithoutTeam); // batch save
|
||||
usersWithoutTeam.forEach(teamMembershipService::syncMembership);
|
||||
if (usersWithoutTeam != null && !usersWithoutTeam.isEmpty()) {
|
||||
log.info(
|
||||
"Assigned {} user(s) without a team to the default team.",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user