mirror of
https://github.com/Stirling-Tools/Stirling-PDF.git
synced 2026-09-03 05:10:16 +03:00
Fix WebKit PDF-engine and storage failures, and catch them in cross-browser CI (#7366)
# Description of Changes Follow-up to #7314, which fixed the IndexedDB blob rejection itself. This one fixes the remaining WebKit engine gaps, fixes the ways that class of failure surfaced to the user, and adds the cross-browser signal that would have caught them on the PR instead of six weeks later. ## Why this exists Two total WebKit outages sat on `main` for weeks: 1. pdf.js reads its text stream with `for await (… of readableStream)`, and WebKit has no `ReadableStream[Symbol.asyncIterator]`. **All** pdf.js text extraction threw `TypeError: undefined is not a function` — Compare, read-aloud and the PDF text editor were dead on Safari. 2. IndexedDB in WebKit rejects Blob/File values with `UnknownError: Error preparing Blob/File data to be stored in object store`, so nothing persisted and every reload came back empty. Neither was caught, because the existing specs never did the work. The Compare specs filled both slots and asserted the button was enabled; none of them clicked it. The persistence specs asserted a *filename* reappeared after a reload, which only needs the metadata record, not the bytes. Every failure here **looked like success** — empty panes, blank thumbnails, a `src` that was set but empty. That shapes the tests more than the fixes. ## WebKit engine gaps - **`ReadableStream[Symbol.asyncIterator]`**, installed at the entry point before any PDF work starts. The lock discipline is the subtle part: releasing is idempotent, is *not* done after a successful read, and *is* done in the read's error steps — `for await` never calls `return()` when `next()` rejects, so nothing else would ever unlock an errored stream. - **`requestIdleCallback`**, installed once instead of guarded at each call site. This one wasn't broken, it was mistimed: the local fallbacks fired at 200ms and 1000ms, landing the pdfium WASM compile on top of the app's first renders. The shim honours the caller's full timeout, so `{timeout: 2000}` means 2000ms. - **`convertToBlob()` does not fail on a format it can't encode.** Per spec it silently serialises to PNG, so asking for WebP and getting PNG back looks like success. Canvas output now probes what the engine really produced (once per realm) and uses the best lossy format it honours. PNG of a rendered page is several times the size of the equivalent WebP or JPEG, held as object URLs for every page on screen, on the engine with the tightest renderer memory budget. ## WebKit storage failures These read as generic transaction hygiene. They aren't — a refused blob write **aborts its transaction**, which is the mechanism that turned a WebKit rejection into a hang. - **Blob refusal is remembered from any write**, not just the initial `add`. WebKit reports it when it can't write the blob's *backing file*, which is per-operation — an engine that accepted the add can still refuse the rewrite, and every read-modify-write rewrites the record with its body attached. - **Aborted transactions no longer hang.** Read-modify-write moves to a single `updateRecord` helper that owns its transaction, guards it once, and resolves on **commit** rather than on the put's `onsuccess`. The previous shape — two promises over one shared transaction, with an `await` between the get and the put — put the abort guard on the read, leaving the write with no handler at all. `persistVersionedOutputs` awaits that, and `.catch` can't rescue a promise that never settles, so tool outputs could silently stop persisting. - **Stored blobs are no longer re-wrapped on read.** Since #7175 the record holds the `File` itself; wrapping it in `new Blob([record.data])` can cost WebKit the backing handle, giving you an object that looks valid and reads as empty. - **The file sidebar reaches a resting state** when the library can't be read, instead of spinning forever on a rejection nobody observes. It carries on with the in-memory workbench files: an unreadable library should cost the user their history, not the file they're working on. - **Thumbnail failures are logged.** Three `catch {}` blocks returned `""`, and an empty thumbnail is indistinguishable from "this file has no preview" — which is how outage #1 hid as a cosmetic nicety. ## CI `main` now runs the whole stubbed suite once per engine (#7304), so the new `@engine-capability` specs get chromium, firefox and webkit for free. They assert the primitives actually work — a **counted** comparison, a raster thumbnail data URL with real payload, and a page rendered from a file restored by a reload — rather than that the UI rendered. Deliberately small: anything added there is paid for three times per PR, so add depth, not breadth. Run them alone with `task e2e:cross-browser -- --grep @engine-capability`. The cross-browser projects now share the stubbed project's viewport. At the device presets' default 1280x720 a layout difference would fail these specs on Firefox/WebKit only, which reads as an engine outage. `vite.config.ts` gains a `worker.plugins` entry so `@app/*` resolves inside worker bundles. Worker bundles are a separate Rollup pass and don't inherit `plugins`, so the alias worked in the app and failed in a worker — previously worked around with a relative import plus a lint exemption, which silently bypasses the layer cascade. ## Verification - `task frontend:check` green: typecheck, oxlint, theme lint, stylelint, prettier, 215 test files / 1841 tests. - The `@engine-capability` suite passes on Chromium and WebKit locally. - **Negative control:** with the `ReadableStream` shim removed, the WebKit comparison spec fails at the Deletions/Additions assertion — the exact reported Safari symptom. Restored, and it passes. Both the fix and the test that guards it are load-bearing. - The worker alias change verified both ways: the build inlines the encoding probe into the worker chunk, and removing `worker.plugins` fails with `Rollup failed to resolve import "@app/utils/canvasImageEncoding"`. - The abort regression test aborts the transaction mid-write and asserts `markFileAsProcessed` settles. Before the fix it never settles and the test times out. ## Split out of this PR Two things in earlier revisions of this branch were engine-agnostic — found via the same symptom, not the same cause — and now have their own PRs: - **#7416** — blocked IndexedDB upgrades hanging the file library (multi-tab lifecycle, the concurrent-open race, `onversionchange`). - **#7417** — the thumbnail TTL rewriting the whole library on every listing. `FileSidebar`'s try/catch appears in both this PR and #7416, identically: a WebKit rejection and a blocked-open rejection both have to stop stranding the spinner. Whichever merges second is a no-op for that file. ## Known gaps - The blob-refused **rewrite** recovery in `updateRecord` isn't unit-tested. `fake-indexeddb` never returns Blob values from a read, so the branch that converts to a copy can't be reached there. Noted in the test file. - For the same reason, `fileFromRecord`'s "hand the stored File back untouched" path is only covered on a real engine, by the reload spec. - Nothing asserts that `src/index.tsx` imports the shims. The unit suite installs the same module via `setupTests.ts` (jsdom has the same gaps WebKit does), so a future regression where the entry point drops the import would still be green under vitest. - `FileSidebar`'s resting-state fix loses its E2E coverage until #7416 lands — forcing WebKit's blob refusal from a spec isn't practical, which is why that spec blocks the database instead. --- ## Checklist ### General - [x] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [x] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [x] My changes generate no new warnings ### Documentation - [x] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed)
This commit is contained in:
EthanHealy01
+20
-12
@@ -10,6 +10,18 @@ import tsconfigPaths from "vite-tsconfig-paths";
|
||||
* the portal layer at editor/src/portal/). MDX docs pages live in
|
||||
* editor/src/portal/docs/.
|
||||
*/
|
||||
/**
|
||||
* Editor stories import via `@app/*` (proprietary→core fallback), `@core/*` and
|
||||
* `@proprietary/*`. Resolve them exactly the way the editor's own build does -
|
||||
* through vite-tsconfig-paths against the proprietary vite tsconfig - so the
|
||||
* shared Storybook can host editor components without duplicating the alias map
|
||||
* here. Built per pass: the main bundle and the worker bundle each need their own.
|
||||
*/
|
||||
const editorPathAliases = () =>
|
||||
tsconfigPaths({
|
||||
projects: [resolve(__dirname, "../editor/tsconfig.proprietary.vite.json")],
|
||||
});
|
||||
|
||||
const config: StorybookConfig = {
|
||||
stories: [
|
||||
"../editor/src/portal/**/*.mdx",
|
||||
@@ -47,19 +59,15 @@ const config: StorybookConfig = {
|
||||
// than a relative path.
|
||||
"@public": resolve(__dirname, "../editor/public"),
|
||||
};
|
||||
// Editor stories import via @app/* (proprietary→core fallback), @core/* and
|
||||
// @proprietary/*. Resolve them exactly the way the editor's own build does —
|
||||
// through vite-tsconfig-paths against the proprietary vite tsconfig — so the
|
||||
// shared Storybook can host editor components without duplicating the alias
|
||||
// map here.
|
||||
config.plugins = config.plugins ?? [];
|
||||
config.plugins.push(
|
||||
tsconfigPaths({
|
||||
projects: [
|
||||
resolve(__dirname, "../editor/tsconfig.proprietary.vite.json"),
|
||||
],
|
||||
}),
|
||||
);
|
||||
config.plugins.push(editorPathAliases());
|
||||
// Worker bundles are a separate Rollup pass and do NOT inherit `plugins`, so
|
||||
// without this a worker importing @app/* fails to resolve while the same
|
||||
// import works everywhere else. Mirrors editor/vite.config.ts.
|
||||
config.worker = {
|
||||
...(config.worker ?? {}),
|
||||
plugins: () => [editorPathAliases()],
|
||||
};
|
||||
// Point apiClient.saas at a mock origin so the SaaS-backed billing stories
|
||||
// (SubscribedPlanView, PaymentMethodCard, InvoicesList) resolve a base URL and
|
||||
// their MSW handlers (which match "*/api/v1/payg/...") can intercept. The host
|
||||
|
||||
@@ -17,9 +17,12 @@ import { defineConfig, devices } from "@playwright/test";
|
||||
*
|
||||
* @see https://playwright.dev/docs/test-configuration
|
||||
*/
|
||||
/** Shared by every stubbed project so a spec sees one layout on all engines. */
|
||||
const STUBBED_VIEWPORT = { width: 1920, height: 1080 };
|
||||
|
||||
const chromiumViewport = {
|
||||
...devices["Desktop Chrome"],
|
||||
viewport: { width: 1920, height: 1080 },
|
||||
viewport: STUBBED_VIEWPORT,
|
||||
};
|
||||
|
||||
export default defineConfig({
|
||||
@@ -55,7 +58,8 @@ export default defineConfig({
|
||||
},
|
||||
|
||||
projects: [
|
||||
// Stubbed - no backend required, chromium-only for CI speed
|
||||
// Stubbed - no backend required. The chromium arm of the cross-browser
|
||||
// set below; CI fans all three out, one job per engine.
|
||||
{
|
||||
name: "stubbed",
|
||||
testDir: "./src/core/tests/stubbed",
|
||||
@@ -93,16 +97,17 @@ export default defineConfig({
|
||||
},
|
||||
},
|
||||
|
||||
// Cross-browser coverage for the stubbed suite (opt-in locally)
|
||||
// Cross-browser coverage for the stubbed suite. Same viewport as `stubbed`,
|
||||
// or a layout difference here reads as an engine outage.
|
||||
{
|
||||
name: "stubbed-firefox",
|
||||
testDir: "./src/core/tests/stubbed",
|
||||
use: { ...devices["Desktop Firefox"] },
|
||||
use: { ...devices["Desktop Firefox"], viewport: STUBBED_VIEWPORT },
|
||||
},
|
||||
{
|
||||
name: "stubbed-webkit",
|
||||
testDir: "./src/core/tests/stubbed",
|
||||
use: { ...devices["Desktop Safari"] },
|
||||
use: { ...devices["Desktop Safari"], viewport: STUBBED_VIEWPORT },
|
||||
},
|
||||
],
|
||||
|
||||
|
||||
@@ -3890,6 +3890,8 @@ addFiles = "Add files"
|
||||
addingFiles = "Adding files…"
|
||||
collapse = "Collapse sidebar"
|
||||
customizeGroups = "Customize groups"
|
||||
dataLostBody = "This browser lost this file's contents. Upload it again to keep working with it."
|
||||
dataLostTitle = "File data is unavailable"
|
||||
dropHint = "Open files to get started"
|
||||
dropToAdd = "Drop files to add"
|
||||
expand = "Expand sidebar"
|
||||
@@ -3908,6 +3910,8 @@ viewAll = "View all {{count}} files"
|
||||
|
||||
[fileSidebar.fileItem]
|
||||
closeViewer = "Close viewer"
|
||||
dataLost = "Data lost"
|
||||
dataLostTooltip = "This browser lost this file's contents. Upload it again to keep working with it."
|
||||
delete = "Delete"
|
||||
moreActions = "More actions"
|
||||
openInViewer = "Open in viewer"
|
||||
|
||||
@@ -300,6 +300,10 @@ const FileEditorThumbnail = ({
|
||||
const [showVersionHistory, setShowVersionHistory] = useState(false);
|
||||
|
||||
const policyEnforcing = policies.some((p) => p.enforcing);
|
||||
// The overlay swallows clicks, so a run that never settles would leave the card
|
||||
// unusable with no way out. Dismissible, like the viewer's; resets per run.
|
||||
const [enforcingDismissed, setEnforcingDismissed] = useState(false);
|
||||
if (!policyEnforcing && enforcingDismissed) setEnforcingDismissed(false);
|
||||
// The policy currently enforcing, so the overlay's icon/spinner match that
|
||||
// policy's badge instead of a fixed blue.
|
||||
const enforcingPolicy = policies.find((p) => p.enforcing);
|
||||
@@ -548,8 +552,9 @@ const FileEditorThumbnail = ({
|
||||
|
||||
{/* Policy enforcement overlay — shown while any policy is in-flight */}
|
||||
<PolicyEnforcingOverlay
|
||||
enforcing={policyEnforcing}
|
||||
enforcing={policyEnforcing && !enforcingDismissed}
|
||||
zIndex={2}
|
||||
onDismiss={() => setEnforcingDismissed(true)}
|
||||
accentVar={enforcingPolicy?.accentColor}
|
||||
categoryId={enforcingPolicy?.id}
|
||||
/>
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useState, Suspense, lazy } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import KeyboardArrowDownIcon from "@mui/icons-material/KeyboardArrowDown";
|
||||
import { Box, Loader, Center } from "@mantine/core";
|
||||
import { Box, Loader, Center, Stack, Text } from "@mantine/core";
|
||||
import { Button } from "@app/ui/Button";
|
||||
import { useToolWorkflow } from "@app/contexts/ToolWorkflowContext";
|
||||
import { useFileHandler } from "@app/hooks/useFileHandler";
|
||||
@@ -44,7 +44,7 @@ export default function Workbench() {
|
||||
useCookieConsent({ analyticsEnabled: config?.enableAnalytics === true });
|
||||
|
||||
// Use context-based hooks to eliminate all prop drilling
|
||||
const { files: activeFiles } = useAllFiles();
|
||||
const { files: activeFiles, fileIds } = useAllFiles();
|
||||
const { workbench: currentView } = useNavigationState();
|
||||
const { actions: navActions } = useNavigationActions();
|
||||
const setCurrentView = navActions.setWorkbench;
|
||||
@@ -134,6 +134,20 @@ export default function Workbench() {
|
||||
}
|
||||
|
||||
if (activeFiles.length === 0) {
|
||||
// Files are open but their bytes are still loading (a cold PDF engine can
|
||||
// take seconds). Showing the drop zone here reads as "the click did nothing".
|
||||
if (fileIds.length > 0) {
|
||||
return (
|
||||
<Center h="100%" w="100%">
|
||||
<Stack align="center" gap="md">
|
||||
<Loader size="lg" />
|
||||
<Text c="dimmed" size="sm">
|
||||
{t("fileManager.loadingFiles", "Loading files...")}
|
||||
</Text>
|
||||
</Stack>
|
||||
</Center>
|
||||
);
|
||||
}
|
||||
return <LandingPage />;
|
||||
}
|
||||
|
||||
|
||||
@@ -59,7 +59,8 @@ import {
|
||||
deleteServerFile,
|
||||
type DeleteScope,
|
||||
} from "@app/services/serverStorageDelete";
|
||||
import { fileStorage } from "@app/services/fileStorage";
|
||||
import { fileStorage, onRecordUnreadable } from "@app/services/fileStorage";
|
||||
import { alert } from "@app/components/toast";
|
||||
import { useBulkAddProgress } from "@app/services/bulkAddProgress";
|
||||
import { useFolderMembership } from "@app/hooks/useFolderMembership";
|
||||
import { useAllWatchedFolders } from "@app/hooks/useAllWatchedFolders";
|
||||
@@ -280,6 +281,19 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
|
||||
// Leaf files = user-visible files (excludes intermediate tool outputs)
|
||||
const [allFileStubs, setAllFileStubs] = useState<StirlingFileStub[]>([]);
|
||||
// Files whose stored bytes this session PROVED unreadable. Rows render a
|
||||
// "data lost" state instead of pretending the file can open; storage keeps
|
||||
// the record so a reload re-tests it.
|
||||
const [lostFileIds, setLostFileIds] = useState<ReadonlySet<string>>(
|
||||
() => new Set(),
|
||||
);
|
||||
useEffect(
|
||||
() =>
|
||||
onRecordUnreadable((fileId) =>
|
||||
setLostFileIds((prev) => new Set(prev).add(fileId as string)),
|
||||
),
|
||||
[],
|
||||
);
|
||||
const [stubsLoaded, setStubsLoaded] = useState(false);
|
||||
// Kebab "Save to cloud" target; drives BulkUploadToServerModal.
|
||||
const [saveToServerTarget, setSaveToServerTarget] = useState<
|
||||
@@ -298,32 +312,45 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
const storageEnabled = config?.storageEnabled === true && !isAnonymous;
|
||||
|
||||
const refreshStubs = useCallback(async () => {
|
||||
// Leaf files from IDB - same source as the file selection modal.
|
||||
const stubs = await indexedDB.loadLeafMetadata();
|
||||
const idbIds = new Set(stubs.map((s) => s.id as string));
|
||||
// `stubsLoaded` gates the spinner, so the `finally` below must set it on
|
||||
// every path - callers never await this, so a rejection goes nowhere.
|
||||
let stubs: StirlingFileStub[] = [];
|
||||
try {
|
||||
// Leaf files from IDB - same source as the file selection modal.
|
||||
stubs = await indexedDB.loadLeafMetadata();
|
||||
} catch (error) {
|
||||
// Carry on with the in-memory workbench files: an unreadable library
|
||||
// should cost the user their history, not the file they're working on.
|
||||
console.error("Failed to read the file library from storage:", error);
|
||||
}
|
||||
|
||||
// Also include workbench files not yet flushed to IDB.
|
||||
const pendingStubs = state.files.ids
|
||||
.map((id) => state.files.byId[id])
|
||||
.filter(
|
||||
(stub): stub is NonNullable<typeof stub> =>
|
||||
!!stub && stub.isLeaf !== false && !idbIds.has(stub.id as string),
|
||||
try {
|
||||
const idbIds = new Set(stubs.map((s) => s.id as string));
|
||||
|
||||
// Also include workbench files not yet flushed to IDB.
|
||||
const pendingStubs = state.files.ids
|
||||
.map((id) => state.files.byId[id])
|
||||
.filter(
|
||||
(stub): stub is NonNullable<typeof stub> =>
|
||||
!!stub && stub.isLeaf !== false && !idbIds.has(stub.id as string),
|
||||
);
|
||||
|
||||
const allStubs = [...stubs, ...pendingStubs];
|
||||
// A version swap briefly lists both the old leaf (IDB) and its replacement (workbench); two stubs for one lineage collide on the row key and corrupt React reconciliation, so drop any stub another names as its parent.
|
||||
const superseded = new Set(
|
||||
allStubs.map((s) => s.parentFileId as string | undefined),
|
||||
);
|
||||
|
||||
const allStubs = [...stubs, ...pendingStubs];
|
||||
// A version swap briefly lists both the old leaf (IDB) and its replacement (workbench); two stubs for one lineage collide on the row key and corrupt React reconciliation, so drop any stub another names as its parent.
|
||||
const superseded = new Set(
|
||||
allStubs.map((s) => s.parentFileId as string | undefined),
|
||||
);
|
||||
const currentStubs = allStubs.filter(
|
||||
(s) => !superseded.has(s.id as string),
|
||||
);
|
||||
setAllFileStubs(
|
||||
currentStubs.sort(
|
||||
(a, b) => (b.lastModified ?? 0) - (a.lastModified ?? 0),
|
||||
),
|
||||
);
|
||||
setStubsLoaded(true);
|
||||
const currentStubs = allStubs.filter(
|
||||
(s) => !superseded.has(s.id as string),
|
||||
);
|
||||
setAllFileStubs(
|
||||
currentStubs.sort(
|
||||
(a, b) => (b.lastModified ?? 0) - (a.lastModified ?? 0),
|
||||
),
|
||||
);
|
||||
} finally {
|
||||
setStubsLoaded(true);
|
||||
}
|
||||
}, [indexedDB, state.files.ids, state.files.byId]);
|
||||
|
||||
// Refresh on mount, workbench changes, or external IndexedDB writes —
|
||||
@@ -362,7 +389,9 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
setDeleteTarget(stub);
|
||||
return;
|
||||
}
|
||||
await fileActions.removeFiles([fileId], true);
|
||||
// Its superseded versions go too - see orphanedAncestorIds.
|
||||
const orphans = await fileStorage.orphanedAncestorIds([fileId]);
|
||||
await fileActions.removeFiles([fileId, ...orphans], true);
|
||||
await refreshStubs();
|
||||
},
|
||||
[allFileStubs, fileActions, refreshStubs],
|
||||
@@ -380,7 +409,8 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
await deleteServerFile(stub.remoteStorageId);
|
||||
}
|
||||
if (scope === "device" || scope === "everywhere") {
|
||||
await fileActions.removeFiles([stub.id], true);
|
||||
const orphans = await fileStorage.orphanedAncestorIds([stub.id]);
|
||||
await fileActions.removeFiles([stub.id, ...orphans], true);
|
||||
} else if (scope === "cloud") {
|
||||
// Local copy kept - drop the dead remote pointer so the cloud badge
|
||||
// clears (the sidebar doesn't reconcile with the server itself).
|
||||
@@ -484,6 +514,22 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
const stub = allFileStubs.find((s) => s.id === fileId);
|
||||
if (!stub) return;
|
||||
|
||||
// Its bytes are gone; opening it can only fail. Say so instead of a
|
||||
// click that goes nowhere.
|
||||
if (stub.dataUnavailable || lostFileIds.has(fileId as string)) {
|
||||
alert({
|
||||
alertType: "warning",
|
||||
title: t("fileSidebar.dataLostTitle", "File data is unavailable"),
|
||||
body: t(
|
||||
"fileSidebar.dataLostBody",
|
||||
"This browser lost this file's contents. Upload it again to keep working with it.",
|
||||
),
|
||||
expandable: false,
|
||||
durationMs: 6000,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// In the Watched Folders view a click sends the file into the open folder
|
||||
// (mirrors how a click toggles a file into the active workbench elsewhere).
|
||||
// On the folder list (no folder open) it's a no-op so browsing isn't disrupted.
|
||||
@@ -538,6 +584,8 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
},
|
||||
[
|
||||
allFileStubs,
|
||||
lostFileIds,
|
||||
t,
|
||||
state.files.ids,
|
||||
state.ui.selectedFileIds,
|
||||
fileActions,
|
||||
@@ -725,6 +773,8 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
? state.files.byId[workbenchFileId]?.thumbnailUrl
|
||||
: undefined) || stub.thumbnailUrl;
|
||||
const fileOrigin = getFileOrigin(stub);
|
||||
const dataUnavailable =
|
||||
stub.dataUnavailable === true || lostFileIds.has(stub.id as string);
|
||||
// Key by lineage (originalFileId) so a version swap updates the row in place instead of
|
||||
// remounting. But a 1-input→many-output op (split) yields sibling leaves that share one
|
||||
// originalFileId; those would collide on the key, so fall back to the unique leaf id when a
|
||||
@@ -747,6 +797,7 @@ const FileSidebar = forwardRef<HTMLDivElement, FileSidebarProps>(
|
||||
thumbnailUrl={thumbnailUrl}
|
||||
onClick={handleFileClick}
|
||||
onEyeClick={handleEyeClick}
|
||||
dataUnavailable={dataUnavailable}
|
||||
draggable={isWatchedFoldersActive}
|
||||
onDragStart={handleWatchedFolderDragStart}
|
||||
folders={memberFolders}
|
||||
|
||||
@@ -447,3 +447,13 @@
|
||||
transform: translateY(-50%) scale(1);
|
||||
}
|
||||
}
|
||||
|
||||
/* The stored bytes are gone - the row says so instead of pretending to open. */
|
||||
.file-sidebar-datalost-badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.15rem;
|
||||
color: var(--c-danger);
|
||||
font-size: 0.7rem;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ import ShieldOutlinedIcon from "@mui/icons-material/ShieldOutlined";
|
||||
import MoreVertIcon from "@mui/icons-material/MoreVert";
|
||||
import CloudUploadOutlinedIcon from "@mui/icons-material/CloudUploadOutlined";
|
||||
import CloudDoneIcon from "@mui/icons-material/CloudDone";
|
||||
import ErrorOutlineIcon from "@mui/icons-material/ErrorOutlineOutlined";
|
||||
import DeleteOutlineIcon from "@mui/icons-material/DeleteOutlined";
|
||||
import HistoryIcon from "@mui/icons-material/History";
|
||||
import type { FileId } from "@app/types/file";
|
||||
@@ -163,6 +164,9 @@ export interface FileItemProps {
|
||||
onVersionHistory?: (fileId: FileId) => void;
|
||||
/** Whether this file has more than one version (drives the menu item). */
|
||||
hasVersionHistory?: boolean;
|
||||
/** The stored bytes are gone (WebKit lost the blob's backing store). The row
|
||||
* says so instead of pretending the file can open. */
|
||||
dataUnavailable?: boolean;
|
||||
}
|
||||
|
||||
const MAX_VISIBLE_FOLDER_TAGS = 2;
|
||||
@@ -177,6 +181,7 @@ export const FileItem = React.memo(function FileItem({
|
||||
isSelected,
|
||||
isActive,
|
||||
isViewedInViewer,
|
||||
dataUnavailable,
|
||||
thumbnailUrl,
|
||||
onClick,
|
||||
onEyeClick,
|
||||
@@ -294,6 +299,21 @@ export const FileItem = React.memo(function FileItem({
|
||||
</>
|
||||
)}
|
||||
</span>
|
||||
{dataUnavailable && (
|
||||
<Tooltip
|
||||
label={t(
|
||||
"fileSidebar.fileItem.dataLostTooltip",
|
||||
"This browser lost this file's contents. Upload it again to keep working with it.",
|
||||
)}
|
||||
withArrow
|
||||
position="top"
|
||||
>
|
||||
<span className="file-sidebar-datalost-badge" data-no-select>
|
||||
<ErrorOutlineIcon sx={{ fontSize: "0.85rem" }} />
|
||||
{t("fileSidebar.fileItem.dataLost", "Data lost")}
|
||||
</span>
|
||||
</Tooltip>
|
||||
)}
|
||||
{isUploadedToCloud && (
|
||||
<Tooltip
|
||||
label={t(
|
||||
|
||||
@@ -6,6 +6,9 @@ export function PolicyEnforcingOverlay(_props: {
|
||||
accentVar?: string;
|
||||
/** Category of the enforcing policy — picks its icon in the real overlay. */
|
||||
categoryId?: string;
|
||||
/** Shows a dismiss control in the real overlay, so a run that never settles
|
||||
* can't leave the surface underneath permanently unclickable. */
|
||||
onDismiss?: () => void;
|
||||
}) {
|
||||
return null;
|
||||
}
|
||||
|
||||
@@ -58,6 +58,7 @@ import {
|
||||
IndexedDBProvider,
|
||||
useIndexedDB,
|
||||
} from "@app/contexts/IndexedDBContext";
|
||||
import { onRecordUnreadable } from "@app/services/fileStorage";
|
||||
import { useZipConfirmation } from "@app/hooks/useZipConfirmation";
|
||||
import ZipWarningModal from "@app/components/shared/ZipWarningModal";
|
||||
import EncryptedPdfUnlockModal from "@app/components/shared/EncryptedPdfUnlockModal";
|
||||
@@ -186,6 +187,21 @@ function FileContextInner({
|
||||
setUnlockError(null);
|
||||
}, [activeEncryptedFileId]);
|
||||
|
||||
// Storage proved a file's bytes unreadable (WebKit losing a blob's backing
|
||||
// store). Drop it: the viewer would otherwise spin on a document that can
|
||||
// never load. The record stays, so a reload re-tests it.
|
||||
useEffect(
|
||||
() =>
|
||||
onRecordUnreadable((fileId) => {
|
||||
if (!stateRef.current.files.byId[fileId]) return;
|
||||
console.error(
|
||||
`[FileContext] dropping ${fileId} from the workbench: its stored bytes are unreadable`,
|
||||
);
|
||||
lifecycleManager.removeFiles([fileId], stateRef);
|
||||
}),
|
||||
[lifecycleManager],
|
||||
);
|
||||
|
||||
const handleUnlockSkip = useCallback(() => {
|
||||
if (activeEncryptedFileId) {
|
||||
dismissedEncryptedFilesRef.current.add(activeEncryptedFileId);
|
||||
|
||||
@@ -455,7 +455,10 @@ export function FilesPageProvider({ children }: { children: React.ReactNode }) {
|
||||
})
|
||||
.map((s) => s.id);
|
||||
if (localIds.length > 0) {
|
||||
await fileActions.removeFiles(localIds, true);
|
||||
// Take the superseded versions with it, or their bytes sit in storage
|
||||
// forever - invisible, because listings only show leaves.
|
||||
const orphans = await fileStorage.orphanedAncestorIds(localIds);
|
||||
await fileActions.removeFiles([...localIds, ...orphans], true);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -222,3 +222,40 @@ describe("fileContextReducer — silent CONSUME_FILES (background enforcement)",
|
||||
expect(next.ui.selectedFileIds).toEqual(["b2"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("fileContextReducer — REMOVE_FILES", () => {
|
||||
/** Deleting from the library dispatches this for files that were never in the
|
||||
* workbench; reallocating then re-renders every consumer for nothing. */
|
||||
it("is a true no-op when none of the ids are in the workbench", () => {
|
||||
const state = stateWith([stub("a")]);
|
||||
const next = fileContextReducer(state, {
|
||||
type: "REMOVE_FILES",
|
||||
payload: { fileIds: ["gone" as FileId] },
|
||||
});
|
||||
expect(next).toBe(state);
|
||||
});
|
||||
|
||||
it("still removes the ids it does hold", () => {
|
||||
const state = stateWith([stub("a"), stub("b")]);
|
||||
const next = fileContextReducer(state, {
|
||||
type: "REMOVE_FILES",
|
||||
payload: { fileIds: ["a" as FileId, "gone" as FileId] },
|
||||
});
|
||||
expect(next.files.ids).toEqual(["b"]);
|
||||
expect(next.files.byId["a" as FileId]).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps the files slice when only a selection is cleared", () => {
|
||||
const base = stateWith([stub("a")]);
|
||||
const state: FileContextState = {
|
||||
...base,
|
||||
ui: { ...base.ui, selectedFileIds: ["gone" as FileId] },
|
||||
};
|
||||
const next = fileContextReducer(state, {
|
||||
type: "REMOVE_FILES",
|
||||
payload: { fileIds: ["gone" as FileId] },
|
||||
});
|
||||
expect(next.files).toBe(state.files);
|
||||
expect(next.ui.selectedFileIds).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -183,6 +183,20 @@ export function fileContextReducer(
|
||||
const remainingIds = state.files.ids.filter(
|
||||
(id) => !fileIds.includes(id),
|
||||
);
|
||||
// Clear selections that reference removed files
|
||||
const validSelectedFileIds = state.ui.selectedFileIds.filter(
|
||||
(id) => !fileIds.includes(id),
|
||||
);
|
||||
|
||||
// Deleting a library file that was never in the workbench removes nothing
|
||||
// here, and must not re-render every file and UI consumer.
|
||||
const removedFromWorkbench =
|
||||
remainingIds.length !== state.files.ids.length ||
|
||||
fileIds.some((id) => id in state.files.byId);
|
||||
const deselected =
|
||||
validSelectedFileIds.length !== state.ui.selectedFileIds.length;
|
||||
if (!removedFromWorkbench && !deselected) return state;
|
||||
|
||||
const newById = { ...state.files.byId };
|
||||
|
||||
// Remove files from state (resource cleanup handled by lifecycle manager)
|
||||
@@ -190,21 +204,14 @@ export function fileContextReducer(
|
||||
delete newById[id];
|
||||
});
|
||||
|
||||
// Clear selections that reference removed files
|
||||
const validSelectedFileIds = state.ui.selectedFileIds.filter(
|
||||
(id) => !fileIds.includes(id),
|
||||
);
|
||||
|
||||
return {
|
||||
...state,
|
||||
files: {
|
||||
ids: remainingIds,
|
||||
byId: newById,
|
||||
},
|
||||
ui: {
|
||||
...state.ui,
|
||||
selectedFileIds: validSelectedFileIds,
|
||||
},
|
||||
files: removedFromWorkbench
|
||||
? { ids: remainingIds, byId: newById }
|
||||
: state.files,
|
||||
ui: deselected
|
||||
? { ...state.ui, selectedFileIds: validSelectedFileIds }
|
||||
: state.ui,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -26,6 +26,9 @@ import {
|
||||
clearBulkAddProgress,
|
||||
} from "@app/services/bulkAddProgress";
|
||||
const DEBUG = process.env.NODE_ENV === "development";
|
||||
/** How long a file may sit unhydrated before the console says so. Reporting only:
|
||||
* the read is never abandoned, because large files legitimately take time. */
|
||||
const STALLED_LOAD_MS = 8000;
|
||||
const HYDRATION_CONCURRENCY = 2;
|
||||
let activeHydrations = 0;
|
||||
const hydrationQueue: Array<() => Promise<void>> = [];
|
||||
@@ -854,61 +857,78 @@ export async function addStirlingFileStubs(
|
||||
// Load File object and hydrate metadata in background (non-blocking)
|
||||
const fileId = stub.id;
|
||||
|
||||
// Load File object from IndexedDB asynchronously
|
||||
scheduleMetadataHydration(async () => {
|
||||
const stirlingFile = await fileStorage.getStirlingFile(fileId);
|
||||
// Regenerate page metadata + thumbnails. Queued, because parsing several
|
||||
// PDFs at once is what the concurrency limit exists to bound.
|
||||
const scheduleMetadataFor = (stirlingFile: StirlingFile): void => {
|
||||
scheduleMetadataHydration(async () => {
|
||||
const processedFileMetadata =
|
||||
await generateProcessedFileMetadata(stirlingFile);
|
||||
if (!processedFileMetadata) return;
|
||||
|
||||
const updates: Partial<StirlingFileStub> = {
|
||||
processedFile: processedFileMetadata,
|
||||
};
|
||||
|
||||
// Update thumbnail only if current stub doesn't have one
|
||||
const currentStub = stateRef.current.files.byId[fileId];
|
||||
if (
|
||||
!currentStub?.thumbnailUrl &&
|
||||
processedFileMetadata.thumbnailUrl
|
||||
) {
|
||||
updates.thumbnailUrl = processedFileMetadata.thumbnailUrl;
|
||||
if (processedFileMetadata.thumbnailUrl.startsWith("blob:")) {
|
||||
lifecycleManager.trackBlobUrl(processedFileMetadata.thumbnailUrl);
|
||||
}
|
||||
}
|
||||
|
||||
lifecycleManager.updateStirlingFileStub(fileId, updates, stateRef);
|
||||
});
|
||||
};
|
||||
|
||||
// Load and publish the File, ahead of any parsing. NOT queued: whether a
|
||||
// file opens at all must not wait on other files' parses.
|
||||
void (async () => {
|
||||
// A storage read that never settles renders as a file that silently won't
|
||||
// open. Name it in the console rather than leaving the user guessing.
|
||||
const stall = setTimeout(
|
||||
() =>
|
||||
console.error(
|
||||
`[Hydration] ${stub.name} (${fileId}) has been loading for ${STALLED_LOAD_MS / 1000}s - the IndexedDB read has not settled`,
|
||||
),
|
||||
STALLED_LOAD_MS,
|
||||
);
|
||||
const stirlingFile = await fileStorage
|
||||
.getStirlingFile(fileId)
|
||||
.finally(() => clearTimeout(stall));
|
||||
if (!stirlingFile) {
|
||||
// A row with no bytes renders empty and its clicks look dead, so take it
|
||||
// back out. Storage keeps the record; fileStorage has said why.
|
||||
console.error(
|
||||
`[Hydration] No readable data for ${stub.name} (${fileId}); removing it from the workbench`,
|
||||
);
|
||||
lifecycleManager.removeFiles([fileId], stateRef);
|
||||
return;
|
||||
}
|
||||
|
||||
// Store the loaded file in filesRef
|
||||
filesRef.current.set(fileId, stirlingFile);
|
||||
|
||||
// Check if processedFile data needs regeneration
|
||||
if (stirlingFile.type.startsWith("application/pdf")) {
|
||||
const needsProcessing =
|
||||
!stub.processedFile ||
|
||||
!stub.processedFile.pages ||
|
||||
stub.processedFile.pages.length === 0 ||
|
||||
stub.processedFile.totalPages !== stub.processedFile.pages.length;
|
||||
|
||||
if (needsProcessing) {
|
||||
// Regenerate metadata
|
||||
const processedFileMetadata =
|
||||
await generateProcessedFileMetadata(stirlingFile);
|
||||
|
||||
if (processedFileMetadata) {
|
||||
const updates: Partial<StirlingFileStub> = {
|
||||
processedFile: processedFileMetadata,
|
||||
};
|
||||
|
||||
// Update thumbnail only if current stub doesn't have one
|
||||
const currentStub = stateRef.current.files.byId[fileId];
|
||||
if (
|
||||
!currentStub?.thumbnailUrl &&
|
||||
processedFileMetadata.thumbnailUrl
|
||||
) {
|
||||
updates.thumbnailUrl = processedFileMetadata.thumbnailUrl;
|
||||
if (processedFileMetadata.thumbnailUrl.startsWith("blob:")) {
|
||||
lifecycleManager.trackBlobUrl(
|
||||
processedFileMetadata.thumbnailUrl,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
lifecycleManager.updateStirlingFileStub(
|
||||
fileId,
|
||||
updates,
|
||||
stateRef,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Stub dispatch triggers re-render so the viewer appears (ADD_FILES alone doesn't update selectors).
|
||||
// filesRef is a ref, so the selectors gating the workbench only see the
|
||||
// file once something dispatches. Parsing it can't be a precondition.
|
||||
lifecycleManager.updateStirlingFileStub(fileId, {}, stateRef);
|
||||
});
|
||||
|
||||
const needsProcessing =
|
||||
!stub.processedFile ||
|
||||
!stub.processedFile.pages ||
|
||||
stub.processedFile.pages.length === 0 ||
|
||||
stub.processedFile.totalPages !== stub.processedFile.pages.length;
|
||||
if (
|
||||
stirlingFile.type.startsWith("application/pdf") &&
|
||||
needsProcessing
|
||||
) {
|
||||
scheduleMetadataFor(stirlingFile);
|
||||
}
|
||||
})().catch((error) =>
|
||||
console.error(`[Hydration] Failed to load ${fileId}:`, error),
|
||||
);
|
||||
}
|
||||
|
||||
return loadedFiles;
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
import { describe, expect, test, vi } from "vitest";
|
||||
import type {
|
||||
FileContextState,
|
||||
StirlingFileStub,
|
||||
} from "@app/types/fileContext";
|
||||
import type { FileId } from "@app/types/file";
|
||||
|
||||
/**
|
||||
* A clicked file is only visible once hydration DISPATCHES: the workbench reads
|
||||
* files out of a ref, so `activeFiles` stays empty until then. Parsing must not
|
||||
* gate that - a PDF engine that stalls used to leave the workbench on its empty
|
||||
* state with the row showing as open, and clicks doing nothing.
|
||||
*/
|
||||
|
||||
const getStirlingFile = vi.hoisted(() => vi.fn());
|
||||
vi.mock("@app/services/fileStorage", () => ({
|
||||
fileStorage: { getStirlingFile },
|
||||
}));
|
||||
/** The stall under test: the page parse never settles. */
|
||||
vi.mock("@app/utils/thumbnailUtils", () => ({
|
||||
generateThumbnailPairWithMetadata: () => new Promise(() => {}),
|
||||
}));
|
||||
|
||||
const stub = (id: string): StirlingFileStub =>
|
||||
({
|
||||
id: id as FileId,
|
||||
name: `${id}.pdf`,
|
||||
type: "application/pdf",
|
||||
size: 10,
|
||||
lastModified: 0,
|
||||
}) as StirlingFileStub;
|
||||
|
||||
async function harness(ids: string[]) {
|
||||
vi.resetModules();
|
||||
getStirlingFile.mockImplementation(
|
||||
async (id: FileId) =>
|
||||
new File(["%PDF-1.7"], `${id}.pdf`, { type: "application/pdf" }),
|
||||
);
|
||||
const { addStirlingFileStubs } =
|
||||
await import("@app/contexts/file/fileActions");
|
||||
|
||||
const stubs = ids.map(stub);
|
||||
const state = {
|
||||
files: { ids: [], byId: {} },
|
||||
pinnedFiles: new Set(),
|
||||
ui: { selectedFileIds: [], selectedPageNumbers: [] },
|
||||
} as unknown as FileContextState;
|
||||
const stateRef = { current: state };
|
||||
const filesRef = { current: new Map<FileId, File>() };
|
||||
const published: FileId[] = [];
|
||||
const lifecycleManager = {
|
||||
updateStirlingFileStub: (fileId: FileId) => published.push(fileId),
|
||||
removeFiles: () => {},
|
||||
trackBlobUrl: () => {},
|
||||
};
|
||||
|
||||
await addStirlingFileStubs(
|
||||
stubs,
|
||||
{},
|
||||
stateRef,
|
||||
filesRef,
|
||||
() => {},
|
||||
lifecycleManager as never,
|
||||
);
|
||||
return { filesRef, published };
|
||||
}
|
||||
|
||||
describe("workbench hydration — a stalled parse can't hide the file", () => {
|
||||
test("publishes every file's bytes while their parses hang", async () => {
|
||||
// Three, because the parse queue only runs two at a time: the third proves
|
||||
// loading isn't queued behind parses that never finish.
|
||||
const { filesRef, published } = await harness(["a", "b", "c"]);
|
||||
|
||||
await vi.waitFor(() => expect(published).toHaveLength(3));
|
||||
expect([...filesRef.current.keys()]).toEqual(["a", "b", "c"]);
|
||||
});
|
||||
});
|
||||
@@ -389,6 +389,18 @@ export const useFileManager = () => {
|
||||
// Optimistic update — remove from UI immediately, delete IDB in background
|
||||
setFiles(files.filter((_, i) => i !== index));
|
||||
onRemovedFromWorkbench?.(file.id);
|
||||
// Superseded versions go with it (see orphanedAncestorIds); best-effort,
|
||||
// because failing to tidy history must not fail the delete itself.
|
||||
void fileStorage
|
||||
.orphanedAncestorIds([file.id])
|
||||
.then((orphans) =>
|
||||
orphans.length > 0
|
||||
? fileStorage.deleteMultipleStirlingFiles(orphans)
|
||||
: undefined,
|
||||
)
|
||||
.catch((error) =>
|
||||
console.warn("Failed to remove superseded versions:", error),
|
||||
);
|
||||
indexedDB.deleteFile(file.id).catch((error) => {
|
||||
console.error("Failed to remove file from IndexedDB:", error);
|
||||
// Restore consistency — file is still in IDB so refresh brings it back
|
||||
|
||||
@@ -3,23 +3,26 @@ import "fake-indexeddb/auto";
|
||||
import { expectConsole } from "@app/tests/failOnConsole";
|
||||
|
||||
/**
|
||||
* Regression test for the WebKit nightly breakage introduced with the
|
||||
* large-file OOM fix (#7175): `storeStirlingFile` began putting the `File`
|
||||
* itself into IndexedDB (persisted by reference, so multi-GB uploads never
|
||||
* materialize in JS memory). WebKit refuses blob values whenever it can't write
|
||||
* the blob's backing file and rejects the request with `UnknownError: Error
|
||||
* preparing Blob/File data to be stored in object store`, so on WebKit every
|
||||
* upload silently failed to persist: files vanished on navigation, Compare
|
||||
* slots never filled, and the classification backfill had no bytes to read.
|
||||
* WebKit refuses blob values when it can't write the blob's backing file, so
|
||||
* every upload silently failed to persist after #7175. Retried as a copy now.
|
||||
*
|
||||
* The service now retries such a rejection with an ArrayBuffer copy and stops
|
||||
* offering blobs for the rest of the session.
|
||||
* It can also accept one and then lose the backing store. fake-indexeddb returns no
|
||||
* Blob, so that loss is injected at the read; real round-trips: the e2e spec.
|
||||
*/
|
||||
|
||||
const nativeAdd = IDBObjectStore.prototype.add;
|
||||
const alertMock = vi.hoisted(() => vi.fn());
|
||||
vi.mock("@app/components/toast", () => ({
|
||||
alert: (options: unknown) => alertMock(options),
|
||||
}));
|
||||
|
||||
/** What each `add` attempt carried in `data` — the blob path or the copy path. */
|
||||
const nativeAdd = IDBObjectStore.prototype.add;
|
||||
const nativePut = IDBObjectStore.prototype.put;
|
||||
const nativeGet = IDBObjectStore.prototype.get;
|
||||
|
||||
/** What each `add` attempt carried in `data`: blob path or copy path. */
|
||||
let attempts: Array<"blob" | "copy"> = [];
|
||||
/** The same, for `put` - the rewrite path a lost backing store recovers through. */
|
||||
let putAttempts: Array<"blob" | "copy"> = [];
|
||||
|
||||
/** An IDBRequest that fails asynchronously, the way WebKit rejects blob puts. */
|
||||
class FailingRequest extends EventTarget {
|
||||
@@ -32,10 +35,7 @@ class FailingRequest extends EventTarget {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record every add attempt, optionally failing the blob-valued ones the way an
|
||||
* engine without blob storage does.
|
||||
*/
|
||||
/** Record every add, optionally failing the blob-valued ones. */
|
||||
function instrumentAdd(options: { rejectBlobs: boolean }) {
|
||||
IDBObjectStore.prototype.add = function (
|
||||
this: IDBObjectStore,
|
||||
@@ -58,11 +58,72 @@ function instrumentAdd(options: { rejectBlobs: boolean }) {
|
||||
} as typeof IDBObjectStore.prototype.add;
|
||||
}
|
||||
|
||||
/**
|
||||
* A fresh service per test: whether the engine accepts blobs is remembered for
|
||||
* the process lifetime by design, so tests must not inherit that decision from
|
||||
* each other.
|
||||
*/
|
||||
/** Record every put, so the copy-rewrite recovery can be observed. */
|
||||
function instrumentPut() {
|
||||
IDBObjectStore.prototype.put = function (
|
||||
this: IDBObjectStore,
|
||||
value: unknown,
|
||||
key?: IDBValidKey,
|
||||
) {
|
||||
putAttempts.push(
|
||||
(value as { data?: unknown } | null)?.data instanceof Blob
|
||||
? "blob"
|
||||
: "copy",
|
||||
);
|
||||
return key === undefined
|
||||
? nativePut.call(this, value)
|
||||
: nativePut.call(this, value, key);
|
||||
} as typeof IDBObjectStore.prototype.put;
|
||||
}
|
||||
|
||||
/** A stored blob whose backing store the engine has lost: it still reports a name,
|
||||
* type and size, and every read of its bytes fails the way WebKit's does. */
|
||||
function blobWithLostBackingStore(): Blob {
|
||||
const lost = () => {
|
||||
throw new DOMException(
|
||||
"The object can not be found here.",
|
||||
"NotFoundError",
|
||||
);
|
||||
};
|
||||
return Object.assign(
|
||||
new Blob(["%PDF-1.7 stirling"], { type: "application/pdf" }),
|
||||
{ slice: lost, arrayBuffer: lost, text: lost, stream: lost },
|
||||
);
|
||||
}
|
||||
|
||||
/** The next `deadReads` reads come back with a lost backing store, later ones
|
||||
* untouched - so a repaired record can still be read normally. */
|
||||
function loseBackingStoreOnRead(deadReads: number) {
|
||||
let remaining = deadReads;
|
||||
IDBObjectStore.prototype.get = function (
|
||||
this: IDBObjectStore,
|
||||
key: IDBValidKey | IDBKeyRange,
|
||||
) {
|
||||
const request = nativeGet.call(this, key as IDBValidKey);
|
||||
// One substitution per request, however often `result` is read.
|
||||
let injected = false;
|
||||
return new Proxy(request, {
|
||||
get(target, prop) {
|
||||
// Receiver must be the real request: IDBRequest's accessors are branded.
|
||||
const value = Reflect.get(target, prop, target);
|
||||
if (prop !== "result") {
|
||||
return typeof value === "function" ? value.bind(target) : value;
|
||||
}
|
||||
if (!value || injected || remaining === 0) return value;
|
||||
injected = true;
|
||||
remaining--;
|
||||
return { ...(value as object), data: blobWithLostBackingStore() };
|
||||
},
|
||||
set(target, prop, value) {
|
||||
Reflect.set(target, prop, value, target);
|
||||
return true;
|
||||
},
|
||||
});
|
||||
} as typeof IDBObjectStore.prototype.get;
|
||||
}
|
||||
|
||||
/** A fresh service per test: the blob decision is remembered by design, so
|
||||
* tests must not inherit it from each other. */
|
||||
async function freshFileStorage() {
|
||||
vi.resetModules();
|
||||
const [{ fileStorage }, { createStirlingFile, createNewStirlingFileStub }] =
|
||||
@@ -86,10 +147,58 @@ async function freshFileStorage() {
|
||||
|
||||
beforeEach(() => {
|
||||
attempts = [];
|
||||
putAttempts = [];
|
||||
alertMock.mockClear();
|
||||
// The blob verdict is deliberately durable, so each test must start undecided.
|
||||
localStorage.clear();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
IDBObjectStore.prototype.add = nativeAdd;
|
||||
IDBObjectStore.prototype.put = nativePut;
|
||||
IDBObjectStore.prototype.get = nativeGet;
|
||||
});
|
||||
|
||||
/** Abort the transaction the moment a write is issued over it. */
|
||||
function abortOnPut() {
|
||||
IDBObjectStore.prototype.put = function (this: IDBObjectStore) {
|
||||
const request = new FailingRequest(
|
||||
new DOMException("transaction aborted", "AbortError"),
|
||||
) as unknown as IDBRequest<IDBValidKey>;
|
||||
this.transaction.abort();
|
||||
return request;
|
||||
} as typeof IDBObjectStore.prototype.put;
|
||||
}
|
||||
|
||||
describe("read-modify-write — a refused rewrite must not hang or vanish", () => {
|
||||
/** The abort guard used to sit on the read promise, leaving the write with a
|
||||
* dead reject - and `.catch` can't rescue a promise that never settles. */
|
||||
test("settles instead of hanging when the write transaction aborts", async () => {
|
||||
expectConsole.error(/Failed to mark file as processed/);
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const id = await store("aborts.pdf");
|
||||
|
||||
abortOnPut();
|
||||
|
||||
// Before the fix this never settled and the test timed out.
|
||||
await expect(fileStorage.markFileAsProcessed(id)).resolves.toBe(false);
|
||||
});
|
||||
|
||||
/** The copy-and-retry recovery can't be exercised here: it needs a record that
|
||||
* reads back as a Blob, which fake-indexeddb never returns. */
|
||||
test("a metadata rewrite still commits, and reports commit not put", async () => {
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const id = await store("rewrite.pdf");
|
||||
|
||||
await expect(fileStorage.markFileAsProcessed(id)).resolves.toBe(true);
|
||||
// Missing record: `false`, not a throw and not a claim of success.
|
||||
await expect(
|
||||
fileStorage.markFileAsProcessed("nope" as never),
|
||||
).resolves.toBe(false);
|
||||
expect((await fileStorage.getStirlingFile(id))?.name).toBe("rewrite.pdf");
|
||||
});
|
||||
});
|
||||
|
||||
describe("storeStirlingFile — blob-value fallback", () => {
|
||||
@@ -113,8 +222,7 @@ describe("storeStirlingFile — blob-value fallback", () => {
|
||||
const id = await store("webkit.pdf");
|
||||
|
||||
expect(attempts).toEqual(["blob", "copy"]);
|
||||
// Readable back is what every downstream consumer depends on: rehydration
|
||||
// after navigation, thumbnails, the classification backfill.
|
||||
// Readable back is what rehydration, thumbnails and backfill depend on.
|
||||
expect((await fileStorage.getStirlingFile(id))?.name).toBe("webkit.pdf");
|
||||
});
|
||||
|
||||
@@ -127,12 +235,32 @@ describe("storeStirlingFile — blob-value fallback", () => {
|
||||
attempts = [];
|
||||
const id = await store("second.pdf");
|
||||
|
||||
// Straight to the copy path — no repeated blob probe, and only the single
|
||||
// warning expected above.
|
||||
// Straight to the copy path, and only the one warning expected above.
|
||||
expect(attempts).toEqual(["copy"]);
|
||||
expect((await fileStorage.getStirlingFile(id))?.name).toBe("second.pdf");
|
||||
});
|
||||
|
||||
/** Committing is not evidence the bytes survived, and by the next reload the
|
||||
* source File is gone: without this the upload looks fine and the file is dead. */
|
||||
test("repairs a record whose stored blob loses its backing store", async () => {
|
||||
expectConsole.warn(/could not read its bytes back/);
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
instrumentPut();
|
||||
loseBackingStoreOnRead(1); // only the store's own read-back is dead
|
||||
|
||||
const id = await store("dead-on-arrival.pdf");
|
||||
|
||||
// Accepted as a blob, then rewritten from the file still in hand.
|
||||
expect(attempts).toEqual(["blob"]);
|
||||
expect(putAttempts).toEqual(["copy"]);
|
||||
expect((await fileStorage.getStirlingFile(id))?.name).toBe(
|
||||
"dead-on-arrival.pdf",
|
||||
);
|
||||
// Self-healed, so nothing to tell the user about.
|
||||
expect(alertMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("does not retry a failure a copy can't fix (quota)", async () => {
|
||||
const { store } = await freshFileStorage();
|
||||
IDBObjectStore.prototype.add = function (this: IDBObjectStore) {
|
||||
@@ -144,3 +272,246 @@ describe("storeStirlingFile — blob-value fallback", () => {
|
||||
expect(attempts).toEqual(["blob"]);
|
||||
});
|
||||
});
|
||||
|
||||
/** An earlier session's record can't be repaired, and the user has to be told - but
|
||||
* the telling must never gate the open. Awaiting the probe stalled every file in
|
||||
* Safari, where the probe read of a lost backing store never settles. */
|
||||
describe("reads — a stored blob whose bytes are gone", () => {
|
||||
test("hands the file over and reports the loss out of band", async () => {
|
||||
expectConsole.warn(/could not read its bytes back/);
|
||||
expectConsole.error(/cannot be read/);
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const id = await store("lost.pdf");
|
||||
|
||||
loseBackingStoreOnRead(5); // every read from here on
|
||||
|
||||
// Not null, and not awaited on the probe: the caller is never blocked.
|
||||
expect((await fileStorage.getStirlingFile(id))?.name).toBe("lost.pdf");
|
||||
await new Promise((resolve) => setTimeout(resolve));
|
||||
|
||||
// Told once, not once per reader: every consumer of the file hits this record.
|
||||
expect(alertMock).toHaveBeenCalledTimes(1);
|
||||
expect(alertMock.mock.calls[0][0]).toMatchObject({
|
||||
alertType: "warning",
|
||||
body: expect.stringContaining("lost.pdf"),
|
||||
});
|
||||
});
|
||||
|
||||
/** The loop this closes: a reload re-decides optimistically, writes blobs the
|
||||
* engine loses again, and the browser never settles on a shape that works. */
|
||||
test("remembers across reloads that this browser loses blob values", async () => {
|
||||
expectConsole.warn(/could not read its bytes back/);
|
||||
expectConsole.error(/cannot be read/);
|
||||
const first = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const id = await first.store("lost.pdf");
|
||||
loseBackingStoreOnRead(1);
|
||||
expect(await first.fileStorage.getStirlingFile(id)).not.toBeNull();
|
||||
await new Promise((resolve) => setTimeout(resolve));
|
||||
|
||||
// A new page load: a fresh service, same browser profile.
|
||||
const next = await freshFileStorage();
|
||||
attempts = [];
|
||||
const later = await next.store("after-reload.pdf");
|
||||
|
||||
expect(attempts).toEqual(["copy"]);
|
||||
expect((await next.fileStorage.getStirlingFile(later))?.name).toBe(
|
||||
"after-reload.pdf",
|
||||
);
|
||||
});
|
||||
|
||||
test("stops offering blob values for the rest of the session", async () => {
|
||||
expectConsole.warn(/could not read its bytes back/);
|
||||
expectConsole.error(/cannot be read/);
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const first = await store("lost.pdf");
|
||||
|
||||
loseBackingStoreOnRead(1);
|
||||
expect(await fileStorage.getStirlingFile(first)).not.toBeNull();
|
||||
await new Promise((resolve) => setTimeout(resolve));
|
||||
|
||||
// An engine that loses a blob it accepted can't be trusted with the next one,
|
||||
// so the read failure degrades writes too.
|
||||
attempts = [];
|
||||
const second = await store("later.pdf");
|
||||
expect(attempts).toEqual(["copy"]);
|
||||
expect((await fileStorage.getStirlingFile(second))?.name).toBe("later.pdf");
|
||||
});
|
||||
});
|
||||
|
||||
/** Deleting a file used to leave its superseded versions in storage forever,
|
||||
* invisible (listings filter on isLeaf) and still holding their full bytes. */
|
||||
describe("orphanedAncestorIds", () => {
|
||||
const store = async (
|
||||
fileStorage: { storeStirlingFile: (f: never, s: never) => Promise<void> },
|
||||
id: string,
|
||||
parentFileId: string | undefined,
|
||||
isLeaf: boolean,
|
||||
) => {
|
||||
const { createStirlingFile, createNewStirlingFileStub } =
|
||||
await import("@app/types/fileContext");
|
||||
const file = new File(["%PDF-1.7"], `${id}.pdf`, {
|
||||
type: "application/pdf",
|
||||
});
|
||||
const base = createNewStirlingFileStub(file);
|
||||
await fileStorage.storeStirlingFile(
|
||||
createStirlingFile(file, id as never) as never,
|
||||
{ ...base, id, isLeaf, parentFileId, originalFileId: "v1" } as never,
|
||||
);
|
||||
};
|
||||
|
||||
test("takes the superseded versions with the leaf", async () => {
|
||||
const { fileStorage } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
await store(fileStorage as never, "v1", undefined, false);
|
||||
await store(fileStorage as never, "v2", "v1", true);
|
||||
|
||||
expect(await fileStorage.orphanedAncestorIds(["v2" as never])).toEqual([
|
||||
"v1",
|
||||
]);
|
||||
});
|
||||
|
||||
test("leaves a split sibling's history alone", async () => {
|
||||
const { fileStorage } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
// Distinct ids: the fake database outlives the module reset between tests.
|
||||
await store(fileStorage as never, "split-root", undefined, false);
|
||||
await store(fileStorage as never, "split-a", "split-root", true);
|
||||
await store(fileStorage as never, "split-b", "split-root", true);
|
||||
|
||||
// `split-b` still descends from the root, so deleting `split-a` can't strip it.
|
||||
expect(await fileStorage.orphanedAncestorIds(["split-a" as never])).toEqual(
|
||||
[],
|
||||
);
|
||||
// Once both leaves go, the shared ancestor is genuinely unreachable.
|
||||
expect(
|
||||
await fileStorage.orphanedAncestorIds([
|
||||
"split-a" as never,
|
||||
"split-b" as never,
|
||||
]),
|
||||
).toEqual(["split-root"]);
|
||||
});
|
||||
});
|
||||
|
||||
/** Handing dead bytes over is only safe if whoever holds them is told to let go -
|
||||
* otherwise the viewer renders a document that never loads (an endless spinner). */
|
||||
describe("confirmed-unreadable records", () => {
|
||||
test("notifies listeners and refuses to hand the same file out twice", async () => {
|
||||
expectConsole.warn(/could not read its bytes back/);
|
||||
expectConsole.error(/cannot be read/);
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
const { onRecordUnreadable } = await import("@app/services/fileStorage");
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const id = await store("doomed.pdf");
|
||||
|
||||
const dropped: string[] = [];
|
||||
const unsubscribe = onRecordUnreadable((fileId) => dropped.push(fileId));
|
||||
|
||||
loseBackingStoreOnRead(5);
|
||||
// First read still hands the file over: the probe is out of band.
|
||||
expect(await fileStorage.getStirlingFile(id)).not.toBeNull();
|
||||
await new Promise((resolve) => setTimeout(resolve));
|
||||
|
||||
// The holder is told, so the workbench can drop it instead of spinning.
|
||||
expect(dropped).toEqual([id]);
|
||||
// And a second consumer never gets the same dead bytes.
|
||||
expect(await fileStorage.getStirlingFile(id)).toBeNull();
|
||||
|
||||
unsubscribe();
|
||||
});
|
||||
});
|
||||
|
||||
/** A readable-blob substitute, for the rescue path: fake-indexeddb never returns
|
||||
* Blob values, so a healthy legacy blob record is injected the same way a dead
|
||||
* one is. */
|
||||
function substituteHealthyBlobOnRead(reads: number) {
|
||||
let remaining = reads;
|
||||
IDBObjectStore.prototype.get = function (
|
||||
this: IDBObjectStore,
|
||||
key: IDBValidKey | IDBKeyRange,
|
||||
) {
|
||||
const request = nativeGet.call(this, key as IDBValidKey);
|
||||
let injected = false;
|
||||
return new Proxy(request, {
|
||||
get(target, prop) {
|
||||
const value = Reflect.get(target, prop, target);
|
||||
if (prop !== "result") {
|
||||
return typeof value === "function" ? value.bind(target) : value;
|
||||
}
|
||||
if (!value || injected || remaining === 0) return value;
|
||||
injected = true;
|
||||
remaining--;
|
||||
return {
|
||||
...(value as object),
|
||||
data: new Blob(["%PDF-1.7 stirling"], { type: "application/pdf" }),
|
||||
};
|
||||
},
|
||||
set(target, prop, value) {
|
||||
Reflect.set(target, prop, value, target);
|
||||
return true;
|
||||
},
|
||||
});
|
||||
} as typeof IDBObjectStore.prototype.get;
|
||||
}
|
||||
|
||||
/** The library must tell the truth per row: a record whose bytes are gone lists
|
||||
* as data-lost instead of a file that pretends to open. */
|
||||
describe("stub listings — data-lost auditing", () => {
|
||||
test("flags a dead record on the stub once the audit lands", async () => {
|
||||
expectConsole.warn(/could not read its bytes back/);
|
||||
expectConsole.error(/cannot be read/);
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
const id = await store("husk.pdf");
|
||||
|
||||
loseBackingStoreOnRead(1);
|
||||
// First read schedules the out-of-band audit; unknown is not yet flagged.
|
||||
expect(
|
||||
(await fileStorage.getStirlingFileStub(id))?.dataUnavailable,
|
||||
).toBeUndefined();
|
||||
await new Promise((resolve) => setTimeout(resolve));
|
||||
|
||||
expect((await fileStorage.getStirlingFileStub(id))?.dataUnavailable).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
test("rescues a still-readable legacy blob to a copy on a no-blob browser", async () => {
|
||||
// The durable verdict says this browser loses blob values...
|
||||
localStorage.setItem("stirling.indexeddb.blobValuesUnsupported", "true");
|
||||
const { fileStorage, store } = await freshFileStorage();
|
||||
instrumentAdd({ rejectBlobs: false });
|
||||
instrumentPut();
|
||||
const id = await store("legacy.pdf");
|
||||
|
||||
// ...and a legacy record still holds a READABLE blob: save it while we can.
|
||||
substituteHealthyBlobOnRead(5);
|
||||
await fileStorage.getStirlingFileStub(id);
|
||||
await vi.waitFor(() => expect(putAttempts).toContain("copy"));
|
||||
// Rescued, not condemned: the stub stays openable.
|
||||
expect(
|
||||
(await fileStorage.getStirlingFileStub(id))?.dataUnavailable,
|
||||
).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
/** One hung request inside the TTL bump's readwrite transaction wedged the whole
|
||||
* store: every later read and write queued behind it forever - the infinite
|
||||
* "Loading files..." after a Safari reload. Maintenance must not touch
|
||||
* blob-bodied records on a browser that can't rewrite them anyway. */
|
||||
describe("maintenanceMayRewrite", () => {
|
||||
test("keeps maintenance away from blob records on a no-blob browser", async () => {
|
||||
const { maintenanceMayRewrite } = await import("@app/services/fileStorage");
|
||||
const blobRecord = { data: new Blob(["x"]) };
|
||||
const copyRecord = { data: new ArrayBuffer(1) };
|
||||
|
||||
expect(maintenanceMayRewrite(blobRecord, false)).toBe(false);
|
||||
// Copies never hang and their rewrite is accepted - always safe.
|
||||
expect(maintenanceMayRewrite(copyRecord, false)).toBe(true);
|
||||
// On engines that genuinely support blobs (Chrome), nothing changes.
|
||||
expect(maintenanceMayRewrite(blobRecord, true)).toBe(true);
|
||||
expect(maintenanceMayRewrite(copyRecord, true)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
indexedDBManager,
|
||||
DATABASE_CONFIGS,
|
||||
} from "@app/services/indexedDBManager";
|
||||
import { alert } from "@app/components/toast";
|
||||
|
||||
/**
|
||||
* Storage record - single source of truth
|
||||
@@ -75,15 +76,135 @@ function isBlobValueRejection(error: unknown): boolean {
|
||||
return name === "UnknownError" || name === "DataCloneError";
|
||||
}
|
||||
|
||||
/** This engine loses Blob values, remembered per browser: session-scoped, each
|
||||
* reload re-decides optimistically and writes more files it will lose. */
|
||||
const BLOB_VALUES_UNSUPPORTED_KEY = "stirling.indexeddb.blobValuesUnsupported";
|
||||
|
||||
function readBlobValuesSupported(): boolean {
|
||||
try {
|
||||
return localStorage.getItem(BLOB_VALUES_UNSUPPORTED_KEY) !== "true";
|
||||
} catch {
|
||||
// Storage unavailable (private mode): decide fresh each session.
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
function persistBlobValuesUnsupported(): void {
|
||||
try {
|
||||
localStorage.setItem(BLOB_VALUES_UNSUPPORTED_KEY, "true");
|
||||
} catch {
|
||||
// Storage unavailable: the session-scoped flag still degrades this session.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether maintenance writes (the thumbnail TTL bump) may re-read/re-write this
|
||||
* record. In WebKit, a `get` touching a blob-bodied record whose backing store is
|
||||
* damaged can HANG rather than error - and one pending request wedges the whole
|
||||
* object store: every later transaction, read or write, queues behind it forever.
|
||||
* That was the infinite "Loading files..." after a reload: the TTL bump's
|
||||
* transaction never completed, so nothing else on the store ever ran. On a
|
||||
* browser whose verdict is "blobs unsupported" the rewrite would be refused
|
||||
* anyway, so blob-bodied records are not worth the risk of touching at all.
|
||||
*/
|
||||
export function maintenanceMayRewrite(
|
||||
record: { data: ArrayBuffer | Blob },
|
||||
blobValuesSupported: boolean,
|
||||
): boolean {
|
||||
return !(record.data instanceof Blob) || blobValuesSupported;
|
||||
}
|
||||
|
||||
/** WebKit loses backing stores for blobs it accepted, and only a real read shows
|
||||
* it. One byte is enough: what fails is opening the store, not the length. */
|
||||
async function blobReadFailure(data: Blob): Promise<unknown> {
|
||||
try {
|
||||
await data.slice(0, 1).arrayBuffer();
|
||||
return null;
|
||||
} catch (error) {
|
||||
return error ?? new Error("Reading a stored blob's bytes failed");
|
||||
}
|
||||
}
|
||||
|
||||
/** Notified when a record's bytes are proven unreadable, so whoever is holding the
|
||||
* file can drop it instead of rendering a document that never arrives. */
|
||||
const unreadableListeners = new Set<(fileId: FileId) => void>();
|
||||
|
||||
export function onRecordUnreadable(
|
||||
listener: (fileId: FileId) => void,
|
||||
): () => void {
|
||||
unreadableListeners.add(listener);
|
||||
return () => unreadableListeners.delete(listener);
|
||||
}
|
||||
|
||||
/** The probe read itself can hang in WebKit, so anything that awaits it needs a
|
||||
* deadline. Distinct from a failure: nothing was proven either way. */
|
||||
const PROBE_UNANSWERED = { unanswered: true } as const;
|
||||
const PROBE_DEADLINE_MS = 3000;
|
||||
|
||||
function withProbeDeadline(
|
||||
probe: Promise<unknown>,
|
||||
): Promise<unknown | typeof PROBE_UNANSWERED> {
|
||||
return Promise.race([
|
||||
probe,
|
||||
new Promise<typeof PROBE_UNANSWERED>((resolve) =>
|
||||
setTimeout(() => resolve(PROBE_UNANSWERED), PROBE_DEADLINE_MS),
|
||||
),
|
||||
]);
|
||||
}
|
||||
|
||||
/**
|
||||
* The File for a stored record. Re-wrapping a stored blob can cost WebKit the
|
||||
* backing handle, so hand it back untouched when its identity fields match.
|
||||
*/
|
||||
function fileFromRecord(record: StoredStirlingFileRecord): File {
|
||||
const { data } = record;
|
||||
if (
|
||||
data instanceof File &&
|
||||
data.name === record.name &&
|
||||
data.type === record.type &&
|
||||
data.lastModified === record.lastModified
|
||||
) {
|
||||
return data;
|
||||
}
|
||||
return new File([data], record.name, {
|
||||
type: record.type,
|
||||
lastModified: record.lastModified,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle on abort, for promises whose settle paths (a cursor tick, a request not
|
||||
* yet issued) never arrive. Call ONCE per transaction - there is one slot.
|
||||
*/
|
||||
function settleOnAbort(
|
||||
transaction: IDBTransaction,
|
||||
settle: (reason: Error) => void,
|
||||
): void {
|
||||
transaction.onabort = () =>
|
||||
settle(
|
||||
transaction.error ??
|
||||
new Error("IndexedDB transaction aborted before it completed"),
|
||||
);
|
||||
}
|
||||
|
||||
class FileStorageService {
|
||||
private readonly dbConfig = DATABASE_CONFIGS.FILES;
|
||||
private readonly storeName = "files";
|
||||
/**
|
||||
* Whether this engine accepts Blob/File values in IndexedDB. Optimistic: the
|
||||
* blob path avoids copying multi-GB files into JS memory, so we try it and
|
||||
* remember the answer, rather than pre-emptively degrading everywhere.
|
||||
*/
|
||||
private blobValuesSupported = true;
|
||||
/** Whether this engine takes Blob/File values, which avoid copying multi-GB
|
||||
* files into JS memory. Optimistic; a No outlives the session (see the key). */
|
||||
private blobValuesSupported = readBlobValuesSupported();
|
||||
/** Whether a stored blob's bytes have come back yet. Until they have, each
|
||||
* store proves it: accepting the write is no evidence the bytes survived. */
|
||||
private blobReadbackVerified = false;
|
||||
/** Ids whose TTL write failed. Without this the swallowed failure repeats a
|
||||
* whole-file rewrite on every listing. Session-scoped on purpose. */
|
||||
private readonly unwritableRecords = new Set<FileId>();
|
||||
/** Ids already reported as unreadable, so one dead record is surfaced once
|
||||
* rather than on every read of it. Session-scoped on purpose. */
|
||||
private readonly unreadableRecords = new Set<FileId>();
|
||||
/** Ids whose blob bytes this session has already audited (either way), so
|
||||
* listings don't re-probe every record on every refresh. */
|
||||
private readonly auditedRecords = new Set<FileId>();
|
||||
|
||||
/**
|
||||
* Get database connection using centralized manager
|
||||
@@ -101,7 +222,8 @@ class FileStorageService {
|
||||
|
||||
/** Fire-and-forget: bump thumbnailStoredAt (or clear expired thumbnail) for a set of ids. */
|
||||
private async bumpThumbnailTTL(ids: FileId[], clear = false): Promise<void> {
|
||||
if (ids.length === 0) return;
|
||||
const targets = ids.filter((id) => !this.unwritableRecords.has(id));
|
||||
if (targets.length === 0) return;
|
||||
const db = await this.getDatabase();
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
@@ -112,7 +234,7 @@ class FileStorageService {
|
||||
|
||||
// Issue all gets up front - each onsuccess creates a put before the
|
||||
// transaction can auto-commit, keeping it alive until all puts settle.
|
||||
ids.forEach((id) => {
|
||||
targets.forEach((id) => {
|
||||
const req = store.get(id);
|
||||
req.onsuccess = () => {
|
||||
const record = req.result as StoredStirlingFileRecord | undefined;
|
||||
@@ -123,7 +245,30 @@ class FileStorageService {
|
||||
} else {
|
||||
record.thumbnailStoredAt = Date.now();
|
||||
}
|
||||
store.put(record);
|
||||
// One unwritable record must not take the batch with it: a rejected
|
||||
// put aborts the transaction the other queued gets are still using.
|
||||
try {
|
||||
const put = store.put(record);
|
||||
put.onerror = (event) => {
|
||||
// The write we just swallowed is the one that would have taken
|
||||
// this record out of the expiring set, so stop retrying it.
|
||||
this.unwritableRecords.add(id);
|
||||
this.noteBlobRefusal(put.error);
|
||||
console.warn(
|
||||
`[fileStorage] thumbnail TTL bump skipped for ${id}:`,
|
||||
put.error,
|
||||
);
|
||||
// Swallow it here so the failure doesn't abort the transaction.
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
};
|
||||
} catch (error) {
|
||||
this.unwritableRecords.add(id);
|
||||
console.warn(
|
||||
`[fileStorage] thumbnail TTL bump could not be issued for ${id}:`,
|
||||
error,
|
||||
);
|
||||
}
|
||||
};
|
||||
req.onerror = () => reject(req.error);
|
||||
});
|
||||
@@ -186,18 +331,255 @@ class FileStorageService {
|
||||
} catch (error) {
|
||||
// Recoverable: re-add as a copy, and stop offering blobs this session.
|
||||
// Anything else is the caller's to report.
|
||||
if (!(record.data instanceof Blob) || !isBlobValueRejection(error)) {
|
||||
if (!(record.data instanceof Blob) || !this.noteBlobRefusal(error)) {
|
||||
throw error;
|
||||
}
|
||||
this.blobValuesSupported = false;
|
||||
console.warn(
|
||||
"IndexedDB rejected a Blob value; falling back to in-memory copies for this session. " +
|
||||
"Very large files may now exhaust renderer memory.",
|
||||
error,
|
||||
);
|
||||
record.data = await record.data.arrayBuffer();
|
||||
await this.addFileRecord(db, record);
|
||||
return;
|
||||
}
|
||||
|
||||
// Committed is not retrievable. Prove the round-trip while the source File is
|
||||
// still in hand; after a reload there is nothing left to repair from.
|
||||
if (record.data instanceof Blob && !this.blobReadbackVerified) {
|
||||
await this.verifyStoredBlobReadable(db, record, stirlingFile);
|
||||
}
|
||||
}
|
||||
|
||||
/** Read one stored blob back, rewriting the record from {@code source} if its
|
||||
* bytes don't come with it. Runs until one round-trip succeeds. */
|
||||
private async verifyStoredBlobReadable(
|
||||
db: IDBDatabase,
|
||||
record: StoredStirlingFileRecord,
|
||||
source: File,
|
||||
): Promise<void> {
|
||||
// A record we can't read back at all is the caller's problem, not the probe's.
|
||||
const stored = await this.readRecord(db, record.id).catch(() => undefined);
|
||||
if (!(stored?.data instanceof Blob)) return;
|
||||
|
||||
const failure = await withProbeDeadline(blobReadFailure(stored.data));
|
||||
if (!failure) {
|
||||
this.blobReadbackVerified = true;
|
||||
return;
|
||||
}
|
||||
if (failure === PROBE_UNANSWERED) {
|
||||
// Nothing proven, and an upload must never wait on a probe. Leave the record
|
||||
// as written; the read path reports it if the bytes really are gone.
|
||||
console.warn(
|
||||
`[fileStorage] readability probe for ${record.id} did not answer in ${PROBE_DEADLINE_MS}ms`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
this.noteBlobUnreadable(failure);
|
||||
try {
|
||||
record.data = await source.arrayBuffer();
|
||||
await this.putRecord(db, record);
|
||||
} catch (error) {
|
||||
// The record is unusable either way, and the read path reports that to the
|
||||
// user. Don't turn a write that already committed into a failure.
|
||||
console.warn(
|
||||
`[fileStorage] could not rewrite ${record.id} as an in-memory copy:`,
|
||||
error,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Refused a Blob value? Stop offering blobs on this browser. Any write can flip
|
||||
* this: WebKit refuses per-operation, not per-engine. */
|
||||
private noteBlobRefusal(error: unknown): boolean {
|
||||
if (!isBlobValueRejection(error)) return false;
|
||||
this.disableBlobValues(
|
||||
"IndexedDB rejected a Blob value; falling back to in-memory copies on this browser. " +
|
||||
"Very large files may now exhaust renderer memory.",
|
||||
error,
|
||||
);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** A stored blob whose bytes won't read back means blob values can't be trusted
|
||||
* on this engine either, even though it accepted the write. */
|
||||
private noteBlobUnreadable(error: unknown): void {
|
||||
this.disableBlobValues(
|
||||
"IndexedDB accepted a Blob value but could not read its bytes back; " +
|
||||
"falling back to in-memory copies on this browser. " +
|
||||
"Very large files may now exhaust renderer memory.",
|
||||
error,
|
||||
);
|
||||
}
|
||||
|
||||
private disableBlobValues(message: string, error: unknown): void {
|
||||
if (!this.blobValuesSupported) return;
|
||||
this.blobValuesSupported = false;
|
||||
persistBlobValuesUnsupported();
|
||||
console.warn(message, error);
|
||||
}
|
||||
|
||||
/**
|
||||
* Audit a blob-backed record's bytes WITHOUT gating anything on the answer.
|
||||
* Awaiting this was a mistake: in Safari the probe read of a lost backing store
|
||||
* can stay pending forever, so it stalled every file open instead of the one
|
||||
* consumer that would have failed anyway.
|
||||
*
|
||||
* Two outcomes, both out of band:
|
||||
* - Bytes gone: mark + report, so the library shows "data lost" instead of a
|
||||
* file that pretends to open.
|
||||
* - Bytes readable on a browser whose verdict is "blobs unsupported": RESCUE the
|
||||
* record to an ArrayBuffer copy now, while the bytes still exist. Legacy blob
|
||||
* records on WebKit are one engine hiccup away from being lost for good.
|
||||
*/
|
||||
private reportIfUnreadable(record: StoredStirlingFileRecord): void {
|
||||
if (!(record.data instanceof Blob)) return;
|
||||
if (this.auditedRecords.has(record.id)) return;
|
||||
this.auditedRecords.add(record.id);
|
||||
void blobReadFailure(record.data).then((failure) => {
|
||||
if (!failure) {
|
||||
this.blobReadbackVerified = true;
|
||||
if (!this.blobValuesSupported) void this.rescueBlobRecord(record.id);
|
||||
return;
|
||||
}
|
||||
this.noteBlobUnreadable(failure);
|
||||
this.reportUnreadableRecord(record, failure);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Rewrite one still-readable legacy blob record as an ArrayBuffer copy. Reads
|
||||
* the FULL bytes (the audit only proved the first one) and goes through
|
||||
* {@link updateRecord}'s read-modify-write so a concurrent metadata update
|
||||
* isn't clobbered by a stale snapshot.
|
||||
*/
|
||||
private async rescueBlobRecord(fileId: FileId): Promise<void> {
|
||||
try {
|
||||
const db = await this.getDatabase();
|
||||
const record = await this.readRecord(db, fileId);
|
||||
if (!(record?.data instanceof Blob)) return;
|
||||
const bytes = await withProbeDeadline(record.data.arrayBuffer());
|
||||
if (bytes === PROBE_UNANSWERED || !(bytes instanceof ArrayBuffer)) return;
|
||||
record.data = bytes;
|
||||
await this.putRecord(db, record);
|
||||
console.info(
|
||||
`[fileStorage] rescued "${record.name}" (${fileId}) to an in-memory copy before this browser could lose its blob`,
|
||||
);
|
||||
} catch (error) {
|
||||
// Best-effort: a failed rescue leaves the record exactly as it was.
|
||||
console.warn(`[fileStorage] could not rescue ${fileId}:`, error);
|
||||
}
|
||||
}
|
||||
|
||||
/** One console error and one toast per dead record: every consumer of the file
|
||||
* hits the same record, and the user needs the reason once, not per reader. */
|
||||
private reportUnreadableRecord(
|
||||
record: StoredStirlingFileRecord,
|
||||
failure: unknown,
|
||||
): void {
|
||||
if (this.unreadableRecords.has(record.id)) return;
|
||||
this.unreadableRecords.add(record.id);
|
||||
// Whoever is holding it needs to let go, or the viewer renders a document
|
||||
// whose bytes never arrive - a spinner with no terminal state.
|
||||
for (const listener of unreadableListeners) listener(record.id);
|
||||
console.error(
|
||||
`[fileStorage] stored data for "${record.name}" (${record.id}) cannot be read; ` +
|
||||
"the browser no longer has the blob's backing store",
|
||||
failure,
|
||||
);
|
||||
alert({
|
||||
alertType: "warning",
|
||||
title: "File data is unavailable",
|
||||
body:
|
||||
`"${record.name}" is saved in this browser but its contents can no longer be read. ` +
|
||||
"Upload the file again to keep working on it.",
|
||||
expandable: false,
|
||||
durationMs: 8000,
|
||||
});
|
||||
}
|
||||
|
||||
/** Read-modify-write one record in one transaction, resolving on COMMIT. Split
|
||||
* across two promises, the abort guard covers one and the other hangs. */
|
||||
private async updateRecord(
|
||||
fileId: FileId,
|
||||
mutate: (record: StoredStirlingFileRecord) => boolean | void,
|
||||
): Promise<boolean> {
|
||||
const db = await this.getDatabase();
|
||||
try {
|
||||
return await this.readModifyWrite(db, fileId, mutate);
|
||||
} catch (error) {
|
||||
// The record we read back still carries its Blob body; retry as a copy.
|
||||
if (!this.noteBlobRefusal(error)) throw error;
|
||||
return await this.rewriteRecordAsCopy(db, fileId, mutate);
|
||||
}
|
||||
}
|
||||
|
||||
/** {@link updateRecord}'s happy path: one transaction, resolve on commit. */
|
||||
private readModifyWrite(
|
||||
db: IDBDatabase,
|
||||
fileId: FileId,
|
||||
mutate: (record: StoredStirlingFileRecord) => boolean | void,
|
||||
): Promise<boolean> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
let written = false;
|
||||
settleOnAbort(transaction, reject);
|
||||
transaction.onerror = () => reject(transaction.error);
|
||||
transaction.oncomplete = () => resolve(written);
|
||||
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const getRequest = store.get(fileId);
|
||||
getRequest.onerror = () => reject(getRequest.error);
|
||||
getRequest.onsuccess = () => {
|
||||
const record = getRequest.result as
|
||||
| StoredStirlingFileRecord
|
||||
| undefined;
|
||||
// Nothing to write: let the empty transaction commit and report false.
|
||||
if (!record || mutate(record) === false) return;
|
||||
written = true;
|
||||
store.put(record);
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Recovery path: two transactions, because materializing the copy is async
|
||||
* and a transaction cannot survive an await. Last-write-wins either way. */
|
||||
private async rewriteRecordAsCopy(
|
||||
db: IDBDatabase,
|
||||
fileId: FileId,
|
||||
mutate: (record: StoredStirlingFileRecord) => boolean | void,
|
||||
): Promise<boolean> {
|
||||
const record = await this.readRecord(db, fileId);
|
||||
if (!record || mutate(record) === false) return false;
|
||||
if (record.data instanceof Blob) {
|
||||
record.data = await record.data.arrayBuffer();
|
||||
}
|
||||
await this.putRecord(db, record);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** One record by id, in its own transaction. */
|
||||
private readRecord(
|
||||
db: IDBDatabase,
|
||||
fileId: FileId,
|
||||
): Promise<StoredStirlingFileRecord | undefined> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readonly");
|
||||
settleOnAbort(transaction, reject);
|
||||
const request = transaction.objectStore(this.storeName).get(fileId);
|
||||
request.onerror = () => reject(request.error);
|
||||
request.onsuccess = () => resolve(request.result);
|
||||
});
|
||||
}
|
||||
|
||||
/** One `put`, resolving on commit. */
|
||||
private putRecord(
|
||||
db: IDBDatabase,
|
||||
record: StoredStirlingFileRecord,
|
||||
): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
settleOnAbort(transaction, reject);
|
||||
transaction.onerror = () => reject(transaction.error);
|
||||
transaction.oncomplete = () => resolve();
|
||||
transaction.objectStore(this.storeName).put(record);
|
||||
});
|
||||
}
|
||||
|
||||
/** Single `add` of a file record. Rejects with the underlying IDB error. */
|
||||
@@ -215,6 +597,7 @@ class FileStorageService {
|
||||
}
|
||||
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
settleOnAbort(transaction, reject);
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
|
||||
const request = store.add(record);
|
||||
@@ -227,37 +610,21 @@ class FileStorageService {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Get StirlingFile with full data - for loading into workbench
|
||||
*/
|
||||
/** Get StirlingFile with full data - for loading into workbench. Null covers
|
||||
* both no such record and bytes gone; neither is a file callers can use. */
|
||||
async getStirlingFile(id: FileId): Promise<StirlingFile | null> {
|
||||
// Already proven unreadable this session: don't hand the same dead bytes to
|
||||
// another consumer that will spin on them. Session-scoped, so a reload retries.
|
||||
if (this.unreadableRecords.has(id)) return null;
|
||||
const db = await this.getDatabase();
|
||||
const record = await this.readRecord(db, id);
|
||||
if (!record) return null;
|
||||
// Reporting only, and NEVER awaited: WebKit can leave a read of a lost backing
|
||||
// store pending forever, and this is the path every file open goes through.
|
||||
this.reportIfUnreadable(record);
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readonly");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.get(id);
|
||||
|
||||
request.onerror = () => reject(request.error);
|
||||
request.onsuccess = () => {
|
||||
const record = request.result as StoredStirlingFileRecord | undefined;
|
||||
if (!record) {
|
||||
resolve(null);
|
||||
return;
|
||||
}
|
||||
|
||||
// Create File from stored data
|
||||
const blob = new Blob([record.data], { type: record.type });
|
||||
const file = new File([blob], record.name, {
|
||||
type: record.type,
|
||||
lastModified: record.lastModified,
|
||||
});
|
||||
|
||||
// Convert to StirlingFile with preserved IDs
|
||||
const stirlingFile = createStirlingFile(file, record.fileId);
|
||||
resolve(stirlingFile);
|
||||
};
|
||||
});
|
||||
// Convert to StirlingFile with preserved IDs
|
||||
return createStirlingFile(fileFromRecord(record), record.fileId);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -278,6 +645,7 @@ class FileStorageService {
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readonly");
|
||||
settleOnAbort(transaction, reject);
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.get(id);
|
||||
|
||||
@@ -295,9 +663,13 @@ class FileStorageService {
|
||||
// We still gate thumbnailUrl on freshness so stale thumbnails
|
||||
// don't leak through this read path.
|
||||
const fresh = this.isThumbnailFresh(record);
|
||||
// Out-of-band byte audit, so the library reflects lost data (and rescues
|
||||
// still-readable legacy blobs) instead of listing files that can't open.
|
||||
this.reportIfUnreadable(record);
|
||||
|
||||
const stub: StirlingFileStub = {
|
||||
id: record.id,
|
||||
dataUnavailable: this.unreadableRecords.has(record.id) || undefined,
|
||||
name: record.name,
|
||||
type: record.type,
|
||||
size: record.size,
|
||||
@@ -338,6 +710,7 @@ class FileStorageService {
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readonly");
|
||||
settleOnAbort(transaction, reject);
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.openCursor();
|
||||
const stubs: StirlingFileStub[] = [];
|
||||
@@ -352,12 +725,18 @@ class FileStorageService {
|
||||
const record = cursor.value as StoredStirlingFileRecord;
|
||||
if (record && record.name && typeof record.size === "number") {
|
||||
const fresh = this.isThumbnailFresh(record);
|
||||
if (record.thumbnail) {
|
||||
if (
|
||||
record.thumbnail &&
|
||||
maintenanceMayRewrite(record, this.blobValuesSupported)
|
||||
) {
|
||||
if (fresh) tobump.push(record.id);
|
||||
else toexpire.push(record.id);
|
||||
}
|
||||
this.reportIfUnreadable(record);
|
||||
stubs.push({
|
||||
id: record.id,
|
||||
dataUnavailable:
|
||||
this.unreadableRecords.has(record.id) || undefined,
|
||||
name: record.name,
|
||||
type: record.type,
|
||||
size: record.size,
|
||||
@@ -425,6 +804,7 @@ class FileStorageService {
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readonly");
|
||||
settleOnAbort(transaction, reject);
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.openCursor();
|
||||
const leafStubs: StirlingFileStub[] = [];
|
||||
@@ -444,12 +824,18 @@ class FileStorageService {
|
||||
record.isLeaf !== false
|
||||
) {
|
||||
const fresh = this.isThumbnailFresh(record);
|
||||
if (record.thumbnail) {
|
||||
if (
|
||||
record.thumbnail &&
|
||||
maintenanceMayRewrite(record, this.blobValuesSupported)
|
||||
) {
|
||||
if (fresh) tobump.push(record.id);
|
||||
else toexpire.push(record.id);
|
||||
}
|
||||
this.reportIfUnreadable(record);
|
||||
leafStubs.push({
|
||||
id: record.id,
|
||||
dataUnavailable:
|
||||
this.unreadableRecords.has(record.id) || undefined,
|
||||
name: record.name,
|
||||
type: record.type,
|
||||
size: record.size,
|
||||
@@ -579,6 +965,46 @@ class FileStorageService {
|
||||
return cleared;
|
||||
}
|
||||
|
||||
/**
|
||||
* Superseded versions that nothing else needs once {@code deleting} goes.
|
||||
*
|
||||
* Deleting a file removes one record; its older versions keep their full bytes
|
||||
* and are invisible (listings filter on isLeaf), so they accumulate forever.
|
||||
* Only for user-facing "delete this file" - deleting ONE version from the
|
||||
* history journey must leave the rest of the chain alone.
|
||||
*/
|
||||
async orphanedAncestorIds(deleting: FileId[]): Promise<FileId[]> {
|
||||
if (deleting.length === 0) return [];
|
||||
const stubs = await this.getAllStirlingFileStubs();
|
||||
const byId = new Map(stubs.map((s) => [s.id as string, s]));
|
||||
const doomed = new Set(deleting.map(String));
|
||||
|
||||
// Anything a surviving record descends from has to stay: split siblings
|
||||
// share a lineage, so one leaf's delete must not strip another's history.
|
||||
const keep = new Set<string>();
|
||||
for (const stub of stubs) {
|
||||
if (doomed.has(stub.id as string)) continue;
|
||||
let cursor = stub.parentFileId as string | undefined;
|
||||
while (cursor && !keep.has(cursor)) {
|
||||
keep.add(cursor);
|
||||
cursor = byId.get(cursor)?.parentFileId as string | undefined;
|
||||
}
|
||||
}
|
||||
|
||||
const orphans: FileId[] = [];
|
||||
for (const id of deleting) {
|
||||
let cursor = byId.get(String(id))?.parentFileId as string | undefined;
|
||||
while (cursor) {
|
||||
if (!keep.has(cursor) && !doomed.has(cursor) && byId.has(cursor)) {
|
||||
doomed.add(cursor);
|
||||
orphans.push(cursor as FileId);
|
||||
}
|
||||
cursor = byId.get(cursor)?.parentFileId as string | undefined;
|
||||
}
|
||||
}
|
||||
return orphans;
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete StirlingFile - single operation, no sync issues
|
||||
*/
|
||||
@@ -587,11 +1013,12 @@ class FileStorageService {
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.delete(id);
|
||||
|
||||
request.onerror = () => reject(request.error);
|
||||
request.onsuccess = () => resolve();
|
||||
// On commit, not on the request: callers refresh their list from storage as
|
||||
// soon as this resolves, and an aborted delete would put the row back.
|
||||
settleOnAbort(transaction, reject);
|
||||
transaction.onerror = () => reject(transaction.error);
|
||||
transaction.oncomplete = () => resolve();
|
||||
transaction.objectStore(this.storeName).delete(id);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -617,45 +1044,16 @@ class FileStorageService {
|
||||
* Update thumbnail for existing file
|
||||
*/
|
||||
async updateThumbnail(id: FileId, thumbnail: string): Promise<boolean> {
|
||||
const db = await this.getDatabase();
|
||||
|
||||
return new Promise((resolve, _reject) => {
|
||||
try {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const getRequest = store.get(id);
|
||||
|
||||
getRequest.onsuccess = () => {
|
||||
const record = getRequest.result as StoredStirlingFileRecord;
|
||||
if (record) {
|
||||
record.thumbnail = thumbnail;
|
||||
record.thumbnailStoredAt = Date.now();
|
||||
const updateRequest = store.put(record);
|
||||
|
||||
updateRequest.onsuccess = () => {
|
||||
resolve(true);
|
||||
};
|
||||
updateRequest.onerror = () => {
|
||||
console.error("Failed to update thumbnail:", updateRequest.error);
|
||||
resolve(false);
|
||||
};
|
||||
} else {
|
||||
resolve(false);
|
||||
}
|
||||
};
|
||||
|
||||
getRequest.onerror = () => {
|
||||
console.error(
|
||||
"Failed to get file for thumbnail update:",
|
||||
getRequest.error,
|
||||
);
|
||||
resolve(false);
|
||||
};
|
||||
} catch (error) {
|
||||
console.error("Transaction error during thumbnail update:", error);
|
||||
resolve(false);
|
||||
}
|
||||
});
|
||||
// Reports failure as `false` rather than rejecting; callers just need an answer.
|
||||
try {
|
||||
return await this.updateRecord(id, (record) => {
|
||||
record.thumbnail = thumbnail;
|
||||
record.thumbnailStoredAt = Date.now();
|
||||
});
|
||||
} catch (error) {
|
||||
console.error("Failed to update thumbnail:", error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -666,6 +1064,7 @@ class FileStorageService {
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
settleOnAbort(transaction, reject);
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.clear();
|
||||
|
||||
@@ -720,24 +1119,16 @@ class FileStorageService {
|
||||
async createBlobUrl(id: FileId): Promise<string | null> {
|
||||
try {
|
||||
const db = await this.getDatabase();
|
||||
const record = await this.readRecord(db, id);
|
||||
if (!record) return null;
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readonly");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
const request = store.get(id);
|
||||
|
||||
request.onerror = () => reject(request.error);
|
||||
request.onsuccess = () => {
|
||||
const record = request.result as StoredStirlingFileRecord | undefined;
|
||||
if (record) {
|
||||
const blob = new Blob([record.data], { type: record.type });
|
||||
const url = URL.createObjectURL(blob);
|
||||
resolve(url);
|
||||
} else {
|
||||
resolve(null);
|
||||
}
|
||||
};
|
||||
});
|
||||
// Stored blobs are handed straight to createObjectURL — re-wrapping
|
||||
// one can cost WebKit the backing handle. See fileFromRecord.
|
||||
const blob =
|
||||
record.data instanceof Blob
|
||||
? record.data
|
||||
: new Blob([record.data], { type: record.type });
|
||||
return URL.createObjectURL(blob);
|
||||
} catch (error) {
|
||||
console.warn(`Failed to create blob URL for ${id}:`, error);
|
||||
return null;
|
||||
@@ -750,32 +1141,9 @@ class FileStorageService {
|
||||
*/
|
||||
async markFileAsProcessed(fileId: FileId): Promise<boolean> {
|
||||
try {
|
||||
const db = await this.getDatabase();
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
|
||||
const record = await new Promise<StoredStirlingFileRecord | undefined>(
|
||||
(resolve, reject) => {
|
||||
const request = store.get(fileId);
|
||||
request.onsuccess = () => resolve(request.result);
|
||||
request.onerror = () => reject(request.error);
|
||||
},
|
||||
);
|
||||
|
||||
if (!record) {
|
||||
return false; // File not found
|
||||
}
|
||||
|
||||
// Update the isLeaf flag to false
|
||||
record.isLeaf = false;
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const request = store.put(record);
|
||||
request.onsuccess = () => resolve();
|
||||
request.onerror = () => reject(request.error);
|
||||
return await this.updateRecord(fileId, (record) => {
|
||||
record.isLeaf = false;
|
||||
});
|
||||
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.error("Failed to mark file as processed:", error);
|
||||
return false;
|
||||
@@ -835,32 +1203,9 @@ class FileStorageService {
|
||||
*/
|
||||
async markFileAsLeaf(fileId: FileId): Promise<boolean> {
|
||||
try {
|
||||
const db = await this.getDatabase();
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
|
||||
const record = await new Promise<StoredStirlingFileRecord | undefined>(
|
||||
(resolve, reject) => {
|
||||
const request = store.get(fileId);
|
||||
request.onsuccess = () => resolve(request.result);
|
||||
request.onerror = () => reject(request.error);
|
||||
},
|
||||
);
|
||||
|
||||
if (!record) {
|
||||
return false; // File not found
|
||||
}
|
||||
|
||||
// Update the isLeaf flag to true
|
||||
record.isLeaf = true;
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const request = store.put(record);
|
||||
request.onsuccess = () => resolve();
|
||||
request.onerror = () => reject(request.error);
|
||||
return await this.updateRecord(fileId, (record) => {
|
||||
record.isLeaf = true;
|
||||
});
|
||||
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.error("Failed to mark file as leaf:", error);
|
||||
return false;
|
||||
@@ -870,41 +1215,16 @@ class FileStorageService {
|
||||
/**
|
||||
* Update metadata fields for a stored file record.
|
||||
*
|
||||
* Resolves on transaction.oncomplete, NOT on the individual put's onsuccess,
|
||||
* so callers only receive `true` once the write actually commits. If the
|
||||
* transaction aborts after put() succeeded but before commit, we return false
|
||||
* - the previous behavior incorrectly claimed success in that window.
|
||||
* Returns `true` only once the write commits, never on the put's `onsuccess`.
|
||||
* {@link updateRecord} owns that guarantee for every write in this class.
|
||||
*/
|
||||
async updateFileMetadata(
|
||||
fileId: FileId,
|
||||
updates: Partial<StoredStirlingFileRecord>,
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
const db = await this.getDatabase();
|
||||
return await new Promise<boolean>((resolve, reject) => {
|
||||
const transaction = db.transaction([this.storeName], "readwrite");
|
||||
const store = transaction.objectStore(this.storeName);
|
||||
let recordFound = false;
|
||||
|
||||
const getRequest = store.get(fileId);
|
||||
getRequest.onsuccess = () => {
|
||||
const record = getRequest.result as
|
||||
| StoredStirlingFileRecord
|
||||
| undefined;
|
||||
if (!record) {
|
||||
// Don't commit anything; caller wants false.
|
||||
return;
|
||||
}
|
||||
recordFound = true;
|
||||
const updatedRecord = { ...record, ...updates };
|
||||
store.put(updatedRecord);
|
||||
};
|
||||
getRequest.onerror = () => reject(getRequest.error);
|
||||
|
||||
transaction.oncomplete = () => resolve(recordFound);
|
||||
transaction.onerror = () => reject(transaction.error);
|
||||
transaction.onabort = () =>
|
||||
reject(transaction.error ?? new Error("updateFileMetadata aborted"));
|
||||
return await this.updateRecord(fileId, (record) => {
|
||||
Object.assign(record, updates);
|
||||
});
|
||||
} catch (error) {
|
||||
console.error("Failed to update file metadata:", error);
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
import { afterEach, describe, expect, test, vi } from "vitest";
|
||||
import "fake-indexeddb/auto";
|
||||
import { expectConsole } from "@app/tests/failOnConsole";
|
||||
import type { DatabaseConfig } from "@app/services/indexedDBManager";
|
||||
|
||||
/**
|
||||
* A blocked open fires `blocked` and then nothing at all - no success, no error -
|
||||
* until the other connection goes away. Unguarded, the open promise never settles
|
||||
* and every caller hangs SILENTLY: the file library spun forever with an empty
|
||||
* console, which is why this kept being reported as unreproducible.
|
||||
*/
|
||||
|
||||
const config = (name: string, version: number): DatabaseConfig => ({
|
||||
name,
|
||||
version,
|
||||
stores: [{ name: "things", keyPath: "id" }],
|
||||
});
|
||||
|
||||
/** A raw connection on an older version that never yields, i.e. the other tab. */
|
||||
function holdOlderVersion(name: string): Promise<IDBDatabase> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const request = indexedDB.open(name, 1);
|
||||
request.onupgradeneeded = () => {
|
||||
if (!request.result.objectStoreNames.contains("things")) {
|
||||
request.result.createObjectStore("things", { keyPath: "id" });
|
||||
}
|
||||
};
|
||||
request.onsuccess = () => resolve(request.result);
|
||||
request.onerror = () => reject(request.error);
|
||||
});
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
vi.resetModules();
|
||||
});
|
||||
|
||||
describe("openDatabase — blocked by another connection", () => {
|
||||
test("rejects with something actionable instead of hanging", async () => {
|
||||
expectConsole.warn(/blocked by another connection/);
|
||||
const { indexedDBManager } = await import("@app/services/indexedDBManager");
|
||||
const held = await holdOlderVersion("blocked-db");
|
||||
|
||||
vi.useFakeTimers();
|
||||
const open = indexedDBManager.openDatabase(config("blocked-db", 2));
|
||||
const settled = vi.fn();
|
||||
void open.then(settled, settled);
|
||||
|
||||
// Still pending before the grace period is up: a tab that yields quickly
|
||||
// must not be failed prematurely.
|
||||
await vi.advanceTimersByTimeAsync(4_000);
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
|
||||
await vi.advanceTimersByTimeAsync(2_000);
|
||||
await expect(open).rejects.toThrow(/blocked by another connection/);
|
||||
|
||||
held.close();
|
||||
});
|
||||
|
||||
test("dedupes concurrent callers onto one connection", async () => {
|
||||
const { indexedDBManager } = await import("@app/services/indexedDBManager");
|
||||
const spy = vi.spyOn(indexedDB, "open");
|
||||
|
||||
// Racing in the same tick is the case registration-after-await could not
|
||||
// dedupe, and only the first request would ever receive `blocked`.
|
||||
const [a, b, c] = await Promise.all([
|
||||
indexedDBManager.openDatabase(config("shared-db", 1)),
|
||||
indexedDBManager.openDatabase(config("shared-db", 1)),
|
||||
indexedDBManager.openDatabase(config("shared-db", 1)),
|
||||
]);
|
||||
|
||||
expect(a).toBe(b);
|
||||
expect(b).toBe(c);
|
||||
expect(
|
||||
spy.mock.calls.filter(([name]) => name === "shared-db"),
|
||||
).toHaveLength(1);
|
||||
spy.mockRestore();
|
||||
});
|
||||
});
|
||||
@@ -19,6 +19,11 @@ export interface DatabaseConfig {
|
||||
}[];
|
||||
}
|
||||
|
||||
/** How long to wait out another connection before failing an open with something
|
||||
* the user can act on. Rejecting does NOT cancel the request, so a connection
|
||||
* that arrives later is closed rather than held. */
|
||||
const BLOCKED_GRACE_MS = 5000;
|
||||
|
||||
class IndexedDBManager {
|
||||
private static instance: IndexedDBManager;
|
||||
private databases = new Map<string, IDBDatabase>();
|
||||
@@ -47,6 +52,26 @@ class IndexedDBManager {
|
||||
return existingPromise;
|
||||
}
|
||||
|
||||
// Registered BEFORE anything async. A map written after a yield point can't
|
||||
// dedupe callers racing into it in the same tick, so every context that opened
|
||||
// this database during boot got its own connection - and per spec only the
|
||||
// FIRST request ever receives `blocked`, leaving the rest waiting on an event
|
||||
// that never comes.
|
||||
const initPromise = this.openWithRecovery(config);
|
||||
this.initPromises.set(config.name, initPromise);
|
||||
|
||||
try {
|
||||
const db = await initPromise;
|
||||
this.databases.set(config.name, db);
|
||||
return db;
|
||||
} catch (error) {
|
||||
this.initPromises.delete(config.name);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/** The v6/v7 wipe, kept off {@link openDatabase}'s synchronous registration path. */
|
||||
private async openWithRecovery(config: DatabaseConfig): Promise<IDBDatabase> {
|
||||
// SaaS lineage shipped a v6 and a v7 of stirling-pdf-files whose
|
||||
// upgrade paths corrupted records (separate cursor walks racing in
|
||||
// one versionchange transaction). The SaaS build wipes those
|
||||
@@ -64,18 +89,7 @@ class IndexedDBManager {
|
||||
await this.deleteDatabase(config.name);
|
||||
}
|
||||
}
|
||||
|
||||
const initPromise = this.performDatabaseInit(config);
|
||||
this.initPromises.set(config.name, initPromise);
|
||||
|
||||
try {
|
||||
const db = await initPromise;
|
||||
this.databases.set(config.name, db);
|
||||
return db;
|
||||
} catch (error) {
|
||||
this.initPromises.delete(config.name);
|
||||
throw error;
|
||||
}
|
||||
return this.performDatabaseInit(config);
|
||||
}
|
||||
|
||||
private performDatabaseInit(config: DatabaseConfig): Promise<IDBDatabase> {
|
||||
@@ -83,15 +97,60 @@ class IndexedDBManager {
|
||||
console.log(`Opening IndexedDB: ${config.name} v${config.version}`);
|
||||
const request = indexedDB.open(config.name, config.version);
|
||||
|
||||
// A blocked upgrade fires `blocked` and then NOTHING - no success, no error -
|
||||
// until the other connection goes away. Unguarded, the promise never settles
|
||||
// and every awaiting caller hangs with nothing in the console.
|
||||
let settled = false;
|
||||
let blockedTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
|
||||
request.onblocked = () => {
|
||||
console.warn(
|
||||
`Opening ${config.name} is blocked by another connection (another tab on an older version?). ` +
|
||||
`Giving up in ${BLOCKED_GRACE_MS}ms if it doesn't yield.`,
|
||||
);
|
||||
blockedTimer = setTimeout(() => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
reject(
|
||||
new Error(
|
||||
`Opening ${config.name} was blocked by another connection for ${BLOCKED_GRACE_MS}ms. ` +
|
||||
"Close other tabs of this app and reload.",
|
||||
),
|
||||
);
|
||||
}, BLOCKED_GRACE_MS);
|
||||
};
|
||||
|
||||
request.onerror = () => {
|
||||
clearTimeout(blockedTimer);
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
console.error(`Failed to open ${config.name}:`, request.error);
|
||||
reject(request.error);
|
||||
};
|
||||
|
||||
request.onsuccess = () => {
|
||||
clearTimeout(blockedTimer);
|
||||
const db = request.result;
|
||||
// We already gave up waiting: close it rather than hold a handle nobody
|
||||
// awaits, or we become the next tab's blocker.
|
||||
if (settled) {
|
||||
db.close();
|
||||
return;
|
||||
}
|
||||
settled = true;
|
||||
console.log(`Successfully opened ${config.name}`);
|
||||
|
||||
// Another tab wants a newer schema. Forget BEFORE closing: a cached but
|
||||
// closed handle is worse than none, because every transaction on it throws.
|
||||
db.onversionchange = () => {
|
||||
console.warn(
|
||||
`${config.name}: another tab requested a version change; closing this connection`,
|
||||
);
|
||||
this.databases.delete(config.name);
|
||||
this.initPromises.delete(config.name);
|
||||
db.close();
|
||||
};
|
||||
|
||||
// Set up close handler to clean up our references
|
||||
db.onclose = () => {
|
||||
console.log(`Database ${config.name} closed`);
|
||||
@@ -329,9 +388,36 @@ class IndexedDBManager {
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const deleteRequest = indexedDB.deleteDatabase(name);
|
||||
// A delete blocks exactly like an upgrade, and this one is awaited on the
|
||||
// files open path - so an unguarded block hangs the whole storage layer.
|
||||
let settled = false;
|
||||
let blockedTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
|
||||
deleteRequest.onerror = () => reject(deleteRequest.error);
|
||||
deleteRequest.onblocked = () => {
|
||||
console.warn(
|
||||
`Deleting ${name} is blocked by another connection; giving up in ${BLOCKED_GRACE_MS}ms.`,
|
||||
);
|
||||
blockedTimer = setTimeout(() => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
reject(
|
||||
new Error(
|
||||
`Deleting ${name} was blocked by another connection for ${BLOCKED_GRACE_MS}ms.`,
|
||||
),
|
||||
);
|
||||
}, BLOCKED_GRACE_MS);
|
||||
};
|
||||
|
||||
deleteRequest.onerror = () => {
|
||||
clearTimeout(blockedTimer);
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
reject(deleteRequest.error);
|
||||
};
|
||||
deleteRequest.onsuccess = () => {
|
||||
clearTimeout(blockedTimer);
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
console.log(`Deleted database: ${name}`);
|
||||
resolve();
|
||||
};
|
||||
@@ -343,17 +429,32 @@ class IndexedDBManager {
|
||||
*/
|
||||
async getDatabaseVersion(name: string): Promise<number | null> {
|
||||
return new Promise((resolve) => {
|
||||
// This probe runs BEFORE the guarded open, and a versionless open can be
|
||||
// delayed indefinitely by another tab mid-versionchange. Unknown after the
|
||||
// grace period beats hanging every storage consumer: the real open that
|
||||
// follows has its own blocked guard and a message the user can act on.
|
||||
const giveUp = setTimeout(() => {
|
||||
console.warn(
|
||||
`Version probe for ${name} did not answer in ${BLOCKED_GRACE_MS}ms; proceeding without it.`,
|
||||
);
|
||||
resolve(null);
|
||||
}, BLOCKED_GRACE_MS);
|
||||
const request = indexedDB.open(name);
|
||||
request.onsuccess = () => {
|
||||
clearTimeout(giveUp);
|
||||
const db = request.result;
|
||||
const version = db.version;
|
||||
db.close();
|
||||
resolve(version);
|
||||
};
|
||||
request.onerror = () => resolve(null);
|
||||
request.onerror = () => {
|
||||
clearTimeout(giveUp);
|
||||
resolve(null);
|
||||
};
|
||||
request.onupgradeneeded = () => {
|
||||
// Cancel the upgrade
|
||||
request.transaction?.abort();
|
||||
clearTimeout(giveUp);
|
||||
resolve(null);
|
||||
};
|
||||
});
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { describe, expect, test, vi } from "vitest";
|
||||
|
||||
/**
|
||||
* A WASM instantiate that fails must reject, not hang. `instantiateWasm` reports
|
||||
* success by callback, so a swallowed rejection leaves `init()` pending and takes
|
||||
* every thumbnail, page parse and form read with it - silently.
|
||||
*/
|
||||
|
||||
const init = vi.hoisted(() => vi.fn());
|
||||
vi.mock("@embedpdf/pdfium", () => ({ init }));
|
||||
|
||||
const wasmModule = vi.hoisted(() => ({}) as WebAssembly.Module);
|
||||
vi.mock("@app/services/wasmPrecompiler", () => ({
|
||||
pdfiumWasmModulePromise: Promise.resolve(wasmModule),
|
||||
startEagerWasmCompilation: () => {},
|
||||
pdfiumWasmUrl: "http://localhost/pdfium.wasm",
|
||||
}));
|
||||
|
||||
/** emscripten's contract: it calls instantiateWasm and waits to be called back. */
|
||||
function emscriptenInit(
|
||||
instantiate: (imports: object, ok: () => void) => void,
|
||||
) {
|
||||
return new Promise(() => {
|
||||
instantiate({}, () => {});
|
||||
});
|
||||
}
|
||||
|
||||
async function loadService() {
|
||||
vi.resetModules();
|
||||
return await import("@app/services/pdfiumService");
|
||||
}
|
||||
|
||||
describe("pdfium bootstrap", () => {
|
||||
test("rejects when instantiating the pre-compiled module fails", async () => {
|
||||
const failure = new Error("LinkError: import mismatch");
|
||||
vi.spyOn(WebAssembly, "instantiate").mockRejectedValue(failure as never);
|
||||
init.mockImplementation((overrides: Record<string, never>) =>
|
||||
emscriptenInit(
|
||||
overrides.instantiateWasm as unknown as (
|
||||
imports: object,
|
||||
ok: () => void,
|
||||
) => void,
|
||||
),
|
||||
);
|
||||
|
||||
const { getPdfiumModule } = await loadService();
|
||||
|
||||
// Before the fix this never settled, so the test timed out.
|
||||
await expect(getPdfiumModule()).rejects.toThrow(/LinkError/);
|
||||
});
|
||||
|
||||
test("a failed load isn't cached, so the next call retries", async () => {
|
||||
const instantiate = vi
|
||||
.spyOn(WebAssembly, "instantiate")
|
||||
.mockRejectedValueOnce(new Error("transient") as never)
|
||||
.mockResolvedValue({} as never);
|
||||
const ready = { PDFiumExt_Init: () => {} };
|
||||
init.mockImplementation(
|
||||
(overrides: Record<string, never>) =>
|
||||
new Promise((resolve) => {
|
||||
const instantiateWasm = overrides.instantiateWasm as unknown as (
|
||||
imports: object,
|
||||
ok: () => void,
|
||||
) => void;
|
||||
instantiateWasm({}, () => resolve(ready));
|
||||
}),
|
||||
);
|
||||
|
||||
const { getPdfiumModule } = await loadService();
|
||||
|
||||
await expect(getPdfiumModule()).rejects.toThrow(/transient/);
|
||||
await expect(getPdfiumModule()).resolves.toBe(ready);
|
||||
expect(instantiate).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
@@ -80,17 +80,24 @@ function wasmUrl(): string {
|
||||
* This is the low-level PDFium WASM interface with all C functions wrapped.
|
||||
* Prefer `withDocument()` for document-scoped work.
|
||||
*/
|
||||
export async function getPdfiumModule(): Promise<WrappedPdfiumModule> {
|
||||
if (_module) return _module;
|
||||
if (!_initPromise) {
|
||||
// Ensure eager compilation has started if PDF service is requested before idle timeout
|
||||
startEagerWasmCompilation();
|
||||
/** Reuses the WASM pre-compiled at boot. Every failure must reach this promise:
|
||||
* `instantiateWasm` reports success by callback, so a rejection inside it leaves
|
||||
* `init()` pending forever - and with it every thumbnail, parse and form read. */
|
||||
async function initPdfiumModule(): Promise<WrappedPdfiumModule> {
|
||||
// Ensure eager compilation has started if PDF service is requested before idle timeout
|
||||
startEagerWasmCompilation();
|
||||
|
||||
const overrides: PdfiumModuleOverrides = {
|
||||
locateFile: () => wasmUrl(),
|
||||
};
|
||||
const overrides: PdfiumModuleOverrides = { locateFile: () => wasmUrl() };
|
||||
const precompiled = await pdfiumWasmModulePromise;
|
||||
|
||||
// Eagerly reuse pre-compiled WASM module from app boot if available
|
||||
let reportFailure: (error: unknown) => void = () => {};
|
||||
const instantiateFailed = new Promise<never>((_, reject) => {
|
||||
reportFailure = reject;
|
||||
});
|
||||
|
||||
// No pre-compiled module: leave instantiateWasm alone so emscripten fetches the
|
||||
// WASM itself and rejects init() on failure, instead of a fallback that can't.
|
||||
if (precompiled) {
|
||||
overrides.instantiateWasm = (
|
||||
imports: WebAssembly.Imports,
|
||||
successCallback: (
|
||||
@@ -98,40 +105,34 @@ export async function getPdfiumModule(): Promise<WrappedPdfiumModule> {
|
||||
module: WebAssembly.Module,
|
||||
) => void,
|
||||
) => {
|
||||
pdfiumWasmModulePromise
|
||||
.then((wasmModule) => {
|
||||
if (wasmModule) {
|
||||
return WebAssembly.instantiate(wasmModule, imports).then(
|
||||
(instance) => {
|
||||
successCallback(instance, wasmModule);
|
||||
},
|
||||
);
|
||||
} else {
|
||||
throw new Error("No pre-compiled WASM module found");
|
||||
}
|
||||
})
|
||||
.catch((err: unknown) => {
|
||||
console.warn(
|
||||
"Eager WebAssembly instantiation failed, falling back to streaming compilation:",
|
||||
err,
|
||||
);
|
||||
WebAssembly.instantiateStreaming(fetch(wasmUrl()), imports).then(
|
||||
(result) => {
|
||||
successCallback(result.instance, result.module);
|
||||
},
|
||||
);
|
||||
});
|
||||
WebAssembly.instantiate(precompiled, imports)
|
||||
.then((instance) => successCallback(instance, precompiled))
|
||||
.catch(reportFailure);
|
||||
};
|
||||
}
|
||||
|
||||
_initPromise = init(overrides as Partial<PdfiumModule>).then((m) => {
|
||||
// Call PDFiumExt_Init to ensure extensions (form fill etc.) are set up
|
||||
try {
|
||||
m.PDFiumExt_Init();
|
||||
} catch {
|
||||
/* already initialized */
|
||||
}
|
||||
_module = m;
|
||||
return m;
|
||||
const m = await Promise.race([
|
||||
init(overrides as Partial<PdfiumModule>),
|
||||
instantiateFailed,
|
||||
]);
|
||||
// Call PDFiumExt_Init to ensure extensions (form fill etc.) are set up
|
||||
try {
|
||||
m.PDFiumExt_Init();
|
||||
} catch {
|
||||
/* already initialized */
|
||||
}
|
||||
_module = m;
|
||||
return m;
|
||||
}
|
||||
|
||||
export async function getPdfiumModule(): Promise<WrappedPdfiumModule> {
|
||||
if (_module) return _module;
|
||||
if (!_initPromise) {
|
||||
_initPromise = initPdfiumModule().catch((error: unknown) => {
|
||||
// Don't cache the failure: every PDF feature in the app goes through here,
|
||||
// so a transient WASM fetch would take them all down for the session.
|
||||
_initPromise = null;
|
||||
throw error;
|
||||
});
|
||||
}
|
||||
return _initPromise;
|
||||
|
||||
@@ -2,6 +2,10 @@ import "@testing-library/jest-dom";
|
||||
import { vi } from "vitest";
|
||||
import { installFailOnConsole } from "@app/tests/failOnConsole";
|
||||
|
||||
// jsdom is missing the same APIs WebKit is, so tests must agree with the
|
||||
// browser. Same module `src/index.tsx` installs.
|
||||
import "@app/utils/engineShims";
|
||||
|
||||
installFailOnConsole();
|
||||
|
||||
// Mock localStorage for tests
|
||||
|
||||
@@ -68,6 +68,8 @@ const mockedApiClient = vi.mocked(apiClient);
|
||||
|
||||
// Mock only essential services that are actually called by the tests
|
||||
vi.mock("../../services/fileStorage", () => ({
|
||||
// FileContext subscribes to this to drop files whose bytes are unreadable.
|
||||
onRecordUnreadable: () => () => {},
|
||||
fileStorage: {
|
||||
init: vi.fn().mockResolvedValue(undefined),
|
||||
storeFile: vi.fn().mockImplementation((file, thumbnail) => {
|
||||
|
||||
@@ -66,6 +66,8 @@ const mockedApiClient = vi.mocked(apiClient);
|
||||
|
||||
// Mock only essential services that are actually called by the tests
|
||||
vi.mock("../../services/fileStorage", () => ({
|
||||
// FileContext subscribes to this to drop files whose bytes are unreadable.
|
||||
onRecordUnreadable: () => () => {},
|
||||
fileStorage: {
|
||||
init: vi.fn().mockResolvedValue(undefined),
|
||||
storeFile: vi.fn().mockImplementation((file, thumbnail) => {
|
||||
|
||||
@@ -230,4 +230,7 @@ test.describe("Compare tool slot selection", () => {
|
||||
page.locator('[data-testid="compare-slot-comparison"]'),
|
||||
).toHaveAttribute("data-slot-state", "empty");
|
||||
});
|
||||
|
||||
// These specs stop at slot state. Actually running a comparison lives in
|
||||
// `engine-capabilities.spec.ts`, which is cross-browser in PR CI.
|
||||
});
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
/** Runs on all three engines in PR CI, asserting on evidence that can only exist
|
||||
* if the engine did the work. Keep small - it is paid for three times per PR. */
|
||||
|
||||
import path from "path";
|
||||
import type { Page } from "@playwright/test";
|
||||
import { test, expect } from "@app/tests/helpers/stub-test-base";
|
||||
import { dismissTourTooltip, uploadFiles } from "@app/tests/helpers/ui-helpers";
|
||||
|
||||
const FIXTURES_DIR = path.join(import.meta.dirname, "../test-fixtures");
|
||||
const SAMPLE_PDF = path.join(FIXTURES_DIR, "sample.pdf");
|
||||
const PDF_A = path.join(FIXTURES_DIR, "compare_sample_a.pdf");
|
||||
const PDF_B = path.join(FIXTURES_DIR, "compare_sample_b.pdf");
|
||||
|
||||
/** A missing global or prototype method always surfaces as one of these.
|
||||
* Matching the shape keeps benign engine noise out (console-clean.spec.ts). */
|
||||
const MISSING_API_ERROR =
|
||||
/is not a function|is not a constructor|undefined is not an object|has no method/i;
|
||||
|
||||
/** Collect the "this engine lacks an API we used" errors seen on the page. */
|
||||
function recordMissingApiErrors(page: Page): string[] {
|
||||
const errors: string[] = [];
|
||||
page.on("pageerror", (error: Error) => {
|
||||
const text = String(error);
|
||||
if (MISSING_API_ERROR.test(text)) errors.push(text);
|
||||
});
|
||||
return errors;
|
||||
}
|
||||
|
||||
async function fillCompareSlot(
|
||||
page: Page,
|
||||
role: "base" | "comparison",
|
||||
filePath: string,
|
||||
) {
|
||||
await page
|
||||
.getByTestId(`compare-slot-${role}-add-input`)
|
||||
.setInputFiles(filePath);
|
||||
await expect(
|
||||
page.locator(`[data-testid="compare-slot-${role}"]`),
|
||||
).toHaveAttribute("data-slot-state", "filled", { timeout: 20_000 });
|
||||
// The upload modal's overlay outlives its close transition and eats clicks.
|
||||
await page
|
||||
.locator(".mantine-Modal-overlay")
|
||||
.waitFor({ state: "detached", timeout: 5_000 })
|
||||
.catch(() => {
|
||||
/* already gone */
|
||||
});
|
||||
}
|
||||
|
||||
test.describe("engine capabilities", { tag: "@engine-capability" }, () => {
|
||||
test("extracts PDF text and completes a comparison", async ({ page }) => {
|
||||
test.setTimeout(120_000);
|
||||
const missingApis = recordMissingApiErrors(page);
|
||||
|
||||
await page.locator('[data-tour="tool-button-compare"]').first().click();
|
||||
await page.waitForSelector('[data-testid="compare-slot-base"]', {
|
||||
timeout: 20_000,
|
||||
});
|
||||
|
||||
await fillCompareSlot(page, "base", PDF_A);
|
||||
await fillCompareSlot(page, "comparison", PDF_B);
|
||||
|
||||
// By test id: `name` matches as a substring, so "Compare" also hits the
|
||||
// tool button that opened this panel.
|
||||
await page.getByTestId("compare-execute").click();
|
||||
|
||||
// Counted results, not headings: an extraction returning nothing still
|
||||
// renders empty panes, which is how the WebKit failure looked like success.
|
||||
const deletions = page.getByText(/Deletions \((\d+)\)/);
|
||||
const additions = page.getByText(/Additions \((\d+)\)/);
|
||||
await expect(deletions).toBeVisible({ timeout: 60_000 });
|
||||
await expect(additions).toBeVisible();
|
||||
expect(await deletions.innerText()).not.toMatch(/\(0\)/);
|
||||
expect(await additions.innerText()).not.toMatch(/\(0\)/);
|
||||
|
||||
expect(missingApis, "no missing-API errors during comparison").toEqual([]);
|
||||
});
|
||||
|
||||
test("rasterises page thumbnails via the PDF engine", async ({ page }) => {
|
||||
test.setTimeout(120_000);
|
||||
const missingApis = recordMissingApiErrors(page);
|
||||
|
||||
await uploadFiles(page, SAMPLE_PDF);
|
||||
await dismissTourTooltip(page);
|
||||
|
||||
// A page thumbnail only exists if the WASM engine loaded, rendered and
|
||||
// encoded. When it fails the grid still renders, just with no <img>.
|
||||
await page.getByText("PDF Multi Tool", { exact: true }).first().click();
|
||||
|
||||
const thumbnail = page
|
||||
.locator("[data-page-id] img[data-original-rotation]")
|
||||
.first();
|
||||
await expect(thumbnail).toBeVisible({ timeout: 60_000 });
|
||||
|
||||
// An empty encode still yields a src; require enough payload to be real.
|
||||
const src = await thumbnail.getAttribute("src");
|
||||
expect(src ?? "").toMatch(/^data:image\//);
|
||||
expect(src?.length ?? 0).toBeGreaterThan(1_000);
|
||||
|
||||
expect(missingApis, "no missing-API errors during thumbnailing").toEqual(
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
test("reads a stored file's bytes back after a reload", async ({ page }) => {
|
||||
test.setTimeout(120_000);
|
||||
const missingApis = recordMissingApiErrors(page);
|
||||
|
||||
await uploadFiles(page, SAMPLE_PDF);
|
||||
|
||||
// Full reload: FileContext rehydrates from IndexedDB, not from memory.
|
||||
await page.reload({ waitUntil: "domcontentloaded" });
|
||||
|
||||
const restored = page.locator(".file-sidebar-file-item").first();
|
||||
await expect(restored).toBeVisible({ timeout: 30_000 });
|
||||
|
||||
// Rendering it is the assertion that matters: the metadata record survives
|
||||
// even when the bytes were never stored, so a filename proves nothing.
|
||||
await restored.hover();
|
||||
await restored
|
||||
.locator(".file-sidebar-eye-btn")
|
||||
.click({ timeout: 15_000, force: true });
|
||||
|
||||
const firstPage = page.locator('[data-page-index="0"]').first();
|
||||
await expect(firstPage).toBeVisible({ timeout: 60_000 });
|
||||
|
||||
// A tile that decoded has non-zero naturalWidth. A blob stored but not
|
||||
// readable back resolves to nothing, and renders as an empty page.
|
||||
const tile = firstPage.locator('img[src^="blob:"]').first();
|
||||
await expect(tile).toBeAttached({ timeout: 30_000 });
|
||||
await expect
|
||||
.poll(() => tile.evaluate((img: HTMLImageElement) => img.naturalWidth), {
|
||||
timeout: 30_000,
|
||||
})
|
||||
.toBeGreaterThan(0);
|
||||
|
||||
expect(missingApis, "no missing-API errors after rehydration").toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -35,7 +35,7 @@ import {
|
||||
} from "@app/tools/formFill/FormFillContext";
|
||||
import { useNavigation } from "@app/contexts/NavigationContext";
|
||||
import { useViewer } from "@app/contexts/ViewerContext";
|
||||
import { useFileState } from "@app/contexts/FileContext";
|
||||
import { useAllFiles, useFileState } from "@app/contexts/FileContext";
|
||||
import { Skeleton } from "@mantine/core";
|
||||
import { isStirlingFile, getFormFillFileId } from "@app/types/fileContext";
|
||||
import type { BaseToolProps } from "@app/types/tool";
|
||||
@@ -124,7 +124,7 @@ const _MODE_TABS: ModeTabDef[] = [
|
||||
const FormFill = (_props: BaseToolProps) => {
|
||||
const { t } = useTranslation();
|
||||
const { selectedTool } = useNavigation();
|
||||
const { selectors, state: fileState } = useFileState();
|
||||
const { state: fileState } = useFileState();
|
||||
|
||||
const {
|
||||
state: formState,
|
||||
@@ -178,7 +178,9 @@ const FormFill = (_props: BaseToolProps) => {
|
||||
const isDirtyRef = useRef(formState.isDirty);
|
||||
isDirtyRef.current = formState.isDirty;
|
||||
|
||||
const activeFiles = selectors.getFiles();
|
||||
// Subscribing read: getFiles() during render doesn't re-run when the workbench
|
||||
// changes, so the panel kept showing the pre-hydration (or pre-version) file.
|
||||
const { files: activeFiles } = useAllFiles();
|
||||
const selectedFileIds = fileState.ui.selectedFileIds;
|
||||
const currentFile = useMemo(() => {
|
||||
if (activeFiles.length === 0) return null;
|
||||
|
||||
@@ -61,6 +61,12 @@ export interface StirlingFileStub extends BaseFileMetadata {
|
||||
* unclassified files / non-SaaS builds.
|
||||
*/
|
||||
classificationLabels?: string[];
|
||||
/**
|
||||
* This session proved the stored bytes unreadable (WebKit losing a blob's
|
||||
* backing store). The row renders as "data lost" instead of pretending the
|
||||
* file can open; re-uploading is the only recovery.
|
||||
*/
|
||||
dataUnavailable?: boolean;
|
||||
// Note: File object stored in provider ref, not in state
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import {
|
||||
lossyEncodeOptions,
|
||||
resetCanvasEncodingProbe,
|
||||
} from "@app/utils/canvasImageEncoding";
|
||||
|
||||
/** An engine that can't encode a format returns PNG instead of throwing, so
|
||||
* only the returned Blob's `type` reveals what happened. */
|
||||
|
||||
/** Stand in for OffscreenCanvas, honouring only the given MIME types. */
|
||||
function stubOffscreenCanvas(honoured: string[]): void {
|
||||
class FakeOffscreenCanvas {
|
||||
constructor(
|
||||
public width: number,
|
||||
public height: number,
|
||||
) {}
|
||||
|
||||
getContext() {
|
||||
return { fillStyle: "", fillRect: () => {} };
|
||||
}
|
||||
|
||||
convertToBlob({ type }: { type: string }) {
|
||||
// Per spec: an unsupported type silently serialises as PNG.
|
||||
const actual = honoured.includes(type) ? type : "image/png";
|
||||
return Promise.resolve(new Blob([new Uint8Array([0])], { type: actual }));
|
||||
}
|
||||
}
|
||||
vi.stubGlobal("OffscreenCanvas", FakeOffscreenCanvas);
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
resetCanvasEncodingProbe();
|
||||
});
|
||||
|
||||
describe("lossyEncodeOptions", () => {
|
||||
it("uses WebP when the engine really encodes it", async () => {
|
||||
stubOffscreenCanvas(["image/webp", "image/jpeg", "image/png"]);
|
||||
await expect(lossyEncodeOptions()).resolves.toEqual({
|
||||
type: "image/webp",
|
||||
quality: 0.85,
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to JPEG when a WebP request silently yields PNG", async () => {
|
||||
// WebKit's actual behaviour: no WebP encoder, no error either.
|
||||
stubOffscreenCanvas(["image/jpeg", "image/png"]);
|
||||
await expect(lossyEncodeOptions()).resolves.toEqual({
|
||||
type: "image/jpeg",
|
||||
quality: 0.85,
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to PNG when no lossy format is honoured", async () => {
|
||||
stubOffscreenCanvas(["image/png"]);
|
||||
await expect(lossyEncodeOptions()).resolves.toEqual({
|
||||
type: "image/png",
|
||||
quality: 0.85,
|
||||
});
|
||||
});
|
||||
|
||||
it("passes the caller's quality through", async () => {
|
||||
stubOffscreenCanvas(["image/webp", "image/png"]);
|
||||
await expect(lossyEncodeOptions(0.5)).resolves.toEqual({
|
||||
type: "image/webp",
|
||||
quality: 0.5,
|
||||
});
|
||||
});
|
||||
|
||||
it("probes once and reuses the answer", async () => {
|
||||
stubOffscreenCanvas(["image/webp", "image/png"]);
|
||||
const spy = vi.spyOn(
|
||||
(globalThis as unknown as { OffscreenCanvas: { prototype: object } })
|
||||
.OffscreenCanvas.prototype as { convertToBlob: () => unknown },
|
||||
"convertToBlob",
|
||||
);
|
||||
await lossyEncodeOptions();
|
||||
const afterFirst = spy.mock.calls.length;
|
||||
await lossyEncodeOptions();
|
||||
expect(spy.mock.calls.length).toBe(afterFirst);
|
||||
});
|
||||
|
||||
it("returns PNG where there is no OffscreenCanvas at all", async () => {
|
||||
vi.stubGlobal("OffscreenCanvas", undefined);
|
||||
await expect(lossyEncodeOptions()).resolves.toEqual({
|
||||
type: "image/png",
|
||||
quality: 0.85,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,45 @@
|
||||
/** `convertToBlob` silently serialises to PNG for a format it can't encode
|
||||
* (WebKit, WebP), so only the returned `type` reveals it. Probe once per realm. */
|
||||
|
||||
/** Candidates in preference order, best compression first. PNG always works. */
|
||||
const LOSSY_CANDIDATES = ["image/webp", "image/jpeg"] as const;
|
||||
const PNG_TYPE = "image/png";
|
||||
|
||||
let probe: Promise<string> | null = null;
|
||||
|
||||
async function honoursType(type: string, quality: number): Promise<boolean> {
|
||||
try {
|
||||
const canvas = new OffscreenCanvas(2, 2);
|
||||
const ctx = canvas.getContext("2d");
|
||||
if (!ctx) return false;
|
||||
// JPEG has no alpha, and the pixels are thrown away either way.
|
||||
ctx.fillStyle = "#000000";
|
||||
ctx.fillRect(0, 0, 2, 2);
|
||||
const blob = await canvas.convertToBlob({ type, quality });
|
||||
return blob.type === type;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function detectLossyType(quality: number): Promise<string> {
|
||||
if (typeof OffscreenCanvas === "undefined") return PNG_TYPE;
|
||||
for (const candidate of LOSSY_CANDIDATES) {
|
||||
if (await honoursType(candidate, quality)) return candidate;
|
||||
}
|
||||
return PNG_TYPE;
|
||||
}
|
||||
|
||||
/** Encode options for a page raster: the best lossy format this engine really
|
||||
* supports, PNG only if it supports none. `quality` is ignored by PNG. */
|
||||
export function lossyEncodeOptions(
|
||||
quality = 0.85,
|
||||
): Promise<ImageEncodeOptions> {
|
||||
probe ??= detectLossyType(quality);
|
||||
return probe.then((type) => ({ type, quality }));
|
||||
}
|
||||
|
||||
/** Reset the cached probe. Tests only. */
|
||||
export function resetCanvasEncodingProbe(): void {
|
||||
probe = null;
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
// Browser APIs the app is entitled to assume exist, stood in where an engine
|
||||
// omits them. Import once, first, before anything that might use them.
|
||||
import "@app/utils/patchReadableStreamAsyncIterator";
|
||||
import "@app/utils/patchRequestIdleCallback";
|
||||
@@ -0,0 +1,93 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { patchReadableStreamAsyncIterator } from "@app/utils/patchReadableStreamAsyncIterator";
|
||||
|
||||
/** Node implements the API, so remove it to exercise the shim, then restore. */
|
||||
const KEY: PropertyKey = Symbol.asyncIterator;
|
||||
const proto: object = ReadableStream.prototype;
|
||||
let native: PropertyDescriptor | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
native = Object.getOwnPropertyDescriptor(proto, KEY);
|
||||
Reflect.deleteProperty(proto, KEY);
|
||||
patchReadableStreamAsyncIterator();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (native) Object.defineProperty(proto, KEY, native);
|
||||
});
|
||||
|
||||
function streamOf(...chunks: number[]): ReadableStream<number> {
|
||||
return new ReadableStream<number>({
|
||||
start(controller) {
|
||||
chunks.forEach((c) => controller.enqueue(c));
|
||||
controller.close();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function failingStream(error: Error): ReadableStream<number> {
|
||||
return new ReadableStream<number>({
|
||||
start(controller) {
|
||||
controller.enqueue(1);
|
||||
controller.error(error);
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
describe("patchReadableStreamAsyncIterator", () => {
|
||||
it("drains a stream with for await, then unlocks it", async () => {
|
||||
const stream = streamOf(1, 2, 3);
|
||||
const seen: number[] = [];
|
||||
for await (const chunk of stream) seen.push(chunk);
|
||||
|
||||
expect(seen).toEqual([1, 2, 3]);
|
||||
expect(stream.locked).toBe(false);
|
||||
});
|
||||
|
||||
// The reader must be released in the read's error steps. `for await` does not
|
||||
// call `return()` when `next()` rejects, so nothing else would ever unlock it.
|
||||
it("releases the lock when the stream errors mid-read", async () => {
|
||||
const boom = new Error("boom");
|
||||
const stream = failingStream(boom);
|
||||
|
||||
await expect(
|
||||
(async () => {
|
||||
for await (const _ of stream) {
|
||||
/* drain until it throws */
|
||||
}
|
||||
})(),
|
||||
).rejects.toThrow("boom");
|
||||
|
||||
expect(stream.locked).toBe(false);
|
||||
});
|
||||
|
||||
it("cancels and unlocks when the consumer breaks early", async () => {
|
||||
const stream = streamOf(1, 2, 3);
|
||||
for await (const chunk of stream) {
|
||||
expect(chunk).toBe(1);
|
||||
break;
|
||||
}
|
||||
expect(stream.locked).toBe(false);
|
||||
});
|
||||
|
||||
// Native short-circuits once finished; throwing "reader owned by no readable
|
||||
// stream" would break Array.fromAsync and defensive `return()` in a finally.
|
||||
it("keeps reporting done after the stream is drained", async () => {
|
||||
const iterator = streamOf(1)[Symbol.asyncIterator]();
|
||||
await iterator.next();
|
||||
|
||||
await expect(iterator.next()).resolves.toEqual({
|
||||
done: true,
|
||||
value: undefined,
|
||||
});
|
||||
await expect(iterator.return?.()).resolves.toMatchObject({ done: true });
|
||||
});
|
||||
|
||||
it("leaves a real implementation alone", () => {
|
||||
if (native) Object.defineProperty(proto, KEY, native);
|
||||
const before = Object.getOwnPropertyDescriptor(proto, KEY);
|
||||
patchReadableStreamAsyncIterator();
|
||||
expect(Object.getOwnPropertyDescriptor(proto, KEY)).toEqual(before);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,58 @@
|
||||
// WebKit ships no `ReadableStream[Symbol.asyncIterator]`, and pdf.js reads its
|
||||
// text stream with `for await`, so all text extraction threw on Safari.
|
||||
|
||||
export function patchReadableStreamAsyncIterator(): void {
|
||||
if (typeof ReadableStream === "undefined") return;
|
||||
if (Symbol.asyncIterator in ReadableStream.prototype) return;
|
||||
|
||||
Object.defineProperty(ReadableStream.prototype, Symbol.asyncIterator, {
|
||||
writable: true,
|
||||
configurable: true,
|
||||
value: function <T>(this: ReadableStream<T>): AsyncIterableIterator<T> {
|
||||
const reader = this.getReader();
|
||||
// Releasing must be idempotent and must NOT happen after a successful
|
||||
// read - the next `read()` on a released reader throws.
|
||||
let finished = false;
|
||||
const release = () => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
reader.releaseLock();
|
||||
};
|
||||
return {
|
||||
async next(): Promise<IteratorResult<T>> {
|
||||
// Spec short-circuits once finished; throwing here would break any
|
||||
// consumer that calls `next()` or `return()` defensively.
|
||||
if (finished) return { done: true, value: undefined };
|
||||
try {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) {
|
||||
release();
|
||||
return { done: true, value: undefined };
|
||||
}
|
||||
return { done: false, value };
|
||||
} catch (error) {
|
||||
// The spec releases the reader in the read request's error steps.
|
||||
// Without this an errored stream stays locked for good: `for await`
|
||||
// does not call `return()` when `next()` rejects.
|
||||
release();
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
async return(value?: unknown): Promise<IteratorResult<T>> {
|
||||
if (finished) return { done: true, value: value as T };
|
||||
try {
|
||||
await reader.cancel();
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
return { done: true, value: value as T };
|
||||
},
|
||||
[Symbol.asyncIterator]() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
patchReadableStreamAsyncIterator();
|
||||
@@ -0,0 +1,92 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { patchRequestIdleCallback } from "@app/utils/patchRequestIdleCallback";
|
||||
|
||||
/** Two things must hold: it stands in where the API is absent, and it never
|
||||
* displaces a real implementation. */
|
||||
|
||||
type IdleGlobals = {
|
||||
requestIdleCallback?: unknown;
|
||||
cancelIdleCallback?: unknown;
|
||||
};
|
||||
|
||||
const globals = globalThis as IdleGlobals;
|
||||
|
||||
let saved: IdleGlobals;
|
||||
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
saved = {
|
||||
requestIdleCallback: globals.requestIdleCallback,
|
||||
cancelIdleCallback: globals.cancelIdleCallback,
|
||||
};
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
globals.requestIdleCallback = saved.requestIdleCallback;
|
||||
globals.cancelIdleCallback = saved.cancelIdleCallback;
|
||||
});
|
||||
|
||||
describe("patchRequestIdleCallback", () => {
|
||||
it("stands in where the engine has no requestIdleCallback", () => {
|
||||
delete globals.requestIdleCallback;
|
||||
delete globals.cancelIdleCallback;
|
||||
patchRequestIdleCallback();
|
||||
|
||||
const task = vi.fn();
|
||||
requestIdleCallback(task);
|
||||
|
||||
// Asynchronous: never runs on the caller's own turn.
|
||||
expect(task).not.toHaveBeenCalled();
|
||||
// With no deadline given, it yields briefly and then runs.
|
||||
vi.advanceTimersByTime(200);
|
||||
expect(task).toHaveBeenCalledTimes(1);
|
||||
|
||||
const deadline = task.mock.calls[0][0] as IdleDeadline;
|
||||
expect(deadline.didTimeout).toBe(false);
|
||||
expect(deadline.timeRemaining()).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("never runs later than the deadline the caller asked for", () => {
|
||||
delete globals.requestIdleCallback;
|
||||
patchRequestIdleCallback();
|
||||
|
||||
const task = vi.fn();
|
||||
requestIdleCallback(task, { timeout: 50 });
|
||||
vi.advanceTimersByTime(50);
|
||||
expect(task).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
// `src/index.tsx` asks for 2000ms so the pdfium WASM compile doesn't land on
|
||||
// top of the app's first renders.
|
||||
it("does not run background work earlier than the caller allowed for", () => {
|
||||
delete globals.requestIdleCallback;
|
||||
patchRequestIdleCallback();
|
||||
|
||||
const task = vi.fn();
|
||||
requestIdleCallback(task, { timeout: 2000 });
|
||||
vi.advanceTimersByTime(1999);
|
||||
expect(task).not.toHaveBeenCalled();
|
||||
vi.advanceTimersByTime(1);
|
||||
expect(task).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("cancels a scheduled task by handle", () => {
|
||||
delete globals.requestIdleCallback;
|
||||
delete globals.cancelIdleCallback;
|
||||
patchRequestIdleCallback();
|
||||
|
||||
const task = vi.fn();
|
||||
cancelIdleCallback(requestIdleCallback(task));
|
||||
vi.advanceTimersByTime(1000);
|
||||
expect(task).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("leaves a real implementation alone", () => {
|
||||
const native = vi.fn();
|
||||
globals.requestIdleCallback = native;
|
||||
patchRequestIdleCallback();
|
||||
expect(globals.requestIdleCallback).toBe(native);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,37 @@
|
||||
// WebKit ships no `requestIdleCallback`. The shim honours the async/deadline/
|
||||
// cancel contract, not the *idle* part, and is window-only like the real API.
|
||||
|
||||
/** Short delay before running, so the current turn's work drains first. */
|
||||
const YIELD_MS = 200;
|
||||
|
||||
/** Idle-deadline shape for the timer stand-in. A small positive budget keeps a
|
||||
* `while (timeRemaining() > 0)` loop doing one chunk rather than spinning. */
|
||||
function makeDeadline(): IdleDeadline {
|
||||
return { didTimeout: false, timeRemaining: () => 1 };
|
||||
}
|
||||
|
||||
export function patchRequestIdleCallback(): void {
|
||||
if (typeof globalThis === "undefined") return;
|
||||
const target = globalThis as typeof globalThis & {
|
||||
requestIdleCallback?: typeof requestIdleCallback;
|
||||
cancelIdleCallback?: typeof cancelIdleCallback;
|
||||
};
|
||||
if (typeof target.requestIdleCallback === "function") return;
|
||||
|
||||
target.requestIdleCallback = (
|
||||
callback: IdleRequestCallback,
|
||||
options?: IdleRequestOptions,
|
||||
): number =>
|
||||
setTimeout(
|
||||
() => callback(makeDeadline()),
|
||||
// Honour the caller's deadline: `timeout` is them saying how long this may
|
||||
// wait, so firing earlier lands background work on top of startup.
|
||||
options?.timeout ?? YIELD_MS,
|
||||
) as unknown as number;
|
||||
|
||||
target.cancelIdleCallback = (handle: number): void => {
|
||||
clearTimeout(handle);
|
||||
};
|
||||
}
|
||||
|
||||
patchRequestIdleCallback();
|
||||
@@ -31,6 +31,12 @@ export function calculateScaleFromFileSize(fileSize: number): number {
|
||||
/** PDFium error code 4 = password required (encrypted PDF). */
|
||||
const PDFIUM_ERR_PASSWORD = 4;
|
||||
|
||||
/** Callers still get a placeholder, but log the cause: an empty thumbnail is
|
||||
* indistinguishable from "no raster preview", so an outage hides as a nicety. */
|
||||
function reportThumbnailFailure(file: File, error: unknown): void {
|
||||
console.warn(`Thumbnail generation failed for ${file.name}:`, error);
|
||||
}
|
||||
|
||||
/** PDFs at or above this size never get a full-buffer client-side parse
|
||||
* (renderer OOM) - only the linearized-prefix attempt below. */
|
||||
export const LARGE_PDF_PARSE_LIMIT = 100 * 1024 * 1024;
|
||||
@@ -262,7 +268,7 @@ export async function generateThumbnailForFile(file: File): Promise<string> {
|
||||
const fullArrayBuffer = await file.arrayBuffer();
|
||||
return await generatePDFThumbnail(fullArrayBuffer, scale);
|
||||
} catch (error) {
|
||||
console.warn(`PDF processing failed for ${file.name}:`, error);
|
||||
reportThumbnailFailure(file, error);
|
||||
return "";
|
||||
}
|
||||
}
|
||||
@@ -314,7 +320,8 @@ export async function generateThumbnailWithMetadata(
|
||||
pageRotations: result.pageRotations,
|
||||
pageDimensions: result.pageDimensions,
|
||||
};
|
||||
} catch {
|
||||
} catch (error) {
|
||||
reportThumbnailFailure(file, error);
|
||||
return { thumbnail: "", pageCount: 0 };
|
||||
}
|
||||
}
|
||||
@@ -344,7 +351,8 @@ export async function generateThumbnailWithMetadata(
|
||||
pageRotations: result.pageRotations,
|
||||
pageDimensions: result.pageDimensions,
|
||||
};
|
||||
} catch {
|
||||
} catch (error) {
|
||||
reportThumbnailFailure(file, error);
|
||||
return { thumbnail: "", pageCount: 1 };
|
||||
}
|
||||
}
|
||||
@@ -389,7 +397,8 @@ export async function generateThumbnailPairWithMetadata(file: File): Promise<{
|
||||
unrotated: toPublic(pair.unrotated),
|
||||
rotated: toPublic(pair.rotated),
|
||||
};
|
||||
} catch {
|
||||
} catch (error) {
|
||||
reportThumbnailFailure(file, error);
|
||||
return {
|
||||
unrotated: { thumbnail: "", pageCount: 0 },
|
||||
rotated: { thumbnail: "", pageCount: 0 },
|
||||
|
||||
@@ -14,6 +14,7 @@ import type {
|
||||
PixelCompareWorkerResponse,
|
||||
PixelCompareWorkerWarnings,
|
||||
} from "@app/types/compare";
|
||||
import { lossyEncodeOptions } from "@app/utils/canvasImageEncoding";
|
||||
|
||||
declare const self: DedicatedWorkerGlobalScope;
|
||||
|
||||
@@ -155,7 +156,7 @@ const renderPageToBitmap = async (
|
||||
return { imageData, bitmap };
|
||||
};
|
||||
|
||||
const ENCODE_OPTS: ImageEncodeOptions = { type: "image/webp", quality: 0.85 };
|
||||
const ENCODE_QUALITY = 0.85;
|
||||
|
||||
const bitmapToBlob = async (
|
||||
bitmap: ImageBitmap,
|
||||
@@ -168,7 +169,7 @@ const bitmapToBlob = async (
|
||||
if (!ctx) throw new Error(errorStrings.canvasContextUnavailable);
|
||||
ctx.drawImage(bitmap, 0, 0);
|
||||
bitmap.close();
|
||||
return await canvas.convertToBlob(ENCODE_OPTS);
|
||||
return await canvas.convertToBlob(await lossyEncodeOptions(ENCODE_QUALITY));
|
||||
};
|
||||
|
||||
const diffDataToBlob = async (
|
||||
@@ -183,7 +184,7 @@ const diffDataToBlob = async (
|
||||
ctx.fillStyle = "#ffffff";
|
||||
ctx.fillRect(0, 0, width, height);
|
||||
ctx.putImageData(diff, 0, 0);
|
||||
return await canvas.convertToBlob(ENCODE_OPTS);
|
||||
return await canvas.convertToBlob(await lossyEncodeOptions(ENCODE_QUALITY));
|
||||
};
|
||||
|
||||
interface PageTotals {
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
// (Edge / Google Translate / extensions) from crashing the app via
|
||||
// parent-mismatch DOMExceptions. See the module for details.
|
||||
import "@app/utils/patchDomForTranslators";
|
||||
// WebKit is missing several APIs the app assumes (ReadableStream async
|
||||
// iteration, which pdf.js needs for all text extraction; requestIdleCallback).
|
||||
import "@app/utils/engineShims";
|
||||
import "@mantine/core/styles.css";
|
||||
import "@mantine/dates/styles.css";
|
||||
import "../vite-env.d.ts"; // oxlint-disable-line no-restricted-imports -- Outside app paths
|
||||
@@ -21,13 +24,8 @@ import { startEagerWasmCompilation } from "@app/services/wasmPrecompiler";
|
||||
applyDevWorktreeLabel();
|
||||
|
||||
if (typeof window !== "undefined") {
|
||||
const scheduleCompilation = () => {
|
||||
if (typeof requestIdleCallback === "function") {
|
||||
requestIdleCallback(() => startEagerWasmCompilation(), { timeout: 2000 });
|
||||
} else {
|
||||
setTimeout(startEagerWasmCompilation, 1000);
|
||||
}
|
||||
};
|
||||
const scheduleCompilation = () =>
|
||||
requestIdleCallback(() => startEagerWasmCompilation(), { timeout: 2000 });
|
||||
|
||||
if (document.readyState === "complete") {
|
||||
scheduleCompilation();
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
import "@testing-library/jest-dom";
|
||||
import { vi } from "vitest";
|
||||
|
||||
// The shims `src/index.tsx` installs - see core/setupTests.ts.
|
||||
import "@app/utils/engineShims";
|
||||
|
||||
// Mirrors the editor's setup: jsdom lacks a handful of browser APIs that shared
|
||||
// components (Mantine FocusTrap, responsive helpers) touch on render.
|
||||
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { finishedWithNothingToDeliver } from "@app/components/policies/usePolicyAutoRun";
|
||||
import type { PolicyRunRecord } from "@app/components/policies/policyRunStore";
|
||||
|
||||
/**
|
||||
* A redaction that matches nothing completes with no output file. The import
|
||||
* effect used to skip those runs entirely, so `imported` never flipped and the
|
||||
* file's badge + blocking overlay spun forever - on every engine.
|
||||
*/
|
||||
const run = (overrides: Partial<PolicyRunRecord> = {}): PolicyRunRecord =>
|
||||
({
|
||||
runId: "r",
|
||||
categoryId: "security",
|
||||
fileId: "f",
|
||||
fileName: "f.pdf",
|
||||
fileSize: 1,
|
||||
target: "saas",
|
||||
status: "COMPLETED",
|
||||
outputs: [],
|
||||
error: null,
|
||||
startedAt: 0,
|
||||
...overrides,
|
||||
}) as PolicyRunRecord;
|
||||
|
||||
describe("finishedWithNothingToDeliver", () => {
|
||||
it("settles a completed run that produced no output", () => {
|
||||
expect(finishedWithNothingToDeliver(run())).toBe(true);
|
||||
});
|
||||
|
||||
it("leaves a run with outputs to the import path", () => {
|
||||
expect(
|
||||
finishedWithNothingToDeliver(
|
||||
run({ outputs: [{ fileId: "o", fileName: "o.pdf" }] as never }),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("ignores runs that aren't finished, or are already settled", () => {
|
||||
expect(finishedWithNothingToDeliver(run({ status: "PENDING" }))).toBe(
|
||||
false,
|
||||
);
|
||||
expect(finishedWithNothingToDeliver(run({ status: "FAILED" }))).toBe(false);
|
||||
expect(finishedWithNothingToDeliver(run({ imported: true }))).toBe(false);
|
||||
});
|
||||
|
||||
it("leaves classification alone - it settles via its own label path", () => {
|
||||
expect(
|
||||
finishedWithNothingToDeliver(run({ categoryId: "classification" })),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -124,6 +124,21 @@ function failRun(runId: string, message: string): void {
|
||||
const FILE_WAIT_TRIES = 20;
|
||||
const FILE_WAIT_MS = 250;
|
||||
|
||||
/**
|
||||
* A policy that changed nothing (redaction matched no text, say) completes with no
|
||||
* output: nothing to deliver, but finished. Left unimported, the file's badge and
|
||||
* its blocking overlay spin forever.
|
||||
*/
|
||||
export function finishedWithNothingToDeliver(run: PolicyRunRecord): boolean {
|
||||
return (
|
||||
run.status === "COMPLETED" &&
|
||||
!run.imported &&
|
||||
(run.outputs?.length ?? 0) === 0 &&
|
||||
// Classification has its own settle path: labels, no output file.
|
||||
!isClassificationCategory(run.categoryId)
|
||||
);
|
||||
}
|
||||
|
||||
function isTerminal(status: PolicyRunStatus): boolean {
|
||||
return (
|
||||
status === "COMPLETED" || status === "FAILED" || status === "CANCELLED"
|
||||
@@ -351,13 +366,14 @@ export function usePolicyAutoRun(): void {
|
||||
if (
|
||||
run.status !== "COMPLETED" ||
|
||||
run.imported ||
|
||||
importing.current.has(run.runId) ||
|
||||
// Classification settles even with no outputs (nothing to tag); other
|
||||
// policies need an output to import.
|
||||
(!run.outputs?.length && !classification)
|
||||
importing.current.has(run.runId)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
if (finishedWithNothingToDeliver(run)) {
|
||||
updateRun(run.runId, { imported: true });
|
||||
continue;
|
||||
}
|
||||
importing.current.add(run.runId);
|
||||
// Classification is metadata-only: stamp labels onto the current leaf of
|
||||
// the file it ran on (no version fork). See importClassificationLabels.
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
// Idle-time scheduling shared by the classification/backfill passes.
|
||||
|
||||
/** Schedule work for the browser's idle time (or soon after, as a fallback). */
|
||||
/**
|
||||
* Schedule work for the browser's idle time. Main thread only: the entry-point
|
||||
* shim guarantees `requestIdleCallback`, but not in worker scope.
|
||||
*/
|
||||
export function scheduleIdle(task: () => void): () => void {
|
||||
if (typeof requestIdleCallback === "function") {
|
||||
const handle = requestIdleCallback(task, { timeout: 2000 });
|
||||
return () => cancelIdleCallback(handle);
|
||||
}
|
||||
const timer = window.setTimeout(task, 200);
|
||||
return () => window.clearTimeout(timer);
|
||||
const handle = requestIdleCallback(task, { timeout: 2000 });
|
||||
return () => cancelIdleCallback(handle);
|
||||
}
|
||||
|
||||
@@ -2,6 +2,9 @@ import "@testing-library/jest-dom";
|
||||
import { vi } from "vitest";
|
||||
import { installFailOnConsole } from "@app/tests/failOnConsole";
|
||||
|
||||
// The shims `src/index.tsx` installs - see core/setupTests.ts.
|
||||
import "@app/utils/engineShims";
|
||||
|
||||
installFailOnConsole();
|
||||
|
||||
// Mock localStorage for tests
|
||||
|
||||
@@ -335,6 +335,11 @@ export default defineConfig(async ({ mode, command }) => {
|
||||
compressStaticCopyPlugin(),
|
||||
prerenderOgPlugin(effectiveMode === "saas"),
|
||||
],
|
||||
// Worker bundles are a separate Rollup pass and do NOT inherit `plugins`,
|
||||
// so without this `@app/*` resolves in the app and fails in a worker.
|
||||
worker: {
|
||||
plugins: () => [tsconfigPaths({ projects: [tsconfigProject] })],
|
||||
},
|
||||
server: {
|
||||
host: true,
|
||||
allowedHosts: allowedHosts.length > 0 ? allowedHosts : undefined,
|
||||
|
||||
Reference in New Issue
Block a user