mirror of
https://github.com/Stirling-Tools/Stirling-PDF.git
synced 2026-09-02 21:03:34 +03:00
Add an accessibility regression gate for Storybook (#7086)
## What Follow-up to #7073. Turns the story scan into an accessibility gate: stories run axe in a real browser, and CI flags a change that adds a **new** violation. The app has plenty of existing a11y problems (mostly theme-level colour contrast), so rather than block everything on those, they're recorded in `.storybook/a11y-baseline.json` and grandfathered. The gate cares about three things: - a story breaking a rule it wasn't already breaking - a story that fails to render at all - a scan that didn't cover everything it was asked to Starting point: 839 stories carry a known violation, 1058 story-rule pairs. ## Where it runs - **Pull requests** scan only the stories the branch touches — usually seconds. A full sweep is ~30 minutes, too slow to sit in front of every merge, and the `frontend` path filter is broad enough that unrelated changes would pay for it. - **Nightly** scans every story, so a violation introduced somewhere other than the story itself — a shared component, a theme token — still surfaces within a day. - Both upload their scan reports as artifacts; the reports carry the offending selector and help text, without which a red run can only be understood by reproducing it locally. - **Advisory to start with.** It is deliberately not in `all-checks-passed`, so it reports without blocking. Worth promoting once a few weeks of runs show the pass/fail is stable. ## Using it - **Fixed some violations?** `task frontend:storybook:a11y:record` re-records so the gate locks the improvement in. - **Locally:** `task frontend:storybook:a11y:changed` for your branch, `task frontend:storybook:a11y` for everything. - **New component?** Its story is picked up automatically. ## Testing - Every story — 526 files, ~1,450 stories — runs in a real browser with no render failures, and the gate reports no regressions against the baseline. - Running the gate over a single changed story takes seconds, which is the pull-request path. - The gate's own behaviour is covered against synthetic scan reports: a new rule fails, the same rule on more nodes does not, a crashed story fails, an incomplete scan refuses to report, and re-recording refuses while anything is crashing. - Typecheck (all build variants), ESLint and Prettier pass. ## Notes for reviewers Some of this PR is making the mechanism trustworthy rather than adding features, so it's worth knowing what changed and why: - Rule ids come from the axe docs URL in each violation, not a hand-maintained list of rule names — the old list silently ignored 39 of axe's 104 rules, including `object-alt`, `target-size` and the table rules. - The baseline records **which** rules a story breaks, not how many nodes break them. Node counts drift between runs because stories fetch asynchronously and axe samples whatever has rendered, which made unrelated changes look like regressions. For the same reason the baseline is the union of repeated scans, so a run can only be a subset of it. - A story that fails for a non-a11y reason used to yield no rule id and was recorded as clean, which hid crashes and could mask real violations. Those now fail, and re-recording refuses to run while any story is crashing. - The scan writes a manifest of every story file it intends to cover and the check fails unless all of them reported, so a dropped batch can't read as "no violations". - Vite was pre-bundling the JSX runtime mid-run and reloading the page, which crashed whichever stories were loading; those deps are now named up front and the per-story timeout is above the 5s default. Colour contrast dominates the baseline and is theme-level, tracked separately from this.
This commit is contained in:
Reece Browne
@@ -84,6 +84,7 @@ frontend: &frontend
|
||||
- .taskfiles/frontend.yml
|
||||
- .taskfiles/e2e.yml
|
||||
- .github/workflows/frontend-validation.yml
|
||||
- .github/workflows/frontend-a11y.yml
|
||||
- .github/workflows/e2e-stubbed.yml
|
||||
- .github/workflows/e2e-live.yml
|
||||
|
||||
|
||||
@@ -99,6 +99,17 @@ jobs:
|
||||
uses: ./.github/workflows/frontend-validation.yml
|
||||
secrets: inherit
|
||||
|
||||
# Advisory: deliberately NOT in all-checks-passed. It reports on the stories a
|
||||
# branch touches so a regression is visible in review, but a browser scan is
|
||||
# too new here to block merges on. Promote it once its pass/fail proves stable.
|
||||
frontend-a11y:
|
||||
if: needs.files-changed.outputs.frontend == 'true'
|
||||
needs: [files-changed]
|
||||
permissions:
|
||||
contents: read
|
||||
uses: ./.github/workflows/frontend-a11y.yml
|
||||
secrets: inherit
|
||||
|
||||
playwright-e2e:
|
||||
if: needs.files-changed.outputs.frontend == 'true'
|
||||
needs: [files-changed]
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
name: Frontend a11y regression gate
|
||||
|
||||
# Reusable workflow called from build.yml when frontend sources change.
|
||||
#
|
||||
# Scans the stories this branch touches in real Chromium and runs axe against
|
||||
# each. Existing violations are grandfathered in .storybook/a11y-baseline.json;
|
||||
# the check fails on a NEW violation — a story breaking a rule it wasn't already
|
||||
# breaking — or on a story that fails to render at all.
|
||||
#
|
||||
# Only changed stories, because a full sweep is ~30 minutes: far too slow to sit
|
||||
# in front of every merge. The whole suite is scanned nightly instead
|
||||
# (nightly.yml), which catches anything a branch didn't touch.
|
||||
#
|
||||
# Advisory for now: this is not in build.yml's all-checks-passed list, so a
|
||||
# failure reports without blocking. Promote it once a few weeks of runs show the
|
||||
# pass/fail is stable.
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
frontend-a11y:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
|
||||
with:
|
||||
egress-policy: audit
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
# Need the base branch too, to diff against it.
|
||||
fetch-depth: 0
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: "npm"
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
- name: Install Task
|
||||
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
|
||||
- name: a11y gate (changed stories)
|
||||
run: task frontend:storybook:a11y:changed -- origin/${{ github.base_ref || 'main' }}
|
||||
- name: Upload scan reports
|
||||
# The reports carry the offending selector and help text for each
|
||||
# violation; without them a red run can only be understood by
|
||||
# reproducing the whole scan locally.
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: a11y-scan-${{ github.run_id }}
|
||||
path: frontend/.a11y-scan/
|
||||
retention-days: 7
|
||||
if-no-files-found: ignore
|
||||
@@ -53,6 +53,46 @@ jobs:
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 14
|
||||
|
||||
# Whole-suite accessibility sweep. Pull requests only scan the stories they
|
||||
# touch (frontend-a11y.yml) because a full pass takes ~30 minutes; this covers
|
||||
# everything else, so a violation introduced by a change somewhere other than
|
||||
# the story itself — a shared component, a theme token — still surfaces within
|
||||
# a day.
|
||||
a11y-all-stories:
|
||||
name: a11y (every story)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
steps:
|
||||
- name: Harden the runner (Audit all outbound calls)
|
||||
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
|
||||
with:
|
||||
egress-policy: audit
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: "npm"
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install Task
|
||||
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
|
||||
|
||||
- name: a11y gate (every story)
|
||||
run: task frontend:storybook:a11y
|
||||
|
||||
- name: Upload scan reports
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: a11y-scan-nightly-${{ github.run_id }}
|
||||
path: frontend/.a11y-scan/
|
||||
retention-days: 14
|
||||
if-no-files-found: ignore
|
||||
|
||||
# Builds all desktop platforms on a schedule so the Rust dependency cache is
|
||||
# written on main, where PR and merge-queue tauri builds can restore it.
|
||||
warm-tauri-cache:
|
||||
|
||||
+43
-1
@@ -203,13 +203,55 @@ tasks:
|
||||
- npx playwright install chromium
|
||||
|
||||
storybook:test:
|
||||
desc: "Scan every story in real Chromium — each must mount without throwing"
|
||||
desc: "Scan every story in real Chromium: it must render and pass axe"
|
||||
deps: [install, storybook:browser]
|
||||
cmds:
|
||||
# Runs each story as a browser test. Pass a filter through, e.g.
|
||||
# task frontend:storybook:test -- Button
|
||||
- npx vitest run --config .storybook/vitest.config.ts {{.CLI_ARGS}}
|
||||
|
||||
storybook:a11y:
|
||||
desc: "a11y regression gate over every story: fail only on NEW axe violations"
|
||||
deps: [install, storybook:browser]
|
||||
cmds:
|
||||
- bash .storybook/a11y-scan.sh
|
||||
- node .storybook/a11y-check.mjs --in .a11y-scan --manifest .a11y-scan/manifest.txt
|
||||
|
||||
storybook:a11y:changed:
|
||||
desc: "a11y gate over stories changed vs a base ref (default origin/main)"
|
||||
summary: |
|
||||
Scans only the stories this branch touches, which is what pull requests
|
||||
run — a full scan takes ~30 minutes, far too long to sit in front of every
|
||||
merge. The nightly job covers the rest of the suite.
|
||||
|
||||
Pass a base ref through CLI_ARGS, e.g.
|
||||
task frontend:storybook:a11y:changed -- origin/release
|
||||
deps: [install, storybook:browser]
|
||||
vars:
|
||||
BASE: '{{.CLI_ARGS | default "origin/main"}}'
|
||||
# Stories touched by this branch, plus any not yet committed.
|
||||
CHANGED:
|
||||
sh: |
|
||||
{ git diff --name-only --diff-filter=d {{.CLI_ARGS | default "origin/main"}}...HEAD -- '*.stories.ts' '*.stories.tsx';
|
||||
git diff --name-only --diff-filter=d -- '*.stories.ts' '*.stories.tsx';
|
||||
git ls-files --others --exclude-standard -- '*.stories.ts' '*.stories.tsx'; } \
|
||||
| sed 's|^frontend/||' | sort -u | tr '\n' ' '
|
||||
cmds:
|
||||
- cmd: |
|
||||
if [ -z "{{.CHANGED}}" ]; then
|
||||
echo "a11y: no story files changed vs {{.BASE}} — nothing to check"
|
||||
exit 0
|
||||
fi
|
||||
bash .storybook/a11y-scan.sh {{.CHANGED}}
|
||||
node .storybook/a11y-check.mjs --in .a11y-scan --manifest .a11y-scan/manifest.txt
|
||||
|
||||
storybook:a11y:record:
|
||||
desc: "Re-record the a11y baseline (run after intentionally fixing/adding violations)"
|
||||
deps: [install, storybook:browser]
|
||||
cmds:
|
||||
- bash .storybook/a11y-scan.sh
|
||||
- node .storybook/a11y-check.mjs --in .a11y-scan --manifest .a11y-scan/manifest.txt --record
|
||||
|
||||
# ============================================================
|
||||
# Code quality
|
||||
# ============================================================
|
||||
|
||||
@@ -48,3 +48,4 @@ test-results
|
||||
/scripts/dev-update-test/.update-dist/
|
||||
/scripts/dev-update-test/screenshots/
|
||||
/editor/src-tauri/tauri.conf.dev-update.json
|
||||
.a11y-scan/
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,202 @@
|
||||
// a11y regression gate — baseline diff.
|
||||
//
|
||||
// Consumes the Storybook Vitest scan's JSON reporter output (one or more files
|
||||
// in --in) and compares each story's axe rule violations against
|
||||
// .storybook/a11y-baseline.json. Existing violations are grandfathered; the gate
|
||||
// fails when a story breaks a rule it wasn't already breaking, when a story
|
||||
// crashes, or when the scan didn't cover everything it was supposed to.
|
||||
//
|
||||
// node a11y-check.mjs --in <dir> --manifest <file> # diff (the gate)
|
||||
// node a11y-check.mjs --in <dir> --manifest <file> --record # (re)write baseline
|
||||
// node a11y-check.mjs --in <dir> --merge # union into baseline
|
||||
//
|
||||
// Exit codes: 0 pass · 1 regression · 2 unusable scan (incomplete/no input).
|
||||
//
|
||||
// The baseline records WHICH rules a story breaks, not how many nodes break
|
||||
// them. Node counts drift between runs (stories fetch data asynchronously, so
|
||||
// axe sometimes samples the DOM before late content lands), which made a
|
||||
// count-based gate cry wolf on unrelated changes. Rule presence is stable, and
|
||||
// "story now breaks a rule it didn't before" is the regression worth blocking.
|
||||
//
|
||||
// Run from frontend/. Baseline path defaults next to this script.
|
||||
|
||||
import { readFileSync, readdirSync, writeFileSync, existsSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const args = process.argv.slice(2);
|
||||
const opt = (n, d) => {
|
||||
const i = args.indexOf(n);
|
||||
const v = i >= 0 ? args[i + 1] : undefined;
|
||||
// A flag given without a value is a mistake, not a request for the default.
|
||||
if (i >= 0 && (v === undefined || v.startsWith("--"))) {
|
||||
console.error(`a11y-check: ${n} requires a value`);
|
||||
process.exit(2);
|
||||
}
|
||||
return v ?? d;
|
||||
};
|
||||
const record = args.includes("--record");
|
||||
const merge = args.includes("--merge");
|
||||
const inDir = opt("--in", ".a11y-scan");
|
||||
const manifestFile = opt("--manifest", "");
|
||||
const baselineFile = opt("--baseline", join(here, "a11y-baseline.json"));
|
||||
|
||||
// Every axe violation block in the matcher's message ends with a link to the
|
||||
// rule's docs, e.g. .../rules/axe/4.11/color-contrast?application=axeAPI. That
|
||||
// URL is the only machine-stable rule id in the message — the human-readable
|
||||
// help text around it is free to change between addon and axe versions.
|
||||
const RULE_URL = /dequeuniversity\.com\/rules\/axe\/[\d.]+\/([a-z0-9-]+)/g;
|
||||
|
||||
/** Reads every scan report in `dir`, keyed by "<storyFile> :: <storyName>". */
|
||||
function collect(dir) {
|
||||
const rules = {}; // storyKey -> Set(ruleId)
|
||||
const crashed = []; // storyKey[] — failed for a non-a11y reason
|
||||
const seenFiles = new Set();
|
||||
let scanned = 0;
|
||||
|
||||
for (const cf of readdirSync(dir).filter((f) => /\.json$/.test(f))) {
|
||||
let report;
|
||||
try {
|
||||
report = JSON.parse(readFileSync(join(dir, cf), "utf8"));
|
||||
} catch {
|
||||
console.error(`a11y-check: unreadable scan report: ${cf}`);
|
||||
process.exit(2);
|
||||
}
|
||||
for (const tf of report.testResults || []) {
|
||||
const norm = tf.name.replace(/\\/g, "/");
|
||||
const idx = norm.search(/editor\/src\//);
|
||||
const file = idx >= 0 ? norm.slice(idx) : norm;
|
||||
seenFiles.add(file);
|
||||
for (const a of tf.assertionResults || []) {
|
||||
scanned++;
|
||||
if (a.status === "passed") continue;
|
||||
const key = `${file} :: ${a.title || a.fullName || "?"}`;
|
||||
const found = new Set();
|
||||
for (const m of a.failureMessages || [])
|
||||
for (const hit of m.matchAll(RULE_URL)) found.add(hit[1]);
|
||||
if (found.size) {
|
||||
rules[key] = rules[key] || new Set();
|
||||
for (const id of found) rules[key].add(id);
|
||||
} else {
|
||||
// No rule link anywhere in the message: the story threw, timed out or
|
||||
// otherwise failed. Silently dropping these is how a broken scan can
|
||||
// look clean — and a crashed story reports no violations at all,
|
||||
// which would also mask any it really has.
|
||||
crashed.push(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return { rules, crashed, seenFiles, scanned };
|
||||
}
|
||||
|
||||
if (!existsSync(inDir)) {
|
||||
console.error(`a11y-check: scan dir not found: ${inDir}`);
|
||||
process.exit(2);
|
||||
}
|
||||
const { rules, crashed, seenFiles, scanned } = collect(inDir);
|
||||
const observed = {};
|
||||
for (const [k, set] of Object.entries(rules)) observed[k] = [...set].sort();
|
||||
|
||||
// Completeness: every story file the scan was asked to cover must appear in the
|
||||
// reports. A dropped batch (killed process, empty report) would otherwise read
|
||||
// as "those stories have no violations" and pass.
|
||||
if (manifestFile) {
|
||||
if (!existsSync(manifestFile)) {
|
||||
console.error(`a11y-check: manifest not found: ${manifestFile}`);
|
||||
process.exit(2);
|
||||
}
|
||||
const expected = readFileSync(manifestFile, "utf8")
|
||||
.split("\n")
|
||||
.map((l) => l.trim().replace(/\\/g, "/"))
|
||||
.filter(Boolean);
|
||||
const missing = expected.filter((f) => !seenFiles.has(f));
|
||||
if (missing.length) {
|
||||
console.error(
|
||||
`a11y-check: scan is incomplete — ${missing.length} of ${expected.length} story files produced no results:`,
|
||||
);
|
||||
missing.slice(0, 20).forEach((f) => console.error(` ${f}`));
|
||||
if (missing.length > 20)
|
||||
console.error(` … and ${missing.length - 20} more`);
|
||||
console.error("Refusing to report on a partial scan.");
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
if (record || merge) {
|
||||
const base =
|
||||
merge && existsSync(baselineFile)
|
||||
? JSON.parse(readFileSync(baselineFile, "utf8"))
|
||||
: {};
|
||||
if (crashed.length) {
|
||||
console.error(
|
||||
`a11y-check: refusing to record — ${crashed.length} story(ies) failed for a non-a11y reason:`,
|
||||
);
|
||||
crashed.slice(0, 20).forEach((k) => console.error(` ${k}`));
|
||||
console.error("Fix those first; recording now would bake in blind spots.");
|
||||
process.exit(2);
|
||||
}
|
||||
// Union, so a re-record can only widen what's grandfathered — it never drops
|
||||
// a rule that a slower run happened to miss and then reports it as new.
|
||||
for (const [k, list] of Object.entries(observed))
|
||||
base[k] = [...new Set([...(base[k] || []), ...list])].sort();
|
||||
const sorted = {};
|
||||
for (const k of Object.keys(base).sort()) sorted[k] = base[k];
|
||||
writeFileSync(baselineFile, JSON.stringify(sorted, null, 2) + "\n");
|
||||
const pairs = Object.values(sorted).reduce((s, l) => s + l.length, 0);
|
||||
console.log(
|
||||
`baseline ${merge ? "merged" : "recorded"}: ${scanned} stories scanned, ` +
|
||||
`${Object.keys(sorted).length} with violations, ${pairs} story-rule pairs.`,
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!existsSync(baselineFile)) {
|
||||
console.error("a11y-check: no baseline file:", baselineFile);
|
||||
process.exit(2);
|
||||
}
|
||||
const baseline = JSON.parse(readFileSync(baselineFile, "utf8"));
|
||||
|
||||
const regressions = [];
|
||||
for (const [key, list] of Object.entries(observed)) {
|
||||
const base = baseline[key] || [];
|
||||
for (const id of list)
|
||||
if (!base.includes(id)) regressions.push(`NEW ${key} ${id}`);
|
||||
}
|
||||
// Stories that improved — reported so the baseline can be ratcheted down, never
|
||||
// a failure.
|
||||
const fixed = Object.keys(baseline).filter(
|
||||
(k) => seenFiles.has(k.split(" :: ")[0]) && !observed[k],
|
||||
);
|
||||
|
||||
const pairs = Object.values(observed).reduce((s, l) => s + l.length, 0);
|
||||
console.log(
|
||||
`a11y scan: ${scanned} stories, ${Object.keys(observed).length} with violations, ` +
|
||||
`${pairs} story-rule pairs (baselined).`,
|
||||
);
|
||||
|
||||
if (crashed.length) {
|
||||
console.error(`\n✖ ${crashed.length} story(ies) failed to render:`);
|
||||
crashed.slice(0, 50).forEach((k) => console.error(` ${k}`));
|
||||
if (crashed.length > 50) console.error(` … and ${crashed.length - 50} more`);
|
||||
}
|
||||
if (regressions.length) {
|
||||
console.error(`\n✖ ${regressions.length} a11y regression(s) vs baseline:`);
|
||||
regressions.slice(0, 100).forEach((r) => console.error(` ${r}`));
|
||||
if (regressions.length > 100)
|
||||
console.error(` … and ${regressions.length - 100} more`);
|
||||
console.error(
|
||||
"\nFix the new violation(s). If a story was renamed or moved its old " +
|
||||
"baseline key no longer matches — re-record: task frontend:storybook:a11y:record",
|
||||
);
|
||||
}
|
||||
if (crashed.length || regressions.length) process.exit(1);
|
||||
|
||||
if (fixed.length)
|
||||
console.log(
|
||||
`✓ no a11y regressions. ${fixed.length} baselined story(ies) now clean — ` +
|
||||
`re-record to lock that in: task frontend:storybook:a11y:record`,
|
||||
);
|
||||
else console.log("✓ no a11y regressions vs baseline.");
|
||||
process.exit(0);
|
||||
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run the Storybook Vitest scan in batches and emit one JSON report per batch
|
||||
# into .a11y-scan/, plus a manifest of every story file the run was supposed to
|
||||
# cover and a log of the raw output. Consumed by a11y-check.mjs, which fails if
|
||||
# any manifest entry produced no results. Run from frontend/.
|
||||
#
|
||||
# a11y-scan.sh scan every story
|
||||
# a11y-scan.sh <file> [file…] scan only these story files
|
||||
#
|
||||
# Batching keeps each browser session small: a single run over the whole story
|
||||
# set holds one Chromium context open for the entire scan, so one crash in it
|
||||
# costs every story after that point.
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/.." || exit 1
|
||||
|
||||
OUT=".a11y-scan"
|
||||
rm -rf "$OUT"
|
||||
mkdir -p "$OUT"
|
||||
MANIFEST="$OUT/manifest.txt"
|
||||
LOG="$OUT/scan.log"
|
||||
|
||||
if [ "$#" -gt 0 ]; then
|
||||
# Explicit list (the pull-request path passes just the stories a branch
|
||||
# touched). Anything that no longer exists is dropped, so a deleted story
|
||||
# doesn't fail the manifest check.
|
||||
for f in "$@"; do [ -f "$f" ] && printf '%s\n' "$f"; done | sort -u >"$MANIFEST"
|
||||
else
|
||||
# Tracked story files plus any not yet committed, so a new story can be
|
||||
# checked before it is added to the index.
|
||||
{
|
||||
git ls-files 'editor/src/**/*.stories.ts' 'editor/src/**/*.stories.tsx'
|
||||
git ls-files --others --exclude-standard 'editor/src/**/*.stories.ts' \
|
||||
'editor/src/**/*.stories.tsx'
|
||||
} | sort -u >"$MANIFEST"
|
||||
fi
|
||||
|
||||
mapfile -t FILES <"$MANIFEST"
|
||||
TOTAL=${#FILES[@]}
|
||||
if [ "$TOTAL" -eq 0 ]; then
|
||||
if [ "$#" -gt 0 ]; then
|
||||
echo "a11y-scan: no existing story files in the given list — nothing to scan"
|
||||
exit 0
|
||||
fi
|
||||
echo "a11y-scan: no story files found — check the glob" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
CHUNK=20
|
||||
NB=$(((TOTAL + CHUNK - 1) / CHUNK))
|
||||
echo "a11y-scan: $TOTAL story files, $NB batches of $CHUNK"
|
||||
|
||||
failed=()
|
||||
i=0
|
||||
ci=0
|
||||
while [ "$i" -lt "$TOTAL" ]; do
|
||||
ci=$((ci + 1))
|
||||
batch=("${FILES[@]:i:CHUNK}")
|
||||
i=$((i + CHUNK))
|
||||
out="$OUT/chunk-$ci.json"
|
||||
filters=()
|
||||
for f in "${batch[@]}"; do filters+=("${f%.tsx}"); done
|
||||
# The scan exits non-zero whenever a story has a violation — expected here, so
|
||||
# the report is what matters, not the status. Output is teed to the log so a red
|
||||
# CI run still has the offending selectors and help text to work from.
|
||||
for attempt in 1 2; do
|
||||
timeout 300 npx vitest run --config .storybook/vitest.config.ts \
|
||||
--reporter=json --outputFile="$out" "${filters[@]}" >>"$LOG" 2>&1
|
||||
[ -s "$out" ] && break
|
||||
echo "a11y-scan: batch $ci produced no report (attempt $attempt)" >&2
|
||||
done
|
||||
if [ -s "$out" ]; then
|
||||
echo " batch $ci/$NB done"
|
||||
else
|
||||
failed+=("$ci")
|
||||
echo " batch $ci/$NB FAILED — no report" >&2
|
||||
fi
|
||||
done
|
||||
|
||||
echo "a11y-scan: $(ls "$OUT"/chunk-*.json 2>/dev/null | wc -l)/$NB batches produced reports"
|
||||
if [ ${#failed[@]} -gt 0 ]; then
|
||||
echo "a11y-scan: ${#failed[@]} batch(es) produced no report: ${failed[*]}" >&2
|
||||
echo "a11y-scan: see $LOG. Not reporting on a partial scan." >&2
|
||||
exit 2
|
||||
fi
|
||||
@@ -242,12 +242,10 @@ const preview: Preview = {
|
||||
],
|
||||
},
|
||||
a11y: {
|
||||
// Run axe automatically against the story root; violations show in the
|
||||
// Accessibility panel. `context` replaced `element` in addon-a11y 9.x.
|
||||
context: "#storybook-root",
|
||||
config: {},
|
||||
options: {},
|
||||
test: "todo",
|
||||
// Run axe against the rendered story; `test: "error"` fails the scan on
|
||||
// any violation. Context is left at the addon default (the document root)
|
||||
// so it resolves under both the Storybook UI and the Vitest browser mount.
|
||||
test: "error",
|
||||
},
|
||||
},
|
||||
globalTypes: {
|
||||
|
||||
@@ -4,10 +4,9 @@ import { storybookTest } from "@storybook/addon-vitest/vitest-plugin";
|
||||
|
||||
/**
|
||||
* Dedicated Vitest config that turns every story into a browser test: it mounts
|
||||
* the story in real Chromium as a render/smoke check (a story must mount without
|
||||
* throwing). a11y is currently report-only (preview's `a11y.test: "todo"`) and is
|
||||
* not yet enforced here — flipping it to pass/fail is a follow-up. Kept separate
|
||||
* from editor/vitest.config.ts (the jsdom unit tests) so the two suites don't collide.
|
||||
* the story in real Chromium and runs axe against it, so a story fails if it
|
||||
* throws on mount or trips an accessibility rule. Kept separate from
|
||||
* editor/vitest.config.ts (the jsdom unit tests) so the two suites don't collide.
|
||||
*
|
||||
* The storybook test must live in a `test.projects[]` entry (not a flat config)
|
||||
* so Vitest wires up the browser test runner correctly.
|
||||
@@ -23,8 +22,8 @@ export default defineConfig({
|
||||
// JSX runtime import, so Vite optimizes them lazily mid-run and emits
|
||||
// "optimized dependencies changed, reloading". That reload tears down the
|
||||
// browser worker and whichever stories were mid-load fail with a bogus
|
||||
// "Failed to fetch dynamically imported module" — a result that looks real.
|
||||
// Naming them here keeps a run deterministic.
|
||||
// "Failed to fetch dynamically imported module" — a scan that then looks
|
||||
// like a real result. Naming them here keeps a run deterministic.
|
||||
include: [
|
||||
"react",
|
||||
"react/jsx-runtime",
|
||||
@@ -42,10 +41,11 @@ export default defineConfig({
|
||||
plugins: [storybookTest({ configDir: resolve(__dirname) })],
|
||||
test: {
|
||||
name: "storybook",
|
||||
// Mounting a story takes well over Vitest's 5s default on the heavier
|
||||
// screens, and a story that trips the timeout is reported as a failure
|
||||
// with no message — which reads like a crash. Give it room; a
|
||||
// genuinely hung story still fails, just later.
|
||||
// Mounting a story and running a full axe pass over it takes well
|
||||
// over Vitest's 5s default on the heavier screens, and a story that
|
||||
// trips the timeout is reported as a failure with no message — which
|
||||
// reads like a crash and makes the run non-reproducible. Give it
|
||||
// room; a genuinely hung story still fails, just later.
|
||||
testTimeout: 60_000,
|
||||
hookTimeout: 60_000,
|
||||
browser: {
|
||||
|
||||
@@ -1,10 +1,15 @@
|
||||
import { beforeAll } from "vitest";
|
||||
import { setProjectAnnotations } from "@storybook/react-vite";
|
||||
import * as a11yAddonAnnotations from "@storybook/addon-a11y/preview";
|
||||
// eslint-disable-next-line no-restricted-imports -- Storybook-only: the sibling preview config has no @-alias.
|
||||
import * as projectAnnotations from "./preview";
|
||||
|
||||
// Apply the same decorators/parameters/globals the Storybook UI uses (providers,
|
||||
// i18n, theme) so stories run under Vitest render identically to the browser.
|
||||
const project = setProjectAnnotations([projectAnnotations]);
|
||||
// Include addon-a11y's annotations so its axe checks run under Vitest, not only
|
||||
// in the Storybook UI panel. projectAnnotations supplies the same
|
||||
// decorators/parameters/globals (providers, i18n, theme) as the browser.
|
||||
const project = setProjectAnnotations([
|
||||
a11yAddonAnnotations,
|
||||
projectAnnotations,
|
||||
]);
|
||||
|
||||
beforeAll(project.beforeAll);
|
||||
|
||||
Reference in New Issue
Block a user