a11y job: scan stories when their component changes; fix cold-start false failures (#7191)

## What

Two fixes to the pull-request a11y job (#7086 follow-up), both found on
its first day live.

### It now scans a component's stories when the component changes

The job picked its scan set from changed **story files** alone. But a
story renders the live component — editing `Button.tsx` changes what
every Button story shows without touching a story file, and the job
scanned nothing. That's the common way a11y regressions arrive, and it
was exactly the case the job missed.

The scan set now also includes stories whose **same-named sibling source
file changed**: edit `Button.tsx` or `Button.css` and
`Button.stories.tsx` is scanned. Changes that ripple further than a
component's own stories (shared UI, theme tokens) remain the nightly
sweep's job.

### It no longer fails on cold-start infrastructure noise

The job's first real run (#7163) flagged a story as "failed to render".
The story was fine — on a cold dependency cache (**every** CI run), Vite
discovered the preview's own dependency graph mid-run and reloaded the
page, killing whichever story happened to be loading with `Failed to
fetch dynamically imported module`. Reproduced on a cold cache, passes
on a warm one.

- The preview's deps are named in `optimizeDeps.include`, which removes
the mid-run reload (verified cold).
- A batch whose report contains crash-class failures (failures carrying
no axe rule) is retried once — a one-off infrastructure death passes the
retry, a story that genuinely can't render fails both attempts and is
still reported.

Also: the scan-report artifacts were never actually uploading — they
live in a dot-directory, which `upload-artifact` silently skips as
hidden by default. `include-hidden-files: true` fixes that for the PR
job and the nightly, so a red run finally has its evidence attached.

### The glue is Node now, so tasks work from any shell

Raised in review: the pipeline leaned on `bash`, `sed`, `grep`, `sort`
and `tr`. Task runs its commands in an embedded POSIX interpreter, but
those are external binaries it has to find on PATH — and a Windows dev
calling tasks from **PowerShell** has none of them (`sed`/`tr` missing
outright, `sort` resolves to Windows' own, and `bash` resolves to
*WSL's*). Confirmed broken by running the task from PowerShell before
the change.

The batch runner and affected-story detection are now small Node scripts
(`a11y-scan.mjs`, `a11y-changed.mjs`) — the repo already requires Node,
so one implementation serves PowerShell, git-bash and CI alike, instead
of maintaining `.sh`/`.ps1` twins.

## Testing

- Sibling detection: editing `Tabs.tsx` (component only) pulls
`Tabs.stories.tsx` into the scan set; editing a `.css` sibling does the
same; nothing unrelated leaks in.
- **From PowerShell**: `task frontend:storybook:a11y:changed`
early-exits cleanly with no changes, and with a component edit it
detects the sibling, runs the browser scan and passes the gate — same
result from git-bash.
- Cold cache end-to-end: cleared both Vite caches, ran the scan — no
re-optimize, no reload, stories fail only on their (baselined) axe
results.
- Crash classifier: 1 on a synthetic crash report, 0 on axe-only
failures, 0 on a real report — so the retry can't be triggered by
legitimate violations.
- Full scan + gate run green end-to-end; taskfile parses, workflows are
valid YAML, Prettier/ESLint pass.

#7163's red check needs no action from that PR's author — it should go
green on re-run once this lands.
This commit is contained in:
Reece Browne
2026-07-29 14:18:55 +00:00
committed by GitHub
parent 8a5470dd01
commit 4d207f0c3f
7 changed files with 285 additions and 104 deletions
+14 -14
View File
@@ -214,42 +214,42 @@ tasks:
desc: "a11y regression gate over every story: fail only on NEW axe violations"
deps: [install, storybook:browser]
cmds:
- bash .storybook/a11y-scan.sh
- node .storybook/a11y-scan.mjs
- node .storybook/a11y-check.mjs --in .a11y-scan --manifest .a11y-scan/manifest.txt
storybook:a11y:changed:
desc: "a11y gate over stories changed vs a base ref (default origin/main)"
desc: "a11y gate over the stories this branch affects (default base origin/main)"
summary: |
Scans only the stories this branch touches, which is what pull requests
run — a full scan takes ~30 minutes, far too long to sit in front of every
merge. The nightly job covers the rest of the suite.
Scans the stories a branch affects, which is what pull requests run — a
full scan takes ~30 minutes, far too long to sit in front of every merge.
A story is affected if its file changed, or if a same-named sibling
source file changed (editing Button.tsx or Button.css re-scans
Button.stories.tsx — the story renders the live component, so a component
edit changes what the story shows without touching the story file).
Changes that ripple further than a component's own stories are covered by
the nightly full sweep.
Pass a base ref through CLI_ARGS, e.g.
task frontend:storybook:a11y:changed -- origin/release
deps: [install, storybook:browser]
vars:
BASE: '{{.CLI_ARGS | default "origin/main"}}'
# Stories touched by this branch, plus any not yet committed.
CHANGED:
sh: |
{ git diff --name-only --diff-filter=d {{.CLI_ARGS | default "origin/main"}}...HEAD -- '*.stories.ts' '*.stories.tsx';
git diff --name-only --diff-filter=d -- '*.stories.ts' '*.stories.tsx';
git ls-files --others --exclude-standard -- '*.stories.ts' '*.stories.tsx'; } \
| sed 's|^frontend/||' | sort -u | tr '\n' ' '
sh: node .storybook/a11y-changed.mjs {{.CLI_ARGS | default "origin/main"}}
cmds:
- cmd: |
if [ -z "{{.CHANGED}}" ]; then
echo "a11y: no story files changed vs {{.BASE}} — nothing to check"
echo "a11y: no story files affected vs {{.BASE}} — nothing to check"
exit 0
fi
bash .storybook/a11y-scan.sh {{.CHANGED}}
node .storybook/a11y-scan.mjs {{.CHANGED}}
node .storybook/a11y-check.mjs --in .a11y-scan --manifest .a11y-scan/manifest.txt
storybook:a11y:record:
desc: "Re-record the a11y baseline (run after intentionally fixing/adding violations)"
deps: [install, storybook:browser]
cmds:
- bash .storybook/a11y-scan.sh
- node .storybook/a11y-scan.mjs
- node .storybook/a11y-check.mjs --in .a11y-scan --manifest .a11y-scan/manifest.txt --record
# ============================================================