Add an accessibility regression gate for Storybook (#7086)

## What

Follow-up to #7073. Turns the story scan into an accessibility gate:
stories run axe in a real browser, and CI flags a change that adds a
**new** violation.

The app has plenty of existing a11y problems (mostly theme-level colour
contrast), so rather than block everything on those, they're recorded in
`.storybook/a11y-baseline.json` and grandfathered. The gate cares about
three things:

- a story breaking a rule it wasn't already breaking
- a story that fails to render at all
- a scan that didn't cover everything it was asked to

Starting point: 839 stories carry a known violation, 1058 story-rule
pairs.

## Where it runs

- **Pull requests** scan only the stories the branch touches — usually
seconds. A full sweep is ~30 minutes, too slow to sit in front of every
merge, and the `frontend` path filter is broad enough that unrelated
changes would pay for it.
- **Nightly** scans every story, so a violation introduced somewhere
other than the story itself — a shared component, a theme token — still
surfaces within a day.
- Both upload their scan reports as artifacts; the reports carry the
offending selector and help text, without which a red run can only be
understood by reproducing it locally.
- **Advisory to start with.** It is deliberately not in
`all-checks-passed`, so it reports without blocking. Worth promoting
once a few weeks of runs show the pass/fail is stable.

## Using it

- **Fixed some violations?** `task frontend:storybook:a11y:record`
re-records so the gate locks the improvement in.
- **Locally:** `task frontend:storybook:a11y:changed` for your branch,
`task frontend:storybook:a11y` for everything.
- **New component?** Its story is picked up automatically.

## Testing

- Every story — 526 files, ~1,450 stories — runs in a real browser with
no render failures, and the gate reports no regressions against the
baseline.
- Running the gate over a single changed story takes seconds, which is
the pull-request path.
- The gate's own behaviour is covered against synthetic scan reports: a
new rule fails, the same rule on more nodes does not, a crashed story
fails, an incomplete scan refuses to report, and re-recording refuses
while anything is crashing.
- Typecheck (all build variants), ESLint and Prettier pass.

## Notes for reviewers

Some of this PR is making the mechanism trustworthy rather than adding
features, so it's worth knowing what changed and why:

- Rule ids come from the axe docs URL in each violation, not a
hand-maintained list of rule names — the old list silently ignored 39 of
axe's 104 rules, including `object-alt`, `target-size` and the table
rules.
- The baseline records **which** rules a story breaks, not how many
nodes break them. Node counts drift between runs because stories fetch
asynchronously and axe samples whatever has rendered, which made
unrelated changes look like regressions. For the same reason the
baseline is the union of repeated scans, so a run can only be a subset
of it.
- A story that fails for a non-a11y reason used to yield no rule id and
was recorded as clean, which hid crashes and could mask real violations.
Those now fail, and re-recording refuses to run while any story is
crashing.
- The scan writes a manifest of every story file it intends to cover and
the check fails unless all of them reported, so a dropped batch can't
read as "no violations".
- Vite was pre-bundling the JSX runtime mid-run and reloading the page,
which crashed whichever stories were loading; those deps are now named
up front and the per-story timeout is above the 5s default.

Colour contrast dominates the baseline and is theme-level, tracked
separately from this.
This commit is contained in:
Reece Browne
2026-07-29 11:28:24 +00:00
committed by GitHub
parent 999b5e5995
commit 8a5470dd01
12 changed files with 3083 additions and 20 deletions
+1
View File
@@ -84,6 +84,7 @@ frontend: &frontend
- .taskfiles/frontend.yml
- .taskfiles/e2e.yml
- .github/workflows/frontend-validation.yml
- .github/workflows/frontend-a11y.yml
- .github/workflows/e2e-stubbed.yml
- .github/workflows/e2e-live.yml
+11
View File
@@ -99,6 +99,17 @@ jobs:
uses: ./.github/workflows/frontend-validation.yml
secrets: inherit
# Advisory: deliberately NOT in all-checks-passed. It reports on the stories a
# branch touches so a regression is visible in review, but a browser scan is
# too new here to block merges on. Promote it once its pass/fail proves stable.
frontend-a11y:
if: needs.files-changed.outputs.frontend == 'true'
needs: [files-changed]
permissions:
contents: read
uses: ./.github/workflows/frontend-a11y.yml
secrets: inherit
playwright-e2e:
if: needs.files-changed.outputs.frontend == 'true'
needs: [files-changed]
+57
View File
@@ -0,0 +1,57 @@
name: Frontend a11y regression gate
# Reusable workflow called from build.yml when frontend sources change.
#
# Scans the stories this branch touches in real Chromium and runs axe against
# each. Existing violations are grandfathered in .storybook/a11y-baseline.json;
# the check fails on a NEW violation — a story breaking a rule it wasn't already
# breaking — or on a story that fails to render at all.
#
# Only changed stories, because a full sweep is ~30 minutes: far too slow to sit
# in front of every merge. The whole suite is scanned nightly instead
# (nightly.yml), which catches anything a branch didn't touch.
#
# Advisory for now: this is not in build.yml's all-checks-passed list, so a
# failure reports without blocking. Promote it once a few weeks of runs show the
# pass/fail is stable.
on:
workflow_call:
permissions:
contents: read
jobs:
frontend-a11y:
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- name: Harden Runner
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
# Need the base branch too, to diff against it.
fetch-depth: 0
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "22"
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install Task
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
- name: a11y gate (changed stories)
run: task frontend:storybook:a11y:changed -- origin/${{ github.base_ref || 'main' }}
- name: Upload scan reports
# The reports carry the offending selector and help text for each
# violation; without them a red run can only be understood by
# reproducing the whole scan locally.
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: a11y-scan-${{ github.run_id }}
path: frontend/.a11y-scan/
retention-days: 7
if-no-files-found: ignore
+40
View File
@@ -53,6 +53,46 @@ jobs:
path: frontend/playwright-report/
retention-days: 14
# Whole-suite accessibility sweep. Pull requests only scan the stories they
# touch (frontend-a11y.yml) because a full pass takes ~30 minutes; this covers
# everything else, so a violation introduced by a change somewhere other than
# the story itself — a shared component, a theme token — still surfaces within
# a day.
a11y-all-stories:
name: a11y (every story)
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: "22"
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install Task
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
- name: a11y gate (every story)
run: task frontend:storybook:a11y
- name: Upload scan reports
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: a11y-scan-nightly-${{ github.run_id }}
path: frontend/.a11y-scan/
retention-days: 14
if-no-files-found: ignore
# Builds all desktop platforms on a schedule so the Rust dependency cache is
# written on main, where PR and merge-queue tauri builds can restore it.
warm-tauri-cache: