mirror of
https://github.com/Stirling-Tools/Stirling-PDF.git
synced 2026-09-03 05:10:16 +03:00
Reece Browne
8a5470dd01
## What Follow-up to #7073. Turns the story scan into an accessibility gate: stories run axe in a real browser, and CI flags a change that adds a **new** violation. The app has plenty of existing a11y problems (mostly theme-level colour contrast), so rather than block everything on those, they're recorded in `.storybook/a11y-baseline.json` and grandfathered. The gate cares about three things: - a story breaking a rule it wasn't already breaking - a story that fails to render at all - a scan that didn't cover everything it was asked to Starting point: 839 stories carry a known violation, 1058 story-rule pairs. ## Where it runs - **Pull requests** scan only the stories the branch touches — usually seconds. A full sweep is ~30 minutes, too slow to sit in front of every merge, and the `frontend` path filter is broad enough that unrelated changes would pay for it. - **Nightly** scans every story, so a violation introduced somewhere other than the story itself — a shared component, a theme token — still surfaces within a day. - Both upload their scan reports as artifacts; the reports carry the offending selector and help text, without which a red run can only be understood by reproducing it locally. - **Advisory to start with.** It is deliberately not in `all-checks-passed`, so it reports without blocking. Worth promoting once a few weeks of runs show the pass/fail is stable. ## Using it - **Fixed some violations?** `task frontend:storybook:a11y:record` re-records so the gate locks the improvement in. - **Locally:** `task frontend:storybook:a11y:changed` for your branch, `task frontend:storybook:a11y` for everything. - **New component?** Its story is picked up automatically. ## Testing - Every story — 526 files, ~1,450 stories — runs in a real browser with no render failures, and the gate reports no regressions against the baseline. - Running the gate over a single changed story takes seconds, which is the pull-request path. - The gate's own behaviour is covered against synthetic scan reports: a new rule fails, the same rule on more nodes does not, a crashed story fails, an incomplete scan refuses to report, and re-recording refuses while anything is crashing. - Typecheck (all build variants), ESLint and Prettier pass. ## Notes for reviewers Some of this PR is making the mechanism trustworthy rather than adding features, so it's worth knowing what changed and why: - Rule ids come from the axe docs URL in each violation, not a hand-maintained list of rule names — the old list silently ignored 39 of axe's 104 rules, including `object-alt`, `target-size` and the table rules. - The baseline records **which** rules a story breaks, not how many nodes break them. Node counts drift between runs because stories fetch asynchronously and axe samples whatever has rendered, which made unrelated changes look like regressions. For the same reason the baseline is the union of repeated scans, so a run can only be a subset of it. - A story that fails for a non-a11y reason used to yield no rule id and was recorded as clean, which hid crashes and could mask real violations. Those now fail, and re-recording refuses to run while any story is crashing. - The scan writes a manifest of every story file it intends to cover and the check fails unless all of them reported, so a dropped batch can't read as "no violations". - Vite was pre-bundling the JSX runtime mid-run and reloading the page, which crashed whichever stories were loading; those deps are now named up front and the per-story timeout is above the 5s default. Colour contrast dominates the baseline and is theme-level, tracked separately from this.
108 lines
3.4 KiB
YAML
108 lines
3.4 KiB
YAML
name: Nightly E2E Tests
|
|
|
|
on:
|
|
schedule:
|
|
- cron: "0 2 * * *" # 2 AM UTC every night
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
playwright-all-browsers:
|
|
name: Playwright (chromium + firefox + webkit)
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Harden the runner (Audit all outbound calls)
|
|
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
|
|
with:
|
|
egress-policy: audit
|
|
|
|
- name: Checkout repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
|
with:
|
|
node-version: "22"
|
|
cache: "npm"
|
|
cache-dependency-path: frontend/package-lock.json
|
|
|
|
- name: Install Task
|
|
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
|
|
- name: Install all Playwright browsers
|
|
run: task e2e:install
|
|
|
|
- name: Build frontend (production bundle for vite preview)
|
|
env:
|
|
VITE_BUILD_FOR_PREVIEW: "1"
|
|
run: task frontend:build
|
|
|
|
- name: Run E2E tests (all browsers)
|
|
run: task e2e:cross-browser
|
|
|
|
- name: Upload Playwright report
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
with:
|
|
name: playwright-report-nightly-${{ github.run_id }}
|
|
path: frontend/playwright-report/
|
|
retention-days: 14
|
|
|
|
# Whole-suite accessibility sweep. Pull requests only scan the stories they
|
|
# touch (frontend-a11y.yml) because a full pass takes ~30 minutes; this covers
|
|
# everything else, so a violation introduced by a change somewhere other than
|
|
# the story itself — a shared component, a theme token — still surfaces within
|
|
# a day.
|
|
a11y-all-stories:
|
|
name: a11y (every story)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 60
|
|
steps:
|
|
- name: Harden the runner (Audit all outbound calls)
|
|
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
|
|
with:
|
|
egress-policy: audit
|
|
|
|
- name: Checkout repository
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
|
with:
|
|
node-version: "22"
|
|
cache: "npm"
|
|
cache-dependency-path: frontend/package-lock.json
|
|
|
|
- name: Install Task
|
|
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
|
|
|
|
- name: a11y gate (every story)
|
|
run: task frontend:storybook:a11y
|
|
|
|
- name: Upload scan reports
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
with:
|
|
name: a11y-scan-nightly-${{ github.run_id }}
|
|
path: frontend/.a11y-scan/
|
|
retention-days: 14
|
|
if-no-files-found: ignore
|
|
|
|
# Builds all desktop platforms on a schedule so the Rust dependency cache is
|
|
# written on main, where PR and merge-queue tauri builds can restore it.
|
|
warm-tauri-cache:
|
|
name: Warm Tauri Rust cache
|
|
permissions:
|
|
contents: read
|
|
pull-requests: write
|
|
uses: ./.github/workflows/tauri-build.yml
|
|
with:
|
|
platform: all
|
|
sign: false
|
|
secrets: inherit
|