From 60741e305d46cab2082bbef166ffbe6f1d313596 Mon Sep 17 00:00:00 2001 From: Anthony Stirling <77850077+Frooodle@users.noreply.github.com> Date: Sun, 16 Aug 2026 11:22:19 +0100 Subject: [PATCH 01/59] Update CONTRIBUTING.md to include license information (#7512) Added a section about licensing contributions to the project. # Description of Changes --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details. --- CONTRIBUTING.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9092d5cd72..65fc4bc262 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,6 +2,13 @@ Thank you for your interest in contributing to Stirling-PDF! There are many ways to contribute other than writing code. For example, reporting bugs, creating suggestions, and adding or modifying translations. +## License + +By contributing to this project, you agree that your contributions will be licensed under the project [license](LICENSE), which follows an open-core model. +The codebase is a mix of MIT and source-available code, so your contribution is licensed according to the directory it is committed to. + +PRs are welcome in any directory by any user, just be aware of which license applies to the code you change. + ## Issue Guidelines Issues can be used to report bugs, request features, or ask questions. If you have a question, you could also ask us in our [Discord](https://discord.gg/FJUSXUSYec). @@ -63,7 +70,3 @@ For technical guides, setup instructions, and development resources: For configuration and usage guides, see: - [Database Guide](DATABASE.md) - Database setup and configuration - [OCR Guide](HowToUseOCR.md) - OCR setup and configuration - -## License - -By contributing to this project, you agree that your contributions will be licensed under the [MIT License](LICENSE). From 3bb3b721de206f28346dc410df1512f6a90814f2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:15:34 +0100 Subject: [PATCH 02/59] build(deps): bump logback from 1.5.32 to 1.6.1 (#7339) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps `logback` from 1.5.32 to 1.6.1. Updates `ch.qos.logback:logback-core` from 1.5.32 to 1.6.1
Release notes

Sourced from ch.qos.logback:logback-core's releases.

Logback 1.6.1

2026-07-28 Release of logback version 1.6.1

• In TimeBasedRollingPolicy, when the file option is set, the intermediate file renamed before asynchronous compression now receives the target archive name without the compression suffix (e.g. .gz, .zip, .xz). Previously it used a nanotime-based .tmp suffix. This makes the file easier to identify if compression fails during rollover. (See also the following paragraph.)

• On GZ, ZIP, or XZ compression failure, the original (uncompressed) log file is no longer deleted. Compression strategies now delete the source file only after successful compression and emit a warning that the original was left intact.

• ConsoleAppender with now probes JLine's org.jline.jansi.AnsiConsole first and falls back to the legacy FuseSource org.fusesource.jansi.AnsiConsole class. This keeps ANSI coloring working after Jansi moved under the JLine project. The optional org.jline:jansi-core artifact is declared as a dependency alongside the existing FuseSource jansi dependency. A preferredJansiClassName property was added for tests. This issue was reported in issues/1043 by seonwoo_jung who also provided the relevant PR.

• LayoutWrappingEncoder now reports an error at start() when no layout is set and guards encode() against a null layout. Previously, a missing layout (for example after an ignored // branch) allowed the encoder to start and then fail with a NullPointerException on every event, resulting in silent log loss. This issue was reported in issues/1046 by seonwoo_jung who also provided the relevant PR.

• FileCollisionAnalyser now detects file collisions involving nested appenders of SiftingAppender. When the nested file or fileNamePattern does not textually reference the discriminator key (e.g. ${userId}), a warning is issued at configuration time naming the appender, the key, and the shared target. This closes a gap where statically declared file appenders were checked but sifted nested appenders were not. This enhancement was contributed in [PR #1041](qos-ch/logback#1041) by seonwoo_jung.

• More defensive handling in SyslogOutputStream and SyslogAppenderBase: the close() method now ensures that resources are closed, writes and flushes check that the underlying resources are in a valid state and fallback to no-op otherwise.

• A bit-wise identical binary of this version can be reproduced by building from source code at commit 57759f433000a133088ef0441038963134437fbd associated with the tag v_1.6.1. The release was built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

• See https://logback.qos.ch/news.html#1.6.1 for the original text.

Logback 1.6.0

2026-07-23 Release of logback version 1.6.0

• Removed certain deprecated variables, methods, and classes. For the list of removed members see release_1.6.0.txt.

• In AsyncAppenderBase, the put(ILoggingEvent) method now has the protected modifier to allow access from derived classes. This change was requested by Thomas Skjølberg in pr#1053.

• Bump SLF4J dependency to version 2.0.18.

See also the overview of the 1.6.x series.

• A bit-wise identical binary of this version can be reproduced by building from source code at commit b07adf36019b51a10f824fdd94009985c587b1d3 associated with the tag v_1.6.0. The release was built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

Logback 1.5.38

2026-07-09 Release of logback version 1.5.38

• In HardenedObjectInputStream, fixed a typo preventing Throwable objects from being white-filtered. This issue was reported in [PR #1045](qos-ch/logback#1045) by t0rchwo0d.

• A bitwise identical binary of this version can be reproduced by building from source code at commit d04984a41fce42977466f45a2f076f0ee5cc4207 associated with the tag v_1.5.38. Release built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

Logback 1.5.37

2026-06-26 Release of logback version 1.5.37

  1. • Given the numerous vulnerabilities related to conditional configuration processing based on the evaluation of Java expressions using the Janino library, support for such expressions has been removed. Users are offered the an online migration service or the <condition> element introduced in version 1.5.20. See the relevant documentation for more details.

• A bitwise identical binary of this version can be reproduced by building from source code at commit c1df7f522e648eec7b4ef6a12c8758fec0f00048 associated with the tag v_1.5.37. Release built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

Logback 1.5.36

2026-06-25 Release of logback version 1.5.36

• The 'condition' attribute in <if> elements now reject certain references that are associated with ACE attacks. This issue was reported by "yulate" (yulate531@gmail.com.com) and registered as CVE-2026-13006. Please note that version 1.5.37 provides the full fix to this vulnerability.

... (truncated)

Commits

Updates `ch.qos.logback:logback-classic` from 1.5.32 to 1.6.1
Release notes

Sourced from ch.qos.logback:logback-classic's releases.

Logback 1.6.1

2026-07-28 Release of logback version 1.6.1

• In TimeBasedRollingPolicy, when the file option is set, the intermediate file renamed before asynchronous compression now receives the target archive name without the compression suffix (e.g. .gz, .zip, .xz). Previously it used a nanotime-based .tmp suffix. This makes the file easier to identify if compression fails during rollover. (See also the following paragraph.)

• On GZ, ZIP, or XZ compression failure, the original (uncompressed) log file is no longer deleted. Compression strategies now delete the source file only after successful compression and emit a warning that the original was left intact.

• ConsoleAppender with now probes JLine's org.jline.jansi.AnsiConsole first and falls back to the legacy FuseSource org.fusesource.jansi.AnsiConsole class. This keeps ANSI coloring working after Jansi moved under the JLine project. The optional org.jline:jansi-core artifact is declared as a dependency alongside the existing FuseSource jansi dependency. A preferredJansiClassName property was added for tests. This issue was reported in issues/1043 by seonwoo_jung who also provided the relevant PR.

• LayoutWrappingEncoder now reports an error at start() when no layout is set and guards encode() against a null layout. Previously, a missing layout (for example after an ignored // branch) allowed the encoder to start and then fail with a NullPointerException on every event, resulting in silent log loss. This issue was reported in issues/1046 by seonwoo_jung who also provided the relevant PR.

• FileCollisionAnalyser now detects file collisions involving nested appenders of SiftingAppender. When the nested file or fileNamePattern does not textually reference the discriminator key (e.g. ${userId}), a warning is issued at configuration time naming the appender, the key, and the shared target. This closes a gap where statically declared file appenders were checked but sifted nested appenders were not. This enhancement was contributed in [PR #1041](qos-ch/logback#1041) by seonwoo_jung.

• More defensive handling in SyslogOutputStream and SyslogAppenderBase: the close() method now ensures that resources are closed, writes and flushes check that the underlying resources are in a valid state and fallback to no-op otherwise.

• A bit-wise identical binary of this version can be reproduced by building from source code at commit 57759f433000a133088ef0441038963134437fbd associated with the tag v_1.6.1. The release was built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

• See https://logback.qos.ch/news.html#1.6.1 for the original text.

Logback 1.6.0

2026-07-23 Release of logback version 1.6.0

• Removed certain deprecated variables, methods, and classes. For the list of removed members see release_1.6.0.txt.

• In AsyncAppenderBase, the put(ILoggingEvent) method now has the protected modifier to allow access from derived classes. This change was requested by Thomas Skjølberg in pr#1053.

• Bump SLF4J dependency to version 2.0.18.

See also the overview of the 1.6.x series.

• A bit-wise identical binary of this version can be reproduced by building from source code at commit b07adf36019b51a10f824fdd94009985c587b1d3 associated with the tag v_1.6.0. The release was built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

Logback 1.5.38

2026-07-09 Release of logback version 1.5.38

• In HardenedObjectInputStream, fixed a typo preventing Throwable objects from being white-filtered. This issue was reported in [PR #1045](qos-ch/logback#1045) by t0rchwo0d.

• A bitwise identical binary of this version can be reproduced by building from source code at commit d04984a41fce42977466f45a2f076f0ee5cc4207 associated with the tag v_1.5.38. Release built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

Logback 1.5.37

2026-06-26 Release of logback version 1.5.37

  1. • Given the numerous vulnerabilities related to conditional configuration processing based on the evaluation of Java expressions using the Janino library, support for such expressions has been removed. Users are offered the an online migration service or the <condition> element introduced in version 1.5.20. See the relevant documentation for more details.

• A bitwise identical binary of this version can be reproduced by building from source code at commit c1df7f522e648eec7b4ef6a12c8758fec0f00048 associated with the tag v_1.5.37. Release built using Java "21" 2023-10-17 LTS build 21.0.1.+12-LTS-29 under Linux Debian 11.6.

Logback 1.5.36

2026-06-25 Release of logback version 1.5.36

• The 'condition' attribute in <if> elements now reject certain references that are associated with ACE attacks. This issue was reported by "yulate" (yulate531@gmail.com.com) and registered as CVE-2026-13006. Please note that version 1.5.37 provides the full fix to this vulnerability.

... (truncated)

Commits

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.gradle b/build.gradle index 2b2d0c625f..c7d8eb6f1d 100644 --- a/build.gradle +++ b/build.gradle @@ -30,7 +30,7 @@ ext { openSamlVersion = "5.2.1" commonmarkVersion = "0.28.0" googleJavaFormatVersion = "1.35.0" - logback = "1.5.32" + logback = "1.6.1" commonsIoVersion = "2.22.0" commonsLang3 = "3.20.0" rhinoVersion = "1.9.1" From 88afdd07afa9f16359cf0297957b7bffd2f27db6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:15:58 +0100 Subject: [PATCH 03/59] build(deps): bump awsSdkVersion from 2.44.12 to 2.51.2 (#7371) Bumps `awsSdkVersion` from 2.44.12 to 2.51.2. Updates `software.amazon.awssdk:s3` from 2.44.12 to 2.51.2 Updates `software.amazon.awssdk:url-connection-client` from 2.44.12 to 2.51.2 Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.gradle b/build.gradle index c7d8eb6f1d..5c8f4b47bb 100644 --- a/build.gradle +++ b/build.gradle @@ -44,7 +44,7 @@ ext { batikVersion = "1.19" jpdfiumVersion = "1.0.4" jwtVersion = "0.13.0" - awsSdkVersion = "2.44.12" + awsSdkVersion = "2.51.2" jschVersion = "2.28.6" commonsNetVersion = "3.11.1" smbjVersion = "0.14.0" From fb362f70ab77fcd4914983fa842459f46280915a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:16:15 +0100 Subject: [PATCH 04/59] build(deps): bump github/codeql-action/upload-sarif from 4.37.4 to 4.37.6 (#7467) Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.4 to 4.37.6.
Release notes

Sourced from github/codeql-action/upload-sarif's releases.

v4.37.6

  • Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to .github/codeql-config.yml to align it with the suggested path that is used elsewhere. #4070

v4.37.5

  • Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the init Action instead of falling back to downloading the bundle before extracting it. #4061
Changelog

Sourced from github/codeql-action/upload-sarif's changelog.

CodeQL Action Changelog

See the releases page for the relevant changes to the CodeQL CLI and language packs.

[UNRELEASED]

  • Update default CodeQL bundle version to 2.26.3. #4085

4.37.6 - 04 Aug 2026

  • Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to .github/codeql-config.yml to align it with the suggested path that is used elsewhere. #4070

4.37.5 - 03 Aug 2026

  • Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the init Action instead of falling back to downloading the bundle before extracting it. #4061

4.37.4 - 29 Jul 2026

  • This version of the CodeQL Action adds support for the tools input for the codeql-action/init step to be specified using a github-codeql-tools repository property. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to toolcache to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for tools in the workflow definition always takes precedence unless the value of the repository property starts with !. #4037
  • Update default CodeQL bundle version to 2.26.2. #4051

4.37.3 - 22 Jul 2026

No user facing changes.

4.37.2 - 21 Jul 2026

  • The new address format for the config-file input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the remote= prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. #4023
  • The CodeQL Action can now make use of configured private registries in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. #4007

4.37.1 - 16 Jul 2026

  • Upcoming breaking change: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. #3956
  • Update default CodeQL bundle version to 2.26.1. #4019

4.37.0 - 08 Jul 2026

  • Update default CodeQL bundle version to 2.26.0. #3995
  • In addition to the existing input format, the config-file input for the codeql-action/init step will soon support a new [owner/]repo[@ref][:path] format. All components except the repository name are optional. If omitted, owner defaults to the same owner as the repository the analysis is running for, ref to main, and path to .github/codeql-action.yaml. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. #3973

4.36.3 - 01 Jul 2026

No user facing changes.

4.36.2 - 04 Jun 2026

  • Cache CodeQL CLI version information across Actions steps. #3943
  • Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. #3937
  • Update default CodeQL bundle version to 2.25.6. #3948

... (truncated)

Commits
  • 5595cca Merge pull request #4071 from github/update-v4.37.6-6a9359a1b
  • ec9c757 Add change note for PR 4070
  • 45c8742 Update changelog for v4.37.6
  • 6a9359a Merge pull request #4070 from github/mbg/remote-address/change-file-default
  • 065cdc0 Change DEFAULT_CONFIG_FILE_NAME
  • f99dd5a Merge pull request #4066 from github/dependabot/npm_and_yarn/js-yaml-5.2.2
  • 1804b21 Merge pull request #4068 from github/mergeback/v4.37.5-to-main-d1ba80a1
  • 3020a2f Rebuild
  • 93c3a5a Update changelog and version after v4.37.5
  • d1ba80a Merge pull request #4067 from github/update-v4.37.5-1cd4d01d5
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github/codeql-action/upload-sarif&package-manager=github_actions&previous-version=4.37.4&new-version=4.37.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/scorecards.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/scorecards.yml b/.github/workflows/scorecards.yml index 2eab52b1d4..0e9b27c829 100644 --- a/.github/workflows/scorecards.yml +++ b/.github/workflows/scorecards.yml @@ -75,6 +75,6 @@ jobs: # Upload the results to GitHub's code scanning dashboard. - name: "Upload to code-scanning" - uses: github/codeql-action/upload-sarif@f205ea1c3313d32999d8d6a48b4f6530d4437b38 # v4.37.4 + uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 with: sarif_file: results.sarif From eece9fbdf8f80a09ca66dfd70dabbb65bde141a7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:16:32 +0100 Subject: [PATCH 05/59] build(deps): bump commons-net:commons-net from 3.11.1 to 3.13.0 (#7517) Bumps [commons-net:commons-net](https://github.com/apache/commons-net) from 3.11.1 to 3.13.0.
Changelog

Sourced from commons-net:commons-net's changelog.

Apache Commons Net 3.13.0 Release Notes

The Apache Commons Net team is pleased to announce the release of Apache Commons Net 3.13.0.

Apache Commons Net library contains a collection of network utilities and protocol implementations. Supported protocols include Echo, Finger, FTP, NNTP, NTP, POP3(S), SMTP(S), Telnet, and Whois.

This is a feature and maintenance release. Java 8 or later is required.

For complete information on Apache Commons Net, including instructions on how to submit bug reports, patches, or suggestions for improvement, see the Apache Commons Net website:

https://commons.apache.org/proper/commons-net/

Download page: https://commons.apache.org/proper/commons-net/download_net.cgi

New features

o Add DatagramSocketClient.getDefaultTimeoutDuration() and deprecate getDefaultTimeout(). Thanks to Gary Gregory. o NET-741: Add subnet IPv6 handling with SubnetUtils6 #391. Thanks to Maros Orsak, Gary Gregory.

Fixed Bugs

o DaytimeTCPClientTest now should now pass inside most VPNs. Thanks to Henri Biestro, Gary Gregory. o Migrate tests to JUnit5 #358, #359. Thanks to Jakub Kupczyk, Gary Gregory. o Fix malformed Javadoc comments. Thanks to Gary Gregory. o IMAPExportMbox now restores the current thread's interrupt flag when catching InterruptedException. Thanks to Gary Gregory. o IOUtil.readWrite() now restores the current thread's interrupt flag when catching InterruptedException. Thanks to Gary Gregory. o TelnetInputStream now restores the current thread's interrupt flag when catching InterruptedException. Thanks to Gary Gregory. o NET-740: FTP fails to parse listings for Linux vsftpd in Chinese or Japanese #393. Thanks to Jianwei Guo, Gary Gregory. o TelnetInputStream.read() doesn't preserve the original InterruptedException as the cause of its InterruptedIOException. Thanks to Gary Gregory. o FTPClient._storeFile(String, String, InputStream) doesn't always close it's internal socket when an exception is thrown early in processing. Thanks to Gary Gregory. o ListenerList.removeListener(T) now ignores null input to avoid a NullPointerException. Thanks to Gary Gregory. o ListenerList.addListener(T) now ignores null input. Thanks to Gary Gregory. o Fix typo in FTPConnectionClosedException message from FTP.getReply(boolean). Thanks to Gary Gregory. o Reimplement Util.copyReader() with IOUtils.copyLarge(). Thanks to Gary Gregory. o Reimplement Util.copyStream() with IOUtils.copyLarge(). Thanks to Gary Gregory. o Reimplement Util.copyStream() with IOUtils.copyLarge(). Thanks to Gary Gregory. o Deprecate Util.copyReader(Reader, Writer) in favor of IOUtils.copyLarge(Reader, Writer). Thanks to Gary Gregory.

Changes

o Bump org.apache.commons:commons-parent from 85 to 97 #371, #388, #389. Thanks to Gary Gregory, Dependabot. o Bump org.apache.commons:commons-lang3 from 3.18.0 to 3.19.0. Thanks to Gary Gregory, Dependabot.

... (truncated)

Commits
  • 35fa0df Prepare for the release candidate 3.13.0 RC1
  • 1cc5b6e Prepare for the next release candidate
  • a4b246e Reuse IOUtils.copyLarge()
  • e5ca262 Javadoc
  • 3cb18ec Deprecate Util.copyReader(Reader, Writer) in favor of
  • 3373d77 Reimplement Util.copyReader() with IOUtils.copyLarge()
  • 3be2077 Reimplement Util.copyStream() with IOUtils.copyLarge()
  • 7b41144 Javadoc
  • 1b584b4 Fix typo in FTPConnectionClosedException message from
  • 3792023 Javadoc
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commons-net:commons-net&package-manager=gradle&previous-version=3.11.1&new-version=3.13.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.gradle b/build.gradle index 5c8f4b47bb..b7fcb794cc 100644 --- a/build.gradle +++ b/build.gradle @@ -46,7 +46,7 @@ ext { jwtVersion = "0.13.0" awsSdkVersion = "2.51.2" jschVersion = "2.28.6" - commonsNetVersion = "3.11.1" + commonsNetVersion = "3.13.0" smbjVersion = "0.14.0" tinkVersion = "1.23.0" testcontainersMinioVersion = "1.21.4" From e2700bf3914bac9d855d8f7ed9b77e1ff5398187 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:16:50 +0100 Subject: [PATCH 06/59] build(deps): bump go-task/setup-task from 2.0.0 to 2.1.0 (#7468) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [go-task/setup-task](https://github.com/go-task/setup-task) from 2.0.0 to 2.1.0.
Release notes

Sourced from go-task/setup-task's releases.

v2.1.0

What's Changed

  • Replaced typed-rest-client with @actions/http-client for GitHub API calls to eliminate the Node 24 DEP0169 deprecation warning about url.parse() (#5 by @​vmaerten).
  • Modernized the TypeScript tooling stack (vitest, oxlint, @actions/core@2, @actions/io@2, updated @types/node, @vercel/ncc, prettier, etc.) (#5 by @​vmaerten).
  • Migrated the project to ESM (sources + bundle). Aligns with the new @actions/* ESM-only majors and produces a ~47% smaller dist/index.js (#5 by @​vmaerten).
  • Upgraded @actions/core 2 → 3, @actions/http-client 2 → 4, @actions/io 2 → 3, @actions/tool-cache 2 → 4, typescript 5 → 6, and markdownlint-cli 0.47 → 0.48 (#5 by @​vmaerten).
Changelog

Sourced from go-task/setup-task's changelog.

Changelog

Unreleased

v2.2.0 - 2026-08-09

  • Added a checksum input to verify the SHA256 digest of the downloaded Task archive (#16 by @​illera88).
  • Updated development dependencies: typescript 6 → 7, @vercel/ncc 0.38 → 0.44, markdownlint-cli 0.48 → 0.49, and oxfmt 0.50 → 0.62 (#15).

v2.1.0 - 2026-05-17

  • Replaced typed-rest-client with @actions/http-client for GitHub API calls to eliminate the Node 24 DEP0169 deprecation warning about url.parse().
  • Modernized the TypeScript tooling stack (vitest, oxlint, @actions/core@2, @actions/io@2, updated @types/node, @vercel/ncc, prettier, etc.).
  • Migrated the project to ESM (sources + bundle). Aligns with the new @actions/* ESM-only majors and produces a ~47% smaller dist/index.js.
  • Upgraded @actions/core 2 → 3, @actions/http-client 2 → 4, @actions/io 2 → 3, @actions/tool-cache 2 → 4, typescript 5 → 6, and markdownlint-cli 0.47 → 0.48.

v2.0.0 - 2026-03-18

  • BREAKING: Upgraded to Node 24. Requires a GitHub Actions runner with Node.js 24 support (#10 by @​vmaerten).

v1.1.0 - 2026-03-17

  • Added configurable HTTP retry for API requests (#7 by @​vmaerten).

v1.0.0 - 2025-09-12

Commits

Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/pre_commit.yml | 2 +- .github/workflows/sync_files_v2.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pre_commit.yml b/.github/workflows/pre_commit.yml index 18c4782927..878ce60673 100644 --- a/.github/workflows/pre_commit.yml +++ b/.github/workflows/pre_commit.yml @@ -34,7 +34,7 @@ jobs: cache-suffix: pre-commit - name: Install Task - uses: go-task/setup-task@3be4020d41929789a01026e0e427a4321ce0ad44 # v2.0.0 + uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 - name: Run pre-commit checks run: task pre-commit diff --git a/.github/workflows/sync_files_v2.yml b/.github/workflows/sync_files_v2.yml index 7e6f999618..a21eefffdd 100644 --- a/.github/workflows/sync_files_v2.yml +++ b/.github/workflows/sync_files_v2.yml @@ -66,7 +66,7 @@ jobs: uv sync --project engine --locked --group tools - name: Install Task - uses: go-task/setup-task@3be4020d41929789a01026e0e427a4321ce0ad44 # v2.0.0 + uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 - name: Sync translation TOML files run: | From 42c84a16937666398848332b26e4714066345fb2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:17:01 +0100 Subject: [PATCH 07/59] build(deps): bump base64 from 0.23.0 to 0.23.1 in /frontend/editor/src-tauri (#7460) Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.23.0 to 0.23.1.
Changelog

Sourced from base64's changelog.

0.23.1

  • Make the tests build again on non-SIMD architectures
Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=base64&package-manager=cargo&previous-version=0.23.0&new-version=0.23.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- frontend/editor/src-tauri/Cargo.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/frontend/editor/src-tauri/Cargo.lock b/frontend/editor/src-tauri/Cargo.lock index c7d25757c9..fe8352f5a7 100644 --- a/frontend/editor/src-tauri/Cargo.lock +++ b/frontend/editor/src-tauri/Cargo.lock @@ -276,9 +276,9 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "base64" -version = "0.23.0" +version = "0.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b25655df2c3cdd83c5e5b293b88acd880332b2ddadd7c30ac43144fdc0033da9" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" [[package]] name = "bit-set" @@ -4551,7 +4551,7 @@ checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" name = "stirling-pdf" version = "0.1.0" dependencies = [ - "base64 0.23.0", + "base64 0.23.1", "core-foundation 0.10.1", "core-services", "keyring", From 0deb6fc3d552c25246c1cc4ded5e06ab97413de8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:17:18 +0100 Subject: [PATCH 08/59] build(deps): bump dorny/paths-filter from 4.0.2 to 4.0.3 (#7465) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [dorny/paths-filter](https://github.com/dorny/paths-filter) from 4.0.2 to 4.0.3.
Release notes

Sourced from dorny/paths-filter's releases.

v4.0.3

What's Changed

Security

New Contributors

Full Changelog: https://github.com/dorny/paths-filter/compare/v4...v4.0.3

Changelog

Sourced from dorny/paths-filter's changelog.

Changelog

v4.0.3

v4.0.2

v4.0.1

v4.0.0

v3.0.4

v3.0.3

v3.0.2

v3.0.1

v3.0.0

v2.11.1

v2.11.0

v2.10.2

... (truncated)

Commits
  • ceb8a2b Update CHANGELOG.md for v4.0.3 and v3.0.4 (#327)
  • ef09b88 Document safe handling of file list outputs in workflows (#326)
  • 44adc5b Merge commit from fork
  • 4711b7a feat: add 'some-with-excludes' predicate quantifier (#322)
  • 93c889f fix: escape multi-line filenames in list-files shell and csv output
  • b41dfa9 docs: add contents permission to PR example (#248)
  • 9af6e5a fix: scope base-ignored warning to API path (#319)
  • cae9006 docs: update outputs in readme to account for the 'every' predicate-quantifie...
  • See full diff in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=dorny/paths-filter&package-manager=github_actions&previous-version=4.0.2&new-version=4.0.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/PR-Auto-Deploy-V2.yml | 2 +- .github/workflows/build.yml | 2 +- .github/workflows/frontend-backend-licenses-update.yml | 2 +- .github/workflows/testdriver.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/PR-Auto-Deploy-V2.yml b/.github/workflows/PR-Auto-Deploy-V2.yml index 1ae657fa76..c867929905 100644 --- a/.github/workflows/PR-Auto-Deploy-V2.yml +++ b/.github/workflows/PR-Auto-Deploy-V2.yml @@ -334,7 +334,7 @@ jobs: # under the exact same access rules as the app preview. - name: Detect Storybook changes id: sb-changes - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2 + uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 with: list-files: json filters: | diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 939b701ce9..ee9ebf7bd4 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -55,7 +55,7 @@ jobs: uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Check for file changes - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2 + uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 id: changes with: filters: .github/config/.files.yaml diff --git a/.github/workflows/frontend-backend-licenses-update.yml b/.github/workflows/frontend-backend-licenses-update.yml index f2cca9cee4..69c2a90c63 100644 --- a/.github/workflows/frontend-backend-licenses-update.yml +++ b/.github/workflows/frontend-backend-licenses-update.yml @@ -36,7 +36,7 @@ jobs: uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Check for file changes - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2 + uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 id: changes with: filters: .github/config/.files.yaml diff --git a/.github/workflows/testdriver.yml b/.github/workflows/testdriver.yml index f98d874dfa..fe27e0ba1f 100644 --- a/.github/workflows/testdriver.yml +++ b/.github/workflows/testdriver.yml @@ -159,7 +159,7 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - name: Check for file changes - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2 + uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 id: changes with: filters: ".github/config/.files.yaml" From 88db26b001ebff746442a79a20f696fd15399ac7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:17:29 +0100 Subject: [PATCH 09/59] build(deps): bump the eclipse-temurin group across 3 directories with 1 update (#7453) Bumps the eclipse-temurin group with 1 update in the /docker/backend directory: eclipse-temurin. Bumps the eclipse-temurin group with 1 update in the /docker/base directory: eclipse-temurin. Bumps the eclipse-temurin group with 1 update in the /docker/embedded directory: eclipse-temurin. Updates `eclipse-temurin` from `2f1da10` to `fbcf915` Updates `eclipse-temurin` from `2f1da10` to `fbcf915` Updates `eclipse-temurin` from `2f1da10` to `fbcf915` Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- docker/backend/Dockerfile | 2 +- docker/base/Dockerfile | 2 +- docker/embedded/Dockerfile | 2 +- docker/embedded/Dockerfile.fat | 2 +- docker/embedded/Dockerfile.ultra-lite | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docker/backend/Dockerfile b/docker/backend/Dockerfile index 454fdedcfd..fd26ee2fb7 100644 --- a/docker/backend/Dockerfile +++ b/docker/backend/Dockerfile @@ -45,7 +45,7 @@ RUN JPDFIUM_PLATFORM="$([ "$TARGETARCH" = arm64 ] && echo linux-arm64 || echo li --no-daemon # Stage 2: Extract Spring Boot Layers -FROM eclipse-temurin:25-jre-noble@sha256:2f1da100788559b397bcf48c736169ea5b070bde84e55f203bbee8e83d87a175 AS jar-extract +FROM eclipse-temurin:25-jre-noble@sha256:fbcf915c585659b30eb766ada4d6d7cfc9ec1040bf521e95bf61b10a25af73db AS jar-extract WORKDIR /tmp COPY --from=app-build /app/app/core/build/libs/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination /layers diff --git a/docker/base/Dockerfile b/docker/base/Dockerfile index 6cffbf88f9..95e4e84795 100644 --- a/docker/base/Dockerfile +++ b/docker/base/Dockerfile @@ -364,7 +364,7 @@ RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \ # Final runtime image - the actual base image -FROM eclipse-temurin:25-jre-noble@sha256:2f1da100788559b397bcf48c736169ea5b070bde84e55f203bbee8e83d87a175 AS runtime +FROM eclipse-temurin:25-jre-noble@sha256:fbcf915c585659b30eb766ada4d6d7cfc9ec1040bf521e95bf61b10a25af73db AS runtime SHELL ["/bin/bash", "-o", "pipefail", "-c"] diff --git a/docker/embedded/Dockerfile b/docker/embedded/Dockerfile index de7da3789e..d4ad422476 100644 --- a/docker/embedded/Dockerfile +++ b/docker/embedded/Dockerfile @@ -61,7 +61,7 @@ RUN JPDFIUM_PLATFORM="$([ "$TARGETARCH" = arm64 ] && echo linux-arm64 || echo li --no-daemon # Stage 2: Extract Spring Boot Layers -FROM eclipse-temurin:25-jre-noble@sha256:2f1da100788559b397bcf48c736169ea5b070bde84e55f203bbee8e83d87a175 AS jar-extract +FROM eclipse-temurin:25-jre-noble@sha256:fbcf915c585659b30eb766ada4d6d7cfc9ec1040bf521e95bf61b10a25af73db AS jar-extract WORKDIR /tmp COPY --from=app-build /app/app/core/build/libs/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination /layers diff --git a/docker/embedded/Dockerfile.fat b/docker/embedded/Dockerfile.fat index 300729e560..58eafec851 100644 --- a/docker/embedded/Dockerfile.fat +++ b/docker/embedded/Dockerfile.fat @@ -56,7 +56,7 @@ RUN JPDFIUM_PLATFORM="$([ "$TARGETARCH" = arm64 ] && echo linux-arm64 || echo li --no-daemon # Stage 2: Extract Spring Boot Layers -FROM eclipse-temurin:25-jre-noble@sha256:2f1da100788559b397bcf48c736169ea5b070bde84e55f203bbee8e83d87a175 AS jar-extract +FROM eclipse-temurin:25-jre-noble@sha256:fbcf915c585659b30eb766ada4d6d7cfc9ec1040bf521e95bf61b10a25af73db AS jar-extract WORKDIR /tmp COPY --from=app-build /app/app/core/build/libs/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination /layers diff --git a/docker/embedded/Dockerfile.ultra-lite b/docker/embedded/Dockerfile.ultra-lite index 68a65dea6a..d4a21b7812 100644 --- a/docker/embedded/Dockerfile.ultra-lite +++ b/docker/embedded/Dockerfile.ultra-lite @@ -57,7 +57,7 @@ RUN JPDFIUM_PLATFORM="$([ "$TARGETARCH" = arm64 ] && echo linux-arm64 || echo li # Stage 2: Runtime image # glibc base (not Alpine/musl): JPDFium's PDFium natives are glibc-linked. -FROM eclipse-temurin:25-jre-noble@sha256:2f1da100788559b397bcf48c736169ea5b070bde84e55f203bbee8e83d87a175 +FROM eclipse-temurin:25-jre-noble@sha256:fbcf915c585659b30eb766ada4d6d7cfc9ec1040bf521e95bf61b10a25af73db ENV DEBIAN_FRONTEND=noninteractive \ LANG=C.UTF-8 \ From fb0281fc80ef6baee42f76232fd1bdf99d6f3d9a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:18:23 +0100 Subject: [PATCH 10/59] build(deps-dev): bump codespell from 2.4.2 to 2.4.3 in /engine (#7456) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [codespell](https://github.com/codespell-project/codespell) from 2.4.2 to 2.4.3.
Release notes

Sourced from codespell's releases.

v2.4.3

What's Changed

... (truncated)

Commits
  • 57b2140 Read only [tool.codespell] from TOML config (#3975)
  • 23b8d94 [pre-commit.ci] pre-commit autoupdate
  • 926c4d6 [pre-commit.ci] pre-commit autoupdate
  • 820a301 Merge pull request #3967 from codespell-project/peternewman-patch-1
  • 9fe42c9 Add common misspellings for reseeve->reserve to dictionary
  • 725173c [pre-commit.ci] pre-commit autoupdate
  • 86c13ee [pre-commit.ci] pre-commit autoupdate
  • 7b9f36c Bump actions/checkout from 6 to 7
  • bcc1b80 Add spelling corrections for simpe and variants.
  • 89584cc [pre-commit.ci] pre-commit autoupdate
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=codespell&package-manager=uv&previous-version=2.4.2&new-version=2.4.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- engine/pyproject.toml | 2 +- engine/uv.lock | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/engine/pyproject.toml b/engine/pyproject.toml index 3f2c277fbb..87e7cc99c0 100644 --- a/engine/pyproject.toml +++ b/engine/pyproject.toml @@ -64,7 +64,7 @@ updater-signatures = [ ] # Pinned repository-wide pre-commit tooling. pre-commit = [ - "codespell==2.4.2", + "codespell==2.4.3", "ruff==0.15.5", "tomli-w==1.2.0", ] diff --git a/engine/uv.lock b/engine/uv.lock index 0b25716add..fbfa665bc6 100644 --- a/engine/uv.lock +++ b/engine/uv.lock @@ -414,11 +414,11 @@ wheels = [ [[package]] name = "codespell" -version = "2.4.2" +version = "2.4.3" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/2d/9d/1d0903dff693160f893ca6abcabad545088e7a2ee0a6deae7c24e958be69/codespell-2.4.2.tar.gz", hash = "sha256:3c33be9ae34543807f088aeb4832dfad8cb2dae38da61cac0a7045dd376cfdf3", size = 352058, upload-time = "2026-03-05T18:10:42.936Z" } +sdist = { url = "https://files.pythonhosted.org/packages/80/19/45e941380f69c042b43423513d201e6592346f992394347f5e7174c31407/codespell-2.4.3.tar.gz", hash = "sha256:cbe085e331227b37bb86ef8bddd08dc768c704ee9a07ca869852c093fa2793e2", size = 352773, upload-time = "2026-07-15T11:51:54.159Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl", hash = "sha256:97e0c1060cf46bd1d5db89a936c98db8c2b804e1fdd4b5c645e82a1ec6b1f886", size = 353715, upload-time = "2026-03-05T18:10:41.398Z" }, + { url = "https://files.pythonhosted.org/packages/8b/bf/bdb951d34eb169140b546f44be9ec4525d1acefb9eb5572071f5492b19fc/codespell-2.4.3-py3-none-any.whl", hash = "sha256:af2505b335e8573dbd2d384d1c4ef498f4006f4ba2d6fceca01e55b91f52628a", size = 340736, upload-time = "2026-07-15T11:51:52.925Z" }, ] [[package]] @@ -718,7 +718,7 @@ engine-dev = [ { name = "ruff", specifier = "==0.15.5" }, ] pre-commit = [ - { name = "codespell", specifier = "==2.4.2" }, + { name = "codespell", specifier = "==2.4.3" }, { name = "ruff", specifier = "==0.15.5" }, { name = "tomli-w", specifier = "==1.2.0" }, ] From beea37cb92bcdcd18a53e0311cffb4c91e2f585e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 14:18:51 +0100 Subject: [PATCH 11/59] build(deps): bump step-security/harden-runner from 2.20.0 to 2.20.1 (#7466) Bumps [step-security/harden-runner](https://github.com/step-security/harden-runner) from 2.20.0 to 2.20.1.
Release notes

Sourced from step-security/harden-runner's releases.

v2.20.1

What's Changed

  • AWS CodeBuild-hosted runner support
  • Implicitly allow single-labeled (internal) domains in block-mode

Full Changelog: https://github.com/step-security/harden-runner/compare/v2.20.0...v2.20.1

Commits
  • b09bb98 Merge pull request #680 from step-security/aws-code-build
  • 35cd77b docs: document the Global Block List in the features list
  • bb6dbef chore: rebuild dist with clean dependency install
  • 98f73c5 chore: update eBPF agent to v1.8.14
  • 54193c1 Reapply "feat(runners): detect AWS CodeBuild-hosted runners as third-party pr...
  • d22dd48 Revert "fix(self-hosted): flush agent events at job end when deploy-on-self-h...
  • 0ff0941 fix(self-hosted): flush agent events at job end when deploy-on-self-hosted-vm...
  • a3c333d Revert "feat(runners): detect AWS CodeBuild-hosted runners as third-party pro...
  • bf94c00 feat(runners): detect AWS CodeBuild-hosted runners as third-party provider
  • 514522c fix(self-hosted): resolve runner user when USER env var is unset
  • See full diff in compare view

Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/PR-Auto-Deploy-V2.yml | 6 +++--- .github/workflows/PR-Demo-Comment-with-react.yml | 6 +++--- .github/workflows/PR-Demo-cleanup.yml | 2 +- .github/workflows/_runner-pick.yml | 2 +- .github/workflows/ai-engine.yml | 2 +- .github/workflows/ai_pr_title_review.yml | 2 +- .github/workflows/aur-publish.yml | 4 ++-- .github/workflows/auto-labelerV2.yml | 2 +- .github/workflows/backend-build.yml | 2 +- .github/workflows/build-enterprise.yml | 4 ++-- .github/workflows/build.yml | 6 +++--- .github/workflows/check-generated-models.yml | 2 +- .github/workflows/check-licence.yml | 2 +- .github/workflows/check-openapi.yml | 2 +- .github/workflows/check_toml.yml | 2 +- .github/workflows/coverage-aggregate.yml | 2 +- .github/workflows/db-migration-test.yml | 2 +- .github/workflows/dependency-review.yml | 2 +- .github/workflows/deploy-on-v2-commit.yml | 2 +- .github/workflows/docker-compose-tests.yml | 2 +- .github/workflows/e2e-live.yml | 2 +- .github/workflows/e2e-stubbed.yml | 2 +- .github/workflows/frontend-a11y.yml | 2 +- .github/workflows/frontend-backend-licenses-update.yml | 6 +++--- .github/workflows/frontend-validation.yml | 2 +- .github/workflows/manage-label.yml | 2 +- .github/workflows/multiOSReleases.yml | 8 ++++---- .github/workflows/nightly.yml | 6 +++--- .github/workflows/package-managers.yml | 4 ++-- .github/workflows/pr-conflict-labeler.yml | 2 +- .github/workflows/pre_commit.yml | 2 +- .github/workflows/push-docker-base.yml | 2 +- .github/workflows/push-docker.yml | 2 +- .github/workflows/rollback-latest.yml | 2 +- .github/workflows/scorecards.yml | 2 +- .github/workflows/stale.yml | 2 +- .github/workflows/swagger.yml | 2 +- .github/workflows/sync-portal-docs.yml | 2 +- .github/workflows/sync_files_v2.yml | 2 +- .github/workflows/tauri-build.yml | 8 ++++---- .github/workflows/test-build-docker.yml | 4 ++-- .github/workflows/testdriver.yml | 8 ++++---- 42 files changed, 65 insertions(+), 65 deletions(-) diff --git a/.github/workflows/PR-Auto-Deploy-V2.yml b/.github/workflows/PR-Auto-Deploy-V2.yml index c867929905..0f07aabbe5 100644 --- a/.github/workflows/PR-Auto-Deploy-V2.yml +++ b/.github/workflows/PR-Auto-Deploy-V2.yml @@ -39,7 +39,7 @@ jobs: pr_ref: ${{ steps.resolve.outputs.ref }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -121,7 +121,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -472,7 +472,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/PR-Demo-Comment-with-react.yml b/.github/workflows/PR-Demo-Comment-with-react.yml index 8e0e66032e..fe3f28a637 100644 --- a/.github/workflows/PR-Demo-Comment-with-react.yml +++ b/.github/workflows/PR-Demo-Comment-with-react.yml @@ -70,7 +70,7 @@ jobs: enable_prototypes: ${{ steps.check-prototypes-flag.outputs.enable_prototypes }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -177,7 +177,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -485,7 +485,7 @@ jobs: issues: write # add/remove labels, delete the command comment steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/PR-Demo-cleanup.yml b/.github/workflows/PR-Demo-cleanup.yml index 7b4ee8b3a3..e0032955e3 100644 --- a/.github/workflows/PR-Demo-cleanup.yml +++ b/.github/workflows/PR-Demo-cleanup.yml @@ -23,7 +23,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/_runner-pick.yml b/.github/workflows/_runner-pick.yml index 0b3f76a8cd..d65831c5df 100644 --- a/.github/workflows/_runner-pick.yml +++ b/.github/workflows/_runner-pick.yml @@ -38,7 +38,7 @@ jobs: is_fork: ${{ steps.decide.outputs.is_fork }} steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/ai-engine.yml b/.github/workflows/ai-engine.yml index 056dc35ca3..015934030f 100644 --- a/.github/workflows/ai-engine.yml +++ b/.github/workflows/ai-engine.yml @@ -20,7 +20,7 @@ jobs: pull-requests: write steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/ai_pr_title_review.yml b/.github/workflows/ai_pr_title_review.yml index 9922177b4b..b9b391af0e 100644 --- a/.github/workflows/ai_pr_title_review.yml +++ b/.github/workflows/ai_pr_title_review.yml @@ -21,7 +21,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/aur-publish.yml b/.github/workflows/aur-publish.yml index f1ca2be8ca..af0dc85aa6 100644 --- a/.github/workflows/aur-publish.yml +++ b/.github/workflows/aur-publish.yml @@ -26,7 +26,7 @@ jobs: jar_sha256: ${{ steps.hashes.outputs.jar_sha256 }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -71,7 +71,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/auto-labelerV2.yml b/.github/workflows/auto-labelerV2.yml index bcbd0fcba5..8e6d974f3c 100644 --- a/.github/workflows/auto-labelerV2.yml +++ b/.github/workflows/auto-labelerV2.yml @@ -18,7 +18,7 @@ jobs: issues: write # labels are applied through the issues API steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/backend-build.yml b/.github/workflows/backend-build.yml index 3cba3c8d39..9561833f07 100644 --- a/.github/workflows/backend-build.yml +++ b/.github/workflows/backend-build.yml @@ -28,7 +28,7 @@ jobs: flavor: [core, proprietary, saas] steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/build-enterprise.yml b/.github/workflows/build-enterprise.yml index f1301b6fd2..08849c683b 100644 --- a/.github/workflows/build-enterprise.yml +++ b/.github/workflows/build-enterprise.yml @@ -50,7 +50,7 @@ jobs: SYSTEM_ENABLEANALYTICS: "false" steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository @@ -324,7 +324,7 @@ jobs: MN_COMPOSE: docker-compose-multinode.yml steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ee9ebf7bd4..eee07d599a 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -48,7 +48,7 @@ jobs: proprietary: ${{ steps.changes.outputs.proprietary }} steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository @@ -67,7 +67,7 @@ jobs: timeout-minutes: 15 steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository @@ -306,7 +306,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/check-generated-models.yml b/.github/workflows/check-generated-models.yml index 833b9c8e47..39c6467889 100644 --- a/.github/workflows/check-generated-models.yml +++ b/.github/workflows/check-generated-models.yml @@ -23,7 +23,7 @@ jobs: pull-requests: write steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/check-licence.yml b/.github/workflows/check-licence.yml index b1692c904d..d45a68e860 100644 --- a/.github/workflows/check-licence.yml +++ b/.github/workflows/check-licence.yml @@ -13,7 +13,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/check-openapi.yml b/.github/workflows/check-openapi.yml index c26bf60aba..751a34a33e 100644 --- a/.github/workflows/check-openapi.yml +++ b/.github/workflows/check-openapi.yml @@ -14,7 +14,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/check_toml.yml b/.github/workflows/check_toml.yml index eff416c379..ffb3e01226 100644 --- a/.github/workflows/check_toml.yml +++ b/.github/workflows/check_toml.yml @@ -28,7 +28,7 @@ jobs: pull-requests: write # Allow writing to pull requests steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/coverage-aggregate.yml b/.github/workflows/coverage-aggregate.yml index 507525f254..a97b579f15 100644 --- a/.github/workflows/coverage-aggregate.yml +++ b/.github/workflows/coverage-aggregate.yml @@ -34,7 +34,7 @@ jobs: timeout-minutes: 15 steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/db-migration-test.yml b/.github/workflows/db-migration-test.yml index 377d2daf98..4181390285 100644 --- a/.github/workflows/db-migration-test.yml +++ b/.github/workflows/db-migration-test.yml @@ -17,7 +17,7 @@ jobs: timeout-minutes: 30 steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/dependency-review.yml b/.github/workflows/dependency-review.yml index 07f1f8ce1a..b27596ddfb 100644 --- a/.github/workflows/dependency-review.yml +++ b/.github/workflows/dependency-review.yml @@ -14,7 +14,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/deploy-on-v2-commit.yml b/.github/workflows/deploy-on-v2-commit.yml index 21d12044f5..01114c64b2 100644 --- a/.github/workflows/deploy-on-v2-commit.yml +++ b/.github/workflows/deploy-on-v2-commit.yml @@ -22,7 +22,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/docker-compose-tests.yml b/.github/workflows/docker-compose-tests.yml index 05b66ba34b..ddb35b4e1a 100644 --- a/.github/workflows/docker-compose-tests.yml +++ b/.github/workflows/docker-compose-tests.yml @@ -25,7 +25,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/e2e-live.yml b/.github/workflows/e2e-live.yml index 3c5c12d6e7..4aa2a78c89 100644 --- a/.github/workflows/e2e-live.yml +++ b/.github/workflows/e2e-live.yml @@ -15,7 +15,7 @@ jobs: timeout-minutes: 30 steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/e2e-stubbed.yml b/.github/workflows/e2e-stubbed.yml index df3c5fa82a..d02005936c 100644 --- a/.github/workflows/e2e-stubbed.yml +++ b/.github/workflows/e2e-stubbed.yml @@ -27,7 +27,7 @@ jobs: project: stubbed-webkit steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/frontend-a11y.yml b/.github/workflows/frontend-a11y.yml index 95050a5db3..8f97120d96 100644 --- a/.github/workflows/frontend-a11y.yml +++ b/.github/workflows/frontend-a11y.yml @@ -21,7 +21,7 @@ jobs: timeout-minutes: 25 steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/frontend-backend-licenses-update.yml b/.github/workflows/frontend-backend-licenses-update.yml index 69c2a90c63..1155f01e39 100644 --- a/.github/workflows/frontend-backend-licenses-update.yml +++ b/.github/workflows/frontend-backend-licenses-update.yml @@ -28,7 +28,7 @@ jobs: licenses-backend: ${{ steps.changes.outputs.licenses-backend }} steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -52,7 +52,7 @@ jobs: repository-projects: write # Required for enabling automerge steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -326,7 +326,7 @@ jobs: repository-projects: write # Required for enabling automerge steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/frontend-validation.yml b/.github/workflows/frontend-validation.yml index 7e2602c311..133d940b28 100644 --- a/.github/workflows/frontend-validation.yml +++ b/.github/workflows/frontend-validation.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit - name: Checkout repository diff --git a/.github/workflows/manage-label.yml b/.github/workflows/manage-label.yml index 571479b030..1a8bdd112d 100644 --- a/.github/workflows/manage-label.yml +++ b/.github/workflows/manage-label.yml @@ -15,7 +15,7 @@ jobs: issues: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/multiOSReleases.yml b/.github/workflows/multiOSReleases.yml index c712467e7b..3bb014a82d 100644 --- a/.github/workflows/multiOSReleases.yml +++ b/.github/workflows/multiOSReleases.yml @@ -45,7 +45,7 @@ jobs: version: ${{ steps.versionNumber.outputs.versionNumber }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -137,7 +137,7 @@ jobs: file_suffix: "-server" steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -208,7 +208,7 @@ jobs: RELEASE_GPG_PRIVATE_KEY: ${{ secrets.RELEASE_GPG_PRIVATE_KEY }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit allowed-endpoints: > @@ -740,7 +740,7 @@ jobs: contents: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 48b3299827..70e99d6074 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -23,7 +23,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -74,7 +74,7 @@ jobs: timeout-minutes: 60 steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -137,7 +137,7 @@ jobs: contents: read steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/package-managers.yml b/.github/workflows/package-managers.yml index 751b3c1877..777c8d6ed4 100644 --- a/.github/workflows/package-managers.yml +++ b/.github/workflows/package-managers.yml @@ -28,7 +28,7 @@ jobs: jar_sha256: ${{ steps.hashes.outputs.jar_sha256 }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -80,7 +80,7 @@ jobs: contents: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/pr-conflict-labeler.yml b/.github/workflows/pr-conflict-labeler.yml index 362421d0f8..c642b6cf3d 100644 --- a/.github/workflows/pr-conflict-labeler.yml +++ b/.github/workflows/pr-conflict-labeler.yml @@ -32,7 +32,7 @@ jobs: pull-requests: write # pulls.get/list plus add/remove the label on PRs steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/pre_commit.yml b/.github/workflows/pre_commit.yml index 878ce60673..db67ccefe6 100644 --- a/.github/workflows/pre_commit.yml +++ b/.github/workflows/pre_commit.yml @@ -14,7 +14,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/push-docker-base.yml b/.github/workflows/push-docker-base.yml index f7dd1bcab4..658583ea08 100644 --- a/.github/workflows/push-docker-base.yml +++ b/.github/workflows/push-docker-base.yml @@ -45,7 +45,7 @@ jobs: echo "version=${VERSION}" >> $GITHUB_OUTPUT - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/push-docker.yml b/.github/workflows/push-docker.yml index c9caf1b2f2..b88d69c3c2 100644 --- a/.github/workflows/push-docker.yml +++ b/.github/workflows/push-docker.yml @@ -53,7 +53,7 @@ jobs: RUN_UNOSERVER: ${{ github.event_name != 'workflow_dispatch' || inputs.build_unoserver }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/rollback-latest.yml b/.github/workflows/rollback-latest.yml index 442a6f499e..3e49f727c6 100644 --- a/.github/workflows/rollback-latest.yml +++ b/.github/workflows/rollback-latest.yml @@ -19,7 +19,7 @@ jobs: packages: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/scorecards.yml b/.github/workflows/scorecards.yml index 0e9b27c829..f0129277de 100644 --- a/.github/workflows/scorecards.yml +++ b/.github/workflows/scorecards.yml @@ -35,7 +35,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index ab9078831a..af255de3cf 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -17,7 +17,7 @@ jobs: pull-requests: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/swagger.yml b/.github/workflows/swagger.yml index 733ce9e7ed..38c1985a83 100644 --- a/.github/workflows/swagger.yml +++ b/.github/workflows/swagger.yml @@ -27,7 +27,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/sync-portal-docs.yml b/.github/workflows/sync-portal-docs.yml index 5ff4f465c7..636cd5f2e1 100644 --- a/.github/workflows/sync-portal-docs.yml +++ b/.github/workflows/sync-portal-docs.yml @@ -33,7 +33,7 @@ jobs: pull-requests: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/sync_files_v2.yml b/.github/workflows/sync_files_v2.yml index a21eefffdd..124c992c71 100644 --- a/.github/workflows/sync_files_v2.yml +++ b/.github/workflows/sync_files_v2.yml @@ -37,7 +37,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/tauri-build.yml b/.github/workflows/tauri-build.yml index 5c04f6009f..c25430262e 100644 --- a/.github/workflows/tauri-build.yml +++ b/.github/workflows/tauri-build.yml @@ -63,7 +63,7 @@ jobs: matrix: ${{ steps.set-matrix.outputs.matrix }} steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -120,7 +120,7 @@ jobs: SIGN_BUNDLE: ${{ inputs.sign && (matrix.platform == 'macos-15' && secrets.APPLE_CERTIFICATE != '' || github.ref == 'refs/heads/main') }} steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -703,7 +703,7 @@ jobs: pull-requests: write steps: - name: Harden the runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -795,7 +795,7 @@ jobs: if: always() steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/test-build-docker.yml b/.github/workflows/test-build-docker.yml index 4ce06e6247..660cd2458b 100644 --- a/.github/workflows/test-build-docker.yml +++ b/.github/workflows/test-build-docker.yml @@ -53,7 +53,7 @@ jobs: cache-scope: stirling-pdf-fat steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -191,7 +191,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit diff --git a/.github/workflows/testdriver.yml b/.github/workflows/testdriver.yml index fe27e0ba1f..751eaf44f7 100644 --- a/.github/workflows/testdriver.yml +++ b/.github/workflows/testdriver.yml @@ -29,7 +29,7 @@ jobs: packages: write steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -152,7 +152,7 @@ jobs: frontend: ${{ steps.changes.outputs.frontend }} steps: - name: Harden the runner (Audit all outbound calls) - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -171,7 +171,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit @@ -208,7 +208,7 @@ jobs: steps: - name: Harden Runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1 with: egress-policy: audit From 9c10204cb06780d34c92f394775c5d586d8d4093 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 17:15:37 +0100 Subject: [PATCH 12/59] build(deps): bump Swatinem/rust-cache from 2.9.1 to 2.9.2 (#7531) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.9.1 to 2.9.2.
Release notes

Sourced from Swatinem/rust-cache's releases.

v2.9.2

What's Changed

New Contributors

Full Changelog: https://github.com/Swatinem/rust-cache/compare/v2.9.1...v2.9.2

Changelog

Sourced from Swatinem/rust-cache's changelog.

Changelog

2.9.2

  • Fix credentials.toml cleanup
  • Improvements to cleanup, preserving more valid targets
  • Improvements to cargo install handling
  • Correctly sort/dedupe Rust versions

2.9.1

  • Fix regression in hash calculation

2.9.0

  • Update to node24
  • Support running from within a nix shell
  • Consider all installed toolchains for cache key
  • Use case-insensitive comparison to determine exact cache hit

2.8.2

  • Don't overwrite env for cargo-metadata call

2.8.1

  • Set empty CARGO_ENCODED_RUSTFLAGS when retrieving metadata
  • Various dependency updates

2.8.0

  • Add support for warpbuild cache provider
  • Add new cache-workspace-crates feature

2.7.8

  • Include CPU arch in the cache key

2.7.7

  • Also cache cargo install metadata

2.7.6

  • Allow opting out of caching $CARGO_HOME/bin
  • Add runner OS in cache key
  • Adds an option to do lookup-only of the cache

2.7.5

... (truncated)

Commits
  • 6323deb 2.9.2
  • b16e8d7 bump rollup and rebuild
  • 3bf42ac invert target/profile check in cleanup
  • 6e5b278 correctly sort and dedupe Rust versions
  • 5adc05f Bump the actions group across 1 directory with 3 updates (#368)
  • 66b1e95 fix: support Cargo V2 build dir layout (#371)
  • 72d126e Merge pull request #367 from Swatinem/dependabot/npm_and_yarn/dev-patch-2b495...
  • 48968d2 Bump the dev-patch group with 2 updates
  • 9f151ac update dependencies, rebuild
  • 0e24e5d Bump the actions group across 1 directory with 6 updates (#364)
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Swatinem/rust-cache&package-manager=github_actions&previous-version=2.9.1&new-version=2.9.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/tauri-build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/tauri-build.yml b/.github/workflows/tauri-build.yml index c25430262e..ab6d8aca71 100644 --- a/.github/workflows/tauri-build.yml +++ b/.github/workflows/tauri-build.yml @@ -150,7 +150,7 @@ jobs: # only recompiles the app crate. Written on main; PRs and the merge queue # restore from it. - name: Cache Rust build - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1 + uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 with: workspaces: frontend/editor/src-tauri # Stable key shared across workflows so the nightly warmer. From 9a90797041e7d49201f60b529cd7219db6029c98 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sun, 16 Aug 2026 17:15:47 +0100 Subject: [PATCH 13/59] build(deps): bump awsSdkVersion from 2.51.2 to 2.51.3 (#7529) Bumps `awsSdkVersion` from 2.51.2 to 2.51.3. Updates `software.amazon.awssdk:s3` from 2.51.2 to 2.51.3 Updates `software.amazon.awssdk:url-connection-client` from 2.51.2 to 2.51.3 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.gradle b/build.gradle index b7fcb794cc..cadf4bafa6 100644 --- a/build.gradle +++ b/build.gradle @@ -44,7 +44,7 @@ ext { batikVersion = "1.19" jpdfiumVersion = "1.0.4" jwtVersion = "0.13.0" - awsSdkVersion = "2.51.2" + awsSdkVersion = "2.51.3" jschVersion = "2.28.6" commonsNetVersion = "3.13.0" smbjVersion = "0.14.0" From 2c89f5fe5210df7c93c6feeb74ba19b23afdb635 Mon Sep 17 00:00:00 2001 From: Ludy Date: Sun, 16 Aug 2026 18:17:16 +0200 Subject: [PATCH 14/59] fix(ci): locale path and docs (#7525) # Description of Changes Update the TOML locale check to match the current frontend path and align comments/docs with the relocated editor and proprietary source layout. This keeps CI filtering and internal references accurate after the frontend reorganization. --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details. --- .github/config/.files.yaml | 36 +++++++++---------- .github/workflows/check_toml.yml | 2 +- .../src/core/services/httpErrorHandler.ts | 2 +- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/.github/config/.files.yaml b/.github/config/.files.yaml index e6e4f08230..70a964b020 100644 --- a/.github/config/.files.yaml +++ b/.github/config/.files.yaml @@ -1,7 +1,7 @@ -# CI routing infra. Editing the top-level router (build.yml) or this filter -# config re-runs every area's jobs, so every job-gating filter below includes -# *ci. That makes a change to how jobs are dispatched actually exercise those -# jobs (self-testing), instead of a router edit only matching the project filter. +# CI routing infrastructure. Changes to the top-level router (build.yml) or +# this filter configuration rerun every area's jobs. Every job-gating filter +# therefore includes *ci, so routing changes exercise the jobs they affect +# instead of matching only the project filter. ci: &ci - .github/workflows/build.yml - .github/config/.files.yaml @@ -24,9 +24,9 @@ openapi: &openapi docker-base: &docker-base - docker/base/Dockerfile -# Dockerfiles only (base + embedded + unoserver). Gates the slow multi-arch -# (arm64) leg of the PR docker test build: arm64 is only rebuilt when a -# Dockerfile itself changes, not on every code PR. +# Dockerfiles only (base, embedded, and unoserver). The slow multi-architecture +# (arm64) leg of the PR Docker test build runs only when a Dockerfile changes, +# rather than for every code PR. dockerfiles: &dockerfiles - docker/**/Dockerfile* @@ -88,8 +88,8 @@ frontend: &frontend - .github/workflows/e2e-stubbed.yml - .github/workflows/e2e-live.yml -# Files that affect the Tauri desktop bundle. Gate the multi-OS Tauri build -# job on changes to any of these. +# Files that affect the Tauri desktop bundle. Changes to any of these files +# trigger the multi-OS Tauri build job. tauri: &tauri - *ci - frontend/editor/src-tauri/** @@ -102,9 +102,9 @@ tauri: &tauri - Taskfile.yml - .taskfiles/desktop.yml -# Files that affect the AI engine (Python tool models, fixers, tests). Gate -# the engine validation job on changes to engine sources or to the Java -# tool surfaces it generates models from. +# Files that affect the AI engine, including its Python tool models, fixers, +# and tests. The engine validation job also runs when the Java tool surfaces +# used to generate those models change. engine: &engine - *ci - engine/** @@ -114,10 +114,10 @@ engine: &engine - .taskfiles/engine.yml # Files that can make the committed generated API models (frontend tool API -# types + engine tool models) go stale: the Java tool surfaces they derive from, -# the generators, the generated files themselves (to catch a hand-edit), and the -# tasks that drive generation. Deliberately excludes the broad frontend/docker/ -# testing globs, so a CSS-only PR does not boot the backend to rebuild the spec. +# types and engine tool models) stale: their Java sources, generators, +# generated outputs (to catch hand edits), and generation tasks. Broad +# frontend, Docker, and testing globs are intentionally excluded, so a CSS-only +# PR does not start the backend to rebuild the specification. generated-models: &generated-models - *ci - *openapi @@ -141,8 +141,8 @@ licenses-backend: &licenses-backend - ".github/workflows/frontend-backend-licenses-update.yml" - *build -# Files that can affect premium / enterprise behaviour. Gate the enterprise -# Playwright job on changes to any of these on PRs. +# Files that can affect premium or enterprise behaviour. Changes to any of +# these files trigger the enterprise Playwright job for pull requests. proprietary: &proprietary - *ci - app/proprietary/** diff --git a/.github/workflows/check_toml.yml b/.github/workflows/check_toml.yml index ffb3e01226..1681546e71 100644 --- a/.github/workflows/check_toml.yml +++ b/.github/workflows/check_toml.yml @@ -125,7 +125,7 @@ jobs: const changedFiles = files .filter(file => file.status !== "removed" && - /^frontend\/public\/locales\/[a-zA-Z-]+\/translation\.toml$/.test(file.filename) + /^frontend\/editor\/public\/locales\/[a-zA-Z-]+\/translation\.toml$/.test(file.filename) ) .map(file => file.filename); diff --git a/frontend/editor/src/core/services/httpErrorHandler.ts b/frontend/editor/src/core/services/httpErrorHandler.ts index ed7108b6a6..55b0a425f9 100644 --- a/frontend/editor/src/core/services/httpErrorHandler.ts +++ b/frontend/editor/src/core/services/httpErrorHandler.ts @@ -1,4 +1,4 @@ -// frontend/src/services/httpErrorHandler.ts +// frontend/editor/src/core/services/httpErrorHandler.ts import { alert } from "@app/components/toast"; import { broadcastErroredFiles, From 08b08aa8a1a82d7f7ff2dc14f30559b96b954b80 Mon Sep 17 00:00:00 2001 From: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com> Date: Sun, 16 Aug 2026 22:27:20 +0000 Subject: [PATCH 15/59] Let everyone read the failures they caused (Review Flow PR 3) (#7477) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review Flow PR 3. Stacked on #7296. A recorded failure becomes readable by the person who caused it. ## What changes Before this, reading or triaging a failure required leader permissions: `FileRunEventController.requireFailureReviewAllowed()` returned 403 to anyone who could not edit policies. #7296 lets any user report a failure, so they could file into a queue they could never read. That gate is removed from the endpoints and the decision moves into `FileRunEventService`: | Caller | Reads and closes | |---|---| | Team leader or admin | the whole team's failures (unchanged) | | Anyone else | only failures where `actor` is them | | Team unresolvable | nothing | | Name unresolvable | nothing | `GET /kinds` is also opened. It returns static enum metadata, and a member needs it to render failures they can already see. ## Additions - An `actor` predicate on both list queries in `FileRunEventRepository`, threaded through `FileRunEventStore.list`. - `ReadScope` (permitted, teamId, actor) replacing `TeamScope`, with `wholeTeam` / `mine` / `denied` factories. - An actor filter on `dispatch`, so acting on another person's row answers **404, not 403** — the same response as an id that does not exist. ## Fixes - **`report()` filed rows under the wrong team.** It took the team from the read scope, which returns null for a caller who cannot be named, so such a report landed unteamed in the bucket every team shares. It now uses a dedicated `currentTeamId()`. - **`forgetFiles` narrows to the caller even for a leader.** File ids are minted by each client, so scoping on team alone would let one caller close a colleague's incidents by naming ids. - The controller no longer injects `PolicyManagementAuthority` or `ApplicationProperties`; with the gate gone it decides nothing. ## Team isolation Unchanged and covered by database-backed tests rather than mocks. `FileRunEventStoreDbTest` asserts that a caller with a team sees only their own team's rows and never the unteamed ones, and that the actor predicate narrows within a team without ever widening across one. Delete either clause from the JPQL and one of those tests fails. No endpoint accepts a team parameter; the team always comes from the authenticated principal. **Attribution is fixed here too, because this PR depends on it.** A failure's actor was read from the MDC audit principal, which carries the BILLING identity — for a stored policy, always its owner. Since reads are now narrowed to the rows you are the actor on, a wrong actor means the member who caused a failure and holds the document reads nothing, while the policy owner is handed incidents from runs they never triggered. The triggering user is now carried on the run, separate from the billing principal and the output owner, and is null for a trigger-fired sweep so an unattended failure stays ownerless. `PolicyFailureAttributionTest` runs the real engine, recorder, store and service together. The two sides used to assert independently — the engine's test matched the actor with `any()`, which is how this went unnoticed. ## How to test Needs a proprietary or SaaS build with login enabled and two accounts in the same team, one a leader and one not. `task dev:all` gives you the stack. 1. **As the member**, fail a tool: open a PDF and run **Remove Password** with a wrong password. 2. **Still as the member**, go to `/processor/documents` → **Failures**. Before this PR you got nothing here. Now you see your own row, and only yours. 3. **As the leader**, open the same view. You see the whole team's rows, including the member's. 4. **Member cannot reach a colleague's row.** As the leader, copy a row's id from **Show raw JSON**. As the member, `POST /api/v1/file-run-events/{thatId}/actions/DISMISS`. It answers **404**, and the row is untouched — it must not answer 403, which would confirm the row exists. 5. **Member can close their own.** Dismiss your own row as the member. It leaves the default view. 6. **Deleting a file only closes your own rows.** As the leader, delete a file in your editor. The member's incidents are untouched even if the leader's client happened to name the same ids. ## Migration None. `actor` is an existing column; this only adds predicates to existing queries. --- .../failure/FileRunEventController.java | 57 ++-- .../failure/FileRunEventRepository.java | 11 +- .../failure/FileRunEventService.java | 91 ++++-- .../failure/FileRunEventStore.java | 9 +- .../policy/engine/PolicyEngine.java | 30 +- .../proprietary/policy/model/PolicyRun.java | 20 +- .../failure/FileRunEventControllerTest.java | 112 +++++--- .../FileRunEventHttpIntegrationTest.java | 7 +- .../failure/FileRunEventServiceTest.java | 116 ++++++-- .../failure/FileRunEventStoreDbTest.java | 38 ++- .../failure/FileRunEventStoreTest.java | 12 +- .../InMemoryFileRunEventRepository.java | 23 +- .../failure/PolicyFailureAttributionTest.java | 264 ++++++++++++++++++ .../failure/PolicyFailureRecorderTest.java | 30 +- .../controller/PolicyControllerTest.java | 10 +- .../policy/engine/PolicyEngineTest.java | 152 +++++++++- .../policy/engine/PolicyRunRegistryTest.java | 1 + 17 files changed, 800 insertions(+), 183 deletions(-) create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureAttributionTest.java diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventController.java b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventController.java index a14f243678..aac1c3364b 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventController.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventController.java @@ -22,15 +22,13 @@ import io.swagger.v3.oas.annotations.tags.Tag; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; -import stirling.software.common.model.ApplicationProperties; -import stirling.software.proprietary.policy.config.PolicyManagementAuthority; - /** - * Read and triage recorded failures for the caller's team. Note the absence of a team parameter: - * the team comes from the authenticated principal, never the request. + * Read and triage recorded failures. Note the absence of a team parameter: the team comes from the + * authenticated principal, never the request. * - *

Reviewing failures is a leader-level capability, gated the same way policy editing is: see - * {@link #requireFailureReviewAllowed()}. + *

Every endpoint is open to any authenticated user and scoped in the service instead: a leader + * reads and closes the whole team's failures, everyone else their own. Nothing here decides who may + * do what, so the two cannot drift apart. */ @Slf4j @RestController @@ -46,21 +44,21 @@ public class FileRunEventController { private static final int DEFAULT_LIMIT = 50; private final FileRunEventService service; - private final PolicyManagementAuthority policyManagementAuthority; - private final ApplicationProperties applicationProperties; @GetMapping @Operation( summary = "List recorded failures", description = - "Failures recorded for the caller's team, newest first. Each row carries its" - + " available actions already resolved.") + "Failures the caller may see, newest first: their team's for a leader, their own" + + " for everyone else. Each row carries its available actions already" + + " resolved.") public FileRunEventsResponse list( // Spring's converter 400s on a value outside the enum, so no hand-rolled parse. @RequestParam(required = false) FileRunEventStatus status, @RequestParam(required = false) String kindId, @RequestParam(required = false) Integer limit) { - requireFailureReviewAllowed(); + // No role gate: the service scopes the read instead, so a member gets their own failures + // and a leader the team's. int cappedLimit = Math.min(limit == null ? DEFAULT_LIMIT : Math.max(1, limit), MAX_LIMIT); List events = @@ -82,7 +80,8 @@ public class FileRunEventController { @PathVariable String eventId, @PathVariable String actionId, @RequestBody(required = false) ActionRequest request) { - requireFailureReviewAllowed(); + // No role gate: the service decides, which lets someone close their own failure while + // still keeping a colleague's out of reach. Map inputs = request == null ? Map.of() : request.safeInputs(); try { FileRunEvent updated = service.dispatch(eventId, actionId, inputs); @@ -97,9 +96,9 @@ public class FileRunEventController { summary = "Report a failure hit in the editor", description = "For failures the server never sees, because the editor calls tools directly." - + " Open to any authenticated user, unlike the read and triage endpoints:" - + " whoever's work failed can say so, and a leader reviews it. Rejected" - + " with 400 if it names more files than one report may carry.") + + " Open to any authenticated user: whoever's work failed can say so, and" + + " reads it back scoped to themselves. Rejected with 400 if it names" + + " more files than one report may carry.") public ResponseEntity report(@RequestBody EditorFailureReport report) { if (report == null || !report.hasOperation()) { throw new ResponseStatusException( @@ -128,8 +127,8 @@ public class FileRunEventController { summary = "Close the incidents about files deleted from the editor", description = "Deleting the document leaves nothing to act on, so its incidents drop out of" - + " the queue while the rows stay for audit. Open to any authenticated" - + " user, and applies only to their own editor rows.") + + " the queue while the rows stay for audit. Applies only to the" + + " caller's own editor rows, however senior they are.") public ResponseEntity filesRemoved(@RequestBody(required = false) RemovedFiles request) { service.forgetFiles(request == null ? List.of() : request.safeFileIds()); // No body: the editor is telling the server, not asking it anything. @@ -143,29 +142,11 @@ public class FileRunEventController { "The failure registry. Lets a client describe kinds it was not built with, and" + " doubles as the probe for whether failure tracking exists at all.") public List kinds() { - requireFailureReviewAllowed(); + // The registry is copy and metadata, not anyone's data, and a member needs it to render the + // failures they can already see. return Arrays.stream(FailureKind.values()).map(FailureKindView::of).toList(); } - /** - * Triage is for a team leader (SaaS) or admin (self-hosted), mirroring {@code - * PolicyController.requirePolicyEditingAllowed()} rather than inventing a second notion of who - * manages a team's automation: a member can trigger runs, a leader reviews them. - * - *

Login disabled means a single-user deployment with no roles to tell apart, the same - * carve-out the policy endpoints make. Team scoping is separate, and lives in the service. - */ - private void requireFailureReviewAllowed() { - if (!applicationProperties.getSecurity().isEnableLogin()) { - return; - } - if (!policyManagementAuthority.canEditPolicies()) { - throw new ResponseStatusException( - HttpStatus.FORBIDDEN, - "Recorded failures may only be reviewed by a team leader"); - } - } - /** * A closed row is a conflict rather than a bad request: the request was well-formed and would * have been valid a moment earlier. diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventRepository.java b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventRepository.java index 1bee2398f7..1930b36e6a 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventRepository.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventRepository.java @@ -20,26 +20,33 @@ public interface FileRunEventRepository extends JpaRepository{@code actor} narrows to one person's own failures. Null means the whole team, which only + * a leader ever asks for: see {@code FileRunEventService#readScope}. */ @Query( "select e from FileRunEventEntity e where ((:teamId is null and e.teamId is null) or" + " e.teamId = :teamId) and e.status in :statuses" - + " and (:kindId is null or e.kindId = :kindId) order by e.lastSeenAt desc") + + " and (:kindId is null or e.kindId = :kindId)" + + " and (:actor is null or e.actor = :actor) order by e.lastSeenAt desc") List findByTeamAndStatusIn( @Param("teamId") Long teamId, @Param("statuses") List statuses, @Param("kindId") String kindId, + @Param("actor") String actor, Pageable pageable); /** As {@link #findByTeamAndStatusIn} but for exactly one status, for the surface's filters. */ @Query( "select e from FileRunEventEntity e where ((:teamId is null and e.teamId is null) or" + " e.teamId = :teamId) and e.status = :status" - + " and (:kindId is null or e.kindId = :kindId) order by e.lastSeenAt desc") + + " and (:kindId is null or e.kindId = :kindId)" + + " and (:actor is null or e.actor = :actor) order by e.lastSeenAt desc") List findByTeamAndStatus( @Param("teamId") Long teamId, @Param("status") FileRunEventStatus status, @Param("kindId") String kindId, + @Param("actor") String actor, Pageable pageable); /** diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventService.java b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventService.java index 57eb4698e4..fa858b6e22 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventService.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventService.java @@ -13,12 +13,14 @@ import stirling.software.common.service.UserServiceInterface; import stirling.software.proprietary.policy.config.PolicyManagementAuthority; /** - * Reads and acts on incidents for the calling user's team. + * Reads and acts on the incidents the calling user is allowed to see, which is where that decision + * is made: a leader reads and closes the whole team's failures, everyone else their own. Keeping it + * here rather than on the endpoints means the read and the triage cannot drift apart. * *

Team scoping mirrors {@code PolicyAccessGuard}: everyone sees only their own team's rows, the * team always comes from the authenticated principal, and scoping applies only when login is * enabled so single-user deployments keep working. When the team cannot be resolved the caller - * reads nothing; see {@link #scope()}. + * reads nothing; see {@link #readScope()}. */ @Slf4j @Service @@ -41,7 +43,9 @@ public class FileRunEventService { */ public List report(EditorFailureReport report) { FailureKind kind = FailureKind.byErrorCode(report.errorCode()).orElse(FailureKind.UNKNOWN); - Long teamId = scope().teamId(); + // The caller's team, not their read scope: recording is open to everyone, and a reader who + // may see nothing still has their failure filed under the team it happened in. + Long teamId = currentTeamId(); String actor = currentActor(); String detail = detailFor(report); @@ -81,10 +85,14 @@ public class FileRunEventService { * cleared cache or another device never will. Rows left open that way are retention's problem, * not this method's. * + *

Narrowed to the caller's own rows however senior they are, which is why it passes {@link + * #currentActor()} rather than the read scope's actor: file ids are minted by each client, so a + * leader reading with a null actor would match every unattributed row in the team. + * * @return how many incidents were closed */ public int forgetFiles(List fileIds) { - TeamScope scope = scope(); + ReadScope scope = readScope(); if (!scope.permitted()) { return 0; } @@ -92,13 +100,16 @@ public class FileRunEventService { return store.markFilesRemoved(scope.teamId(), currentActor(), named); } - /** The calling user's events, newest first. Empty when their team cannot be resolved. */ + /** + * The events the caller may read, newest first: the team's for a leader, their own for everyone + * else. Empty when their team cannot be resolved. + */ public List list(FileRunEventStatus status, String kindId, int limit) { - TeamScope scope = scope(); + ReadScope scope = readScope(); if (!scope.permitted()) { return List.of(); } - return store.list(scope.teamId(), status, kindId, limit); + return store.list(scope.teamId(), status, kindId, scope.actor(), limit); } /** @@ -108,7 +119,13 @@ public class FileRunEventService { * event's kind does not declare the action, or the event is already closed */ public FileRunEvent dispatch(String eventId, String actionId, Map inputs) { - TeamScope scope = scope(); + // Whoever can see it can close it: a leader for the whole team, everyone else for the + // failures they caused. Someone who fixes their own problem should not have to ask a leader + // to clear the row. + // + // Closing the row is all this covers. Acting on the document behind it, such as supplying a + // password for a retry, would need its own permission, and no such action exists yet. + ReadScope scope = readScope(); if (!scope.permitted()) { // Reported as "no such event", the same as an id from another team, so the response // does @@ -118,6 +135,12 @@ public class FileRunEventService { } FileRunEvent event = store.find(eventId, scope.teamId()) + // Reported as "no such event" rather than a refusal, so a member cannot + // learn that a colleague's incident exists by trying to close it. + .filter( + found -> + scope.actor() == null + || scope.actor().equals(found.actor())) .orElseThrow( () -> new FailureActionException( @@ -179,31 +202,55 @@ public class FileRunEventService { } /** - * Which rows the caller may touch, since a null team id means two different things. Login - * disabled is the self-hosted setup with no users or teams, where unteamed rows are everyone's, - * as {@code PolicyAccessGuard} also treats them. Login enabled with no resolvable team reads - * nothing, because unteamed rows there are shared by every team's ad-hoc runs. + * Which rows the caller may read. A leader reviews the whole team's, as before. Everyone else + * reads the failures they caused themselves: a member can already report one, so letting them + * see their own back is what makes telling them about it worth anything, and it exposes nothing + * of a colleague's. + * + *

A null team id means two different things. Login disabled is the self-hosted setup with no + * users or teams, where unteamed rows are everyone's, as {@code PolicyAccessGuard} also treats + * them. Login enabled with no resolvable team reads nothing, because unteamed rows there are + * shared by every team's ad-hoc runs. */ - private TeamScope scope() { + private ReadScope readScope() { if (!enforced()) { - return TeamScope.of(null); + return ReadScope.wholeTeam(null); } - Long teamId = policyManagementAuthority.currentUserTeamId(); - return teamId == null ? TeamScope.denied() : TeamScope.of(teamId); + Long teamId = currentTeamId(); + if (teamId == null) { + return ReadScope.denied(); + } + if (policyManagementAuthority.canEditPolicies()) { + return ReadScope.wholeTeam(teamId); + } + // Narrowing to "mine" needs a name to narrow by. Without one the filter would be dropped + // and a member would read the whole team, so refuse rather than widen. + String actor = currentActor(); + return actor == null ? ReadScope.denied() : ReadScope.mine(teamId, actor); } /** - * The caller's readable team, or a refusal. {@code teamId} is only meaningful when permitted. + * What the caller may read. {@code actor} is the person to narrow to, or null for the whole + * team; both are only meaningful when permitted. */ - private record TeamScope(boolean permitted, Long teamId) { + private record ReadScope(boolean permitted, Long teamId, String actor) { - static TeamScope of(Long teamId) { - return new TeamScope(true, teamId); + static ReadScope wholeTeam(Long teamId) { + return new ReadScope(true, teamId, null); } - static TeamScope denied() { - return new TeamScope(false, null); + static ReadScope mine(Long teamId, String actor) { + return new ReadScope(true, teamId, actor); } + + static ReadScope denied() { + return new ReadScope(false, null, null); + } + } + + /** The team a row belongs to, which is nobody's when there are no teams to belong to. */ + private Long currentTeamId() { + return enforced() ? policyManagementAuthority.currentUserTeamId() : null; } private String currentActor() { diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventStore.java b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventStore.java index cf336ab76e..2ed86e61b2 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventStore.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/failure/FileRunEventStore.java @@ -121,16 +121,19 @@ public class FileRunEventStore { * *

Both filters live in the query, before the limit: filtering an already-limited page could * return nothing while matching rows exist. + * + *

{@code actor} narrows to one person's own failures, or reads the whole team when null. Who + * gets which is the service's decision, not this method's. */ @Transactional(readOnly = true) public List list( - Long teamId, FileRunEventStatus status, String kindId, int limit) { + Long teamId, FileRunEventStatus status, String kindId, String actor, int limit) { Pageable page = PageRequest.of(0, Math.max(1, limit)); List rows = status == null ? repository.findByTeamAndStatusIn( - teamId, FileRunEventStatus.open(), kindId, page) - : repository.findByTeamAndStatus(teamId, status, kindId, page); + teamId, FileRunEventStatus.open(), kindId, actor, page) + : repository.findByTeamAndStatus(teamId, status, kindId, actor, page); return rows.stream().map(FileRunEvent::of).toList(); } diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyEngine.java b/app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyEngine.java index 1a424176ce..0d75866a6f 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyEngine.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/policy/engine/PolicyEngine.java @@ -130,7 +130,15 @@ public class PolicyEngine { // worker. String principal = currentActingPrincipal(); return submitForPrincipal( - principal, principal, policyId, definition, inputs, listener, null, null); + principal, + principal, + principal, + policyId, + definition, + inputs, + listener, + null, + null); } /** Run a stored policy on demand. {@code enabled} gates triggers, not explicit runs. */ @@ -158,6 +166,10 @@ public class PolicyEngine { // they can download their enforced file; otherwise an org-wide policy's output is owned by // the admin and the triggering user is denied it. Trigger-fired runs have no such user, so // the owner owns those outputs. + // + // The triggering user is also carried on the run, as the actor of any failure it records: + // null for a trigger-fired run, which is what makes an unattended incident ownerless rather + // than the owner's problem. Three identities, deliberately not interchangeable. String triggeringUser = currentActingPrincipal(); String fileOwner = triggeringUser != null ? triggeringUser : policy.owner(); // Stored supporting files (certificates, watermark images, ...) load here, before the @@ -172,6 +184,7 @@ public class PolicyEngine { return submitForPrincipal( policy.owner(), fileOwner, + triggeringUser, policy.id(), definition, // main's asset-resolved inputs, not the raw ones: stored certificates and watermark @@ -185,6 +198,7 @@ public class PolicyEngine { private PolicyRunHandle submitForPrincipal( String billingPrincipal, String fileOwner, + String triggeringUser, String policyId, PipelineDefinition definition, PolicyInputs inputs, @@ -199,7 +213,8 @@ public class PolicyEngine { if (policyId != null) { taskManager.putMetadata(runId, "policyId", policyId); } - PolicyRun run = new PolicyRun(runId, policyId, definition, sourceId, fileIdentity); + PolicyRun run = + new PolicyRun(runId, policyId, definition, sourceId, fileIdentity, triggeringUser); registry.register(run); CompletableFuture completion = new CompletableFuture<>(); PolicyProgressListener tracking = trackingListener(runId, run, listener); @@ -364,12 +379,14 @@ public class PolicyEngine { taskManager.setError(run.getRunId(), message); // No exception to classify here: nothing was thrown by a tool, the run simply was not // admitted. Record it explicitly so a run lost to load pressure is still accounted for. + // Attributed like any other failure: a user whose run was refused is still the person + // holding that document, and an unattended sweep's run carries no triggering user. failureRecorder.recordRunFailureAs( FailureKind.UNKNOWN, run.getRunId(), run.getPolicyId(), run.getSourceId(), - null, + run.getTriggeringUser(), message); completion.complete(run); } @@ -379,6 +396,11 @@ public class PolicyEngine { /** * Record why a run failed. Called after the run's own state transition and task-manager update, * so a recording problem cannot change the outcome the caller observes. + * + *

The actor is the run's triggering user, not the MDC audit principal: that carries the + * BILLING identity, which for a stored policy is always its owner. Reading it here filed every + * failure under the owner — hiding an attended failure from the member who caused it and holds + * the document, and leaving an unattended sweep's failure looking attended. */ private void recordFailure(PolicyRun run, String message, Throwable cause) { failureRecorder.recordRunFailure( @@ -386,7 +408,7 @@ public class PolicyEngine { run.getPolicyId(), run.getSourceId(), run.getFileIdentity(), - MDC.get(AUDIT_PRINCIPAL_MDC_KEY), + run.getTriggeringUser(), message, cause); } diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/policy/model/PolicyRun.java b/app/proprietary/src/main/java/stirling/software/proprietary/policy/model/PolicyRun.java index 65cfa658ca..8eb0cfcc73 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/policy/model/PolicyRun.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/policy/model/PolicyRun.java @@ -35,6 +35,14 @@ public class PolicyRun { */ private final String fileIdentity; + /** + * The user who triggered this run, or null when nothing attended it (a trigger-fired sweep). + * Recorded as a failure's actor, so an attended failure is handed to the person holding the + * document. Deliberately not the billing principal: a shared policy is billed to its owner, who + * may never have touched the file. + */ + private final String triggeringUser; + private final Instant createdAt = Instant.now(); private volatile PolicyRunStatus status = PolicyRunStatus.PENDING; @@ -63,22 +71,24 @@ public class PolicyRun { private volatile Instant updatedAt = Instant.now(); /** - * Both references are required rather than defaulted: a run with neither is a real case (a - * user's upload, an ad-hoc pipeline), but it should be stated at the call site. Overloads that - * omitted them would make losing the attribution the frictionless option, which is how both - * fields went unpopulated in the first place. + * All three attribution references are required rather than defaulted: a run with none is a + * real case (an unattended sweep of a generator pipeline), but it should be stated at the call + * site. Overloads that omitted them would make losing the attribution the frictionless option, + * which is how {@code sourceId} and {@code fileIdentity} went unpopulated in the first place. */ public PolicyRun( String runId, String policyId, PipelineDefinition definition, String sourceId, - String fileIdentity) { + String fileIdentity, + String triggeringUser) { this.runId = runId; this.policyId = policyId; this.sourceId = sourceId; this.definition = definition; this.fileIdentity = fileIdentity; + this.triggeringUser = triggeringUser; } public int stepCount() { diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventControllerTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventControllerTest.java index 088cffe8ec..58b8d1b408 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventControllerTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventControllerTest.java @@ -51,9 +51,7 @@ class FileRunEventControllerTest { List.of(new AcknowledgeAction(store), new DismissAction(store))); controller = new FileRunEventController( - new FileRunEventService(store, registry, authority, userService, props), - authority, - props); + new FileRunEventService(store, registry, authority, userService, props)); lenient().when(authority.canEditPolicies()).thenReturn(true); lenient().when(authority.currentUserTeamId()).thenReturn(TEAM); @@ -61,14 +59,27 @@ class FileRunEventControllerTest { } private FileRunEvent given(FailureKind kind, Long teamId, String fileId) { + return recorded("author@example.com", kind, teamId, fileId, "run-1"); + } + + /** + * As {@link #given} but naming who hit it, in its own run. A RUN-scoped kind keys on the run, + * so two rows sharing one run id are one incident, however they differ otherwise. + */ + private FileRunEvent givenHitBy(String actor, FailureKind kind, Long teamId, String fileId) { + return recorded(actor, kind, teamId, fileId, "run-" + fileId); + } + + private FileRunEvent recorded( + String actor, FailureKind kind, Long teamId, String fileId, String runId) { return store.record( new RecordFailure( kind, FailureOrigin.POLICY, teamId, - "author@example.com", + actor, "policy-1", - "run-1", + runId, null, fileId, "the raw failure message")); @@ -281,48 +292,68 @@ class FileRunEventControllerTest { } @Nested - @DisplayName("only a team leader may review failures") + @DisplayName("a leader reviews the team's failures, everyone else their own") class Authorization { @Test - void aMemberCannotListThem() { + void aMemberSeesTheirOwnFailuresAndNobodyElses() { + // A member can report a failure, so they get to see it back. It must not widen to a + // colleague's. + givenHitBy("reviewer@example.com", FailureKind.UNKNOWN, TEAM, "mine"); + givenHitBy("colleague@example.com", FailureKind.UNKNOWN, TEAM, "theirs"); when(authority.canEditPolicies()).thenReturn(false); - assertThatThrownBy(() -> controller.list(null, null, null)) - .isInstanceOf(ResponseStatusException.class) - .satisfies( - e -> - assertThat(((ResponseStatusException) e).getStatusCode()) - .isEqualTo(HttpStatus.FORBIDDEN)); + assertThat(controller.list(null, null, null).events()) + .extracting(FileRunEventView::fileId) + .containsExactly("mine"); } @Test - void aMemberCannotDispatchAnAction() { - // The read being refused is not enough on its own: an id learned any other way must - // not let a member close another user's failure. - FileRunEvent event = given(FailureKind.UNKNOWN, TEAM, "f-1"); - when(authority.canEditPolicies()).thenReturn(false); + void aLeaderSeesTheWholeTeams() { + givenHitBy("reviewer@example.com", FailureKind.UNKNOWN, TEAM, "mine"); + givenHitBy("colleague@example.com", FailureKind.UNKNOWN, TEAM, "theirs"); + when(authority.canEditPolicies()).thenReturn(true); - assertThatThrownBy(() -> controller.act(event.id(), "DISMISS", null)) - .isInstanceOf(ResponseStatusException.class) - .satisfies( - e -> - assertThat(((ResponseStatusException) e).getStatusCode()) - .isEqualTo(HttpStatus.FORBIDDEN)); + assertThat(controller.list(null, null, null).events()) + .extracting(FileRunEventView::fileId) + .containsExactlyInAnyOrder("mine", "theirs"); } @Test - void theRegistryIsAlsoLeaderOnly() { + void aMemberMayCloseTheirOwn() { + // Someone who fixes their own problem should not have to ask a leader to clear the row. + FileRunEvent mine = + givenHitBy("reviewer@example.com", FailureKind.UNKNOWN, TEAM, "mine"); when(authority.canEditPolicies()).thenReturn(false); - assertThatThrownBy(() -> controller.kinds()) - .isInstanceOf(ResponseStatusException.class); + assertThat(controller.act(mine.id(), "DISMISS", null).status()) + .isEqualTo(FileRunEventStatus.DISMISSED); + } + + @Test + void aMemberCannotCloseAColleaguesEvenKnowingTheId() { + // Refusing the read is not enough on its own: an id learned any other way must not work + // either. Answered as not-found rather than forbidden, so trying does not confirm the + // row exists. + FileRunEvent theirs = + givenHitBy("colleague@example.com", FailureKind.UNKNOWN, TEAM, "theirs"); + when(authority.canEditPolicies()).thenReturn(false); + + assertThat(statusOf(() -> controller.act(theirs.id(), "DISMISS", null))) + .isEqualTo(HttpStatus.NOT_FOUND); + } + + @Test + void theRegistryIsOpenBecauseItIsCopyNotData() { + // A member renders the failures they can see, so they need the labels for them. No role + // stub: the point is that kinds() never asks. + assertThat(controller.kinds()).isNotEmpty(); } @Test void loginDisabledTrustsTheLocalOperator() { - // A single-user deployment has no roles to distinguish, so the role gate must not lock - // the only user out of their own failures. + // A single-user deployment has no roles to distinguish, so the narrowing must not leave + // the only user reading nothing. ApplicationProperties unsecured = new ApplicationProperties(); unsecured.getSecurity().setEnableLogin(false); FileRunEventController noLogin = @@ -335,9 +366,7 @@ class FileRunEventControllerTest { new DismissAction(store))), authority, userService, - unsecured), - authority, - unsecured); + unsecured)); assertThatCode(() -> noLogin.list(null, null, null)).doesNotThrowAnyException(); // Not merely permitted: the role is never consulted at all, which is what makes the @@ -382,9 +411,7 @@ class FileRunEventControllerTest { new DismissAction(store))), authority, userService, - unsecured), - authority, - unsecured); + unsecured)); given(FailureKind.UNKNOWN, null, "unteamed"); given(FailureKind.UNKNOWN, TEAM, "teamed"); @@ -399,9 +426,9 @@ class FileRunEventControllerTest { class Reporting { @Test - void aMemberMayReportEvenThoughTheyMayNotRead() { - // The asymmetry is the point: anyone whose work failed can say so, but only a leader - // reviews the queue. + void aMemberMayReportAndThenSeeTheirOwnReport() { + // Reporting was always open to a member; reading their own back is the round trip that + // makes the report worth anything to them. when(authority.canEditPolicies()).thenReturn(false); assertThatCode( @@ -410,8 +437,9 @@ class FileRunEventControllerTest { new EditorFailureReport( "compress", "E004", List.of("f-1"), "boom"))) .doesNotThrowAnyException(); - assertThatThrownBy(() -> controller.list(null, null, null)) - .isInstanceOf(ResponseStatusException.class); + assertThat(controller.list(null, null, null).events()) + .extracting(FileRunEventView::fileId) + .containsExactly("f-1"); } @Test @@ -441,7 +469,7 @@ class FileRunEventControllerTest { new EditorFailureReport("compress", "E004", atLimit, "boom"); assertThat(controller.report(report).getStatusCode()).isEqualTo(HttpStatus.NO_CONTENT); - assertThat(store.list(TEAM, null, null, EditorFailureReport.MAX_FILE_IDS + 10)) + assertThat(store.list(TEAM, null, null, null, EditorFailureReport.MAX_FILE_IDS + 10)) .hasSize(EditorFailureReport.MAX_FILE_IDS); } @@ -463,7 +491,7 @@ class FileRunEventControllerTest { overLimit, "boom")))) .isEqualTo(HttpStatus.BAD_REQUEST); - assertThat(store.list(TEAM, null, null, EditorFailureReport.MAX_FILE_IDS + 10)) + assertThat(store.list(TEAM, null, null, null, EditorFailureReport.MAX_FILE_IDS + 10)) .isEmpty(); } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventHttpIntegrationTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventHttpIntegrationTest.java index b3ee4b7d15..375e46f4ad 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventHttpIntegrationTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventHttpIntegrationTest.java @@ -438,11 +438,8 @@ class FileRunEventHttpIntegrationTest { } @Bean - FileRunEventController fileRunEventController( - FileRunEventService service, - PolicyManagementAuthority authority, - ApplicationProperties props) { - return new FileRunEventController(service, authority, props); + FileRunEventController fileRunEventController(FileRunEventService service) { + return new FileRunEventController(service); } } } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventServiceTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventServiceTest.java index fbbe8e6359..4e6508bed9 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventServiceTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventServiceTest.java @@ -54,6 +54,9 @@ class FileRunEventServiceTest { lenient().when(authority.currentUserTeamId()).thenReturn(TEAM); lenient().when(userService.getCurrentUsername()).thenReturn(ACTOR); + // A leader unless a test says otherwise: most of these are about team scoping, which is + // what a leader sees. The member narrowing has its own tests. + lenient().when(authority.canEditPolicies()).thenReturn(true); } private FileRunEvent given(FailureKind kind, Long teamId, String fileId) { @@ -291,7 +294,7 @@ class FileRunEventServiceTest { } @Nested - @DisplayName("team scoping") + @DisplayName("read scoping") class Scoping { @Test @@ -304,6 +307,68 @@ class FileRunEventServiceTest { .containsExactly("mine"); } + @Test + void aMemberReadsOnlyTheFailuresTheyCaused() { + // Reporting is open to a member, so reading their own back is what lets us tell them + // anything at all. A colleague's must not come with it. + store.record(RecordFailure.forEditor(FailureKind.UNKNOWN, TEAM, ACTOR, "mine", "boom")); + store.record( + RecordFailure.forEditor( + FailureKind.UNKNOWN, TEAM, "colleague@example.com", "theirs", "boom")); + when(authority.canEditPolicies()).thenReturn(false); + + assertThat(service.list(null, null, 50)) + .extracting(FileRunEvent::fileId) + .containsExactly("mine"); + } + + @Test + void aMemberWithNoResolvableNameReadsNothingRatherThanEverything() { + // Narrowing to "mine" needs a name to narrow by. Dropping the filter would hand the + // whole team to someone who may not have it. + given(FailureKind.UNKNOWN, TEAM, "mine"); + when(authority.canEditPolicies()).thenReturn(false); + when(userService.getCurrentUsername()).thenReturn(null); + + assertThat(service.list(null, null, 50)).isEmpty(); + } + + @Test + void aMemberCannotActOnAColleaguesRowEvenKnowingItsId() { + // Refusing the read is not enough on its own: an id learned any other way must not work + // either. Reported as not-found, so trying does not confirm the row exists. + FileRunEvent theirs = + store.record( + RecordFailure.forEditor( + FailureKind.UNKNOWN, + TEAM, + "colleague@example.com", + "theirs", + "boom")); + when(authority.canEditPolicies()).thenReturn(false); + + assertThatThrownBy(() -> service.dispatch(theirs.id(), "DISMISS", Map.of())) + .isInstanceOf(FailureActionException.class) + .extracting(e -> ((FailureActionException) e).getReason()) + .isEqualTo(FailureActionException.Reason.EVENT_NOT_FOUND); + + assertThat(store.find(theirs.id(), TEAM).orElseThrow().status()) + .isEqualTo(FileRunEventStatus.NEW); + } + + @Test + void aMemberMayCloseTheirOwn() { + // Someone who fixes their own problem should not have to ask a leader to clear the row. + FileRunEvent mine = + store.record( + RecordFailure.forEditor( + FailureKind.UNKNOWN, TEAM, ACTOR, "mine", "boom")); + when(authority.canEditPolicies()).thenReturn(false); + + assertThat(service.dispatch(mine.id(), "DISMISS", Map.of()).status()) + .isEqualTo(FileRunEventStatus.DISMISSED); + } + @Test void aCallerWhoseTeamCannotBeResolvedReadsNothing() { // A run with no stored policy is recorded unteamed, and those rows are shared by every @@ -399,7 +464,7 @@ class FileRunEventServiceTest { service.report( new EditorFailureReport("remove-password", "E004", List.of("f-1"), "boom")); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.kind()).isEqualTo(FailureKind.INPUT_PASSWORD_PROTECTED); assertThat(event.origin()).isEqualTo(FailureOrigin.TOOL); assertThat(event.fileId()).isEqualTo("f-1"); @@ -412,16 +477,32 @@ class FileRunEventServiceTest { // session. service.report(new EditorFailureReport("compress", "E004", List.of("f-1"), "boom")); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.teamId()).isEqualTo(TEAM); assertThat(event.actor()).isEqualTo(ACTOR); } + @Test + void stillFilesTheRowUnderTheTeamWhenTheReporterCannotBeNamed() { + // Recording is open to everyone and takes the caller's team, not their read scope: a + // reporter who cannot be named reads nothing back, but the row is still the team's + // rather than dropping into the unteamed bucket every team shares. No role stub either, + // since recording never asks. + when(userService.getCurrentUsername()).thenReturn(null); + + service.report(new EditorFailureReport("compress", "E004", List.of("f-1"), "boom")); + + assertThat(store.list(TEAM, null, null, null, 10)) + .singleElement() + .extracting(FileRunEvent::teamId) + .isEqualTo(TEAM); + } + @Test void recordsAnUnrecognisedCodeAsUnknownRatherThanDroppingIt() { service.report(new EditorFailureReport("ocr", "E999", List.of("f-1"), "no idea")); - assertThat(store.list(TEAM, null, null, 10).getFirst().kind()) + assertThat(store.list(TEAM, null, null, null, 10).getFirst().kind()) .isEqualTo(FailureKind.UNKNOWN); } @@ -429,7 +510,7 @@ class FileRunEventServiceTest { void recordsAnAbsentCodeAsUnknown() { service.report(new EditorFailureReport("ocr", null, List.of("f-1"), "network died")); - assertThat(store.list(TEAM, null, null, 10).getFirst().kind()) + assertThat(store.list(TEAM, null, null, null, 10).getFirst().kind()) .isEqualTo(FailureKind.UNKNOWN); } @@ -439,7 +520,7 @@ class FileRunEventServiceTest { new EditorFailureReport( "compress", "E004", List.of("f-1", "f-2", "f-3"), "boom")); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .hasSize(3) .extracting(FileRunEvent::fileId) .containsExactlyInAnyOrder("f-1", "f-2", "f-3"); @@ -451,7 +532,7 @@ class FileRunEventServiceTest { service.report( new EditorFailureReport("compress", "E004", List.of("f-1"), "boom again")); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .singleElement() .extracting(FileRunEvent::occurrences) .isEqualTo(2); @@ -461,7 +542,7 @@ class FileRunEventServiceTest { void recordsOneUnattributedIncidentWhenNoFileWasNamed() { service.report(new EditorFailureReport("compress", "E004", List.of(), "boom")); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .singleElement() .extracting(FileRunEvent::fileId) .isNull(); @@ -476,7 +557,7 @@ class FileRunEventServiceTest { service.report(new EditorFailureReport("compress", "E004", many, "boom")); - assertThat(store.list(TEAM, null, null, 200)).hasSize(60); + assertThat(store.list(TEAM, null, null, null, 200)).hasSize(60); } @Test @@ -486,7 +567,7 @@ class FileRunEventServiceTest { service.report( new EditorFailureReport("remove-password", "E004", List.of("f-1"), "boom")); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.detail()).contains("remove-password"); assertThat(event.fileId()).isEqualTo("f-1"); } @@ -499,7 +580,7 @@ class FileRunEventServiceTest { new EditorFailureReport( "compress", "E004", List.of("f-1"), "Failed on Q4 report.pdf")); - assertThat(store.list(TEAM, null, null, 10).getFirst().detail()) + assertThat(store.list(TEAM, null, null, null, 10).getFirst().detail()) .isEqualTo("compress: Failed on Q4 report.pdf"); } @@ -540,7 +621,7 @@ class FileRunEventServiceTest { reportedBy("alice@example.com", "a-1"); reportedBy("bob@example.com", "b-1"); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .extracting(FileRunEvent::actor) .containsExactlyInAnyOrder("alice@example.com", "bob@example.com"); } @@ -550,7 +631,7 @@ class FileRunEventServiceTest { reportedBy("alice@example.com", "a-1"); reportedBy("alice@example.com", "a-2"); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .extracting(FileRunEvent::fileId) .containsExactlyInAnyOrder("a-1", "a-2"); } @@ -560,7 +641,7 @@ class FileRunEventServiceTest { reportedBy("alice@example.com", "a-1"); reportedBy("alice@example.com", "a-1"); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .singleElement() .extracting(FileRunEvent::occurrences) .isEqualTo(2); @@ -612,8 +693,11 @@ class FileRunEventServiceTest { } @Test - void aColleaguesIncidentIsUntouched() { - // File ids come from the client, so naming one must not close someone else's row. + void aColleaguesIncidentIsUntouchedEvenForALeader() { + // File ids come from the client, so naming one must not close someone else's row. The + // caller here is a leader, who reads the whole team: this path narrows to their own + // rows + // regardless, since a null actor would otherwise match every unattributed row. store.record( RecordFailure.forEditor( FailureKind.UNKNOWN, TEAM, "employee@example.com", "f-1", "theirs")); diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreDbTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreDbTest.java index 949904dfb9..982d804354 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreDbTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreDbTest.java @@ -55,11 +55,16 @@ class FileRunEventStoreDbTest { } private RecordFailure failure(FailureKind kind, Long teamId, String fileId) { + return failure(kind, teamId, "author@example.com", fileId); + } + + /** As {@link #failure} but naming the actor, which is what the read scope narrows by. */ + private RecordFailure failure(FailureKind kind, Long teamId, String actor, String fileId) { return new RecordFailure( kind, FailureOrigin.POLICY, teamId, - "author@example.com", + actor, "policy-1", "run-1", null, @@ -74,16 +79,37 @@ class FileRunEventStoreDbTest { store.record(failure(FailureKind.UNKNOWN, OTHER_TEAM, "theirs")); store.record(failure(FailureKind.UNKNOWN, null, "unteamed")); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .extracting(FileRunEvent::fileId) .containsExactly("ours"); // A plain `e.teamId = :teamId` would return nothing here: SQL equality against NULL is // never true, which is what the explicit null branch in the JPQL exists for. - assertThat(store.list(null, null, null, 10)) + assertThat(store.list(null, null, null, null, 10)) .extracting(FileRunEvent::fileId) .containsExactly("unteamed"); } + @Test + @DisplayName("actor narrowing is enforced by the query, within the team") + void actorNarrowingIsEnforcedBySql() { + // The clause that makes a member read only their own rows. Exercised here rather than only + // against the in-memory repository, which reimplements the filter in Java and would agree + // with a query that had lost it. + store.record(failure(FailureKind.INPUT_PASSWORD_PROTECTED, TEAM, "mine@example.com", "f1")); + store.record( + failure(FailureKind.INPUT_PASSWORD_PROTECTED, TEAM, "theirs@example.com", "f2")); + store.record(failure(FailureKind.INPUT_PASSWORD_PROTECTED, TEAM, null, "f3")); + + assertThat(store.list(TEAM, null, null, "mine@example.com", 10)) + .extracting(FileRunEvent::fileId) + .containsExactly("f1"); + // A null actor is "no filter", which is what a leader reads with: the whole team, including + // the rows nobody is named on. + assertThat(store.list(TEAM, null, null, null, 10)) + .extracting(FileRunEvent::fileId) + .containsExactlyInAnyOrder("f1", "f2", "f3"); + } + @Test @DisplayName("a fold lands on the row's current state, not the caller's snapshot") void foldTargetsTheCurrentRowNotACallersSnapshot() { @@ -170,7 +196,7 @@ class FileRunEventStoreDbTest { assertThat(replacement.id()).isNotEqualTo(first.id()); assertThat(replacement.occurrences()).isEqualTo(1); - assertThat(store.list(TEAM, null, null, 10)).hasSize(1); + assertThat(store.list(TEAM, null, null, null, 10)).hasSize(1); } @Test @@ -188,7 +214,7 @@ class FileRunEventStoreDbTest { FileRunEvent folded = store.record(secondSweep); assertThat(folded.occurrences()).isEqualTo(2); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .as("one incident per document, however many runs it failed in") .extracting(FileRunEvent::fileId) .containsExactlyInAnyOrder("file-hash-a", "file-hash-b"); @@ -267,7 +293,7 @@ class FileRunEventStoreDbTest { store.record(failure(FailureKind.INPUT_PASSWORD_PROTECTED, TEAM, "newer-" + i)); } - assertThat(store.list(TEAM, null, "UNKNOWN", 1)) + assertThat(store.list(TEAM, null, "UNKNOWN", null, 1)) .extracting(FileRunEvent::fileId) .containsExactly("old-unknown"); } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreTest.java index 714d65c29c..97de1952e9 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/FileRunEventStoreTest.java @@ -281,7 +281,7 @@ class FileRunEventStoreTest { store.record(failure(FailureKind.UNKNOWN, TEAM, "mine", "a")); store.record(failure(FailureKind.UNKNOWN, OTHER_TEAM, "theirs", "b")); - assertThat(store.list(TEAM, null, null, 50)) + assertThat(store.list(TEAM, null, null, null, 50)) .extracting(FileRunEvent::fileId) .containsExactly("mine"); } @@ -293,7 +293,7 @@ class FileRunEventStoreTest { store.record(failure(FailureKind.UNKNOWN, null, "unteamed", "a")); store.record(failure(FailureKind.UNKNOWN, TEAM, "teamed", "b")); - assertThat(store.list(null, null, null, 50)) + assertThat(store.list(null, null, null, null, 50)) .extracting(FileRunEvent::fileId) .containsExactly("unteamed"); } @@ -305,8 +305,8 @@ class FileRunEventStoreTest { FileRunEvent event = store.record(failure(FailureKind.UNKNOWN, TEAM, "f1", "boom")); store.applyStatus(event.id(), TEAM, FileRunEventStatus.DISMISSED, "reviewer"); - assertThat(store.list(TEAM, null, null, 10)).isEmpty(); - assertThat(store.list(TEAM, FileRunEventStatus.DISMISSED, null, 10)).hasSize(1); + assertThat(store.list(TEAM, null, null, null, 10)).isEmpty(); + assertThat(store.list(TEAM, FileRunEventStatus.DISMISSED, null, null, 10)).hasSize(1); } @Test @@ -314,7 +314,7 @@ class FileRunEventStoreTest { FileRunEvent event = store.record(failure(FailureKind.UNKNOWN, TEAM, "f1", "boom")); store.applyStatus(event.id(), TEAM, FileRunEventStatus.ACKNOWLEDGED, "reviewer"); - assertThat(store.list(TEAM, null, null, 10)).hasSize(1); + assertThat(store.list(TEAM, null, null, null, 10)).hasSize(1); } @Test @@ -323,7 +323,7 @@ class FileRunEventStoreTest { store.record(failure(FailureKind.INPUT_PASSWORD_PROTECTED, TEAM, "closed", "b")); store.applyStatus(open.id(), TEAM, FileRunEventStatus.ACKNOWLEDGED, "me"); - assertThat(store.list(TEAM, FileRunEventStatus.ACKNOWLEDGED, null, 50)) + assertThat(store.list(TEAM, FileRunEventStatus.ACKNOWLEDGED, null, null, 50)) .extracting(FileRunEvent::fileId) .containsExactly("open"); } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/InMemoryFileRunEventRepository.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/InMemoryFileRunEventRepository.java index 1873f6265b..6dbb16322b 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/InMemoryFileRunEventRepository.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/InMemoryFileRunEventRepository.java @@ -54,9 +54,18 @@ class InMemoryFileRunEventRepository implements FileRunEventRepository { return kindId == null || kindId.equals(entity.getKindId()); } + /** Null means the whole team, matching the JPQL's {@code :actor is null} branch. */ + private static boolean sameActor(FileRunEventEntity entity, String actor) { + return actor == null || actor.equals(entity.getActor()); + } + @Override public List findByTeamAndStatus( - Long teamId, FileRunEventStatus status, String kindId, Pageable pageable) { + Long teamId, + FileRunEventStatus status, + String kindId, + String actor, + Pageable pageable) { return page( newestFirst( rows.values().stream() @@ -64,7 +73,8 @@ class InMemoryFileRunEventRepository implements FileRunEventRepository { e -> sameTeam(e, teamId) && e.getStatus() == status - && sameKind(e, kindId)) + && sameKind(e, kindId) + && sameActor(e, actor)) .toList()), pageable); } @@ -142,7 +152,11 @@ class InMemoryFileRunEventRepository implements FileRunEventRepository { @Override public List findByTeamAndStatusIn( - Long teamId, List statuses, String kindId, Pageable pageable) { + Long teamId, + List statuses, + String kindId, + String actor, + Pageable pageable) { return page( newestFirst( rows.values().stream() @@ -150,7 +164,8 @@ class InMemoryFileRunEventRepository implements FileRunEventRepository { e -> sameTeam(e, teamId) && statuses.contains(e.getStatus()) - && sameKind(e, kindId)) + && sameKind(e, kindId) + && sameActor(e, actor)) .toList()), pageable); } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureAttributionTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureAttributionTest.java new file mode 100644 index 0000000000..2a71c236a7 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureAttributionTest.java @@ -0,0 +1,264 @@ +package stirling.software.proprietary.failure; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.when; + +import java.nio.file.Path; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.TimeUnit; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.slf4j.MDC; +import org.springframework.core.io.ByteArrayResource; + +import stirling.software.common.model.ApplicationProperties; +import stirling.software.common.service.FileStorage; +import stirling.software.common.service.InternalApiClient; +import stirling.software.common.service.JobOwnershipService; +import stirling.software.common.service.JobQueue; +import stirling.software.common.service.ResourceMonitor; +import stirling.software.common.service.TaskManager; +import stirling.software.common.service.ToolMetadataService; +import stirling.software.common.service.UserServiceInterface; +import stirling.software.common.util.TempFileManager; +import stirling.software.common.util.TempFileRegistry; +import stirling.software.proprietary.policy.asset.InProcessPolicyAssetStore; +import stirling.software.proprietary.policy.asset.PolicyAssetResolver; +import stirling.software.proprietary.policy.config.PolicyManagementAuthority; +import stirling.software.proprietary.policy.engine.PolicyEngine; +import stirling.software.proprietary.policy.engine.PolicyExecutor; +import stirling.software.proprietary.policy.engine.PolicyRunRegistry; +import stirling.software.proprietary.policy.model.OutputSpec; +import stirling.software.proprietary.policy.model.PipelineStep; +import stirling.software.proprietary.policy.model.Policy; +import stirling.software.proprietary.policy.model.PolicyInputs; +import stirling.software.proprietary.policy.output.InlineOutputSink; +import stirling.software.proprietary.policy.output.PolicyOutputResolver; +import stirling.software.proprietary.policy.progress.PolicyProgressListener; +import stirling.software.proprietary.policy.source.InProcessSourceStore; +import stirling.software.proprietary.policy.store.PolicyStore; + +import tools.jackson.databind.json.JsonMapper; + +/** + * Pins what the engine records as a failure's actor against what a reader gets back, because the + * two sides used to assert independently: the engine's test passed {@code any()} for the actor, and + * the service's fixtures assumed an actor the engine never actually produced. Every collaborator + * between the failing tool call and the read is real here, so a regression in either one fails. + * + *

This is what makes this PR's promise hold. Reads are narrowed to the rows the caller is the + * actor on, so if the engine names the wrong person, a member reads nothing at all. + * + *

The bug it exists for: the engine recorded the BILLING principal as the actor, which for a + * stored policy is always its owner. So an attended failure was filed under someone who never + * touched the document, and the member who did could not see it. + */ +@ExtendWith(MockitoExtension.class) +class PolicyFailureAttributionTest { + + private static final String ROTATE = "/api/v1/general/rotate-pdf"; + private static final Long TEAM = 3L; + + @Mock private InternalApiClient internalApiClient; + @Mock private ToolMetadataService toolMetadataService; + @Mock private TaskManager taskManager; + @Mock private FileStorage fileStorage; + @Mock private JobOwnershipService jobOwnershipService; + @Mock private ResourceMonitor resourceMonitor; + @Mock private JobQueue jobQueue; + @Mock private PolicyStore policyStore; + @Mock private PolicyManagementAuthority authority; + @Mock private UserServiceInterface userService; + + @TempDir Path tempDir; + + private PolicyEngine engine; + private FileRunEventService service; + + @BeforeEach + void setUp() { + ApplicationProperties props = new ApplicationProperties(); + props.getSecurity().setEnableLogin(true); + props.getSystem().getTempFileManagement().setBaseTmpDir(tempDir.toString()); + props.getSystem().getTempFileManagement().setPrefix("failure-attribution-test-"); + + FileRunEventStore store = new FileRunEventStore(new InMemoryFileRunEventRepository()); + service = + new FileRunEventService( + store, + new FailureActionRegistry( + List.of(new AcknowledgeAction(store), new DismissAction(store))), + authority, + userService, + props); + + PolicyFailureRecorder recorder = + new PolicyFailureRecorder( + new FailureClassifier(JsonMapper.builder().build()), store, policyStore); + PolicyExecutor executor = + new PolicyExecutor( + internalApiClient, + toolMetadataService, + new TempFileManager(new TempFileRegistry(), props), + JsonMapper.builder().build()); + engine = + new PolicyEngine( + executor, + taskManager, + new PolicyRunRegistry(new ApplicationProperties()), + recorder, + fileStorage, + jobOwnershipService, + List.of(new InlineOutputSink(fileStorage)), + new PolicyOutputResolver(new InProcessSourceStore()), + resourceMonitor, + jobQueue, + new PolicyAssetResolver(new InProcessPolicyAssetStore())); + + lenient() + .when(jobOwnershipService.createScopedJobKey(anyString())) + .thenAnswer(invocation -> invocation.getArgument(0)); + lenient().when(resourceMonitor.shouldQueueJob(anyInt())).thenReturn(false); + lenient().when(toolMetadataService.isMultiInput(anyString())).thenReturn(false); + // The team is resolved from the policy, so the recorded row lands in the reader's team. + lenient().when(policyStore.get(anyString())).thenReturn(Optional.of(sharedPolicy())); + lenient().when(authority.currentUserTeamId()).thenReturn(TEAM); + } + + /** Alice's policy, shared with her team. Bob is a member of it and does not own it. */ + private static Policy sharedPolicy() { + return new Policy( + "p1", + "rotate", + "alice", + true, + List.of(), + List.of(new PipelineStep(ROTATE, Map.of())), + OutputSpec.inline(), + TEAM); + } + + /** + * Run the shared policy so its single tool step fails, as {@code triggeredBy} (null = sweep). + */ + private void runAndFail(String triggeredBy, String sourceId, String fileIdentity) + throws Exception { + when(internalApiClient.post(eq(ROTATE), any())).thenThrow(new RuntimeException("boom")); + if (triggeredBy != null) { + MDC.put("auditPrincipal", triggeredBy); + } + try { + engine.runPolicy( + sharedPolicy(), + PolicyInputs.of(List.of(pdf())), + PolicyProgressListener.NOOP, + sourceId, + fileIdentity) + .completion() + .get(10, TimeUnit.SECONDS); + } finally { + MDC.remove("auditPrincipal"); + } + } + + private static ByteArrayResource pdf() { + return new ByteArrayResource("input".getBytes()) { + @Override + public String getFilename() { + return "input.pdf"; + } + }; + } + + /** Read as a plain member, who is narrowed to the rows they are the actor on. */ + private FileRunEvent asMember(String reader) { + lenient().when(userService.getCurrentUsername()).thenReturn(reader); + lenient().when(authority.canEditPolicies()).thenReturn(false); + List visible = service.list(null, null, 10); + return visible.isEmpty() ? null : visible.getFirst(); + } + + /** Read as a team leader, who reviews the whole team's incidents. */ + private FileRunEvent asReviewer(String reader) { + lenient().when(userService.getCurrentUsername()).thenReturn(reader); + lenient().when(authority.canEditPolicies()).thenReturn(true); + return service.list(null, null, 10).getFirst(); + } + + @Nested + @DisplayName("a non-owner runs a shared policy on their own upload") + class AttendedByANonOwner { + + @Test + void theTriggeringUserCanReadTheFailureTheyCaused() throws Exception { + runAndFail("bob", null, "bob-doc-1"); + + // The whole point: Bob's read scope narrows to his own rows, so the row only reaches + // him if the engine named him. Before the fix this list was empty. + FileRunEvent mine = asMember("bob"); + assertThat(mine).as("bob must be able to see the failure he caused").isNotNull(); + assertThat(mine.actor()).isEqualTo("bob"); + } + + @Test + void thePolicyOwnerIsNotNamedAsTheActorMerelyForBeingBilled() throws Exception { + runAndFail("bob", null, "bob-doc-1"); + + // Alice owns the policy and pays for the run, but she never touched the document. + assertThat(asReviewer("alice").actor()).isEqualTo("bob"); + } + + @Test + void aColleagueWhoDidNotTriggerItCannotSeeItAtAll() throws Exception { + runAndFail("bob", null, "bob-doc-1"); + + assertThat(asMember("carol")).isNull(); + } + } + + @Nested + @DisplayName("an unattended sweep pulls a file from a source") + class UnattendedSweep { + + @Test + void theRowIsRecordedWithNoActorWhileStillBillingTheOwner() throws Exception { + runAndFail(null, "src-watched-folder", "file-hash-1"); + + assertThat(asReviewer("alice").actor()) + .as("a trigger-fired run has no user to name") + .isNull(); + } + + @Test + void theSourceThatFedItIsStillRecorded() throws Exception { + runAndFail(null, "src-watched-folder", "file-hash-1"); + + FileRunEvent unattended = asReviewer("alice"); + assertThat(unattended.sourceId()).isEqualTo("src-watched-folder"); + assertThat(unattended.fileId()).isEqualTo("file-hash-1"); + } + + @Test + void aMemberDoesNotInheritAnUnattendedFailureAsTheirOwn() throws Exception { + // An unowned row must not fall to whoever happens to be reading: with no actor there is + // nothing for a member's narrowed read to match. + runAndFail(null, "src-watched-folder", "file-hash-1"); + + assertThat(asMember("bob")).isNull(); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureRecorderTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureRecorderTest.java index e1a17843f0..73c5add6ad 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureRecorderTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/failure/PolicyFailureRecorderTest.java @@ -95,7 +95,7 @@ class PolicyFailureRecorderTest { "Policy run failed: locked", passwordFailure()); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.kind()).isEqualTo(FailureKind.INPUT_PASSWORD_PROTECTED); assertThat(event.runId()).isEqualTo("run-1"); assertThat(event.policyId()).isEqualTo("policy-1"); @@ -121,7 +121,7 @@ class PolicyFailureRecorderTest { "Policy run failed: something we do not recognise", new RuntimeException("boom")); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.kind()).isEqualTo(FailureKind.UNKNOWN); assertThat(event.detail()).contains("something we do not recognise"); } @@ -139,7 +139,7 @@ class PolicyFailureRecorderTest { "Policy run failed: java.lang.NullPointerException", new RuntimeException("npe")); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.kind()).isEqualTo(FailureKind.UNKNOWN); assertThat(event.detail()).contains("NullPointerException"); } @@ -152,7 +152,7 @@ class PolicyFailureRecorderTest { recorder.recordRunFailureAs( FailureKind.UNKNOWN, "run-3", "policy-1", null, null, "could not be queued"); - assertThat(store.list(TEAM, null, null, 10)).hasSize(1); + assertThat(store.list(TEAM, null, null, null, 10)).hasSize(1); } @Test @@ -180,7 +180,7 @@ class PolicyFailureRecorderTest { "locked", passwordFailure()); - List events = store.list(TEAM, null, null, 10); + List events = store.list(TEAM, null, null, null, 10); assertThat(events).hasSize(2); assertThat(events).allMatch(event -> event.occurrences() == 1); assertThat(events) @@ -203,7 +203,7 @@ class PolicyFailureRecorderTest { "locked", passwordFailure()); - FileRunEvent event = store.list(TEAM, null, null, 10).getFirst(); + FileRunEvent event = store.list(TEAM, null, null, null, 10).getFirst(); assertThat(event.sourceId()).isEqualTo("src-s3-invoices"); assertThat(event.actor()).isNull(); } @@ -218,7 +218,7 @@ class PolicyFailureRecorderTest { recorder.recordRunFailureAs( FailureKind.UNKNOWN, "run-2", "policy-1", "src-b", null, "unreachable"); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .hasSize(2) .extracting(FileRunEvent::sourceId) .containsExactlyInAnyOrder("src-a", "src-b"); @@ -245,7 +245,7 @@ class PolicyFailureRecorderTest { "locked", passwordFailure()); - assertThat(store.list(TEAM, null, null, 10)) + assertThat(store.list(TEAM, null, null, null, 10)) .singleElement() .extracting(FileRunEvent::occurrences) .isEqualTo(2); @@ -263,7 +263,7 @@ class PolicyFailureRecorderTest { recorder.recordRunFailure( "run-1", "policy-1", null, null, null, "boom", new RuntimeException()); - assertThat(store.list(TEAM, null, null, 10)).hasSize(1); + assertThat(store.list(TEAM, null, null, null, 10)).hasSize(1); } @Test @@ -273,8 +273,8 @@ class PolicyFailureRecorderTest { recorder.recordRunFailure( "run-1", null, null, null, null, "boom", new RuntimeException()); - assertThat(store.list(null, null, null, 10)).hasSize(1); - assertThat(store.list(TEAM, null, null, 10)).isEmpty(); + assertThat(store.list(null, null, null, null, 10)).hasSize(1); + assertThat(store.list(TEAM, null, null, null, 10)).isEmpty(); } @Test @@ -293,7 +293,7 @@ class PolicyFailureRecorderTest { new RuntimeException())) .doesNotThrowAnyException(); // Still recorded, just unteamed: a lookup problem must not lose the incident. - assertThat(store.list(null, null, null, 10)).hasSize(1); + assertThat(store.list(null, null, null, null, 10)).hasSize(1); } } @@ -344,7 +344,7 @@ class PolicyFailureRecorderTest { "no cause", null)) .doesNotThrowAnyException(); - assertThat(store.list(TEAM, null, null, 10).getFirst().kind()) + assertThat(store.list(TEAM, null, null, null, 10).getFirst().kind()) .isEqualTo(FailureKind.UNKNOWN); } } @@ -362,7 +362,7 @@ class PolicyFailureRecorderTest { recorder.recordRunFailure( "run-1", "policy-1", null, null, null, "boom", new IOException("x")); - List events = store.list(TEAM, null, null, 10); + List events = store.list(TEAM, null, null, null, 10); assertThat(events).hasSize(1); assertThat(events.getFirst().occurrences()).isEqualTo(2); } @@ -377,7 +377,7 @@ class PolicyFailureRecorderTest { recorder.recordRunFailure( "run-2", "policy-1", null, null, null, "boom", new IOException("x")); - assertThat(store.list(TEAM, null, null, 10)).hasSize(2); + assertThat(store.list(TEAM, null, null, null, 10)).hasSize(2); } } } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java index c38f5c7d89..84e9998b90 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java @@ -216,7 +216,7 @@ class PolicyControllerTest { } private static PolicyRunHandle handle(String runId) { - PolicyRun run = new PolicyRun(runId, null, definitionWithStep(), null, null); + PolicyRun run = new PolicyRun(runId, null, definitionWithStep(), null, null, null); return new PolicyRunHandle(runId, CompletableFuture.completedFuture(run)); } @@ -317,7 +317,7 @@ class PolicyControllerTest { @Test @DisplayName("returns the run view when present") void found() { - PolicyRun run = new PolicyRun("run-3", null, definitionWithStep(), null, null); + PolicyRun run = new PolicyRun("run-3", null, definitionWithStep(), null, null, null); when(runRegistry.get("run-3")).thenReturn(run); ResponseEntity response = controller.status("run-3"); @@ -346,11 +346,11 @@ class PolicyControllerTest { @Test @DisplayName("excludes ad-hoc runs and runs owned by others") void filtersRuns() { - PolicyRun adHoc = new PolicyRun("adhoc", null, definitionWithStep(), null, null); + PolicyRun adHoc = new PolicyRun("adhoc", null, definitionWithStep(), null, null, null); PolicyRun ownedStored = - new PolicyRun("owned", "policy-A", definitionWithStep(), null, null); + new PolicyRun("owned", "policy-A", definitionWithStep(), null, null, null); PolicyRun otherStored = - new PolicyRun("other", "policy-B", definitionWithStep(), null, null); + new PolicyRun("other", "policy-B", definitionWithStep(), null, null, null); when(runRegistry.all()).thenReturn(List.of(adHoc, ownedStored, otherStored)); // ownedByCurrentUser: strip then re-apply scope reproduces the key only for the owned diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyEngineTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyEngineTest.java index b3b45dcd96..1efe094fe4 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyEngineTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyEngineTest.java @@ -9,6 +9,7 @@ import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.ArgumentMatchers.anyLong; import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.ArgumentMatchers.isNull; import static org.mockito.Mockito.atLeastOnce; import static org.mockito.Mockito.doReturn; import static org.mockito.Mockito.doThrow; @@ -231,20 +232,15 @@ class PolicyEngineTest { @Test void recordsWhichSourceFedAFailedRun() throws Exception { // The source is threaded onto the run so an unattended failure is attributable: there is no - // user to name for a file that arrived from a bucket. + // user to name for a file that arrived from a bucket. The actor is asserted null rather + // than + // any(): a loose matcher here is what let the owner be recorded as the actor unnoticed. when(toolMetadataService.isMultiInput(ROTATE)).thenReturn(false); when(internalApiClient.post(eq(ROTATE), any())).thenThrow(new RuntimeException("boom")); PolicyRunHandle handle = engine.runPolicy( - new Policy( - "p1", - "rotate", - "owner", - true, - List.of(), - List.of(new PipelineStep(ROTATE, Map.of())), - OutputSpec.inline()), + policyOwnedBy("owner"), PolicyInputs.of(List.of(pdf("input", "input.pdf"))), PolicyProgressListener.NOOP, "src-s3-invoices", @@ -257,11 +253,136 @@ class PolicyEngineTest { any(), eq("src-s3-invoices"), eq("file-hash-1"), - any(), + isNull(), anyString(), any(Throwable.class)); } + @Test + void anAttendedFailureIsRecordedAgainstWhoTriggeredItNotThePolicysOwner() throws Exception { + // Bob runs Alice's shared policy on his own upload and it fails. The row must name Bob: he + // is the one whose browser holds the document, and a member's read scope narrows to their + // own rows, so filing it under Alice hides it from the only person who can act on it. + when(toolMetadataService.isMultiInput(ROTATE)).thenReturn(false); + when(internalApiClient.post(eq(ROTATE), any())).thenThrow(new RuntimeException("boom")); + + MDC.put("auditPrincipal", "bob"); // the request thread's acting user + try { + engine.runPolicy( + policyOwnedBy("alice"), + PolicyInputs.of(List.of(pdf("input", "input.pdf"))), + PolicyProgressListener.NOOP, + null, + "bob-doc-1") + .completion() + .get(10, TimeUnit.SECONDS); + } finally { + MDC.remove("auditPrincipal"); + } + + verify(failureRecorder) + .recordRunFailure( + anyString(), + any(), + isNull(), + eq("bob-doc-1"), + eq("bob"), + anyString(), + any(Throwable.class)); + } + + @Test + void anUnattendedFailureIsRecordedWithNoActorWhileStillBillingTheOwner() throws Exception { + // The two identities are deliberately different, and this pins both at once: usage is + // charged to the owner (MDC audit principal on the worker), but the failure has no actor, + // which is what makes it UNOWNED and hands the owner actions to the team's reviewer. + when(toolMetadataService.isMultiInput(ROTATE)).thenReturn(false); + String[] principalAtDispatch = {""}; + when(internalApiClient.post(eq(ROTATE), any())) + .thenAnswer( + invocation -> { + principalAtDispatch[0] = MDC.get("auditPrincipal"); + throw new RuntimeException("boom"); + }); + + // No MDC and no security context: exactly a trigger-fired sweep. + engine.runPolicy( + policyOwnedBy("alice"), + PolicyInputs.of(List.of(pdf("input", "input.pdf"))), + PolicyProgressListener.NOOP, + "src-watched-folder", + "file-hash-1") + .completion() + .get(10, TimeUnit.SECONDS); + + assertEquals("alice", principalAtDispatch[0], "billing must still be the policy owner"); + verify(failureRecorder) + .recordRunFailure( + anyString(), + any(), + eq("src-watched-folder"), + eq("file-hash-1"), + isNull(), + anyString(), + any(Throwable.class)); + } + + @Test + void anAdHocFailureIsRecordedAgainstTheSubmittingUser() throws Exception { + // An ad-hoc run has no stored policy, so the submitter is both payer and actor. Asserted so + // the two entry points cannot drift apart. + when(toolMetadataService.isMultiInput(ROTATE)).thenReturn(false); + when(internalApiClient.post(eq(ROTATE), any())).thenThrow(new RuntimeException("boom")); + + MDC.put("auditPrincipal", "bob"); + try { + engine.submit( + definition(new PipelineStep(ROTATE, Map.of())), + PolicyInputs.of(List.of(pdf("input", "input.pdf"))), + PolicyProgressListener.NOOP) + .completion() + .get(10, TimeUnit.SECONDS); + } finally { + MDC.remove("auditPrincipal"); + } + + verify(failureRecorder) + .recordRunFailure( + anyString(), + any(), + any(), + any(), + eq("bob"), + anyString(), + any(Throwable.class)); + } + + @Test + void aRunRefusedAtAdmissionIsRecordedAgainstWhoeverTriggeredIt() throws Exception { + // The queue-full path records its own row, and it is attended: the user is still holding + // the + // document, so it must reach them rather than landing as an ownerless incident. + when(resourceMonitor.shouldQueueJob(anyInt())).thenReturn(true); + CompletableFuture rejected = new CompletableFuture<>(); + rejected.completeExceptionally(new RuntimeException("Job queue full")); + doReturn(rejected).when(jobQueue).queueJob(anyString(), anyInt(), any(), anyLong()); + + MDC.put("auditPrincipal", "bob"); + try { + engine.runPolicy( + policyOwnedBy("alice"), + PolicyInputs.of(List.of(pdf("input", "input.pdf"))), + PolicyProgressListener.NOOP, + null, + "bob-doc-1"); + } finally { + MDC.remove("auditPrincipal"); + } + + verify(failureRecorder) + .recordRunFailureAs(any(), anyString(), any(), isNull(), eq("bob"), anyString()); + } + @Test void recordingAFailureNeverChangesTheRunsOutcome() throws Exception { // Recording is best-effort: losing the incident row is bad, but turning a classified @@ -540,6 +661,17 @@ class PolicyEngineTest { return new PipelineDefinition("test", List.of(steps), OutputSpec.inline()); } + private static Policy policyOwnedBy(String owner) { + return new Policy( + "p1", + "rotate", + owner, + true, + List.of(), + List.of(new PipelineStep(ROTATE, Map.of())), + OutputSpec.inline()); + } + private void stubEndpoint(String endpoint, Resource body) { when(internalApiClient.post(eq(endpoint), any())).thenReturn(ResponseEntity.ok(body)); } diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyRunRegistryTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyRunRegistryTest.java index 1eb0a1a1fe..692fe84c37 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyRunRegistryTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/policy/engine/PolicyRunRegistryTest.java @@ -101,6 +101,7 @@ class PolicyRunRegistryTest { null, new PipelineDefinition(runId, List.of(), List.of()), null, + null, null); registry.register(run); return run; From f15832b2bb1aec1aa8547ea774184e3c3296d855 Mon Sep 17 00:00:00 2001 From: ConnorYoh <40631091+ConnorYoh@users.noreply.github.com> Date: Mon, 17 Aug 2026 10:08:09 +0000 Subject: [PATCH 16/59] chore(saas): make schema ownership explicit and enforce it (#7489) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## The problem The SaaS database has two writers and always has: the Supabase migrations in the SaaS repo, and Hibernate's `ddl-auto`. That was a convention rather than a rule, and it leaked twice. - An older `ddl-auto` run widened `team_memberships.role` to varchar(255), which needed [a dedicated migration](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/blob/v3/supabase/migrations/20260804000000_fix_team_memberships_role_varchar50.sql) to repair, because RLS policies depended on the column. - `payg_instance_usage` shipped with an entity and **no migration**, and nobody noticed for months — staging already had the table from an earlier `ddl-auto` run. It surfaced only when a fresh preview branch, built from migrations alone, threw `relation does not exist`. Both are the same bug: nobody had to *say* who owned a table, so the answer got decided by accident. ## The fix `SaasSchemaOwnership` is the register — **29 migration-owned, 29 inherited** and left to Hibernate. `MigrationOwnedSchemaFilter` applies it via Hibernate's `hbm2ddl.schema_filter_provider`, wired on the **saas profile only**. Hibernate is never shown a migration-owned table, so it cannot create, alter, drop or truncate one whatever `ddl-auto` is set to. Inherited tables stay managed, so a fresh preview branch still heals itself on first boot. Self-hosted is untouched — there Hibernate rightly owns everything. **Why a filter rather than just `ddl-auto=none`:** off, and a fresh branch is missing the 29 inherited tables. On, and Hibernate can reach the other 29. The filter is what lets both be true at once. **Why per-table, not per-schema:** Hibernate's schema management runs over every mapped entity regardless of namespace. Moving SaaS tables to their own schema would *not* by itself keep Hibernate out of them — worth knowing, because that was the intuitive fix and it doesn't work. ## The part that makes it stick `SaasSchemaOwnershipTest` makes the register binding: every `@Entity` on the SaaS classpath must appear in exactly one set, so **a new entity fails the build until someone states who owns its table**. That's the forcing function that would have caught `payg_instance_usage`. I verified it bites rather than assuming it — removing a single entry fails with: ``` These entity tables are not declared in SaasSchemaOwnership, so nobody owns them. Offending tables -> entities: [policies (stirling.software.proprietary.policy.store.PolicyEntity)] ``` naming both the table and the class, which is what the next person actually needs. ## One debatable call The **validate** filter excludes them too. Letting validation through would flag drift, which is genuinely useful — but `ddl-auto=validate` fails startup, and it would fail on differences we've deliberately accepted (`ai_create_sessions` carries columns from a reverted Typst feature that nothing maps). A boot failure over a table we chose not to manage is noise. Argued in the javadoc; happy to flip it if you'd rather have the signal. ## Dependency Depends on [Stirling-PDF-SaaS#324](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/324), which adds migrations for the four SaaS-owned tables that had none. They're listed here as migration-owned on that basis, so #324 should land first. Companion to [#7483](https://github.com/Stirling-Tools/Stirling-PDF/pull/7483) (dev/staging profiles with per-profile `ddl-auto`). ## Verification `:saas:test` green including the 5 new tests, `spotlessCheck` green, and the mutation check above. --- .../config/MigrationOwnedSchemaFilter.java | 108 +++++++++++ .../saas/config/SaasSchemaOwnership.java | 117 ++++++++++++ .../resources/application-saas.properties | 7 + .../MigrationOwnedSchemaFilterTest.java | 129 ++++++++++++++ .../saas/config/SaasSchemaOwnershipTest.java | 167 ++++++++++++++++++ 5 files changed, 528 insertions(+) create mode 100644 app/saas/src/main/java/stirling/software/saas/config/MigrationOwnedSchemaFilter.java create mode 100644 app/saas/src/main/java/stirling/software/saas/config/SaasSchemaOwnership.java create mode 100644 app/saas/src/test/java/stirling/software/saas/config/MigrationOwnedSchemaFilterTest.java create mode 100644 app/saas/src/test/java/stirling/software/saas/config/SaasSchemaOwnershipTest.java diff --git a/app/saas/src/main/java/stirling/software/saas/config/MigrationOwnedSchemaFilter.java b/app/saas/src/main/java/stirling/software/saas/config/MigrationOwnedSchemaFilter.java new file mode 100644 index 0000000000..4b2496bf15 --- /dev/null +++ b/app/saas/src/main/java/stirling/software/saas/config/MigrationOwnedSchemaFilter.java @@ -0,0 +1,108 @@ +package stirling.software.saas.config; + +import org.hibernate.boot.model.relational.Namespace; +import org.hibernate.boot.model.relational.Sequence; +import org.hibernate.mapping.Table; +import org.hibernate.tool.schema.spi.SchemaFilter; +import org.hibernate.tool.schema.spi.SchemaFilterProvider; + +/** + * Hides the migration-owned tables from Hibernate's schema management. + * + *

Wired on the SaaS profile only, via {@code hibernate.hbm2ddl.schema_filter_provider}. + * Self-hosted is untouched: there Hibernate rightly owns everything. + * + *

Why a filter rather than simply turning {@code ddl-auto} off: the SaaS database has two + * writers. The Supabase migrations own the SaaS tables, and Hibernate owns roughly thirty tables + * inherited from the self-hosted app that no migration has ever created. Turn {@code ddl-auto} off + * and a fresh preview branch is missing that second half; leave it on and Hibernate is free to + * reconcile migration-owned tables, which is how {@code team_memberships.role} ended up widened to + * varchar(255) and needed a migration to put back. A filter keeps the first half working and makes + * the second impossible. + * + *

Note that this is a per-table filter, not a per-schema one. Hibernate's schema management runs + * over every mapped entity regardless of namespace, so moving SaaS tables to their own schema would + * not by itself keep Hibernate out of them. {@link SaasSchemaOwnership} is the register; this class + * only applies it. + * + *

Foreign keys still cross the line, on purpose. Several inherited tables reference + * migration-owned ones — {@code folders}, {@code stored_files} and {@code file_shares} all point at + * {@code users}/{@code teams}. Hibernate's {@code SchemaCreatorImpl.createForeignKeys} and {@code + * AbstractSchemaMigrator.applyForeignKeys} check {@code includeTable} against the *owning* table + * only and then emit every foreign key on it, without consulting the referenced table. So excluding + * {@code users} does not cost the branch its referential integrity, and a branch ends up matching + * staging. It does mean the referenced tables have to exist by the time Hibernate runs, which holds + * because a Supabase branch applies its migrations at build time and the app connects afterwards. + * + *

Known gap: this cannot detect drift. Filtering means Hibernate never inspects these + * tables, and {@link #getValidateFilter()} extends that to {@code validate}, so nothing here + * compares a migration-owned table against its entity. Combined with the register being a + * hand-maintained list of another repo's contents (see {@link SaasSchemaOwnership}), there is + * currently no automated signal when the register, the entities and the database disagree. That is + * a deliberate trade for a boot that does not fail on differences we accept, not a claim that drift + * cannot happen; a non-fatal drift report is the missing piece and belongs outside this class. + */ +public class MigrationOwnedSchemaFilter implements SchemaFilterProvider, SchemaFilter { + + /** + * The one decision this class makes. Everything Hibernate might do to a table it does not own — + * create, alter, drop, truncate — is refused. + */ + @Override + public boolean includeTable(Table table) { + return !SaasSchemaOwnership.isMigrationOwned(table.getName()); + } + + /** + * Namespaces are never filtered. The inherited tables and the migration-owned ones share {@code + * stirling_pdf}, so excluding the namespace would take both with it. + */ + @Override + public boolean includeNamespace(Namespace namespace) { + return true; + } + + /** + * Sequences are left alone. Every id here is an identity column rather than a mapped generator, + * so there is nothing for Hibernate to create; filtering them would be dead code pretending to + * be a safeguard. + */ + @Override + public boolean includeSequence(Sequence sequence) { + return true; + } + + @Override + public SchemaFilter getCreateFilter() { + return this; + } + + @Override + public SchemaFilter getMigrateFilter() { + return this; + } + + @Override + public SchemaFilter getDropFilter() { + return this; + } + + @Override + public SchemaFilter getTruncatorFilter() { + return this; + } + + /** + * Validation is filtered too, which is the one debatable call here. + * + *

Letting it through would give a useful signal when a migration-owned table drifts from its + * entity. But {@code ddl-auto=validate} fails startup, and it would fail on differences we have + * deliberately accepted — {@code ai_create_sessions} carries columns from a reverted feature + * that nothing maps, for instance. A boot failure over a table we have chosen not to manage is + * noise, so the rule stays uniform: Hibernate does not concern itself with these tables at all. + */ + @Override + public SchemaFilter getValidateFilter() { + return this; + } +} diff --git a/app/saas/src/main/java/stirling/software/saas/config/SaasSchemaOwnership.java b/app/saas/src/main/java/stirling/software/saas/config/SaasSchemaOwnership.java new file mode 100644 index 0000000000..dffd1eccf1 --- /dev/null +++ b/app/saas/src/main/java/stirling/software/saas/config/SaasSchemaOwnership.java @@ -0,0 +1,117 @@ +package stirling.software.saas.config; + +import java.util.Set; + +/** + * Which side owns each table in the SaaS database. + * + *

The SaaS schema has two writers and always has: the Supabase migrations in the + * Stirling-PDF-SaaS repo, and Hibernate's {@code ddl-auto}. That was a convention rather than a + * rule, and it leaked twice. An older {@code ddl-auto} run widened {@code team_memberships.role} to + * varchar(255), which needed a dedicated migration to repair because RLS policies depended on the + * column. Separately {@code payg_instance_usage} went months with an entity and no migration, so it + * simply did not exist on a fresh preview branch. + * + *

This class makes the boundary explicit and {@code SaasSchemaOwnershipTest} makes it binding: + * every {@code @Entity} the SaaS app maps must appear in exactly one of these two sets. A new + * entity fails the build until someone states who owns its table, which is the decision that was + * previously made by accident. + * + *

{@link MigrationOwnedSchemaFilter} enforces it at runtime: Hibernate is never shown the + * migration-owned tables, so it cannot create, alter or drop them whatever {@code ddl-auto} says. + * Inherited tables stay under Hibernate, so a preview branch built from migrations alone still + * heals itself on first boot. + * + *

What this does not catch. The register is a hand-maintained copy of what lives in + * another repository, and only one direction is enforced. The test fails when a *new* entity + * appears with no owner. It cannot notice a table changing sides: write a migration for {@code + * folders} in Stirling-PDF-SaaS and nothing here changes, the test still passes, and Hibernate + * carries on managing a table the migrations now own — which is precisely how {@code + * team_memberships.role} got widened. Adding a migration for anything in {@link #HIBERNATE_MANAGED} + * therefore means moving it to {@link #MIGRATION_OWNED} in the same change; nothing will remind + * you. Making that structural rather than remembered is what moving the SaaS tables into their own + * schema would buy, and is the reason this class is a stepping stone rather than the answer. + */ +public final class SaasSchemaOwnership { + + /** + * Created and altered by the Supabase migrations. Hibernate must not touch these: the + * migrations carry constraints, defaults and RLS policies it knows nothing about and would + * reconcile away. + */ + public static final Set MIGRATION_OWNED = + Set.of( + "ai_create_sessions", + "audit_events", + "authorities", + "billing_subscriptions", + "job_artifact_hash", + "legal_consent", + "linked_instance", + "payg_instance_usage", + "payg_meter_event_log", + "payg_prepaid_bundle", + "payg_shadow_charge", + "payg_team_extensions", + "persistent_logins", + "pricing_policy", + "processing_job", + "processing_job_step", + "procurement_agreement_signature", + "procurement_deal", + "procurement_quote", + "saas_team_extensions", + "saas_user_extensions", + "sessions", + "team_invitations", + "team_memberships", + "teams", + "users", + "wallet_entitlement_snapshot", + "wallet_ledger", + "wallet_policy"); + + /** + * Inherited from the self-hosted app, where {@code ddl-auto} owns the schema and no Supabase + * migration exists. Deliberately left under Hibernate so a fresh branch gets them on first + * boot. + */ + public static final Set HIBERNATE_MANAGED = + Set.of( + "account_link_device_credential", + "account_link_metered_signature", + "account_link_sync_state", + "account_link_usage_counter", + "api_key_daily_usage", + "api_keys", + "file_encryption_keys", + "file_run_events", + "file_share_accesses", + "file_shares", + "folders", + "integration_configs", + "invite_tokens", + "jwt_signing_keys", + "policies", + "policy_assets", + "policy_completed_migrations", + "policy_processed_files", + "policy_source_doc_counts", + "policy_source_doc_totals", + "policy_sources", + "resource_grants", + "storage_cleanup_entries", + "stored_file_blobs", + "stored_files", + "user_license_settings", + "user_server_certificates", + "workflow_participants", + "workflow_sessions"); + + private SaasSchemaOwnership() {} + + /** Case-insensitive: Hibernate hands us whatever casing the mapping used. */ + public static boolean isMigrationOwned(String tableName) { + return tableName != null && MIGRATION_OWNED.contains(tableName.toLowerCase()); + } +} diff --git a/app/saas/src/main/resources/application-saas.properties b/app/saas/src/main/resources/application-saas.properties index f0630aaac1..e3c32c429e 100644 --- a/app/saas/src/main/resources/application-saas.properties +++ b/app/saas/src/main/resources/application-saas.properties @@ -27,6 +27,13 @@ spring.jpa.properties.hibernate.hbm2ddl.create_namespaces=true spring.jpa.hibernate.ddl-auto=update +# ...but only over the tables Hibernate actually owns. The SaaS database has two writers: the +# Supabase migrations own the SaaS tables, Hibernate owns ~30 inherited from the self-hosted app that +# no migration has ever created. This filter hides the former from schema management, so ddl-auto can +# still heal a fresh preview branch without being free to reconcile a migration-owned table — which +# is how team_memberships.role ended up widened to varchar(255). Register: SaasSchemaOwnership. +spring.jpa.properties.hibernate.hbm2ddl.schema_filter_provider=stirling.software.saas.config.MigrationOwnedSchemaFilter + # ---------- Supabase JWT auth ---------- # Required: set SAAS_DB_PROJECT_REF via env. app.supabase.project-ref=${SAAS_DB_PROJECT_REF:} diff --git a/app/saas/src/test/java/stirling/software/saas/config/MigrationOwnedSchemaFilterTest.java b/app/saas/src/test/java/stirling/software/saas/config/MigrationOwnedSchemaFilterTest.java new file mode 100644 index 0000000000..956654384e --- /dev/null +++ b/app/saas/src/test/java/stirling/software/saas/config/MigrationOwnedSchemaFilterTest.java @@ -0,0 +1,129 @@ +package stirling.software.saas.config; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.InputStream; +import java.util.Properties; + +import org.hibernate.mapping.Table; +import org.hibernate.tool.schema.spi.SchemaFilter; +import org.hibernate.tool.schema.spi.SchemaFilterProvider; +import org.junit.jupiter.api.Test; + +/** + * Covers {@link MigrationOwnedSchemaFilter} and, just as importantly, its wiring. + * + *

{@link SaasSchemaOwnershipTest} proves the register is complete; nothing proved the filter + * applies it, or that Hibernate is even asking. A typo in the {@code + * hibernate.hbm2ddl.schema_filter_provider} key, a stale fully-qualified name after a package move, + * or a getter returning null would all leave every migration-owned table exposed to {@code + * ddl-auto} with a fully green build. Hence the property assertion below, which is the only thing + * here that would catch that. + */ +class MigrationOwnedSchemaFilterTest { + + private static final String FILTER_PROPERTY = + "spring.jpa.properties.hibernate.hbm2ddl.schema_filter_provider"; + + private final MigrationOwnedSchemaFilter filter = new MigrationOwnedSchemaFilter(); + + /** "orm" is Hibernate's own default contributor; the value is irrelevant to the filter. */ + private static Table table(String name) { + return new Table("orm", name); + } + + @Test + void migrationOwnedTablesAreHiddenFromHibernate() { + assertThat(filter.includeTable(table("teams"))).isFalse(); + assertThat(filter.includeTable(table("users"))).isFalse(); + assertThat(filter.includeTable(table("team_memberships"))).isFalse(); + assertThat(filter.includeTable(table("payg_instance_usage"))).isFalse(); + } + + @Test + void inheritedTablesStayUnderHibernate() { + assertThat(filter.includeTable(table("folders"))).isTrue(); + assertThat(filter.includeTable(table("stored_files"))).isTrue(); + assertThat(filter.includeTable(table("api_keys"))).isTrue(); + } + + @Test + void anUnknownTableIsLeftToHibernate() { + // Fail-open is the right default: an unrecognised table is either brand new or from a + // module + // we do not know about, and SaasSchemaOwnershipTest is what stops it staying unrecognised. + assertThat(filter.includeTable(table("no_such_table"))).isTrue(); + } + + @Test + void casingDoesNotDefeatTheFilter() { + assertThat(filter.includeTable(table("TEAMS"))).isFalse(); + assertThat(filter.includeTable(table("Team_Memberships"))).isFalse(); + } + + /** + * Foreign keys from an inherited table into a migration-owned one survive the filter. + * + *

Worth pinning, because it is not obvious and it decides whether a preview branch keeps + * referential integrity. {@code folders}, {@code stored_files} and {@code file_shares} all + * reference {@code users}/{@code teams}, which the filter excludes. Hibernate 7.2's {@code + * SchemaCreatorImpl.createForeignKeys} (and {@code AbstractSchemaMigrator.applyForeignKeys}) + * tests {@code includeTable} against the *owning* table only, then emits every foreign key on + * it; the referenced table is never consulted. So the constraints are still created and a + * branch matches staging. + * + *

The one thing this depends on is ordering: the referenced tables have to exist first. They + * do, because a Supabase branch runs its migrations at build time and the app connects after. + */ + @Test + void foreignKeysIntoMigrationOwnedTablesAreStillEmitted() { + assertThat(filter.includeTable(table("folders"))).isTrue(); + assertThat(filter.includeTable(table("file_shares"))).isTrue(); + assertThat(filter.includeTable(table("users"))).isFalse(); + assertThat(filter.includeTable(table("teams"))).isFalse(); + } + + @Test + void everySchemaActionGetsTheSameFilter() { + assertThat(filter.getCreateFilter()).isSameAs(filter); + assertThat(filter.getMigrateFilter()).isSameAs(filter); + assertThat(filter.getDropFilter()).isSameAs(filter); + assertThat(filter.getTruncatorFilter()).isSameAs(filter); + assertThat(filter.getValidateFilter()).isSameAs(filter); + } + + @Test + void namespacesAndSequencesAreNeverFiltered() { + // Both share the stirling_pdf namespace, so filtering it would take the inherited tables + // with it. Neither argument is read, so nulls are fine and keep the test free of Hibernate + // bootstrap machinery. + assertThat(filter.includeNamespace(null)).isTrue(); + assertThat(filter.includeSequence(null)).isTrue(); + } + + @Test + void theFilterIsActuallyWiredIntoHibernate() throws Exception { + Properties properties = new Properties(); + try (InputStream in = getClass().getResourceAsStream("/application-saas.properties")) { + assertThat(in) + .as("application-saas.properties must be on the test classpath to check wiring") + .isNotNull(); + properties.load(in); + } + + String configured = properties.getProperty(FILTER_PROPERTY); + assertThat(configured) + .as( + "%s is unset, so Hibernate installs its default filter and every" + + " migration-owned table is back under ddl-auto", + FILTER_PROPERTY) + .isNotBlank(); + + Class wired = Class.forName(configured.trim()); + assertThat(SchemaFilterProvider.class) + .as("Hibernate only accepts a SchemaFilterProvider here") + .isAssignableFrom(wired); + assertThat(SchemaFilter.class).isAssignableFrom(wired); + assertThat(wired).isEqualTo(MigrationOwnedSchemaFilter.class); + } +} diff --git a/app/saas/src/test/java/stirling/software/saas/config/SaasSchemaOwnershipTest.java b/app/saas/src/test/java/stirling/software/saas/config/SaasSchemaOwnershipTest.java new file mode 100644 index 0000000000..e07b1042e7 --- /dev/null +++ b/app/saas/src/test/java/stirling/software/saas/config/SaasSchemaOwnershipTest.java @@ -0,0 +1,167 @@ +package stirling.software.saas.config; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.Arrays; +import java.util.List; +import java.util.Set; +import java.util.TreeMap; +import java.util.TreeSet; + +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.config.BeanDefinition; +import org.springframework.boot.persistence.autoconfigure.EntityScan; +import org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider; +import org.springframework.core.type.filter.AnnotationTypeFilter; +import org.springframework.util.ClassUtils; + +import jakarta.persistence.Entity; +import jakarta.persistence.Table; + +import stirling.software.proprietary.security.configuration.DatabaseConfig; + +/** + * Makes {@link SaasSchemaOwnership} binding rather than decorative. + * + *

Every {@code @Entity} the SaaS app maps has to be declared as owned by either the Supabase + * migrations or Hibernate. Adding an entity without saying which fails here, at build time, instead + * of months later on a preview branch that has no such table. That is not hypothetical: {@code + * payg_instance_usage} shipped with an entity and no migration and went unnoticed until a branch + * tried to use it. + * + *

"Maps" is meant precisely: the scan covers the packages named by the {@code @EntityScan} + * declarations the app actually boots with, not everything under {@code stirling.software}. See + * {@link #mappedPackages()}. Note this only enforces one direction — {@link SaasSchemaOwnership} + * documents the drift it cannot see. + */ +class SaasSchemaOwnershipTest { + + /** + * The packages the running app actually maps, read off the two {@code @EntityScan} declarations + * that define them rather than hardcoded. + * + *

Scanning all of {@code stirling.software} would be easier and wrong in a quiet way: it is + * a superset, so it would force ownership declarations for entities Hibernate never sees and + * let the register claim tables that do not exist as far as the SaaS app is concerned. Deriving + * the list means this test measures the same set Hibernate does, and follows a package being + * added or moved without anyone updating it here. + */ + private static Set mappedPackages() { + Set packages = new TreeSet<>(); + for (Class config : List.of(SaasJpaConfig.class, DatabaseConfig.class)) { + EntityScan scan = config.getAnnotation(EntityScan.class); + assertThat(scan) + .as("%s must carry @EntityScan, or its entities are not mapped", config) + .isNotNull(); + packages.addAll(Arrays.asList(scan.value())); + } + return packages; + } + + private static TreeMap mappedTables() { + ClassPathScanningCandidateComponentProvider scanner = + new ClassPathScanningCandidateComponentProvider(false); + scanner.addIncludeFilter(new AnnotationTypeFilter(Entity.class)); + TreeMap byTable = new TreeMap<>(); + for (String basePackage : mappedPackages()) { + for (BeanDefinition bd : scanner.findCandidateComponents(basePackage)) { + String className = bd.getBeanClassName(); + Class type; + try { + type = + ClassUtils.forName( + className, SaasSchemaOwnershipTest.class.getClassLoader()); + } catch (ClassNotFoundException | LinkageError e) { + continue; // not on this module's runtime classpath; nothing to own + } + Table table = type.getAnnotation(Table.class); + String name = + table != null && !table.name().isBlank() + ? table.name() + : camelToSnake(type.getSimpleName()); + byTable.put(name.toLowerCase(), className); + } + } + return byTable; + } + + /** Mirrors Spring Boot's default CamelCaseToUnderscoresNamingStrategy for an unnamed @Table. */ + private static String camelToSnake(String name) { + return name.replaceAll("([a-z0-9])([A-Z])", "$1_$2").toLowerCase(); + } + + @Test + void everyEntityTableIsOwnedByExactlyOneSide() { + TreeMap mapped = mappedTables(); + assertThat(mapped) + .as("entity scan found nothing, so this test proves nothing") + .isNotEmpty(); + // The scan is derived from @EntityScan now, so a package quietly dropped from either + // declaration would shrink it and weaken this test rather than fail it. These four straddle + // the two declarations, so losing either side fails here instead of silently checking less. + assertThat(mapped.keySet()) + .as("both @EntityScan declarations must have contributed to the scan") + .contains("users", "teams", "payg_instance_usage", "folders"); + + Set undeclared = new TreeSet<>(); + Set both = new TreeSet<>(); + for (String table : mapped.keySet()) { + boolean migration = SaasSchemaOwnership.MIGRATION_OWNED.contains(table); + boolean hibernate = SaasSchemaOwnership.HIBERNATE_MANAGED.contains(table); + if (migration && hibernate) both.add(table); + if (!migration && !hibernate) undeclared.add(table); + } + + assertThat(undeclared) + .as( + """ + These entity tables are not declared in SaasSchemaOwnership, so nobody owns \ + them. Decide and add each to exactly one set: + - MIGRATION_OWNED: also add a migration in Stirling-PDF-SaaS, or the table \ + will not exist on a fresh preview branch. + - HIBERNATE_MANAGED: only correct for a table inherited from the \ + self-hosted app that no Supabase migration creates. + Offending tables -> entities: %s""" + .formatted( + undeclared.stream() + .map(t -> t + " (" + mapped.get(t) + ")") + .toList())) + .isEmpty(); + + assertThat(both) + .as("declared as owned by both sides, which is the one thing it cannot be") + .isEmpty(); + } + + @Test + void theTwoSetsDoNotOverlap() { + Set overlap = new TreeSet<>(SaasSchemaOwnership.MIGRATION_OWNED); + overlap.retainAll(SaasSchemaOwnership.HIBERNATE_MANAGED); + assertThat(overlap).isEmpty(); + } + + @Test + void tableNamesAreLowercaseSoLookupsCannotMiss() { + // isMigrationOwned() lowercases its input; a capital in either set would be unreachable. + assertThat(SaasSchemaOwnership.MIGRATION_OWNED) + .allSatisfy(t -> assertThat(t).isEqualTo(t.toLowerCase())); + assertThat(SaasSchemaOwnership.HIBERNATE_MANAGED) + .allSatisfy(t -> assertThat(t).isEqualTo(t.toLowerCase())); + } + + @Test + void migrationOwnedTablesIncludeTheOnesThatBitUs() { + // team_memberships is the table an old ddl-auto run widened; payg_instance_usage is the one + // that had an entity and no migration. Both must be on the migrations' side of the line. + assertThat(SaasSchemaOwnership.MIGRATION_OWNED) + .contains("team_memberships", "payg_instance_usage", "teams", "users"); + } + + @Test + void isMigrationOwnedIsCaseInsensitiveAndNullSafe() { + assertThat(SaasSchemaOwnership.isMigrationOwned("TEAM_MEMBERSHIPS")).isTrue(); + assertThat(SaasSchemaOwnership.isMigrationOwned("team_memberships")).isTrue(); + assertThat(SaasSchemaOwnership.isMigrationOwned(null)).isFalse(); + assertThat(SaasSchemaOwnership.isMigrationOwned("no_such_table")).isFalse(); + } +} From 99dda7a9f6160652050e26279f1ff72a0b1af123 Mon Sep 17 00:00:00 2001 From: James Brunton Date: Mon, 17 Aug 2026 11:47:08 +0000 Subject: [PATCH 17/59] Convert Processor tables to be consistent with each other (#7312) # Description of Changes Currently, the tables in the processor are inconsistently styled and function differently to each other because they don't all share the same table component. This is a big problem since the Processor is a largely table-based UI and most of most pages in it are tables. This PR creates a new common `DataTable` component and converts all existing tables in the Processor (other than those in the docs pages) to use it, tweaking the individual tables content as necessary so they all work the same. The `DataTable` component is designed to take minimal JSX as arguments. Instead, it takes typed data from the caller and it converts that into JSX, which makes it a lot easier to guarantee that the tables will all render the same and if we want to update the table styling in the future, we just need to do it in one place. I also added sorting capabilities to many of the tables since it was easy with the library and it was appropriate for many of the existing table columns. ## Table Comparisons AuditTab-comparison DocumentExtractions-comparison InstanceHealthTable-comparison Integrations-comparison InvoicesList-comparison LinkedInstancesTable-comparison PendingInvitations-comparison PipelinesTable-comparison PolicyCatalogue-comparison ReviewQueueTable-comparison SourcesTable-comparison UsersDirectory-comparison --- .../public/locales/en-US/translation.toml | 32 +- frontend/editor/src/core/theme/colors.css | 5 + frontend/editor/src/core/theme/dimensions.css | 1 + frontend/editor/src/core/ui/DataTable.css | 376 +++++++++ .../editor/src/core/ui/DataTable.stories.tsx | 262 +++++++ frontend/editor/src/core/ui/DataTable.tsx | 516 ++++++++++++ frontend/editor/src/core/ui/Dropdown.css | 12 +- frontend/editor/src/core/ui/Dropdown.tsx | 91 ++- frontend/editor/src/core/ui/Table.css | 73 -- frontend/editor/src/core/ui/Table.stories.tsx | 109 --- frontend/editor/src/core/ui/Table.tsx | 141 ---- .../editor/src/core/ui/dataTableColumns.tsx | 569 ++++++++++++++ frontend/editor/src/core/ui/index.ts | 2 +- frontend/editor/src/portal/api/documents.ts | 6 - .../account-link/LinkedInstancesTable.tsx | 128 ++- .../components/billing/InvoicesList.tsx | 143 ++-- .../documents/DocumentExtractions.stories.tsx | 34 +- .../documents/DocumentExtractions.tsx | 42 +- .../components/documents/ReviewQueueTable.tsx | 204 ++--- .../editor-admin/InstanceHealthTable.tsx | 102 +-- .../components/infrastructure/AuditTab.tsx | 130 ++- .../infrastructure/TableSkeleton.stories.tsx | 23 - .../infrastructure/TableSkeleton.tsx | 16 - .../components/infrastructure/infraFormat.ts | 9 - .../components/pipelines/PipelinesTable.tsx | 103 +-- .../policies/PolicyCatalogueTable.tsx | 134 ++-- .../components/sources/SourcesTable.tsx | 119 +-- .../users/PendingInvitations.test.tsx | 2 +- .../components/users/PendingInvitations.tsx | 95 ++- .../users/UsersDirectory.stories.tsx | 3 +- .../components/users/UsersDirectory.tsx | 740 ++++++++---------- .../src/portal/views/Integrations.test.tsx | 25 +- .../editor/src/portal/views/Integrations.tsx | 401 +++++----- .../src/portal/views/Users.saas.test.tsx | 2 +- frontend/editor/src/portal/views/Users.tsx | 56 +- frontend/package-lock.json | 65 ++ frontend/package.json | 1 + 37 files changed, 2983 insertions(+), 1789 deletions(-) create mode 100644 frontend/editor/src/core/ui/DataTable.css create mode 100644 frontend/editor/src/core/ui/DataTable.stories.tsx create mode 100644 frontend/editor/src/core/ui/DataTable.tsx delete mode 100644 frontend/editor/src/core/ui/Table.css delete mode 100644 frontend/editor/src/core/ui/Table.stories.tsx delete mode 100644 frontend/editor/src/core/ui/Table.tsx create mode 100644 frontend/editor/src/core/ui/dataTableColumns.tsx delete mode 100644 frontend/editor/src/portal/components/infrastructure/TableSkeleton.stories.tsx delete mode 100644 frontend/editor/src/portal/components/infrastructure/TableSkeleton.tsx diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml index 4d5e271103..99d68c35de 100644 --- a/frontend/editor/public/locales/en-US/translation.toml +++ b/frontend/editor/public/locales/en-US/translation.toml @@ -3010,6 +3010,7 @@ saturation = "Saturation and brightness" title = "Choose color" [common] +actions = "Actions" back = "Back" cancel = "Cancel" close = "Close" @@ -3025,6 +3026,7 @@ error = "Error" expand = "Expand" loading = "Loading..." next = "Next" +open = "Open" preview = "Preview" previous = "Previous" refresh = "Refresh" @@ -6339,7 +6341,6 @@ revoked = "Revoked" unnamed = "Unnamed instance" [portal.accountLink.instances.columns] -actions = "Actions" instance = "Instance" lastSeen = "Last seen" linked = "Linked" @@ -7262,12 +7263,11 @@ editorAction = "Editor" empty = "No documents match this filter." rowActions = "Row actions" sensitiveLabel = "Sensitive" -sensitiveTitle = "Sensitive — access required" [portal.documents.table.columns] action = "Pipeline / Action" -actions = "Actions" document = "Document" +labels = "Labels" product = "Product" status = "Status" time = "Time" @@ -7283,6 +7283,7 @@ host = "Host" lastSeen = "Last seen" region = "Region" status = "Status" +target = "Target" version = "Version" [portal.editorAdmin.health.empty] @@ -7602,11 +7603,9 @@ security = "Security" storage = "Storage" [portal.integrations] -addAnother = "Add another" availableHeading = "Available" comingSoonHeading = "Coming soon" connect = "Connect" -connectedHeading = "Connected" connectionCount_one = "{{count}} connection" connectionCount_other = "{{count}} connections" customApi = "Custom API" @@ -7624,6 +7623,10 @@ security = "Security" signing = "Signing" storage = "Storage" +[portal.integrations.noResults] +description = "No integrations match your filters. Try a different category or search." +title = "No matches" + [portal.integrations.status] connected = "Connected" @@ -7826,10 +7829,10 @@ paused = "Paused" [portal.pipelines.table] name = "Pipeline" -open = "Open" sources = "Sources" status = "Status" steps = "Steps" +trigger = "Trigger" [portal.pipelines.trigger] folder-watch = "Folder watch" @@ -8722,9 +8725,9 @@ unused = "Unused" [portal.sources.table] documents = "Documents" -open = "Open" source = "Source" status = "Status" +type = "Type" usedBy = "Policies" [portal.sources.types.box] @@ -10948,6 +10951,13 @@ approver = "Approves policy" editor = "Editor" processor = "Processor" +[users.columns] +capabilities = "Capabilities" +email = "Email" +person = "Person" +role = "Role" +status = "Status" + [users.confirm] cancelInviteBody = "Cancel the invitation to {{email}}? They won't be able to join with the current link." cancelInviteTitle = "Cancel invitation" @@ -10967,10 +10977,8 @@ title = "No members yet" addToTeam = "Add to team" guestCount = "{{count}} guest" guests = "Guests" -guestsDesc = "External collaborators, scoped to what you shared. Editor only." ledBy = "led by {{owner}}" org = "Organization" -orgDesc = "Owners with org-wide authority and policy approval" owners = "{{count}} owner" team = "{{name}} team" teamMeta = "{{count}} people" @@ -11010,13 +11018,15 @@ usernamePlaceholder = "jsmith" [users.invites] by = "Invited by {{who}}" cancel = "Cancel" -count = "{{count}} pending" -desc = "Invited people who haven't joined yet. They hold a seat until they accept." expiresInDays_one = "Expires in {{count}} day" expiresInDays_other = "Expires in {{count}} days" expiresToday = "Expires today" title = "Pending invitations" +[users.invites.columns] +expires = "Expires" +invitee = "Invitee" + [users.loadError] description = "Something went wrong reaching the backend, or you don't have access. Try again." title = "Couldn't load members" diff --git a/frontend/editor/src/core/theme/colors.css b/frontend/editor/src/core/theme/colors.css index 3caf240da4..dffd6beaba 100644 --- a/frontend/editor/src/core/theme/colors.css +++ b/frontend/editor/src/core/theme/colors.css @@ -32,6 +32,9 @@ html[data-app-theme="light"] { --c-primary: var(--p-blue-500); --c-primary-hover: var(--p-blue-600); --c-primary-subtle: color-mix(in srgb, var(--p-blue-500) 10%, transparent); + /* Accent used as TEXT (links, toggles): a deeper hue that clears 4.5:1 on + light surfaces, where --c-primary itself does not. */ + --c-accent-text: var(--p-blue-700); --c-success: var(--p-green-600); --c-danger: var(--p-red-600); @@ -144,6 +147,8 @@ html[data-app-theme="midnight"] { --c-text-muted: var(--p-zinc-200); --c-text-subtle: var(--p-zinc-250); --c-text-on-primary: var(--p-white); + /* Accent used as TEXT on dark surfaces: a lighter step for 4.5:1. */ + --c-accent-text: var(--p-blue-400); --c-btn-solid: var(--c-text); --c-btn-inverse: var(--p-ink); --c-btn-secondary: var(--p-c-1a1a1d); diff --git a/frontend/editor/src/core/theme/dimensions.css b/frontend/editor/src/core/theme/dimensions.css index aa0a1b38cc..554baddcca 100644 --- a/frontend/editor/src/core/theme/dimensions.css +++ b/frontend/editor/src/core/theme/dimensions.css @@ -62,6 +62,7 @@ /* ── Z-index ladder ── */ --z-dropdown: 25; --z-drawer: 50; + --z-popover: 150; --z-toast: 200; /* Fullscreen tool-picker surfaces (editor) */ --z-fullscreen-icon-svg: 1; diff --git a/frontend/editor/src/core/ui/DataTable.css b/frontend/editor/src/core/ui/DataTable.css new file mode 100644 index 0000000000..c7572efa90 --- /dev/null +++ b/frontend/editor/src/core/ui/DataTable.css @@ -0,0 +1,376 @@ +/* + * DataTable: the one Stirling table. Appearance is owned entirely here: the + * canonical surface, the header/row grammar, the standardized loading / empty / + * error states, and every cell KIND (`sui-dtc__*`). Call-sites choose a kind and + * supply data; they never style a cell. This is what lets every table look and + * behave the same. + */ + +.sui-datatable { + width: 100%; +} + +/* The one canonical surface: every table wears it; call-sites don't wrap it. */ +.sui-datatable__frame { + border: 1px solid var(--c-border); + border-radius: var(--radius-md); + background: var(--c-surface); + overflow: hidden; +} + +.sui-datatable__toolbar { + display: flex; + align-items: center; + gap: 0.75rem; + flex-wrap: wrap; + padding: 0.75rem 0.875rem; + border-bottom: 1px solid var(--c-border); +} + +.sui-datatable__scroll { + width: 100%; + overflow-x: auto; +} + +.sui-datatable__table { + width: 100%; + border-collapse: collapse; + font-size: 0.8125rem; +} + +.sui-datatable__caption { + caption-side: top; + text-align: left; + padding: 0.625rem 0.875rem; + color: var(--c-text-subtle); + font-size: 0.75rem; +} + +/* Header */ +.sui-datatable__th { + text-align: left; + font-weight: 600; + color: var(--c-text-subtle); + font-size: 0.6875rem; + text-transform: uppercase; + letter-spacing: 0.04em; + padding: 0.625rem 0.875rem; + border-bottom: 1px solid var(--c-border); + white-space: nowrap; +} +.sui-datatable__th--right { + text-align: right; +} +.sui-datatable__th--fit { + width: 1%; +} +/* Accessible-only header text for blank affordance/action columns. */ +.sui-datatable__th-sr { + position: absolute; + width: 1px; + height: 1px; + padding: 0; + margin: -1px; + overflow: hidden; + clip: rect(0, 0, 0, 0); + white-space: nowrap; + border: 0; +} + +.sui-datatable__sort { + display: inline-flex; + align-items: center; + gap: 0.25rem; + background: none; + border: none; + padding: 0; + margin: 0; + cursor: pointer; + font: inherit; + letter-spacing: inherit; + text-transform: inherit; + color: inherit; +} +.sui-datatable__th--right .sui-datatable__sort { + flex-direction: row-reverse; +} +.sui-datatable__sort:hover { + color: var(--c-text); +} +.sui-datatable__sort:focus-visible { + outline: 0.125rem solid var(--c-primary); + outline-offset: 0.125rem; + border-radius: var(--radius-sm); +} +.sui-datatable__sort-icon { + display: inline-flex; + color: var(--c-text-subtle); + opacity: 0; + transition: opacity var(--motion-fast); +} +.sui-datatable__sort:hover .sui-datatable__sort-icon { + opacity: 0.5; +} +.sui-datatable__sort-icon--asc, +.sui-datatable__sort-icon--desc { + opacity: 1; + color: var(--c-primary); +} +.sui-datatable__sort-icon--desc svg { + transform: rotate(180deg); +} + +/* Body */ +.sui-datatable__td { + padding: 0.625rem 0.875rem; + color: var(--c-text-muted); + border-bottom: 1px solid var(--c-border-subtle); + vertical-align: middle; +} +.sui-datatable__table tbody tr:last-child .sui-datatable__td { + border-bottom: none; +} +.sui-datatable__td--right { + text-align: right; +} +.sui-datatable__td--nowrap { + white-space: nowrap; +} +.sui-datatable__td--fit { + width: 1%; + white-space: nowrap; +} + +/* Density (the compact variant) */ +.sui-datatable--compact .sui-datatable__th, +.sui-datatable--compact .sui-datatable__td { + padding: 0.375rem 0.625rem; +} + +/* Interactive rows */ +.sui-datatable__row--interactive { + cursor: pointer; + transition: background var(--motion-fast); +} +.sui-datatable__row--interactive:hover { + background: var(--c-hover); +} +.sui-datatable__row--interactive:focus-visible { + outline: 0.125rem solid var(--c-primary); + outline-offset: -0.125rem; +} + +/* Non-actionable rows (e.g. coming-soon): de-emphasized to read as disabled. + Muting is done with accessible colours + a faded (non-text) icon rather than + row opacity, which would blend text under the WCAG contrast floor. */ +.sui-datatable__row--muted .sui-dtc__entity-name { + color: var(--c-text-muted); + font-weight: 500; +} +.sui-datatable__row--muted .sui-dtc__entity-icon { + opacity: 0.4; +} + +.sui-datatable__chevron { + display: inline-flex; + color: var(--c-text-subtle); +} + +/* Full-width state cell (empty / error) */ +.sui-datatable__state { + padding: 2rem; + text-align: center; + color: var(--c-text-subtle); +} +.sui-datatable__state--error { + color: var(--c-danger); +} +.sui-datatable__state--node { + padding: 0; +} + +/* + * Cell kinds (sui-dtc__*): the locked cell vocabulary. These absorb the + * per-namespace portal-*__ cell styles (mono / muted / cell-stack / name-cell / + * caret …) that every table used to re-implement. + */ +.sui-dtc__text { + color: var(--c-text-muted); +} +/* Labeled value ("Category: action"): full-strength text, bold label. */ +.sui-dtc__text--labeled { + color: var(--c-text); +} +.sui-dtc__text-label { + font-weight: 600; +} +.sui-dtc__mono { + font-family: var(--font-mono); + font-size: 0.75rem; + color: var(--c-text-muted); +} +.sui-dtc__muted { + color: var(--c-text-subtle); +} +.sui-dtc__num { + font-variant-numeric: tabular-nums; +} +.sui-dtc__labels { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: 0.375rem; +} + +/* entity: leading icon + bold name + optional muted note */ +.sui-dtc__entity { + display: flex; + align-items: center; + gap: 0.625rem; + min-width: 0; +} +.sui-dtc__entity-icon { + display: inline-flex; + align-items: center; + flex: none; + color: var(--c-text-muted); +} +/* The component sizes bare icons; self-styled marks (Avatar) render as-is. */ +.sui-dtc__entity-icon svg { + width: 1.2rem; + height: 1.2rem; +} +.sui-dtc__entity-body { + display: flex; + flex-direction: column; + gap: 0.1875rem; + min-width: 0; +} +.sui-dtc__entity-head { + display: flex; + align-items: baseline; + flex-wrap: wrap; + gap: 0.25rem; +} +.sui-dtc__entity-name { + font-weight: 600; + color: var(--c-text); +} +.sui-dtc__entity-suffix { + font-size: 0.75rem; + font-weight: 400; + color: var(--c-text-subtle); +} +.sui-dtc__note { + font-size: 0.75rem; + color: var(--c-text-subtle); +} + +.sui-dtc__actions { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 0.25rem; +} +.sui-dtc__menu-item--danger { + color: var(--c-danger); +} + +/* Grouped tables: section header row + per-group "show all" toggle. */ +.sui-datatable__group-cell { + padding: 0.5rem 0.875rem; + background: var(--c-surface-sunken, var(--c-hover)); + border-bottom: 1px solid var(--c-border); +} +.sui-datatable__table + tbody + tr.sui-datatable__group:not(:first-child) + .sui-datatable__group-cell { + border-top: 1px solid var(--c-border); +} +.sui-datatable__group-head { + display: flex; + align-items: center; + justify-content: space-between; + gap: 0.75rem; +} +.sui-datatable__group-title { + display: flex; + align-items: baseline; + flex-wrap: wrap; + gap: 0.5rem; +} +.sui-datatable__group-title strong { + font-size: 0.8125rem; + color: var(--c-text); +} +.sui-datatable__group-meta { + font-size: 0.75rem; + color: var(--c-text-subtle); +} +.sui-datatable__group-more { + padding: 0.5rem 0.875rem; +} +.sui-datatable__show-all { + background: none; + border: none; + padding: 0; + cursor: pointer; + font: inherit; + font-size: 0.75rem; + color: var(--c-accent-text); +} +.sui-datatable__show-all:hover { + text-decoration: underline; +} + +.sui-dtc__progress { + display: flex; + align-items: center; + gap: 0.5rem; +} +.sui-dtc__progress-bar { + display: block; + width: 5rem; +} +.sui-dtc__progress-pct { + font-variant-numeric: tabular-nums; + color: var(--c-text-muted); + min-width: 2.5rem; +} + +.sui-dtc__links { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 0.75rem; +} +/* Neutral chip-style link (matches the invoice-link look): the colour is not an + * accent - the trailing glyph + hover fill carry the affordance. */ +.sui-dtc__link { + display: inline-flex; + align-items: center; + gap: 0.25rem; + padding: 0.25rem 0.5rem; + border-radius: var(--radius-sm); + color: var(--c-text); + text-decoration: none; + font-weight: 500; + border: 1px solid transparent; + transition: + background var(--motion-fast), + border-color var(--motion-fast); +} +.sui-dtc__link:hover { + background: var(--c-surface-sunken); + border-color: var(--c-border); +} +.sui-dtc__link:focus-visible { + outline: none; + border-color: var(--c-primary); + background: var(--c-surface-sunken); +} + +.sui-dtc__select { + min-width: 13rem; +} diff --git a/frontend/editor/src/core/ui/DataTable.stories.tsx b/frontend/editor/src/core/ui/DataTable.stories.tsx new file mode 100644 index 0000000000..630537ace9 --- /dev/null +++ b/frontend/editor/src/core/ui/DataTable.stories.tsx @@ -0,0 +1,262 @@ +import { useState } from "react"; +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { Button } from "@app/ui/Button"; +import { column, DataTable, type DataTableColumn } from "@app/ui/DataTable"; + +interface Region { + id: string; + name: string; + code: string; + status: "healthy" | "degraded"; + docs: number; + latency: number; + auto: boolean; +} + +const REGIONS: Region[] = [ + { + id: "1", + name: "US East", + code: "us-east-1", + status: "healthy", + docs: 12481, + latency: 41, + auto: true, + }, + { + id: "2", + name: "US West", + code: "us-west-2", + status: "healthy", + docs: 8210, + latency: 63, + auto: false, + }, + { + id: "3", + name: "EU West", + code: "eu-west-1", + status: "degraded", + docs: 3044, + latency: 190, + auto: true, + }, + { + id: "4", + name: "AP South", + code: "ap-south-1", + status: "healthy", + docs: 5622, + latency: 88, + auto: false, + }, +]; + +const tone = (r: Region) => ({ + tone: r.status === "healthy" ? ("success" as const) : ("warning" as const), + label: r.status, +}); + +const COLUMNS: DataTableColumn[] = [ + column.entity({ + key: "name", + header: "Region", + primary: (r) => r.name, + }), + column.mono({ key: "code", header: "Code", get: (r) => r.code }), + column.badge({ key: "status", header: "Status", get: tone }), + column.number({ + key: "docs", + header: "Docs 24h", + get: (r) => r.docs, + format: (n) => n.toLocaleString(), + }), + column.number({ + key: "latency", + header: "P95", + get: (r) => r.latency, + format: (n) => `${n} ms`, + }), +]; + +const SORTABLE_COLUMNS: DataTableColumn[] = [ + column.entity({ + key: "name", + header: "Region", + primary: (r) => r.name, + sortable: true, + }), + column.mono({ + key: "code", + header: "Code", + get: (r) => r.code, + sortable: true, + }), + column.badge({ key: "status", header: "Status", get: tone, sortable: true }), + column.number({ + key: "docs", + header: "Docs 24h", + get: (r) => r.docs, + format: (n) => n.toLocaleString(), + sortable: true, + }), + column.number({ + key: "latency", + header: "P95", + get: (r) => r.latency, + format: (n) => `${n} ms`, + sortable: true, + }), +]; + +const meta: Meta = { + title: "Compound/DataTable", + component: DataTable, + tags: ["autodocs"], + parameters: { layout: "padded" }, +}; +export default meta; +type Story = StoryObj; + +/** Columns come from the `column` vocabulary; call-sites never style a cell. */ +export const Basic: Story = { + render: () => ( + columns={COLUMNS} rows={REGIONS} rowKey={(r) => r.id} /> + ), +}; + +/** Sorting is opt-in per column (`sortable: true`); click a header. */ +export const Sortable: Story = { + render: () => ( + + columns={SORTABLE_COLUMNS} + rows={REGIONS} + rowKey={(r) => r.id} + defaultSort={{ key: "docs", direction: "desc" }} + /> + ), +}; + +/** `onRowClick` + a chevron affordance. The line above shows the click land. */ +export const Interactive: Story = { + render: () => { + function Bound() { + const [clicked, setClicked] = useState(null); + return ( + <> +

+ {clicked + ? `Clicked: ${clicked.name} (${clicked.code})` + : "Click a row to fire onRowClick."} +

+ + columns={COLUMNS} + rows={REGIONS} + rowKey={(r) => r.id} + onRowClick={setClicked} + rowAffordance="chevron" + /> + + ); + } + return ; + }, +}; + +/** A trailing action column (icon-only kebab), locked to the design system. */ +export const WithActions: Story = { + render: () => ( + + columns={[ + ...COLUMNS, + column.actions({ + key: "actions", + get: () => [ + { + label: "Row actions", + glyph: "kebab", + iconOnly: true, + onClick: () => {}, + }, + ], + }), + ]} + rows={REGIONS} + rowKey={(r) => r.id} + /> + ), +}; + +/** First-load skeleton mirrors the real column layout. */ +export const Loading: Story = { + render: () => ( + + columns={COLUMNS} + rows={[]} + rowKey={(r) => r.id} + loading + /> + ), +}; + +/** Standardized empty slot. */ +export const Empty: Story = { + render: () => ( + + columns={COLUMNS} + rows={[]} + rowKey={(r) => r.id} + empty="No regions deployed yet." + /> + ), +}; + +/** Standardized error slot (announced as an alert). */ +export const ErrorState: Story = { + render: () => ( + + columns={COLUMNS} + rows={[]} + rowKey={(r) => r.id} + error="Couldn't load regions. Try again." + /> + ), +}; + +/** Optional toolbar slot above the table, inside the surface. */ +export const WithToolbar: Story = { + render: () => ( + + columns={COLUMNS} + rows={REGIONS} + rowKey={(r) => r.id} + toolbar={ + <> + Regions + + + + } + /> + ), +}; + +/** The one look choice: the `compact` variant. */ +export const Compact: Story = { + render: () => ( + + columns={COLUMNS} + rows={REGIONS} + rowKey={(r) => r.id} + variant="compact" + /> + ), +}; diff --git a/frontend/editor/src/core/ui/DataTable.tsx b/frontend/editor/src/core/ui/DataTable.tsx new file mode 100644 index 0000000000..ccb5578016 --- /dev/null +++ b/frontend/editor/src/core/ui/DataTable.tsx @@ -0,0 +1,516 @@ +import { type KeyboardEvent, type ReactNode, useMemo, useState } from "react"; +import { + type ColumnDef, + createColumnHelper, + createSortedRowModel, + flexRender, + type RowData, + rowSortingFeature, + sortFn_alphanumeric, + sortFn_basic, + type SortingState, + tableFeatures, + useTable, +} from "@tanstack/react-table"; +import { useTranslation } from "react-i18next"; +import { Skeleton } from "@app/ui/Skeleton"; +import { + type CellAction, + type DataTableColumn, + renderCellActions, +} from "@app/ui/dataTableColumns"; +import "@app/ui/DataTable.css"; + +export * from "@app/ui/dataTableColumns"; + +/** Per-column presentation carried through TanStack's typed `meta` slot. */ +interface ColumnMeta { + align: "left" | "right"; + nowrap: boolean; + fit: boolean; + /** Visually-hidden header text for blank affordance/action columns, so the + * column still has an accessible name (avoids axe `empty-table-header`). */ + srHeader?: string; +} + +/** + * Feature registry for every DataTable, built once. Sorting is always + * registered so any column can opt in; the core row model defaults in. + */ +const DATA_TABLE_FEATURES = tableFeatures({ + rowSortingFeature, + sortedRowModel: createSortedRowModel(), + columnMeta: {} as ColumnMeta, + // Register the comparators the column vocabulary uses. Without this v9 falls + // back to a case-sensitive `basic` sort and warns per column. + sortFns: { alphanumeric: sortFn_alphanumeric, basic: sortFn_basic }, +}); +type DataTableFeatures = typeof DATA_TABLE_FEATURES; + +/** Closed appearance dial — the only look choice a call-site may make. */ +export type DataTableVariant = "default" | "compact"; + +/** + * A collapsible section of rows under a locked header. Group headers are + * structured (title + muted meta + optional right-aligned actions), never raw + * markup, so grouped tables stay as opinionated as flat ones. Provide `groups` + * instead of `rows`. + */ +export interface DataTableGroup { + key: string; + title: string; + /** Muted sub-text on the header (e.g. "5 people · led by Dana"). */ + meta?: string; + /** Right-aligned header actions (e.g. "Add to team", a kebab menu). */ + actions?: CellAction[]; + rows: T[]; + /** Collapse rows past this count behind a "Show all N" toggle. */ + collapseAfter?: number; + /** Render the group's rows greyed/disabled (non-actionable, e.g. coming-soon). */ + muted?: boolean; +} + +export interface DataTableProps { + /** Columns built with the `column` vocabulary — never raw JSX. */ + columns: DataTableColumn[]; + /** Flat rows. Provide this OR `groups`, not both. */ + rows?: T[]; + /** Grouped rows with section headers. Takes precedence over `rows`. */ + groups?: DataTableGroup[]; + rowKey: (row: T) => string; + + /** Makes rows interactive (hover + click + keyboard). */ + onRowClick?: (row: T) => void; + /** Per-row interactivity gate, checked only when `onRowClick` is set. */ + isRowInteractive?: (row: T) => boolean; + /** Trailing affordance drawn on interactive rows. */ + rowAffordance?: "none" | "chevron"; + + /** Initial sort, applied to the matching sortable column. */ + defaultSort?: { key: string; direction?: "asc" | "desc" }; + + /** First-load state: renders column-shaped skeleton rows under the header. */ + loading?: boolean; + /** Skeleton row count while loading. Defaults to 6. */ + skeletonRows?: number; + /** Error slot — replaces the rows with an alert message row. */ + error?: ReactNode; + /** Shown when there are no rows (and not loading / no error). Text or a node. */ + empty?: ReactNode; + + /** Content above the table (filters, search, actions), inside the surface. */ + toolbar?: ReactNode; + /** The only appearance choice. */ + variant?: DataTableVariant; + /** Accessible caption for the table. */ + caption?: string; + /** Labels for a group's "show all / show less" toggle (pass translated). + * `showAll` receives the group's total row count. */ + collapseLabels?: { showAll: (total: number) => string; showLess: string }; +} + +function ChevronGlyph() { + return ( + + + + ); +} + +function SortGlyph() { + return ( + + + + ); +} + +const CHEVRON_COLUMN_KEY = "__affordance"; + +/** + * The shared Stirling table. Call-sites supply data + behaviour; the component + * owns 100% of the appearance. Columns come from the `column` vocabulary (typed + * cell kinds, no raw markup), the surface / density / states are standardized + * here, and the only look choice exposed is the closed `variant`. Behaviour - + * sorting today, more later - is opt-in per column or via props. + */ +export function DataTable({ + columns, + rows = [], + groups, + rowKey, + onRowClick, + isRowInteractive, + rowAffordance = "none", + defaultSort, + loading = false, + skeletonRows = 6, + error, + empty, + toolbar, + variant = "default", + caption, + collapseLabels = { + showAll: (n) => `Show all ${n}`, + showLess: "Show less", + }, +}: DataTableProps) { + const { t } = useTranslation(); + const [sorting, setSorting] = useState( + defaultSort + ? [{ id: defaultSort.key, desc: defaultSort.direction === "desc" }] + : [], + ); + const [openGroups, setOpenGroups] = useState>(new Set()); + const toggleGroup = (key: string) => + setOpenGroups((prev) => { + const next = new Set(prev); + if (next.has(key)) next.delete(key); + else next.add(key); + return next; + }); + + // The data source is either grouped or flat; TanStack (headers, sorting for + // the flat path) is fed the flattened rows. + const flatRows = useMemo( + () => (groups ? groups.flatMap((g) => g.rows) : rows), + [groups, rows], + ); + + const interactive = Boolean(onRowClick); + const showChevron = interactive && rowAffordance === "chevron"; + // A row that holds its own controls (actions/links/select/caps) can't also be + // a `role="button"` (a button may not contain interactive descendants); it + // keeps the click as a mouse shortcut, and the inner control is the keyboard path. + const rowsContainControls = columns.some((c) => c.interactive); + + const effectiveColumns = useMemo[]>(() => { + if (!showChevron) return columns; + return [ + ...columns, + { + key: CHEVRON_COLUMN_KEY, + header: "", + align: "right", + nowrap: true, + fit: true, + sortable: false, + renderCell: (row) => + (isRowInteractive?.(row) ?? true) ? ( + + + + ) : null, + }, + ]; + }, [columns, showChevron, isRowInteractive]); + + const tanstackColumns = useMemo[]>(() => { + const helper = createColumnHelper(); + return effectiveColumns.map((c) => { + // A blank header (trailing affordance/action columns) still needs an + // accessible name for assistive tech. + const srHeader = c.header + ? undefined + : c.key === CHEVRON_COLUMN_KEY + ? t("common.open", "Open") + : t("common.actions", "Actions"); + const meta: ColumnMeta = { + align: c.align, + nowrap: c.nowrap, + fit: c.fit, + srHeader, + }; + if (c.sortable && c.sortValue) { + const sortValue = c.sortValue; + return helper.accessor((row: T): unknown => sortValue(row), { + id: c.key, + header: () => c.header, + cell: (ctx) => c.renderCell(ctx.row.original), + enableSorting: true, + sortUndefined: "last", + sortFn: c.sortFn ?? "basic", + meta, + }); + } + return helper.display({ + id: c.key, + header: () => c.header, + cell: (ctx) => c.renderCell(ctx.row.original), + meta, + }); + }); + }, [effectiveColumns, t]); + + const table = useTable({ + features: DATA_TABLE_FEATURES, + data: flatRows, + columns: tanstackColumns, + state: { sorting }, + onSortingChange: setSorting, + getRowId: (row) => rowKey(row), + }); + + const colCount = effectiveColumns.length; + + // Shared row wiring so grouped rows behave like flat ones (interactivity + + // the affordance column) instead of being a second-class path. + const rowProps = (original: T, muted?: boolean) => { + const rowInteractive = + interactive && (isRowInteractive?.(original) ?? true); + // A row that owns the whole interaction takes the button role + keyboard + // handling; a row with its own controls keeps just the mouse click. + const asButton = rowInteractive && !rowsContainControls; + return { + className: [ + "sui-datatable__row", + rowInteractive ? "sui-datatable__row--interactive" : "", + muted ? "sui-datatable__row--muted" : "", + ] + .filter(Boolean) + .join(" "), + onClick: rowInteractive ? () => onRowClick?.(original) : undefined, + tabIndex: asButton ? 0 : undefined, + role: asButton ? ("button" as const) : undefined, + onKeyDown: asButton + ? (e: KeyboardEvent) => { + if (e.key === "Enter" || e.key === " ") { + e.preventDefault(); + onRowClick?.(original); + } + } + : undefined, + }; + }; + + // No rows at all - covers a grouped table whose groups are all empty (or an + // empty groups list), which would otherwise render a header-only table. + const noRows = groups + ? groups.every((g) => g.rows.length === 0) + : rows.length === 0; + + let body: ReactNode; + if (loading) { + body = Array.from({ length: skeletonRows }).map((_, r) => ( + + {effectiveColumns.map((c) => ( + + + + ))} + + )); + } else if (error != null) { + body = ( + + + {error} + + + ); + } else if (noRows) { + const isNode = typeof empty === "object" && empty !== null; + body = ( + + + {empty ?? "No data"} + + + ); + } else if (groups) { + body = groups.flatMap((g) => { + const limit = g.collapseAfter ?? Infinity; + const open = openGroups.has(g.key); + const overflow = g.rows.length > limit; + const shown = overflow && !open ? g.rows.slice(0, limit) : g.rows; + const header = ( + + +
+
+ {g.title} + {g.meta && ( + {g.meta} + )} +
+ {g.actions && + g.actions.length > 0 && + renderCellActions(g.actions)} +
+ + + ); + const rowEls = shown.map((row) => ( + + {effectiveColumns.map((c) => ( + + {c.renderCell(row)} + + ))} + + )); + const moreEl = overflow ? ( + + + + + + ) : null; + return moreEl ? [header, ...rowEls, moreEl] : [header, ...rowEls]; + }); + } else { + body = table.getRowModel().rows.map((row) => ( + + {row.getAllCells().map((cell) => { + const meta = cell.column.columnDef.meta; + return ( + + {flexRender(cell.column.columnDef.cell, cell.getContext())} + + ); + })} + + )); + } + + return ( +
+
+ {toolbar &&
{toolbar}
} +
+ + {caption && ( + + )} + + {table.getHeaderGroups().map((headerGroup) => ( + + {headerGroup.headers.map((header) => { + const meta = header.column.columnDef.meta; + const align = meta?.align ?? "left"; + const canSort = header.column.getCanSort(); + const sorted = header.column.getIsSorted(); + const label = header.isPlaceholder + ? null + : flexRender( + header.column.columnDef.header, + header.getContext(), + ); + return ( + + ); + })} + + ))} + + {body} +
{caption}
+ {canSort ? ( + + ) : meta?.srHeader ? ( + + {meta.srHeader} + + ) : ( + label + )} +
+
+
+
+ ); +} + +function cellClass( + align: "left" | "right", + nowrap: boolean, + fit: boolean, +): string { + return [ + "sui-datatable__td", + `sui-datatable__td--${align}`, + nowrap ? "sui-datatable__td--nowrap" : "", + fit ? "sui-datatable__td--fit" : "", + ] + .filter(Boolean) + .join(" "); +} + +function headerClass(align: "left" | "right", fit: boolean): string { + return [ + "sui-datatable__th", + `sui-datatable__th--${align}`, + fit ? "sui-datatable__th--fit" : "", + ] + .filter(Boolean) + .join(" "); +} diff --git a/frontend/editor/src/core/ui/Dropdown.css b/frontend/editor/src/core/ui/Dropdown.css index 8b84af29f1..d71026d881 100644 --- a/frontend/editor/src/core/ui/Dropdown.css +++ b/frontend/editor/src/core/ui/Dropdown.css @@ -4,28 +4,20 @@ } .sui-dd__menu { - position: absolute; - top: calc(100% + var(--space-1)); + /* Positioned (fixed, portaled to ) entirely by the Menu component. */ min-width: 12rem; padding: var(--space-1); background: var(--c-surface); border: 1px solid var(--c-border); border-radius: var(--radius-md); box-shadow: var(--shadow-lg); - z-index: var(--z-dropdown); + z-index: var(--z-popover); animation: fadeInUp var(--motion-enter) both; display: flex; flex-direction: column; gap: 0.0625rem; } -.sui-dd__menu--start { - left: 0; -} -.sui-dd__menu--end { - right: 0; -} - .sui-dd__item { display: flex; align-items: center; diff --git a/frontend/editor/src/core/ui/Dropdown.tsx b/frontend/editor/src/core/ui/Dropdown.tsx index bf481c9ea5..25019d7c3c 100644 --- a/frontend/editor/src/core/ui/Dropdown.tsx +++ b/frontend/editor/src/core/ui/Dropdown.tsx @@ -6,12 +6,14 @@ import { useContext, useEffect, useId, + useLayoutEffect, useMemo, useRef, useState, type ReactElement, type ReactNode, } from "react"; +import { createPortal } from "react-dom"; import "@app/ui/Dropdown.css"; type Alignment = "start" | "end"; @@ -20,6 +22,8 @@ interface DropdownContextValue { open: boolean; setOpen: (open: boolean) => void; triggerRef: React.RefObject; + /** The portaled menu element, so click-outside can exclude it. */ + menuRef: React.RefObject; menuId: string; align: Alignment; } @@ -68,15 +72,20 @@ function Root({ const triggerRef = useRef(null); const containerRef = useRef(null); + const menuRef = useRef(null); const menuId = useId(); - // Click-outside + Escape close. + // Click-outside + Escape close. The menu is portaled to , so it is not + // inside containerRef - check it separately or a click on it would close the + // menu before the item's handler runs. useEffect(() => { if (!open) return; function onDocClick(e: MouseEvent) { + const target = e.target as Node; if ( containerRef.current && - !containerRef.current.contains(e.target as Node) + !containerRef.current.contains(target) && + !(menuRef.current && menuRef.current.contains(target)) ) { setOpen(false); } @@ -96,7 +105,7 @@ function Root({ }, [open, setOpen]); const value = useMemo( - () => ({ open, setOpen, triggerRef, menuId, align }), + () => ({ open, setOpen, triggerRef, menuRef, menuId, align }), [open, setOpen, menuId, align], ); @@ -150,23 +159,79 @@ export interface DropdownMenuProps { } function Menu({ children, className, width }: DropdownMenuProps) { - const { open, menuId, align } = useDropdownCtx(); - if (!open) return null; - const style = - width !== undefined + const { open, menuId, align, triggerRef, menuRef } = useDropdownCtx(); + // Fixed position tracked to the trigger. Portaling to keeps the menu + // out of any `overflow` ancestor (e.g. a table's horizontal scroll area), + // which would otherwise clip it and add a scrollbar. + const [pos, setPos] = useState<{ + top?: number; + bottom?: number; + left?: number; + right?: number; + maxHeight: number; + } | null>(null); + + useLayoutEffect(() => { + if (!open) return; + const place = () => { + const el = triggerRef.current; + if (!el) return; + const r = el.getBoundingClientRect(); + const gap = 4; + const margin = 8; + const spaceBelow = window.innerHeight - r.bottom - margin; + const spaceAbove = r.top - margin; + // Flip above when there's more room there, so a trigger near the viewport + // bottom doesn't open a fixed menu that runs off-screen and can't scroll. + const below = spaceBelow >= spaceAbove; + const horizontal = + align === "end" + ? { right: window.innerWidth - r.right } + : { left: r.left }; + setPos({ + ...horizontal, + ...(below + ? { top: r.bottom + gap } + : { bottom: window.innerHeight - r.top + gap }), + maxHeight: Math.max(0, (below ? spaceBelow : spaceAbove) - gap), + }); + }; + place(); + // Track the trigger while scrolling/resizing (capture catches inner scrollers). + window.addEventListener("scroll", place, true); + window.addEventListener("resize", place); + return () => { + window.removeEventListener("scroll", place, true); + window.removeEventListener("resize", place); + }; + }, [open, align, triggerRef]); + + if (!open || !pos) return null; + const style: React.CSSProperties = { + position: "fixed", + // Explicit auto (not undefined) so the CSS fallback `top`/`left` can't leak + // in on the axis this placement isn't pinning. + top: pos.top ?? "auto", + bottom: pos.bottom ?? "auto", + left: pos.left ?? "auto", + right: pos.right ?? "auto", + maxHeight: pos.maxHeight, + overflowY: "auto", + ...(width !== undefined ? { minWidth: typeof width === "number" ? `${width}px` : width } - : undefined; - return ( + : {}), + }; + return createPortal( + , + document.body, ); } diff --git a/frontend/editor/src/core/ui/Table.css b/frontend/editor/src/core/ui/Table.css deleted file mode 100644 index fa89fd04d6..0000000000 --- a/frontend/editor/src/core/ui/Table.css +++ /dev/null @@ -1,73 +0,0 @@ -.sui-table-wrap { - width: 100%; - overflow-x: auto; -} - -.sui-table { - width: 100%; - border-collapse: collapse; - font-size: 0.8125rem; -} - -/* Header text kept for assistive tech only, so a column of controls can be named - without putting a heading above it. Defined here rather than borrowing a global - utility, since the portal loads its own stylesheet. */ -.sui-table__th-sr { - position: absolute; - width: 1px; - height: 1px; - padding: 0; - margin: -1px; - overflow: hidden; - clip-path: inset(50%); - white-space: nowrap; - border: 0; -} - -.sui-table__th { - text-align: left; - font-weight: 600; - color: var(--c-text-subtle); - font-size: 0.6875rem; - text-transform: uppercase; - letter-spacing: 0.04em; - padding: 0.625rem 0.875rem; - border-bottom: 1px solid var(--c-border); - white-space: nowrap; -} -.sui-table__th--right, -.sui-table__td--right { - text-align: right; -} -.sui-table__th--center, -.sui-table__td--center { - text-align: center; -} - -.sui-table__td { - padding: 0.625rem 0.875rem; - color: var(--c-text-muted); - border-bottom: 1px solid var(--c-border-subtle); - vertical-align: middle; -} -.sui-table tbody tr:last-child .sui-table__td { - border-bottom: none; -} - -.sui-table__row--interactive { - cursor: pointer; - transition: background var(--motion-fast); -} -.sui-table__row--interactive:hover { - background: var(--c-hover); -} -.sui-table__row--interactive:focus-visible { - outline: 0.125rem solid var(--c-primary); - outline-offset: -0.125rem; -} - -.sui-table__empty { - padding: 2rem; - text-align: center; - color: var(--c-text-subtle); -} diff --git a/frontend/editor/src/core/ui/Table.stories.tsx b/frontend/editor/src/core/ui/Table.stories.tsx deleted file mode 100644 index 2e7e0f64f1..0000000000 --- a/frontend/editor/src/core/ui/Table.stories.tsx +++ /dev/null @@ -1,109 +0,0 @@ -import type { Meta, StoryObj } from "@storybook/react-vite"; -import { Table, type TableColumn } from "@app/ui/Table"; -import { StatusBadge } from "@app/ui/StatusBadge"; - -interface Region { - id: string; - name: string; - code: string; - status: "healthy" | "degraded"; - docs: number; - latency: string; -} - -const REGIONS: Region[] = [ - { - id: "1", - name: "US East", - code: "us-east-1", - status: "healthy", - docs: 12481, - latency: "41 ms", - }, - { - id: "2", - name: "US West", - code: "us-west-2", - status: "healthy", - docs: 8210, - latency: "63 ms", - }, - { - id: "3", - name: "EU West", - code: "eu-west-1", - status: "degraded", - docs: 3044, - latency: "190 ms", - }, -]; - -const COLUMNS: TableColumn[] = [ - { key: "name", header: "Region", render: (r) => r.name }, - { - key: "code", - header: "Code", - render: (r) => ( - {r.code} - ), - }, - { - key: "status", - header: "Status", - render: (r) => ( - - {r.status} - - ), - }, - { - key: "docs", - header: "Docs 24h", - align: "right", - render: (r) => r.docs.toLocaleString(), - }, - { key: "latency", header: "P95", align: "right", render: (r) => r.latency }, -]; - -const meta: Meta = { - title: "Compound/Table", - component: Table, - tags: ["autodocs"], - parameters: { layout: "padded" }, -}; -export default meta; -type Story = StoryObj; - -/** Presentational table — columns own their cell renderers; pass pre-sorted rows. */ -export const Basic: Story = { - render: () => ( - columns={COLUMNS} rows={REGIONS} rowKey={(r) => r.id} /> - ), -}; - -/** With `onRowClick`, rows become focusable + hoverable (keyboard: Enter/Space). */ -export const Interactive: Story = { - render: () => ( - - columns={COLUMNS} - rows={REGIONS} - rowKey={(r) => r.id} - onRowClick={() => {}} - /> - ), -}; - -/** Empty body slot. */ -export const Empty: Story = { - render: () => ( - - columns={COLUMNS} - rows={[]} - rowKey={(r) => r.id} - empty="No regions deployed yet." - /> - ), -}; diff --git a/frontend/editor/src/core/ui/Table.tsx b/frontend/editor/src/core/ui/Table.tsx deleted file mode 100644 index 417df3d6c1..0000000000 --- a/frontend/editor/src/core/ui/Table.tsx +++ /dev/null @@ -1,141 +0,0 @@ -import type { ReactNode } from "react"; -import "@app/ui/Surface.css"; -import "@app/ui/Table.css"; - -export interface TableColumn { - /** Stable column id. */ - key: string; - header: ReactNode; - /** - * Hides the header visually but keeps it for assistive tech. For a trailing column of controls - * or chevrons, where a visible heading would be noise but a blank one leaves the cells below it - * unlabelled. - */ - headerHidden?: boolean; - /** Cell renderer for a row. */ - render: (row: T) => ReactNode; - align?: "left" | "right" | "center"; - /** Optional fixed/min width (any CSS length). */ - width?: string; -} - -export interface TableProps { - columns: TableColumn[]; - rows: T[]; - /** Stable key per row. */ - rowKey: (row: T) => string; - /** Makes rows interactive (hover + click + keyboard). */ - onRowClick?: (row: T) => void; - /** - * Per-row gate for interactivity, checked only when {@link onRowClick} is set. A row for which - * this returns false is inert: no click/keyboard, and not announced as a button. Defaults to - * all rows interactive. - */ - isRowInteractive?: (row: T) => boolean; - /** - * Set when rows render controls of their own. The row keeps its click as a mouse shortcut but - * stops announcing itself as a button, because a button may not contain other controls and a - * {@code } is no longer a row to a screen reader. That row control is then the - * keyboard path to the same action, so nothing is lost by leaving the row itself inert. - */ - rowsContainControls?: boolean; - /** Rendered in place of the body when there are no rows. */ - empty?: ReactNode; - className?: string; -} - -/** - * Minimal data table primitive. Columns own their own cell renderers, so the - * table stays presentational — callers pre-sort/filter and pass the rows they - * want shown. Rows become focusable buttons-in-disguise when `onRowClick` is - * set. - */ -export function Table({ - columns, - rows, - rowKey, - onRowClick, - isRowInteractive, - rowsContainControls = false, - empty, - className, -}: TableProps) { - const interactive = Boolean(onRowClick); - return ( -
- - - - {columns.map((c) => ( - - ))} - - - - {rows.length === 0 ? ( - - - - ) : ( - rows.map((row) => { - const rowInteractive = - interactive && (isRowInteractive?.(row) ?? true); - // Only a row that owns the whole interaction takes the button role and the keyboard - // handling that goes with it; see rowsContainControls. - const rowIsControl = rowInteractive && !rowsContainControls; - return ( - onRowClick?.(row) : undefined} - tabIndex={rowIsControl ? 0 : undefined} - role={rowIsControl ? "button" : undefined} - onKeyDown={ - rowIsControl - ? (e) => { - if (e.key === "Enter" || e.key === " ") { - e.preventDefault(); - onRowClick?.(row); - } - } - : undefined - } - > - {columns.map((c) => ( - - ))} - - ); - }) - )} - -
- {c.headerHidden ? ( - {c.header} - ) : ( - c.header - )} -
- {empty ?? "No data"} -
- {c.render(row)} -
-
- ); -} diff --git a/frontend/editor/src/core/ui/dataTableColumns.tsx b/frontend/editor/src/core/ui/dataTableColumns.tsx new file mode 100644 index 0000000000..df31dc384a --- /dev/null +++ b/frontend/editor/src/core/ui/dataTableColumns.tsx @@ -0,0 +1,569 @@ +import { Fragment, type ReactNode } from "react"; +import { StatusBadge, type StatusTone } from "@app/ui/StatusBadge"; +import { Chip, type ChipAccent } from "@app/ui/Chip"; +import { Button } from "@app/ui/Button"; +import { Dropdown } from "@app/ui/Dropdown"; +import { ProgressBar } from "@app/ui/ProgressBar"; +import { Select, type SelectOption } from "@app/ui/Select"; + +/** + * The column vocabulary for {@link DataTable}. Call-sites pick a cell KIND and + * supply the data + semantics; the component owns 100% of the appearance. There + * is no raw-JSX / className escape hatch by design; a cell can only look the way + * the design system draws its kind, so every table looks and behaves the same. + */ + +type Align = "left" | "right"; +type SortValue = string | number | boolean | null | undefined; + +/** + * Which built-in comparator sorts a column. Set by the builder from the cell's + * data type - `alphanumeric` (case-insensitive, natural: `v2` before `v10`) for + * text, `basic` (raw numeric) for numbers. Call-sites never choose this. + */ +export type DataTableSortFn = "alphanumeric" | "basic"; + +/** Opaque, fully-resolved column. Produced only by the {@link column} builders. */ +export interface DataTableColumn { + key: string; + header: ReactNode; + align: Align; + /** Prevent wrapping (mono/number values). */ + nowrap: boolean; + /** Shrink the column to its content (actions / affordances). */ + fit: boolean; + sortable: boolean; + sortValue?: (row: T) => SortValue; + /** Comparator kind, derived from the cell type. Only set when sortable. */ + sortFn?: DataTableSortFn; + /** Cell renders its own interactive control (button/link/select/chip). Rows + * containing one drop their `role="button"` so a button never nests inside a + * button - the control is the keyboard path instead. */ + interactive?: boolean; + /** Internal, design-system-owned renderer. Call-sites never supply this. */ + renderCell: (row: T) => ReactNode; +} + +/** The only design-system glyph a cell may use (icon-only actions). */ +export type CellGlyph = "kebab"; + +function KebabGlyph() { + return ( + + + + + + ); +} + +/** An item in a kebab action menu. */ +export interface CellMenuItem { + label: string; + tone?: "default" | "danger"; + disabled?: boolean; + onClick: () => void; + /** Draw a divider above this item. */ + dividerBefore?: boolean; +} + +/** A row/group action. A locked button, or a kebab menu when `menu` is set. */ +export interface CellAction { + label: string; + glyph?: CellGlyph; + /** Icon-only (uses `label` as the accessible name). */ + iconOnly?: boolean; + tone?: "default" | "danger"; + onClick?: () => void; + loading?: boolean; + disabled?: boolean; + /** When set, the button opens this menu instead of firing `onClick`. */ + menu?: CellMenuItem[]; +} + +/** Renders a row of locked action buttons / kebab menus. Shared by the + * `actions` cell kind and grouped-table headers. */ +export function renderCellActions(actions: CellAction[]): ReactNode { + return ( +
e.stopPropagation()}> + {actions.map((a) => + a.menu ? ( + + + + + + {a.menu.map((m) => ( + + {m.dividerBefore && } + + {m.label} + + + ))} + + + ) : ( + + ), + )} +
+ ); +} + +/** An external link inside a cell. */ +export interface CellLink { + label: string; + href: string; + ariaLabel?: string; +} + +interface Common { + key: string; + header: ReactNode; + sortable?: boolean; +} + +function base( + o: Common, + extra: Pick, "align" | "nowrap" | "fit" | "renderCell"> & { + sortValue?: (row: T) => SortValue; + sortFn?: DataTableSortFn; + interactive?: boolean; + }, +): DataTableColumn { + return { + key: o.key, + header: o.header, + align: extra.align, + nowrap: extra.nowrap, + fit: extra.fit, + sortable: !!o.sortable, + sortValue: o.sortable ? extra.sortValue : undefined, + sortFn: o.sortable ? extra.sortFn : undefined, + interactive: extra.interactive, + renderCell: extra.renderCell, + }; +} + +function text( + o: Common & { + get: (row: T) => string; + /** Optional bold label rendered before the value as "Label: value". */ + label?: (row: T) => string | null | undefined; + sortBy?: (row: T) => SortValue; + }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: false, + fit: false, + sortValue: o.sortBy ?? ((r) => o.get(r)), + sortFn: "alphanumeric", + renderCell: (r) => { + const label = o.label?.(r); + return label ? ( + + {label}: {o.get(r)} + + ) : ( + {o.get(r)} + ); + }, + }); +} + +function mono( + o: Common & { get: (row: T) => string; sortBy?: (row: T) => SortValue }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: true, + fit: false, + sortValue: o.sortBy ?? ((r) => o.get(r)), + sortFn: "alphanumeric", + renderCell: (r) => {o.get(r)}, + }); +} + +function muted( + o: Common & { + get: (row: T) => string | null | undefined; + placeholder?: string; + /** Override the sort key (e.g. an ISO date behind a "3 days ago" label). */ + sortBy?: (row: T) => SortValue; + }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: false, + fit: false, + sortValue: o.sortBy ?? ((r) => o.get(r) ?? undefined), + sortFn: "alphanumeric", + renderCell: (r) => ( + + {o.get(r) || (o.placeholder ?? "-")} + + ), + }); +} + +function number( + o: Common & { + get: (row: T) => number | null | undefined; + format?: (n: number, row: T) => string; + placeholder?: string; + /** Override the sort key (e.g. a raw count behind a formatted label). */ + sortBy?: (row: T) => SortValue; + }, +): DataTableColumn { + return base(o, { + align: "right", + nowrap: true, + fit: false, + sortValue: o.sortBy ?? ((r) => o.get(r) ?? undefined), + sortFn: "basic", + renderCell: (r) => { + const n = o.get(r); + if (n == null) { + return ( + + {o.placeholder ?? "-"} + + ); + } + return ( + + {o.format ? o.format(n, r) : String(n)} + + ); + }, + }); +} + +function badge( + o: Common & { + get: (row: T) => { tone: StatusTone; label: string }; + sortBy?: (row: T) => SortValue; + }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: true, + fit: false, + sortValue: o.sortBy ?? ((r) => o.get(r).label), + sortFn: "alphanumeric", + renderCell: (r) => { + const b = o.get(r); + return ( + + {b.label} + + ); + }, + }); +} + +/** + * A user-defined label, rendered as a dot-less pill. Use this ONLY for labels + * that come from data / the user (e.g. a document's classification). Values from + * a fixed set we define (types, environments, providers) are `text`, not pills. + */ +export interface CellLabel { + label: string; + accent?: ChipAccent; +} + +function labels( + o: Common & { get: (row: T) => CellLabel[] }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: false, + fit: false, + sortValue: (r) => o.get(r)[0]?.label ?? undefined, + sortFn: "alphanumeric", + renderCell: (r) => ( +
+ {o.get(r).map((l) => ( + + {l.label} + + ))} +
+ ), + }); +} + +/** + * An interactive capability chip: click to grant, remove to revoke, dashed to + * offer adding. A functional cell (it toggles state), distinct from static + * `labels`. + */ +export interface CellCap { + label: string; + accent?: ChipAccent; + onClick?: () => void; + onRemove?: () => void; + dashed?: boolean; +} + +function caps( + o: Common & { get: (row: T) => CellCap[] }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: false, + fit: false, + interactive: true, + renderCell: (r) => ( +
+ {o.get(r).map((c) => ( + + {c.label} + + ))} +
+ ), + }); +} + +function entity( + o: Common & { + /** Semantic leading icon (component owns its size + colour container). */ + icon?: (row: T) => ReactNode; + primary: (row: T) => string; + /** Muted inline suffix after the name, its own node (e.g. "(you)"). */ + suffix?: (row: T) => string | null | undefined; + /** Secondary muted line under the name. */ + note?: (row: T) => string | null | undefined; + sortBy?: (row: T) => SortValue; + }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: false, + fit: false, + sortValue: o.sortBy ?? ((r) => o.primary(r)), + sortFn: "alphanumeric", + renderCell: (r) => { + const icon = o.icon?.(r); + const suffix = o.suffix?.(r); + const note = o.note?.(r); + return ( +
+ {icon != null && ( + + {icon} + + )} +
+ + {o.primary(r)} + {suffix && ( + {suffix} + )} + + {note && {note}} +
+
+ ); + }, + }); +} + +function actions(o: { + key: string; + header?: ReactNode; + get: (row: T) => CellAction[]; +}): DataTableColumn { + return { + key: o.key, + header: o.header ?? "", + align: "right", + nowrap: true, + fit: true, + sortable: false, + interactive: true, + renderCell: (r) => renderCellActions(o.get(r)), + }; +} + +function progress( + o: Common & { + get: (row: T) => { value: number; label?: string }; + /** Accessible name for the bar (it has no visible text). Defaults to the + * shown percent; pass a description like "Load for us-east-1" when useful. */ + ariaLabel?: (row: T) => string; + }, +): DataTableColumn { + return base(o, { + align: "left", + nowrap: true, + fit: false, + sortValue: (r) => o.get(r).value, + sortFn: "basic", + renderCell: (r) => { + const p = o.get(r); + const shown = p.label ?? `${Math.round(p.value * 100)}%`; + return ( +
+ + + + {shown} +
+ ); + }, + }); +} + +function links(o: { + key: string; + header?: ReactNode; + get: (row: T) => CellLink[]; +}): DataTableColumn { + return { + key: o.key, + header: o.header ?? "", + align: "right", + nowrap: true, + fit: true, + sortable: false, + interactive: true, + renderCell: (r) => ( +
+ {o.get(r).map((l) => ( + + {l.label} + + ))} +
+ ), + }; +} + +function select(o: { + key: string; + header: ReactNode; + get: (row: T) => { + value?: string | null; + defaultValue?: string; + options: SelectOption[]; + ariaLabel?: string; + disabled?: boolean; + }; + /** Omit for an uncontrolled select (local UI state only). */ + onChange?: (row: T, value: string | null) => void; +}): DataTableColumn { + return { + key: o.key, + header: o.header, + align: "left", + nowrap: true, + fit: false, + sortable: false, + interactive: true, + renderCell: (r) => { + const s = o.get(r); + const change = o.onChange; + return ( +
+ onChangeRole(m, (value ?? m.role) as RoleId)} - /> -
- )} + const groups = useMemo[]>(() => { + function ownerNames(owners: string[]): string { + return owners.map((u) => nameByUsername.get(u) ?? u).join(", "); + } + // A team whose name/membership is system-managed - no rename/delete. + function isManagedTeam(team: TeamGroup): boolean { + return SYSTEM_TEAMS.has(team.name) || team.isPersonal === true; + } + function teamKebabHasItems(team: TeamGroup): boolean { + return ( + capabilities.manageGrants || + (!isManagedTeam(team) && + (capabilities.renameTeam || capabilities.deleteTeam)) + ); + } + function teamActions(team: TeamGroup): CellAction[] { + const acts: CellAction[] = [ + { + label: t("users.group.addToTeam", "Add to team"), + onClick: () => onAddToTeam(team), + }, + ]; + if (teamKebabHasItems(team)) { + const items: CellMenuItem[] = []; + if (capabilities.manageGrants) { + items.push( + processorTeamIds.has(team.id) + ? { + label: t( + "users.team.revokeProcessor", + "Revoke Processor from team", + ), + onClick: () => onRevokeTeamProcessor(team), + } + : { + label: t( + "users.team.grantProcessor", + "Grant Processor to team", + ), + onClick: () => onGrantTeamProcessor(team), + }, + ); + } + if (!isManagedTeam(team)) { + const divider = capabilities.manageGrants; + if (capabilities.renameTeam) { + items.push({ + label: t("users.action.rename", "Rename team"), + onClick: () => onRenameTeam(team), + dividerBefore: divider, + }); + } + if (capabilities.deleteTeam) { + items.push({ + label: t("users.action.deleteTeam", "Delete team"), + tone: "danger", + onClick: () => onDeleteTeam(team), + dividerBefore: divider && !capabilities.renameTeam, + }); + } + } + acts.push({ + label: t("users.teamActions", "Team actions"), + glyph: "kebab", + iconOnly: true, + menu: items, + }); + } + return acts; + } - {rowKebab(m)} - - ); - } - - /** Rows for a group, collapsing past COLLAPSED_LIMIT behind a toggle. */ - function renderMembers(list: Member[], key: string) { - const isOpen = expanded.has(key); - const overflow = list.length > COLLAPSED_LIMIT; - const shown = overflow && !isOpen ? list.slice(0, COLLAPSED_LIMIT) : list; - return ( - <> - {shown.map(renderRow)} - {overflow && ( - - )} - - ); - } + const gs: DataTableGroup[] = []; + if (capabilities.orgGroup && dir.organization.length > 0) { + gs.push({ + key: "org", + title: t("users.group.org", "Organization"), + meta: t("users.group.owners", "{{count}} owner", { + count: dir.organization.length, + }), + rows: dir.organization, + collapseAfter: COLLAPSED_LIMIT, + }); + } + for (const team of dir.teams) { + const led = + team.owners.length > 0 + ? ` · ${t("users.group.ledBy", "led by {{owner}}", { + owner: ownerNames(team.owners), + })}` + : ""; + gs.push({ + key: `team-${team.id}`, + title: t("users.group.team", "{{name}} team", { name: team.name }), + meta: + t("users.group.teamMeta", "{{count}} people", { + count: team.members.length, + }) + led, + actions: teamActions(team), + rows: team.members, + collapseAfter: COLLAPSED_LIMIT, + }); + } + if (showGuests && dir.guests.length > 0) { + gs.push({ + key: "guests", + title: t("users.group.guests", "Guests"), + meta: t("users.group.guestCount", "{{count}} guest", { + count: dir.guests.length, + }), + rows: dir.guests, + collapseAfter: COLLAPSED_LIMIT, + }); + } + return gs; + }, [ + t, + dir, + nameByUsername, + capabilities, + showGuests, + processorTeamIds, + onAddToTeam, + onGrantTeamProcessor, + onRevokeTeamProcessor, + onRenameTeam, + onDeleteTeam, + ]); return ( -
- {/* Organization (a single-org deployment only; SaaS has no org). */} - {capabilities.orgGroup && dir.organization.length > 0 && ( -
-
-
- {t("users.group.org", "Organization")} - - {t( - "users.group.orgDesc", - "Owners with org-wide authority and policy approval", - )} - -
- - {t("users.group.owners", "{{count}} owner", { - count: dir.organization.length, - })} - -
- {renderMembers(dir.organization, "org")} -
- )} - - {/* Teams */} - {dir.teams.map((team) => ( -
-
-
- - {t("users.group.team", "{{name}} team", { name: team.name })} - - - {t("users.group.teamMeta", "{{count}} people", { - count: team.members.length, - })} - {team.owners.length > 0 && - ` · ${t("users.group.ledBy", "led by {{owner}}", { - owner: ownerNames(team.owners), - })}`} - -
-
- - {teamKebabHasItems(team) && ( - - - - - - {capabilities.manageGrants && - (processorTeamIds.has(team.id) ? ( - onRevokeTeamProcessor(team)}> - {t( - "users.team.revokeProcessor", - "Revoke Processor from team", - )} - - ) : ( - onGrantTeamProcessor(team)}> - {t( - "users.team.grantProcessor", - "Grant Processor to team", - )} - - ))} - {!isManagedTeam(team) && - (capabilities.renameTeam || capabilities.deleteTeam) && ( - <> - {capabilities.manageGrants && } - {capabilities.renameTeam && ( - onRenameTeam(team)}> - {t("users.action.rename", "Rename team")} - - )} - {capabilities.deleteTeam && ( - onDeleteTeam(team)} - > - {t("users.action.deleteTeam", "Delete team")} - - )} - - )} - - - )} -
-
- {renderMembers(team.members, `team-${team.id}`)} -
- ))} - - {/* Guests (parked in the live app; shown when showGuests is set). */} - {showGuests && dir.guests.length > 0 && ( -
-
-
- {t("users.group.guests", "Guests")} - - {t( - "users.group.guestsDesc", - "External collaborators, scoped to what you shared. Editor only.", - )} - -
- - {t("users.group.guestCount", "{{count}} guest", { - count: dir.guests.length, - })} - -
- {renderMembers(dir.guests, "guests")} -
- )} -
+ + columns={columns} + groups={groups} + rowKey={(m) => String(m.id)} + collapseLabels={{ + showAll: (count) => t("users.showAll", "Show all {{count}}", { count }), + showLess: t("users.showLess", "Show less"), + }} + /> ); } diff --git a/frontend/editor/src/portal/views/Integrations.test.tsx b/frontend/editor/src/portal/views/Integrations.test.tsx index c5e96d8666..8c4e684bfa 100644 --- a/frontend/editor/src/portal/views/Integrations.test.tsx +++ b/frontend/editor/src/portal/views/Integrations.test.tsx @@ -68,39 +68,35 @@ describe("Integrations view", () => { ).toBeInTheDocument(); }); - it("groups connections of the same type and expands to the instances", async () => { + it("groups connections of the same type, instances shown as rows (no expand)", async () => { fetchIntegrations.mockResolvedValue([ bucket(1, "Claims"), bucket(2, "Archive"), ]); render(); - // One connected group row for S3 with the instance count, not two rows. - const group = await screen.findByText( - "portal.integrations.connectionCount", - ); - expect(group).toBeInTheDocument(); - - fireEvent.click(screen.getByText("portal.connections.types.s3.label")); + // Instances are rows directly under the S3 vendor group - no expand click. expect(await screen.findByText("Claims")).toBeInTheDocument(); expect(screen.getByText("Archive")).toBeInTheDocument(); + // Vendor group header shows the instance count and the "add another" action. expect( - screen.getByText("portal.integrations.addAnother"), + screen.getByText("portal.integrations.connectionCount"), ).toBeInTheDocument(); + // Each connected vendor group offers a Connect action (to add another). + expect( + screen.getAllByText("portal.integrations.connect").length, + ).toBeGreaterThan(0); // The available band remains for the other, unconnected vendors. expect( screen.getByText(/portal\.integrations\.availableHeading/), ).toBeInTheDocument(); }); - it("deletes an instance from the expanded group", async () => { + it("deletes an instance directly from its row", async () => { fetchIntegrations.mockResolvedValueOnce([bucket(5, "Claims")]); fetchIntegrations.mockResolvedValueOnce([]); render(); - fireEvent.click( - await screen.findByText("portal.connections.types.s3.label"), - ); fireEvent.click(await screen.findByText("portal.connections.delete")); await waitFor(() => expect(deleteIntegration).toHaveBeenCalledWith(5)); @@ -115,9 +111,6 @@ describe("Integrations view", () => { ); render(); - fireEvent.click( - await screen.findByText("portal.connections.types.s3.label"), - ); fireEvent.click(await screen.findByText("portal.connections.delete")); expect( diff --git a/frontend/editor/src/portal/views/Integrations.tsx b/frontend/editor/src/portal/views/Integrations.tsx index 993583a364..793c21f880 100644 --- a/frontend/editor/src/portal/views/Integrations.tsx +++ b/frontend/editor/src/portal/views/Integrations.tsx @@ -2,8 +2,15 @@ import { useCallback, useEffect, useMemo, useState } from "react"; import { useTranslation } from "react-i18next"; import AddRoundedIcon from "@mui/icons-material/AddRounded"; import SearchRoundedIcon from "@mui/icons-material/SearchRounded"; -import ExpandMoreRoundedIcon from "@mui/icons-material/ExpandMoreRounded"; -import { Banner, Button, Skeleton } from "@app/ui"; +import { + Banner, + Button, + column, + DataTable, + type DataTableColumn, + type DataTableGroup, + EmptyState, +} from "@app/ui"; import { errorMessage } from "@portal/api/http"; import { deleteIntegration, @@ -30,12 +37,11 @@ import "@portal/views/Integrations.css"; /** * The integrations catalogue: everything Stirling can talk to, in one place. * - * Three bands in one list. Connected first — stored connections grouped by - * vendor, expandable when a vendor has several (two S3 buckets is normal, not - * an error), each instance editable and one click from "add another". Then - * Available — the supported vendors, each saying what it works with (sources, - * policies, pipelines) so it's obvious whether a vendor feeds documents in or - * receives them. Coming-soon source connectors close the list greyed out, so + * Three bands, one grouped table. Connected first - stored connections grouped + * by vendor (two S3 buckets is normal, not an error), every instance a row you + * can edit or remove, with "add another" on the vendor's group header. Then + * Available - the supported vendors, each saying what it works with (sources, + * policies, pipelines). Coming-soon source connectors close the list so * "do you support X?" is answered honestly instead of hidden. * * Setup itself stays in the shared {@link ConnectionModal}; every entry point @@ -72,6 +78,20 @@ interface TypeGroup { connections: IntegrationConfig[]; } +/** One normalized row across the three bands, so a single grouped table renders + * connected instances, available vendors, and coming-soon vendors alike. */ +type IntegrationRow = { + key: string; + brandId: string; + title: string; + subtitle: string; + worksWith: WorksWith[]; +} & ( + | { kind: "instance"; connection: IntegrationConfig; canManage: boolean } + | { kind: "available"; typeId: string } + | { kind: "soon" } +); + export function Integrations() { const { t } = useTranslation(); const [connections, setConnections] = useState( @@ -82,13 +102,13 @@ export function Integrations() { >(undefined); const [filter, setFilter] = useState("all"); const [query, setQuery] = useState(""); - const [expanded, setExpanded] = useState>(new Set()); const [modal, setModal] = useState<{ open: boolean; editing: IntegrationConfig | null; fixedTypeId?: string; }>({ open: false, editing: null }); const [busy, setBusy] = useState(false); + const [deletingId, setDeletingId] = useState(null); const [error, setError] = useState(null); const refresh = useCallback(async () => { @@ -192,45 +212,152 @@ export function Integrations() { return counts; }, [catalogue]); - function toggleExpand(typeId: string) { - setExpanded((current) => { - const next = new Set(current); - if (next.has(typeId)) next.delete(typeId); - else next.add(typeId); - return next; - }); - } - - function openCreate(typeId: string) { + const openCreate = useCallback((typeId: string) => { setModal({ open: true, editing: null, fixedTypeId: typeId }); - } + }, []); - function openEdit(connection: IntegrationConfig) { + const openEdit = useCallback((connection: IntegrationConfig) => { setModal({ open: true, editing: connection }); - } + }, []); - async function remove(connection: IntegrationConfig) { - if (busy) return; - setBusy(true); - setError(null); - try { - await deleteIntegration(connection.id); - await refresh(); - } catch (e) { - setError(errorMessage(e)); - } finally { - setBusy(false); - } - } + const remove = useCallback( + async (connection: IntegrationConfig) => { + if (busy) return; + setBusy(true); + setDeletingId(connection.id); + setError(null); + try { + await deleteIntegration(connection.id); + await refresh(); + } catch (e) { + setError(errorMessage(e)); + } finally { + setBusy(false); + setDeletingId(null); + } + }, + [busy, refresh], + ); const isLoading = connections === null; - const chip = (kind: WorksWith) => ( - - {t(`portal.integrations.worksWith.${kind}`)} - + const worksWithText = useCallback( + (list: WorksWith[]) => + list.map((w) => t(`portal.integrations.worksWith.${w}`)).join(", "), + [t], ); + const columns = useMemo[]>( + () => [ + column.entity({ + key: "integration", + header: t("portal.integrations.table.integration"), + icon: (r) => , + primary: (r) => r.title, + note: (r) => r.subtitle || undefined, + }), + column.text({ + key: "worksWith", + header: t("portal.integrations.table.worksWith"), + get: (r) => worksWithText(r.worksWith), + }), + column.actions({ + key: "actions", + get: (r) => { + if (r.kind === "instance") { + return r.canManage + ? [ + { + label: t("portal.connections.edit"), + disabled: busy, + onClick: () => openEdit(r.connection), + }, + { + label: t("portal.connections.delete"), + tone: "danger", + loading: busy && deletingId === r.connection.id, + disabled: busy, + onClick: () => void remove(r.connection), + }, + ] + : []; + } + if (r.kind === "available") { + return [ + { + label: t("portal.integrations.connect"), + onClick: () => openCreate(r.typeId), + }, + ]; + } + return []; + }, + }), + ], + [t, busy, deletingId, remove, openEdit, openCreate, worksWithText], + ); + + const tableGroups = useMemo[]>(() => { + const gs: DataTableGroup[] = []; + for (const { type, connections: list } of connectedGroups) { + gs.push({ + key: `connected-${type.id}`, + title: t(type.labelKey), + meta: + list.length > 1 + ? t("portal.integrations.connectionCount", { count: list.length }) + : t("portal.integrations.status.connected"), + actions: [ + { + label: t("portal.integrations.connect"), + onClick: () => openCreate(type.id), + }, + ], + rows: list.map((c) => ({ + kind: "instance" as const, + key: `i-${c.id}`, + brandId: type.id, + title: c.name, + subtitle: connectionDetail(c), + worksWith: worksWith(type), + connection: c, + canManage: !!c.canManage, + })), + }); + } + if (availableTypes.length > 0) { + gs.push({ + key: "available", + title: t("portal.integrations.availableHeading"), + rows: availableTypes.map((type) => ({ + kind: "available" as const, + key: `a-${type.id}`, + brandId: type.id, + title: t(type.labelKey), + subtitle: t(type.descriptionKey), + worksWith: worksWith(type), + typeId: type.id, + })), + }); + } + if (comingSoon.length > 0) { + gs.push({ + key: "soon", + title: t("portal.integrations.comingSoonHeading"), + muted: true, + rows: comingSoon.map((entry) => ({ + kind: "soon" as const, + key: `s-${entry.type}`, + brandId: entry.type, + title: t(entry.labelKey), + subtitle: t(entry.descriptionKey), + worksWith: ["sources"], + })), + }); + } + return gs; + }, [connectedGroups, availableTypes, comingSoon, t, openCreate]); + return (
@@ -301,187 +428,23 @@ export function Integrations() { {error && } - {isLoading ? ( -
- {Array.from({ length: 4 }).map((_, i) => ( - - ))} -
+ {!isLoading && tableGroups.length === 0 ? ( + ) : ( -
-
- {t("portal.integrations.table.integration")} - {t("portal.integrations.table.worksWith")} - -
- - {connectedGroups.length > 0 && ( -
- {t("portal.integrations.connectedHeading")} ·{" "} - {connectedGroups.length} -
- )} - {connectedGroups.map(({ type, connections: list }) => { - const open = expanded.has(type.id); - return ( -
- - {open && ( -
- {list.map((connection) => ( -
- - {connection.name} - - - {connectionDetail(connection)} - - {connection.canManage && ( - - - - - )} -
- ))} -
- -
-
- )} -
- ); - })} - - {availableTypes.length > 0 && ( -
- {t("portal.integrations.availableHeading")} ·{" "} - {availableTypes.length} -
- )} - {availableTypes.map((type) => ( -
- - - - - {t(type.labelKey)} - - - {t(type.descriptionKey)} - - - - - {worksWith(type).map(chip)} - - - - -
- ))} - - {comingSoon.length > 0 && ( -
- {t("portal.integrations.comingSoonHeading")} · {comingSoon.length} -
- )} - {comingSoon.map((entry) => ( -
- - - - - {t(entry.labelKey)} - - - {t(entry.descriptionKey)} - - - - - {chip("sources")} - - - - {t("portal.sources.builder.comingSoon")} - - -
- ))} -
+ + columns={columns} + groups={tableGroups} + rowKey={(r) => r.key} + loading={isLoading} + skeletonRows={5} + /> )} DELETE /invitations/{id} -> refetch drops the invite. fireEvent.click( diff --git a/frontend/editor/src/portal/views/Users.tsx b/frontend/editor/src/portal/views/Users.tsx index 535bf3e627..79c6c11611 100644 --- a/frontend/editor/src/portal/views/Users.tsx +++ b/frontend/editor/src/portal/views/Users.tsx @@ -1,4 +1,4 @@ -import { useEffect, useMemo, useState } from "react"; +import { useEffect, useMemo, useRef, useState } from "react"; import { useSearchParams } from "react-router-dom"; import { useTranslation } from "react-i18next"; import { Button, EmptyState, Skeleton } from "@app/ui"; @@ -71,12 +71,14 @@ export function Users() { }, [searchParams, setSearchParams]); // Scroll to and flash the row for ?member= (deep link from the super - // search), once the roster has rendered; then strip the param. + // search), once the roster has rendered; then strip the param. Scoped to the + // roster so a pending-invitation row sharing the id can't match first. + const rosterRef = useRef(null); useEffect(() => { const memberId = searchParams.get("member"); if (memberId === null || usersState.loading) return; - const row = document.querySelector( - `[data-member-id="${CSS.escape(memberId)}"]`, + const row = rosterRef.current?.querySelector( + `[data-row-key="${CSS.escape(memberId)}"]`, ); if (row) { row.scrollIntoView({ block: "center" }); @@ -343,28 +345,30 @@ export function Users() { )} {!loading && members.length > 0 && ( - openInvite(team.id)} - onResetPassword={setResetPwMember} - onMoveToTeam={setMoveMember} - onToggleEnabled={toggleEnabled} - onUnlock={unlock} - onDisableMfa={disableMfa} - onRemove={removeUser} - onRenameTeam={(team) => - setRenameTarget({ id: team.id, name: team.name }) - } - onDeleteTeam={deleteTeamAction} - /> +
+ openInvite(team.id)} + onResetPassword={setResetPwMember} + onMoveToTeam={setMoveMember} + onToggleEnabled={toggleEnabled} + onUnlock={unlock} + onDisableMfa={disableMfa} + onRemove={removeUser} + onRenameTeam={(team) => + setRenameTarget({ id: team.id, name: team.name }) + } + onDeleteTeam={deleteTeamAction} + /> +
)} =20" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/tannerlinsley" + }, + "peerDependencies": { + "react": ">=18" + } + }, "node_modules/@tanstack/react-virtual": { "version": "3.13.23", "resolved": "https://registry.npmjs.org/@tanstack/react-virtual/-/react-virtual-3.13.23.tgz", @@ -4953,6 +4992,32 @@ "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" } }, + "node_modules/@tanstack/store": { + "version": "0.11.1", + "resolved": "https://registry.npmjs.org/@tanstack/store/-/store-0.11.1.tgz", + "integrity": "sha512-mzTOBhypOuDJAy/D8n2MfUZ1HFkXnmSETviRyhqEC8LUE7/IZQExOTxMANj3KjTofYTkFNpBY67qaVrT41YccA==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/tannerlinsley" + } + }, + "node_modules/@tanstack/table-core": { + "version": "9.1.2", + "resolved": "https://registry.npmjs.org/@tanstack/table-core/-/table-core-9.1.2.tgz", + "integrity": "sha512-ONpWQeass1sfg80CWF1NSwQ8r3GiqxA2lT/EdqIcrDEPZ0Z+0mM94eQoFYLPN0Kztzj8TQVb2+PrSZSItqA61g==", + "license": "MIT", + "dependencies": { + "@tanstack/store": "^0.11.0" + }, + "engines": { + "node": ">=20" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/tannerlinsley" + } + }, "node_modules/@tanstack/virtual-core": { "version": "3.13.23", "resolved": "https://registry.npmjs.org/@tanstack/virtual-core/-/virtual-core-3.13.23.tgz", diff --git a/frontend/package.json b/frontend/package.json index 3fd1f8e106..90a0e10b05 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -47,6 +47,7 @@ "@supabase/supabase-js": "^2.47.13", "@tailwindcss/postcss": "^4.1.13", "@tanstack/react-query": "^5.101.4", + "@tanstack/react-table": "^9.1.2", "@tanstack/react-virtual": "^3.13.12", "@tauri-apps/api": "^2.10.1", "@tauri-apps/plugin-dialog": "2.7.0", From 526bb85e17e5224f264af8d891399675291268f1 Mon Sep 17 00:00:00 2001 From: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com> Date: Mon, 17 Aug 2026 17:00:35 +0000 Subject: [PATCH 18/59] Translate the failures debug panel strings (#7500) Follow-up to #7296, addressing a missing translation. --- .../public/locales/en-US/translation.toml | 8 ++++++++ .../components/failures/FileRunEventList.tsx | 18 ++++++++++++++---- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml index 99d68c35de..70d3d03bf4 100644 --- a/frontend/editor/public/locales/en-US/translation.toml +++ b/frontend/editor/public/locales/en-US/translation.toml @@ -7372,6 +7372,14 @@ confirm = "Are you sure?" dismiss = "Dismiss" dismissSkipFile = "Skip this file" +[portal.failures.debug] +copyJson = "Copy JSON" +dismissAll = "Dismiss all ({{total}})" +dismissing = "Dismissing..." +hideJson = "Hide raw JSON ({{total}})" +refresh = "Refresh failures" +showJson = "Show raw JSON ({{total}})" + [portal.failures.disabled] closed = "This failure is already closed." unavailable = "Not available for this failure." diff --git a/frontend/editor/src/portal/components/failures/FileRunEventList.tsx b/frontend/editor/src/portal/components/failures/FileRunEventList.tsx index 477af32e50..2330546655 100644 --- a/frontend/editor/src/portal/components/failures/FileRunEventList.tsx +++ b/frontend/editor/src/portal/components/failures/FileRunEventList.tsx @@ -69,7 +69,7 @@ export function FileRunEventList() { const debugPanel = !import.meta.env.DEV ? null : (
{showJson && (

From 89d8ffec5d1266251b7feeb05a3d998a6d5c747f Mon Sep 17 00:00:00 2001
From: Anthony Stirling <77850077+Frooodle@users.noreply.github.com>
Date: Mon, 17 Aug 2026 17:49:13 +0000
Subject: [PATCH 19/59] Ci/environments cleanups, new envs and master to
 release naming (#7511)

# Description of Changes

Ci/environments cleanups, new envs and master to release naming

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
---
 .github/config/.files.yaml                    |   1 -
 .github/workflows/PR-Demo-cleanup.yml         |   4 -
 .github/workflows/ai_pr_title_review.yml      | 221 ----------------
 .github/workflows/backend-build.yml           |   1 +
 .github/workflows/build-enterprise.yml        |   2 +
 .github/workflows/build.yml                   |   1 +
 .github/workflows/check-licence.yml           |   1 +
 .github/workflows/check-openapi.yml           |   1 +
 .github/workflows/db-migration-test.yml       |   1 +
 .github/workflows/deploy-on-v2-commit.yml     | 209 ----------------
 .github/workflows/docker-compose-tests.yml    |   1 +
 .github/workflows/e2e-live.yml                |   1 +
 .../frontend-backend-licenses-update.yml      |   4 +
 .github/workflows/multiOSReleases.yml         |  63 ++---
 .github/workflows/nightly.yml                 |   1 +
 .github/workflows/push-docker-base.yml        |   3 +
 .github/workflows/push-docker.yml             |  29 ++-
 .github/workflows/swagger.yml                 |   5 +-
 .github/workflows/tauri-build.yml             |  42 +---
 .github/workflows/test-build-docker.yml       |   1 +
 .github/workflows/testdriver.yml              | 235 ------------------
 WINDOWS_SIGNING.md                            |  71 +++---
 .../editor/src/core/services/updateService.ts |   2 +-
 23 files changed, 95 insertions(+), 805 deletions(-)
 delete mode 100644 .github/workflows/ai_pr_title_review.yml
 delete mode 100644 .github/workflows/deploy-on-v2-commit.yml
 delete mode 100644 .github/workflows/testdriver.yml

diff --git a/.github/config/.files.yaml b/.github/config/.files.yaml
index 70a964b020..b5cc0527b0 100644
--- a/.github/config/.files.yaml
+++ b/.github/config/.files.yaml
@@ -68,7 +68,6 @@ project: &project
 frontend: &frontend
   - *ci
   - frontend/**
-  - .github/workflows/testdriver.yml
   - testing/**
   - docker/**
   - scripts/translations/*.py
diff --git a/.github/workflows/PR-Demo-cleanup.yml b/.github/workflows/PR-Demo-cleanup.yml
index e0032955e3..1407939994 100644
--- a/.github/workflows/PR-Demo-cleanup.yml
+++ b/.github/workflows/PR-Demo-cleanup.yml
@@ -7,10 +7,6 @@ on:
 permissions:
   contents: read
 
-env:
-  SERVER_IP: ${{ secrets.NEW_VPS_IP }} # Add this to your GitHub secrets
-  CLEANUP_PERFORMED: "false" # Add flag to track if cleanup occurred
-
 jobs:
   cleanup:
     environment: pr-preview
diff --git a/.github/workflows/ai_pr_title_review.yml b/.github/workflows/ai_pr_title_review.yml
deleted file mode 100644
index b9b391af0e..0000000000
--- a/.github/workflows/ai_pr_title_review.yml
+++ /dev/null
@@ -1,221 +0,0 @@
-name: AI - PR Title Review
-
-on:
-  pull_request:
-    types: [opened, edited]
-    branches: [main]
-
-permissions: # required for secure-repo hardening
-  contents: read
-
-jobs:
-  ai-title-review:
-    # GITHUB_TOKEN obeys this block, so it must cover every API call made below.
-    permissions:
-      contents: read # actions/checkout, git fetch/diff
-      issues: write # issues.listComments / createComment / updateComment on the PR
-      pull-requests: write # same endpoints when the target is a pull request
-      models: read # actions/ai-inference
-
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
-        with:
-          egress-policy: audit
-
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-        with:
-          fetch-depth: 0
-
-      - name: Configure Git to suppress detached HEAD warning
-        run: git config --global advice.detachedHead false
-
-      - name: Check if actor is repo developer
-        id: actor
-        run: |
-          if [[ "${{ github.actor }}" == *"[bot]" ]]; then
-            echo "PR opened by a bot – skipping AI title review."
-            echo "is_repo_dev=false" >> $GITHUB_OUTPUT
-            exit 0
-          fi
-          if [ ! -f .github/config/repo_devs.json ]; then
-            echo "Error: .github/config/repo_devs.json not found" >&2
-            exit 1
-          fi
-          # Validate JSON and extract repo_devs
-          REPO_DEVS=$(jq -r '.repo_devs[]' .github/config/repo_devs.json 2>/dev/null || { echo "Error: Invalid JSON in repo_devs.json" >&2; exit 1; })
-          # Convert developer list into Bash array
-          mapfile -t DEVS_ARRAY <<< "$REPO_DEVS"
-          if [[ " ${DEVS_ARRAY[*]} " == *" ${{ github.actor }} "* ]]; then
-            echo "is_repo_dev=true" >> $GITHUB_OUTPUT
-          else
-            echo "is_repo_dev=false" >> $GITHUB_OUTPUT
-          fi
-
-      - name: Get PR diff
-        if: steps.actor.outputs.is_repo_dev == 'true'
-        id: get_diff
-        run: |
-          git fetch origin ${{ github.base_ref }}
-          git diff origin/${{ github.base_ref }}...HEAD | head -n 10000 | grep -vP '[\x00-\x08\x0B\x0C\x0E-\x1F\x7F\x{202E}\x{200B}]' > pr.diff
-          echo "diff<> $GITHUB_OUTPUT
-          cat pr.diff >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
-
-      - name: Check and sanitize PR title
-        if: steps.actor.outputs.is_repo_dev == 'true'
-        id: sanitize_pr_title
-        env:
-          PR_TITLE_RAW: ${{ github.event.pull_request.title }}
-        run: |
-          # Sanitize PR title: max 72 characters, only printable characters
-          PR_TITLE=$(echo "$PR_TITLE_RAW" | tr -d '\n\r' | head -c 72 | sed 's/[^[:print:]]//g')
-          if [[ ${#PR_TITLE} -lt 5 ]]; then
-            echo "PR title is too short. Must be at least 5 characters." >&2
-          fi
-          echo "pr_title=$PR_TITLE" >> $GITHUB_OUTPUT
-
-      - name: AI PR Title Analysis
-        if: steps.actor.outputs.is_repo_dev == 'true'
-        id: ai-title-analysis
-        uses: actions/ai-inference@a7805884c80886efc241e94a5351df715968a0ad # v2.1.1
-        with:
-          model: openai/gpt-4o
-          system-prompt-file: ".github/config/system-prompt.txt"
-          prompt: |
-            Based on the following input data:
-
-            {
-              "diff": "${{ steps.get_diff.outputs.diff }}",
-              "pr_title": "${{ steps.sanitize_pr_title.outputs.pr_title }}"
-            }
-
-            Respond ONLY with valid JSON in the format:
-            {
-              "improved_rating": <0-10>,
-              "improved_ai_title_rating": <0-10>,
-              "improved_title": ""
-            }
-
-      - name: Validate and set SCRIPT_OUTPUT
-        if: steps.actor.outputs.is_repo_dev == 'true'
-        run: |
-          cat < ai_response.json
-          ${{ steps.ai-title-analysis.outputs.response }}
-          EOF
-
-          # Validate JSON structure
-          jq -e '
-            (keys | sort) == ["improved_ai_title_rating", "improved_rating", "improved_title"] and
-            (.improved_rating | type == "number" and . >= 0 and . <= 10) and
-            (.improved_ai_title_rating | type == "number" and . >= 0 and . <= 10) and
-            (.improved_title | type == "string")
-          ' ai_response.json
-          if [ $? -ne 0 ]; then
-            echo "Invalid AI response format" >&2
-            cat ai_response.json >&2
-            exit 1
-          fi
-          # Parse JSON fields
-          IMPROVED_RATING=$(jq -r '.improved_rating' ai_response.json)
-          IMPROVED_TITLE=$(jq -r '.improved_title' ai_response.json)
-          # Limit comment length to 1000 characters
-          COMMENT=$(cat < /tmp/ai-title-comment.md
-          # Log input and output to the GitHub Step Summary
-          echo "### 🤖 AI PR Title Analysis" >> $GITHUB_STEP_SUMMARY
-          echo "### Input PR Title" >> $GITHUB_STEP_SUMMARY
-          echo '```bash' >> $GITHUB_STEP_SUMMARY
-          echo "${{ steps.sanitize_pr_title.outputs.pr_title }}" >> $GITHUB_STEP_SUMMARY
-          echo '```' >> $GITHUB_STEP_SUMMARY
-          echo '### AI Response (raw JSON)' >> $GITHUB_STEP_SUMMARY
-          echo '```json' >> $GITHUB_STEP_SUMMARY
-          cat ai_response.json >> $GITHUB_STEP_SUMMARY
-          echo '```' >> $GITHUB_STEP_SUMMARY
-
-      - name: Post comment on PR if needed
-        if: steps.actor.outputs.is_repo_dev == 'true'
-        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
-        continue-on-error: true
-        with:
-          github-token: ${{ github.token }}
-          script: |
-            const fs = require('fs');
-            const body = fs.readFileSync('/tmp/ai-title-comment.md', 'utf8');
-            const { GITHUB_REPOSITORY } = process.env;
-            const [owner, repo] = GITHUB_REPOSITORY.split('/');
-            const issue_number = context.issue.number;
-
-            const ratingMatch = body.match(/\*\*PR-Title Rating\*\*: (\d+)\/10/);
-            const rating = ratingMatch ? parseInt(ratingMatch[1], 10) : null;
-
-            const expectedActor = "github-actions[bot]";
-            const comments = await github.rest.issues.listComments({ owner, repo, issue_number });
-
-            const existing = comments.data.find(c =>
-              c.user?.login === expectedActor &&
-              c.body.includes("## 🤖 AI PR Title Suggestion")
-            );
-
-            if (rating === null) {
-              console.log("No rating found in AI response – skipping.");
-              return;
-            }
-
-            if (rating <= 5) {
-              if (existing) {
-                await github.rest.issues.updateComment({
-                  owner, repo,
-                  comment_id: existing.id,
-                  body
-                });
-                console.log("Updated existing suggestion comment.");
-              } else {
-                await github.rest.issues.createComment({
-                  owner, repo, issue_number,
-                  body
-                });
-                console.log("Created new suggestion comment.");
-              }
-            } else {
-              const praise = `## 🤖 AI PR Title Suggestion\n\nGreat job! The current PR title is clear and well-structured.\n\n✅ No suggestions needed.\n\n---\n*Generated by GitHub Models AI*`;
-
-              if (existing) {
-                await github.rest.issues.updateComment({
-                  owner, repo,
-                  comment_id: existing.id,
-                  body: praise
-                });
-                console.log("Replaced suggestion with praise.");
-              } else {
-                console.log("Rating > 5 and no existing comment – skipping comment.");
-              }
-            }
-
-      - name: is not repo dev
-        if: steps.actor.outputs.is_repo_dev != 'true'
-        run: |
-          exit 0 # Skip the AI title review for non-repo developers
-
-      - name: Clean up
-        if: always()
-        run: |
-          rm -f pr.diff ai_response.json /tmp/ai-title-comment.md
-          echo "Cleaned up temporary files."
-        continue-on-error: true # Ensure cleanup runs even if previous steps fail
diff --git a/.github/workflows/backend-build.yml b/.github/workflows/backend-build.yml
index 9561833f07..54bd4cb907 100644
--- a/.github/workflows/backend-build.yml
+++ b/.github/workflows/backend-build.yml
@@ -20,6 +20,7 @@ permissions:
 
 jobs:
   build:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
diff --git a/.github/workflows/build-enterprise.yml b/.github/workflows/build-enterprise.yml
index 08849c683b..0604f7176f 100644
--- a/.github/workflows/build-enterprise.yml
+++ b/.github/workflows/build-enterprise.yml
@@ -37,6 +37,7 @@ jobs:
     uses: ./.github/workflows/_runner-pick.yml
 
   playwright-e2e-enterprise:
+    environment: ci-unsigned
     needs: pick
     # Skip on fork PRs / untrusted authors: they have no PREMIUM_KEY_ENTERPRISE,
     # so the suite can't boot premium and would fail. See the header comment.
@@ -309,6 +310,7 @@ jobs:
   # Multi-node regression: builds + seeds the clustered stack (testing/compose/docker-compose-multinode.yml)
   # and runs behave features/multinode. Licence-gated, so it runs after the Playwright job (not in parallel).
   multinode-e2e:
+    environment: ci-unsigned
     needs: [pick, playwright-e2e-enterprise]
     # Nightly cron + manual dispatch only (heavy build), fork-gated for the licence secret.
     if: >-
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index eee07d599a..2f50249099 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -61,6 +61,7 @@ jobs:
           filters: .github/config/.files.yaml
 
   gradle-cache-prime:
+    environment: ci-unsigned
     name: Prime shared Gradle cache
     needs: [files-changed]
     runs-on: ubuntu-latest
diff --git a/.github/workflows/check-licence.yml b/.github/workflows/check-licence.yml
index d45a68e860..2eec970b8f 100644
--- a/.github/workflows/check-licence.yml
+++ b/.github/workflows/check-licence.yml
@@ -10,6 +10,7 @@ permissions:
 
 jobs:
   check-licence:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     steps:
       - name: Harden Runner
diff --git a/.github/workflows/check-openapi.yml b/.github/workflows/check-openapi.yml
index 751a34a33e..f224ce18cf 100644
--- a/.github/workflows/check-openapi.yml
+++ b/.github/workflows/check-openapi.yml
@@ -11,6 +11,7 @@ permissions:
 
 jobs:
   check-generate-openapi-docs:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     steps:
       - name: Harden Runner
diff --git a/.github/workflows/db-migration-test.yml b/.github/workflows/db-migration-test.yml
index 4181390285..d6a61b45c4 100644
--- a/.github/workflows/db-migration-test.yml
+++ b/.github/workflows/db-migration-test.yml
@@ -13,6 +13,7 @@ permissions:
 
 jobs:
   migration-test:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
diff --git a/.github/workflows/deploy-on-v2-commit.yml b/.github/workflows/deploy-on-v2-commit.yml
deleted file mode 100644
index 01114c64b2..0000000000
--- a/.github/workflows/deploy-on-v2-commit.yml
+++ /dev/null
@@ -1,209 +0,0 @@
-name: Auto V2 Deploy on Push
-
-on:
-  push:
-    branches:
-      - V2
-      - deploy-on-v2-commit
-
-permissions:
-  contents: read
-
-jobs:
-  deploy-v2-on-push:
-    environment: pr-preview
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      packages: write
-    concurrency:
-      group: deploy-v2-push-V2
-      cancel-in-progress: true
-
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
-        with:
-          egress-policy: audit
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
-
-      - name: Get commit hashes for frontend and backend
-        id: commit-hashes
-        run: |
-          # Get last commit that touched the frontend folder, docker/frontend, or docker/compose
-          FRONTEND_HASH=$(git log -1 --format="%H" -- frontend/ docker/frontend/ docker/compose/ 2>/dev/null || echo "")
-          if [ -z "$FRONTEND_HASH" ]; then
-            FRONTEND_HASH="no-frontend-changes"
-          fi
-
-          # Get last commit that touched backend code, docker/backend, or docker/compose
-          BACKEND_HASH=$(git log -1 --format="%H" -- app/ docker/backend/ docker/compose/ 2>/dev/null || echo "")
-          if [ -z "$BACKEND_HASH" ]; then
-            BACKEND_HASH="no-backend-changes"
-          fi
-
-          echo "Frontend hash: $FRONTEND_HASH"
-          echo "Backend hash: $BACKEND_HASH"
-
-          echo "frontend_hash=$FRONTEND_HASH" >> $GITHUB_OUTPUT
-          echo "backend_hash=$BACKEND_HASH" >> $GITHUB_OUTPUT
-
-          # Short hashes for tags
-          if [ "$FRONTEND_HASH" = "no-frontend-changes" ]; then
-            echo "frontend_short=no-frontend" >> $GITHUB_OUTPUT
-          else
-            echo "frontend_short=${FRONTEND_HASH:0:8}" >> $GITHUB_OUTPUT
-          fi
-
-          if [ "$BACKEND_HASH" = "no-backend-changes" ]; then
-            echo "backend_short=no-backend" >> $GITHUB_OUTPUT
-          else
-            echo "backend_short=${BACKEND_HASH:0:8}" >> $GITHUB_OUTPUT
-          fi
-
-      - name: Convert repository owner to lowercase
-        id: repoowner
-        run: echo "lowercase=$(echo ${{ github.repository_owner }} | awk '{print tolower($0)}')" >> $GITHUB_OUTPUT
-
-      - name: Login to GitHub Container Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ github.token }}
-
-      - name: Check if frontend image exists
-        id: check-frontend
-        run: |
-          if docker manifest inspect ${IMAGE_BASE}:v2-frontend-${{ steps.commit-hashes.outputs.frontend_short }} >/dev/null 2>&1; then
-            echo "exists=true" >> $GITHUB_OUTPUT
-            echo "Frontend image already exists, skipping build"
-          else
-            echo "exists=false" >> $GITHUB_OUTPUT
-            echo "Frontend image needs to be built"
-          fi
-
-        env:
-          IMAGE_BASE: ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test
-      - name: Check if backend image exists
-        id: check-backend
-        run: |
-          if docker manifest inspect ${IMAGE_BASE}:v2-backend-${{ steps.commit-hashes.outputs.backend_short }} >/dev/null 2>&1; then
-            echo "exists=true" >> $GITHUB_OUTPUT
-            echo "Backend image already exists, skipping build"
-          else
-            echo "exists=false" >> $GITHUB_OUTPUT
-            echo "Backend image needs to be built"
-          fi
-
-        env:
-          IMAGE_BASE: ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test
-
-      - name: Build and push frontend image
-        if: steps.check-frontend.outputs.exists == 'false'
-        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
-        with:
-          context: .
-          file: ./docker/frontend/Dockerfile
-          push: true
-          cache-from: type=gha,scope=stirling-v2-frontend
-          cache-to: type=gha,mode=max,scope=stirling-v2-frontend
-          tags: |
-            ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test:v2-frontend-${{ steps.commit-hashes.outputs.frontend_short }}
-            ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test:v2-frontend-latest
-          build-args: VERSION_TAG=v2-alpha
-          platforms: linux/amd64
-
-      - name: Build and push backend image
-        if: steps.check-backend.outputs.exists == 'false'
-        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
-        with:
-          context: .
-          file: ./docker/backend/Dockerfile
-          push: true
-          cache-from: type=gha,scope=stirling-v2-backend
-          cache-to: type=gha,mode=max,scope=stirling-v2-backend
-          tags: |
-            ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test:v2-backend-${{ steps.commit-hashes.outputs.backend_short }}
-            ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test:v2-backend-latest
-          build-args: VERSION_TAG=v2-alpha
-          platforms: linux/amd64
-
-      - name: Set up SSH
-        run: |
-          mkdir -p ~/.ssh/
-          echo "${NEW_VPS_SSH_KEY}" > ../private.key
-          chmod 600 ../private.key
-
-        env:
-          NEW_VPS_SSH_KEY: ${{ secrets.NEW_VPS_SSH_KEY }}
-      - name: Deploy to VPS on port 3000
-        run: |
-          export UNIQUE_NAME=docker-compose-v2-$GITHUB_RUN_ID.yml
-
-          cat > $UNIQUE_NAME << EOF
-          version: '3.3'
-          services:
-            backend:
-              container_name: stirling-v2-backend
-              image: ${IMAGE_BASE}:v2-backend-${{ steps.commit-hashes.outputs.backend_short }}
-              ports:
-                - "13000:8080"
-              volumes:
-                - /stirling/V2/data:/usr/share/tessdata:rw
-                - /stirling/V2/config:/configs:rw
-                - /stirling/V2/logs:/logs:rw
-              environment:
-                DISABLE_ADDITIONAL_FEATURES: "true"
-                SECURITY_ENABLELOGIN: "false"
-                SYSTEM_DEFAULTLOCALE: en-US
-                UI_APPNAME: "Stirling-PDF V2"
-                UI_HOMEDESCRIPTION: "V2 Frontend/Backend Split"
-                UI_APPNAMENAVBAR: "V2 Deployment"
-                SYSTEM_MAXFILESIZE: "100"
-                METRICS_ENABLED: "true"
-                SYSTEM_GOOGLEVISIBILITY: "false"
-                SWAGGER_SERVER_URL: "https://demo.stirlingpdf.cloud"
-                baseUrl: "https://demo.stirlingpdf.cloud"
-              restart: on-failure:5
-
-            frontend:
-              container_name: stirling-v2-frontend
-              image: ${IMAGE_BASE}:v2-frontend-${{ steps.commit-hashes.outputs.frontend_short }}
-              ports:
-                - "3000:80"
-              environment:
-                VITE_API_BASE_URL: "http://${NEW_VPS_HOST}:13000"
-              depends_on:
-                - backend
-              restart: on-failure:5
-          EOF
-
-          # Copy to remote with unique name
-          scp -i ../private.key -o StrictHostKeyChecking=no $UNIQUE_NAME ${NEW_VPS_USERNAME}@${NEW_VPS_HOST}:/tmp/$UNIQUE_NAME
-
-          # SSH and rename/move atomically to avoid interference
-          ssh -i ../private.key -o StrictHostKeyChecking=no ${NEW_VPS_USERNAME}@${NEW_VPS_HOST} << ENDSSH
-            mkdir -p /stirling/V2/{data,config,logs}
-            mv /tmp/$UNIQUE_NAME /stirling/V2/docker-compose.yml
-            cd /stirling/V2
-            docker-compose down || true
-            docker-compose pull
-            docker-compose up -d
-            docker system prune -af --volumes || true
-            docker image prune -af --filter "until=336h" --filter "label!=keep=true" || true
-          ENDSSH
-
-        env:
-          IMAGE_BASE: ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test
-          NEW_VPS_HOST: ${{ secrets.NEW_VPS_HOST }}
-          NEW_VPS_USERNAME: ${{ secrets.NEW_VPS_USERNAME }}
-      - name: Cleanup temporary files
-        if: always()
-        run: |
-          rm -f ../private.key
diff --git a/.github/workflows/docker-compose-tests.yml b/.github/workflows/docker-compose-tests.yml
index ddb35b4e1a..9d5911404f 100644
--- a/.github/workflows/docker-compose-tests.yml
+++ b/.github/workflows/docker-compose-tests.yml
@@ -17,6 +17,7 @@ permissions:
 
 jobs:
   docker-compose-tests:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     permissions:
       actions: write
diff --git a/.github/workflows/e2e-live.yml b/.github/workflows/e2e-live.yml
index 4aa2a78c89..7bc95df05e 100644
--- a/.github/workflows/e2e-live.yml
+++ b/.github/workflows/e2e-live.yml
@@ -11,6 +11,7 @@ permissions:
 
 jobs:
   playwright-e2e-live:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
diff --git a/.github/workflows/frontend-backend-licenses-update.yml b/.github/workflows/frontend-backend-licenses-update.yml
index 1155f01e39..458766660f 100644
--- a/.github/workflows/frontend-backend-licenses-update.yml
+++ b/.github/workflows/frontend-backend-licenses-update.yml
@@ -42,6 +42,8 @@ jobs:
           filters: .github/config/.files.yaml
 
   generate-frontend-license-report:
+    # ci-bot, not bot-identity: this job runs on PRs too, and bot-identity is main-only.
+    environment: ci-bot
     if: needs.files-changed.outputs.licenses-frontend == 'true'
     name: Generate Frontend License Report
     needs: files-changed
@@ -316,6 +318,8 @@ jobs:
           GH_TOKEN: ${{ steps.setup-bot.outputs.token }}
 
   generate-backend-license-report:
+    # ci-bot, not bot-identity: this job runs on PRs too, and bot-identity is main-only.
+    environment: ci-bot
     if: needs.files-changed.outputs.licenses-backend == 'true'
     needs: files-changed
     name: Generate Backend License Report
diff --git a/.github/workflows/multiOSReleases.yml b/.github/workflows/multiOSReleases.yml
index 3bb014a82d..d9477f722f 100644
--- a/.github/workflows/multiOSReleases.yml
+++ b/.github/workflows/multiOSReleases.yml
@@ -38,6 +38,7 @@ permissions:
 
 jobs:
   determine-matrix:
+    environment: ci-unsigned
     if: ${{ vars.CI_PROFILE != 'lite' }}
     runs-on: ubuntu-latest
     outputs:
@@ -118,6 +119,7 @@ jobs:
         env:
           INPUT_PLATFORM: ${{ github.event.inputs.platform }}
   build-jars:
+    environment: ci-unsigned
     needs: determine-matrix
     runs-on: ubuntu-latest
     strategy:
@@ -204,7 +206,6 @@ jobs:
     runs-on: ${{ matrix.platform }}
     env:
       SM_API_KEY: ${{ secrets.SM_API_KEY }}
-      WINDOWS_CERTIFICATE: ${{ secrets.WINDOWS_CERTIFICATE }}
       RELEASE_GPG_PRIVATE_KEY: ${{ secrets.RELEASE_GPG_PRIVATE_KEY }}
     steps:
       - name: Harden Runner
@@ -295,7 +296,7 @@ jobs:
       # DigiCert KeyLocker Setup (Cloud HSM)
       - name: Setup DigiCert KeyLocker
         id: digicert-setup
-        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master') }}
+        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release') }}
         uses: digicert/ssm-code-signing@1d820463733701cf1484c7eb5d7d24a15ca2c454 # v1.2.1
         env:
           SM_API_KEY: ${{ secrets.SM_API_KEY }}
@@ -305,7 +306,7 @@ jobs:
           SM_HOST: ${{ secrets.SM_HOST }}
 
       - name: Setup DigiCert KeyLocker Certificate
-        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master') }}
+        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release') }}
         shell: pwsh
         run: |
           Write-Host "Setting up DigiCert KeyLocker environment..."
@@ -344,40 +345,8 @@ jobs:
           SM_API_KEY: ${{ secrets.SM_API_KEY }}
           SM_CLIENT_CERT_PASSWORD: ${{ secrets.SM_CLIENT_CERT_PASSWORD }}
           SM_KEYPAIR_ALIAS: ${{ secrets.SM_KEYPAIR_ALIAS }}
-      # Traditional PFX Certificate Import (fallback if KeyLocker not configured)
-      - name: Import Windows Code Signing Certificate
-        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY == '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master') }}
-        env:
-          WINDOWS_CERTIFICATE: ${{ secrets.WINDOWS_CERTIFICATE }}
-          WINDOWS_CERTIFICATE_PASSWORD: ${{ secrets.WINDOWS_CERTIFICATE_PASSWORD }}
-        shell: powershell
-        run: |
-          if ($env:WINDOWS_CERTIFICATE) {
-            Write-Host "Importing Windows Code Signing Certificate..."
-
-            # Decode base64 certificate and save to file
-            $certBytes = [Convert]::FromBase64String($env:WINDOWS_CERTIFICATE)
-            $certPath = Join-Path $env:RUNNER_TEMP "certificate.pfx"
-            [IO.File]::WriteAllBytes($certPath, $certBytes)
-
-            # Import certificate to CurrentUser\My store
-            $cert = Import-PfxCertificate -FilePath $certPath -CertStoreLocation Cert:\CurrentUser\My -Password (ConvertTo-SecureString -String $env:WINDOWS_CERTIFICATE_PASSWORD -AsPlainText -Force)
-
-            # Extract and set thumbprint as environment variable
-            $thumbprint = $cert.Thumbprint
-            Write-Host "Certificate imported with thumbprint: $thumbprint"
-            echo "WINDOWS_CERTIFICATE_THUMBPRINT=$thumbprint" >> $env:GITHUB_ENV
-
-            # Clean up certificate file
-            Remove-Item $certPath
-
-            Write-Host "Windows certificate import completed."
-          } else {
-            Write-Host "⚠️ WINDOWS_CERTIFICATE secret not set - building unsigned binary"
-          }
-
       - name: Import Apple Developer Certificate
-        if: matrix.platform == 'macos-15' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master')
+        if: matrix.platform == 'macos-15' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release')
         env:
           APPLE_CERTIFICATE: ${{ secrets.APPLE_CERTIFICATE }}
           APPLE_CERTIFICATE_PASSWORD: ${{ secrets.APPLE_CERTIFICATE_PASSWORD }}
@@ -398,7 +367,7 @@ jobs:
           rm certificate.p12
 
       - name: Verify Certificate
-        if: matrix.platform == 'macos-15' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master')
+        if: matrix.platform == 'macos-15' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release')
         run: |
           echo "Verifying Apple Developer Certificate..."
           KEYCHAIN_PATH=$RUNNER_TEMP/app-signing.keychain-db
@@ -414,7 +383,7 @@ jobs:
       # Without this, signCommand failures are opaque (Tauri captures but drops
       # smctl's stderr) - running these loudly surfaces auth/env/keypair issues.
       - name: Preflight smctl
-        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master') }}
+        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release') }}
         shell: pwsh
         env:
           KEYPAIR_ALIAS: ${{ secrets.SM_KEYPAIR_ALIAS }}
@@ -445,7 +414,7 @@ jobs:
       # smctl reads SM_HOST, SM_API_KEY, SM_CLIENT_CERT_FILE, SM_CLIENT_CERT_PASSWORD
       # from env (set by prior DigiCert setup step). No --config-file needed.
       - name: Configure Windows code signing
-        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master') }}
+        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release') }}
         shell: bash
         env:
           KEYPAIR_ALIAS: ${{ secrets.SM_KEYPAIR_ALIAS }}
@@ -466,7 +435,7 @@ jobs:
           sed "s/${KEYPAIR_ALIAS}/***/g" ./frontend/editor/src-tauri/tauri.windows.conf.json
 
       - name: Import release GPG signing key (Linux)
-        if: matrix.platform == 'ubuntu-22.04' && env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master')
+        if: matrix.platform == 'ubuntu-22.04' && env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release')
         run: |
           echo "$RELEASE_GPG_PRIVATE_KEY" | gpg --batch --import
           gpg --list-secret-keys --keyid-format=long
@@ -498,8 +467,8 @@ jobs:
           #   APPIMAGETOOL_SIGN_PASSPHRASE  appimagetool uses this to unlock the GPG key non-interactively
           #   SIGN_KEY                      appimagetool picks the key matching this fingerprint
           # Without SIGN=1, the other two are ignored and the AppImage is built unsigned even if a key is present.
-          # Mirror the Windows/macOS gate: only sign on a real release/dispatch+sign or V2-master, when secret is present.
-          SIGN: ${{ (env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master')) && '1' || '0' }}
+          # Mirror the Windows/macOS gate: only sign on a real release/dispatch+sign or the release branch, when secret is present.
+          SIGN: ${{ (env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release')) && '1' || '0' }}
           APPIMAGETOOL_SIGN_PASSPHRASE: ${{ secrets.RELEASE_GPG_PASSPHRASE }}
           SIGN_KEY: ${{ vars.RELEASE_GPG_FINGERPRINT }}
           TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
@@ -525,7 +494,7 @@ jobs:
         env:
           TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
           TAURI_SIGNING_PRIVATE_KEY_PASSWORD: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY_PASSWORD }}
-          GPG_SIGN: ${{ (env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master')) && '1' || '0' }}
+          GPG_SIGN: ${{ (env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release')) && '1' || '0' }}
           SIGN_KEY: ${{ vars.RELEASE_GPG_FINGERPRINT }}
           APPIMAGETOOL_SIGN_PASSPHRASE: ${{ secrets.RELEASE_GPG_PASSPHRASE }}
         run: |
@@ -564,7 +533,7 @@ jobs:
           echo "Stripped bundled libwayland from $(basename "$AI")"
 
       - name: Clear release GPG key from runner keyring (Linux)
-        if: always() && matrix.platform == 'ubuntu-22.04' && env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master')
+        if: always() && matrix.platform == 'ubuntu-22.04' && env.RELEASE_GPG_PRIVATE_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release')
         env:
           RELEASE_GPG_FINGERPRINT: ${{ vars.RELEASE_GPG_FINGERPRINT }}
         run: |
@@ -579,7 +548,7 @@ jobs:
       # artifact. Tauri signs a COPY when bundling into the MSI and leaves the raw
       # cargo output unsigned, so checking it produces false negatives.
       - name: Verify Windows Code Signature
-        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/V2-master') }}
+        if: ${{ startsWith(matrix.platform, 'windows') && env.SM_API_KEY != '' && (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && github.event.inputs.sign != 'false') || github.ref == 'refs/heads/release') }}
         timeout-minutes: 15
         shell: pwsh
         run: |
@@ -911,11 +880,11 @@ jobs:
       # workflow_dispatch path requires platform=='all' so a single-platform
       # dispatch can't overwrite an existing release's full latest.json with a
       # partial one (action-gh-release defaults overwrite_files:true).
-      # release / V2-master always build the full matrix so no extra guard needed.
+      # release event / release branch always build the full matrix so no extra guard needed.
       # fail_on_unmatched_files makes a missing latest.json or installer fail loudly
       # instead of silently shipping a broken auto-update.
       - name: Upload binaries to Release
-        if: (github.event_name == 'workflow_dispatch' && github.event.inputs.test_mode != 'true' && github.event.inputs.platform == 'all') || github.event_name == 'release' || github.ref == 'refs/heads/V2-master'
+        if: (github.event_name == 'workflow_dispatch' && github.event.inputs.test_mode != 'true' && github.event.inputs.platform == 'all') || github.event_name == 'release' || github.ref == 'refs/heads/release'
         uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3.0.2
         with:
           tag_name: v${{ needs.determine-matrix.outputs.version }}
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
index 70e99d6074..c92d17f027 100644
--- a/.github/workflows/nightly.yml
+++ b/.github/workflows/nightly.yml
@@ -127,6 +127,7 @@ jobs:
   # Runs the @nightly tag (conversion scenarios) plus a 10-shard concurrency run
   # of every other feature.
   cucumber-nightly:
+    environment: ci-unsigned
     name: Cucumber (nightly scenarios + full concurrency)
     runs-on: ubuntu-latest
     # Fork pull requests get no MAVEN_* secrets, so the image build cannot work.
diff --git a/.github/workflows/push-docker-base.yml b/.github/workflows/push-docker-base.yml
index 658583ea08..97c227f23c 100644
--- a/.github/workflows/push-docker-base.yml
+++ b/.github/workflows/push-docker-base.yml
@@ -17,6 +17,9 @@ permissions:
 
 jobs:
   push-base:
+    # Own environment: docker-publish is branch-locked to release/main,
+    # which excludes the baseDockerImage/accessIssueFix branches this runs on.
+    environment: docker-base-publish
     if: ${{ vars.CI_PROFILE != 'lite' && github.actor == 'Frooodle' }}
     runs-on: ubuntu-24.04-8core
     permissions:
diff --git a/.github/workflows/push-docker.yml b/.github/workflows/push-docker.yml
index b88d69c3c2..ec9d14822c 100644
--- a/.github/workflows/push-docker.yml
+++ b/.github/workflows/push-docker.yml
@@ -20,9 +20,8 @@ on:
         default: false
   push:
     branches:
-      - master
+      - release
       - main
-      - V2-master
 
 # cancel in-progress jobs if a new job is triggered
 # This is useful to avoid running multiple builds for the same branch if a new commit is pushed
@@ -91,13 +90,13 @@ jobs:
           MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
 
       - name: Install cosign
-        if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master'
+        if: github.ref == 'refs/heads/release'
         uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
         with:
           cosign-release: "v2.4.1"
 
       - name: Install cosign
-        if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master'
+        if: github.ref == 'refs/heads/release'
         uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
         with:
           cosign-release: "v2.4.1"
@@ -133,8 +132,8 @@ jobs:
             ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf
             ${{ secrets.DOCKER_HUB_ORG_USERNAME }}/stirling-pdf
           tags: |
-            type=raw,value=${{ steps.versionNumber.outputs.versionNumber }},enable=${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master' }}
-            type=raw,value=latest,enable=${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master' }}
+            type=raw,value=${{ steps.versionNumber.outputs.versionNumber }},enable=${{ github.ref == 'refs/heads/release' }}
+            type=raw,value=latest,enable=${{ github.ref == 'refs/heads/release' }}
 
       - name: Build and push Unified Dockerfile (latest variant)
         id: build-push-latest
@@ -158,7 +157,7 @@ jobs:
           sbom: true
 
       - name: Sign regular images
-        if: env.RUN_MAIN_APP == 'true' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master') && steps.build-push-latest.outputs.digest != ''
+        if: env.RUN_MAIN_APP == 'true' && (github.ref == 'refs/heads/release') && steps.build-push-latest.outputs.digest != ''
         env:
           DIGEST: ${{ steps.build-push-latest.outputs.digest }}
           TAGS: ${{ steps.meta.outputs.tags }}
@@ -182,8 +181,8 @@ jobs:
             ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf
             ${{ secrets.DOCKER_HUB_ORG_USERNAME }}/stirling-pdf
           tags: |
-            type=raw,value=${{ steps.versionNumber.outputs.versionNumber }}-fat,enable=${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master' }}
-            type=raw,value=latest-fat,enable=${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master' }}
+            type=raw,value=${{ steps.versionNumber.outputs.versionNumber }}-fat,enable=${{ github.ref == 'refs/heads/release' }}
+            type=raw,value=latest-fat,enable=${{ github.ref == 'refs/heads/release' }}
 
       - name: Build and push Unified Dockerfile (fat variant)
         id: build-push-fat
@@ -204,7 +203,7 @@ jobs:
           sbom: true
 
       - name: Sign fat images
-        if: env.RUN_MAIN_APP == 'true' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master') && steps.build-push-fat.outputs.digest != ''
+        if: env.RUN_MAIN_APP == 'true' && (github.ref == 'refs/heads/release') && steps.build-push-fat.outputs.digest != ''
         env:
           DIGEST: ${{ steps.build-push-fat.outputs.digest }}
           TAGS: ${{ steps.meta-fat.outputs.tags }}
@@ -226,8 +225,8 @@ jobs:
             ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf
             ${{ secrets.DOCKER_HUB_ORG_USERNAME }}/stirling-pdf
           tags: |
-            type=raw,value=${{ steps.versionNumber.outputs.versionNumber }}-ultra-lite,enable=${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master' }}
-            type=raw,value=latest-ultra-lite,enable=${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master' }}
+            type=raw,value=${{ steps.versionNumber.outputs.versionNumber }}-ultra-lite,enable=${{ github.ref == 'refs/heads/release' }}
+            type=raw,value=latest-ultra-lite,enable=${{ github.ref == 'refs/heads/release' }}
 
       - name: Build and push Unified Dockerfile (ultra-lite variant)
         id: build-push-lite
@@ -248,7 +247,7 @@ jobs:
           sbom: true
 
       - name: Sign ultra-lite images
-        if: env.RUN_MAIN_APP == 'true' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/V2-master') && steps.build-push-lite.outputs.digest != ''
+        if: env.RUN_MAIN_APP == 'true' && (github.ref == 'refs/heads/release') && steps.build-push-lite.outputs.digest != ''
         env:
           DIGEST: ${{ steps.build-push-lite.outputs.digest }}
           TAGS: ${{ steps.meta-lite.outputs.tags }}
@@ -260,7 +259,7 @@ jobs:
           done
 
       # Standalone unoserver image — versioned independently via
-      # docker/unoserver/VERSION. master/V2-master: publish +latest
+      # docker/unoserver/VERSION. release: publish +latest
       # only when the version is new. main/testMain: republish :alpha only
       # when the source hash differs from the published image's annotation.
       - name: Read unoserver image version
@@ -319,7 +318,7 @@ jobs:
           fi
 
           case "$EFFECTIVE_REF" in
-            refs/heads/master|refs/heads/V2-master)
+            refs/heads/release)
               if [ "${FORCE_REBUILD}" = "true" ]; then
                 echo "force_unoserver_rebuild=true — building stable regardless"
                 mode="stable"
diff --git a/.github/workflows/swagger.yml b/.github/workflows/swagger.yml
index 38c1985a83..115de87d4e 100644
--- a/.github/workflows/swagger.yml
+++ b/.github/workflows/swagger.yml
@@ -4,7 +4,7 @@ on:
   workflow_dispatch:
   push:
     branches:
-      - master
+      - release
 
 # cancel in-progress jobs if a new job is triggered
 # This is useful to avoid running multiple builds for the same branch if a new commit is pushed
@@ -23,6 +23,9 @@ permissions:
 
 jobs:
   push:
+    # package-publish holds SWAGGERHUB_API_KEY. It requires reviewer approval and
+    # is limited to main / release / v* tags, so every push to release waits on one.
+    environment: package-publish
     if: ${{ vars.CI_PROFILE != 'lite' }}
     runs-on: ubuntu-latest
     steps:
diff --git a/.github/workflows/tauri-build.yml b/.github/workflows/tauri-build.yml
index ab6d8aca71..ddf1104bac 100644
--- a/.github/workflows/tauri-build.yml
+++ b/.github/workflows/tauri-build.yml
@@ -57,6 +57,9 @@ permissions:
 
 jobs:
   determine-matrix:
+    # Only probes APPLE_CERTIFICATE for presence, so it stays on the unrestricted
+    # signing environment - release-signing would block every PR run.
+    environment: ci-signing
     if: ${{ vars.CI_PROFILE != 'lite' }}
     runs-on: ubuntu-latest
     outputs:
@@ -103,6 +106,12 @@ jobs:
           echo "matrix={\"include\":[$JOINED]}" >> $GITHUB_OUTPUT
 
   build:
+    # Windows/GPG signing only runs on main (see the per-step gates below), so only
+    # that path needs the reviewer-gated release-signing environment. Everything else
+    # (PRs, merge queue, nightly) signs macOS only and uses ci-signing, which has no
+    # approval or branch restriction.
+    environment:
+      name: ${{ (inputs.sign && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v'))) && 'release-signing' || 'ci-signing' }}
     needs: determine-matrix
     strategy:
       fail-fast: false
@@ -110,7 +119,6 @@ jobs:
     runs-on: ${{ matrix.platform }}
     env:
       SM_API_KEY: ${{ secrets.SM_API_KEY }}
-      WINDOWS_CERTIFICATE: ${{ secrets.WINDOWS_CERTIFICATE }}
       APPLE_CERTIFICATE: ${{ secrets.APPLE_CERTIFICATE }}
       RELEASE_GPG_PRIVATE_KEY: ${{ secrets.RELEASE_GPG_PRIVATE_KEY }}
       # Per-platform sign gate. macOS signs on any run with the cert available,
@@ -264,38 +272,6 @@ jobs:
             }
           }
 
-      # Traditional PFX Certificate Import (fallback if KeyLocker not configured)
-      - name: Import Windows Code Signing Certificate
-        if: ${{ inputs.sign && startsWith(matrix.platform, 'windows') && env.SM_API_KEY == '' && github.ref == 'refs/heads/main' }}
-        env:
-          WINDOWS_CERTIFICATE: ${{ secrets.WINDOWS_CERTIFICATE }}
-          WINDOWS_CERTIFICATE_PASSWORD: ${{ secrets.WINDOWS_CERTIFICATE_PASSWORD }}
-        shell: powershell
-        run: |
-          if ($env:WINDOWS_CERTIFICATE) {
-            Write-Host "Importing Windows Code Signing Certificate..."
-
-            # Decode base64 certificate and save to file
-            $certBytes = [Convert]::FromBase64String($env:WINDOWS_CERTIFICATE)
-            $certPath = Join-Path $env:RUNNER_TEMP "certificate.pfx"
-            [IO.File]::WriteAllBytes($certPath, $certBytes)
-
-            # Import certificate to CurrentUser\My store
-            $cert = Import-PfxCertificate -FilePath $certPath -CertStoreLocation Cert:\CurrentUser\My -Password (ConvertTo-SecureString -String $env:WINDOWS_CERTIFICATE_PASSWORD -AsPlainText -Force)
-
-            # Extract and set thumbprint as environment variable
-            $thumbprint = $cert.Thumbprint
-            Write-Host "Certificate imported with thumbprint: $thumbprint"
-            echo "WINDOWS_CERTIFICATE_THUMBPRINT=$thumbprint" >> $env:GITHUB_ENV
-
-            # Clean up certificate file
-            Remove-Item $certPath
-
-            Write-Host "Windows certificate import completed."
-          } else {
-            Write-Host "⚠️ WINDOWS_CERTIFICATE secret not set - building unsigned binary"
-          }
-
       - name: Import Apple Developer Certificate
         if: env.SIGN_BUNDLE == 'true' && matrix.platform == 'macos-15'
         env:
diff --git a/.github/workflows/test-build-docker.yml b/.github/workflows/test-build-docker.yml
index 660cd2458b..4a79cb3733 100644
--- a/.github/workflows/test-build-docker.yml
+++ b/.github/workflows/test-build-docker.yml
@@ -37,6 +37,7 @@ jobs:
   # spring-security=true matrix entry if `task backend:build` and
   # `task backend:build:ci` produce equivalent JARs (verify before wiring).
   test-build-docker-images:
+    environment: ci-unsigned
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
diff --git a/.github/workflows/testdriver.yml b/.github/workflows/testdriver.yml
deleted file mode 100644
index 751eaf44f7..0000000000
--- a/.github/workflows/testdriver.yml
+++ /dev/null
@@ -1,235 +0,0 @@
-name: UI test with TestDriverAI
-
-on:
-  push:
-    branches: ["master", "UITest", "testdriver"]
-
-# cancel in-progress jobs if a new job is triggered
-# This is useful to avoid running multiple builds for the same branch if a new commit is pushed
-# or a pull request is updated.
-# It helps to save resources and time by ensuring that only the latest commit is built and tested
-# This is particularly useful for long-running jobs that may take a while to complete.
-# The `group` is set to a combination of the workflow name, event name, and branch name.
-# This ensures that jobs are grouped by the workflow and branch, allowing for cancellation of
-# in-progress jobs when a new commit is pushed to the same branch or a new pull request is opened.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref_name || github.ref }}
-  cancel-in-progress: true
-
-permissions:
-  contents: read
-
-jobs:
-  deploy:
-    environment: pr-preview
-    if: ${{ vars.CI_PROFILE != 'lite' }}
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      packages: write
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
-        with:
-          egress-policy: audit
-
-      - name: Checkout repository
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-
-      - name: Set up JDK 25
-        uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0
-        with:
-          java-version: "25"
-          distribution: "temurin"
-
-      - name: Cache Gradle User Home
-        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
-        with:
-          path: |
-            ~/.gradle/caches
-            ~/.gradle/wrapper
-          key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }}
-          restore-keys: |
-            gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-
-            gradle-${{ runner.os }}-${{ runner.arch }}-
-
-      - name: Build with Gradle
-        run: ./gradlew build
-        env:
-          MAVEN_USER: ${{ secrets.MAVEN_USER }}
-          MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
-          MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
-          DISABLE_ADDITIONAL_FEATURES: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
-
-      - name: Get version number
-        id: versionNumber
-        run: |
-          VERSION=$(grep "^version =" build.gradle | awk -F'"' '{print $2}')
-          echo "versionNumber=$VERSION" >> $GITHUB_OUTPUT
-
-      - name: Convert repository owner to lowercase
-        id: repoowner
-        run: echo "lowercase=$(echo ${{ github.repository_owner }} | awk '{print tolower($0)}')" >> $GITHUB_OUTPUT
-
-      - name: Login to GitHub Container Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ github.token }}
-
-      - name: Build and push test image
-        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
-        with:
-          context: .
-          file: ./docker/embedded/Dockerfile
-          push: true
-          cache-from: type=gha,scope=stirling-pdf-latest
-          cache-to: type=gha,mode=max,scope=stirling-pdf-latest
-          tags: ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test:test-${{ github.sha }}
-          build-args: VERSION_TAG=${{ steps.versionNumber.outputs.versionNumber }}
-          platforms: linux/amd64
-
-      - name: Set up SSH
-        run: |
-          mkdir -p ~/.ssh/
-          echo "${NEW_VPS_SSH_KEY}" > ../private.key
-          sudo chmod 600 ../private.key
-
-        env:
-          NEW_VPS_SSH_KEY: ${{ secrets.NEW_VPS_SSH_KEY }}
-      - name: Deploy to VPS
-        run: |
-          cat > docker-compose.yml << EOF
-          version: '3.3'
-          services:
-            stirling-pdf:
-              container_name: stirling-pdf-test-${{ github.sha }}
-              image: ${IMAGE_BASE}:test-${{ github.sha }}
-              ports:
-                - "1337:8080"
-              volumes:
-                - /stirling/test-${{ github.sha }}/data:/usr/share/tessdata:rw
-                - /stirling/test-${{ github.sha }}/config:/configs:rw
-                - /stirling/test-${{ github.sha }}/logs:/logs:rw
-              environment:
-                DISABLE_ADDITIONAL_FEATURES: "true"
-                SECURITY_ENABLELOGIN: "false"
-                SYSTEM_DEFAULTLOCALE: en-US
-                UI_APPNAME: "Stirling-PDF Test"
-                UI_HOMEDESCRIPTION: "Test Deployment"
-                UI_APPNAMENAVBAR: "Test"
-                SYSTEM_MAXFILESIZE: "100"
-                METRICS_ENABLED: "true"
-                SYSTEM_GOOGLEVISIBILITY: "false"
-                SYSTEM_ENABLEANALYTICS: "false"
-              restart: on-failure:5
-          EOF
-
-          scp -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker-compose.yml ${NEW_VPS_USERNAME}@${NEW_VPS_HOST}:/tmp/docker-compose.yml
-
-          ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${NEW_VPS_USERNAME}@${NEW_VPS_HOST} << EOF
-            mkdir -p /stirling/test-${{ github.sha }}/{data,config,logs}
-            mv /tmp/docker-compose.yml /stirling/test-${{ github.sha }}/docker-compose.yml
-            cd /stirling/test-${{ github.sha }}
-            docker-compose pull
-            docker-compose up -d
-          EOF
-
-        env:
-          IMAGE_BASE: ghcr.io/${{ steps.repoowner.outputs.lowercase }}/stirling-pdf-test
-          NEW_VPS_USERNAME: ${{ secrets.NEW_VPS_USERNAME }}
-          NEW_VPS_HOST: ${{ secrets.NEW_VPS_HOST }}
-  files-changed:
-    if: always()
-    name: detect what files changed
-    runs-on: ubuntu-latest
-    timeout-minutes: 3
-    outputs:
-      frontend: ${{ steps.changes.outputs.frontend }}
-    steps:
-      - name: Harden the runner (Audit all outbound calls)
-        uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
-        with:
-          egress-policy: audit
-
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-
-      - name: Check for file changes
-        uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
-        id: changes
-        with:
-          filters: ".github/config/.files.yaml"
-
-  test:
-    environment: pr-preview
-    if: needs.files-changed.outputs.frontend == 'true'
-    needs: [deploy, files-changed]
-    runs-on: ubuntu-latest
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
-        with:
-          egress-policy: audit
-
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-
-      - name: Set up Node
-        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
-        with:
-          cache: "npm"
-          cache-dependency-path: frontend/package-lock.json
-
-      - name: Run TestDriver.ai
-        uses: testdriverai/action@f0d0f45fdd684db628baa843fe9313f3ca3a8aa8 #1.1.3
-        with:
-          key: ${{secrets.TESTDRIVER_API_KEY}}
-          prerun: |
-            choco install go-task -y
-            task frontend:build
-            cd frontend
-            npm install dashcam-chrome --save
-            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "http://${{ secrets.NEW_VPS_HOST }}:1337"
-            Start-Sleep -Seconds 20
-          prompt: |
-            1. /run testing/testdriver/test.yml
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          FORCE_COLOR: "3"
-
-  cleanup:
-    environment: pr-preview
-    needs: [deploy, test]
-    runs-on: ubuntu-latest
-    if: always()
-
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
-        with:
-          egress-policy: audit
-
-      - name: Set up SSH
-        run: |
-          mkdir -p ~/.ssh/
-          echo "${NEW_VPS_SSH_KEY}" > ../private.key
-          sudo chmod 600 ../private.key
-
-        env:
-          NEW_VPS_SSH_KEY: ${{ secrets.NEW_VPS_SSH_KEY }}
-      - name: Cleanup deployment
-        if: always()
-        run: |
-          ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${NEW_VPS_USERNAME}@${NEW_VPS_HOST} << EOF
-            cd /stirling/test-${{ github.sha }}
-            docker-compose down
-            cd /stirling
-            rm -rf test-${{ github.sha }}
-          EOF
-        env:
-          NEW_VPS_USERNAME: ${{ secrets.NEW_VPS_USERNAME }}
-          NEW_VPS_HOST: ${{ secrets.NEW_VPS_HOST }}
-        continue-on-error: true # Ensure cleanup runs even if previous steps fail
diff --git a/WINDOWS_SIGNING.md b/WINDOWS_SIGNING.md
index 58ffd6e657..95cbbd24e2 100644
--- a/WINDOWS_SIGNING.md
+++ b/WINDOWS_SIGNING.md
@@ -4,6 +4,11 @@ This guide explains how to set up Windows code signing for Stirling-PDF desktop
 
 ## Overview
 
+Releases are signed with **DigiCert KeyLocker**, a cloud HSM: the private key never
+leaves DigiCert, and the runner signs through a PKCS#11 provider. The older approach
+of uploading a base64 `.pfx` to a repository secret has been removed from the
+workflows - the sections below describe KeyLocker, which is what actually runs.
+
 Windows code signing is essential for:
 - Preventing Windows SmartScreen warnings
 - Building trust with users
@@ -49,29 +54,19 @@ openssl pkcs12 -export -out certificate.pfx -inkey private-key.key -in certifica
 
 ### Required Secrets
 
-Navigate to your GitHub repository → Settings → Secrets and variables → Actions
+Navigate to your GitHub repository → Settings → Environments → `release-signing`.
 
-Add the following secrets:
+These live in the `release-signing` environment, not at repository scope. That
+environment requires reviewer approval and is limited to `main`, `release`,
+`hotfix/*` and `v*` tags. All five come from the DigiCert ONE console.
 
-#### 1. `WINDOWS_CERTIFICATE`
-- **Description**: Base64-encoded .pfx certificate file
-- **How to create**:
-
-**On macOS/Linux:**
-```bash
-base64 -i certificate.pfx | pbcopy  # Copies to clipboard
-```
-
-**On Windows (PowerShell):**
-```powershell
-[Convert]::ToBase64String([IO.File]::ReadAllBytes("certificate.pfx")) | Set-Clipboard
-```
-
-Paste the entire base64 string into the GitHub secret.
-
-#### 2. `WINDOWS_CERTIFICATE_PASSWORD`
-- **Description**: Password for the .pfx certificate
-- **Value**: The password you set when creating/exporting the .pfx file
+| Secret | Description |
+| --- | --- |
+| `SM_API_KEY` | KeyLocker API key. Also acts as the on/off switch: signing steps are gated on it being non-empty. |
+| `SM_CLIENT_CERT_FILE_B64` | Base64-encoded PKCS#12 client authentication certificate. |
+| `SM_CLIENT_CERT_PASSWORD` | Password for that client certificate. |
+| `SM_KEYPAIR_ALIAS` | Alias of the signing keypair to use. |
+| `SM_HOST` | DigiCert ONE host, e.g. `https://clientauth.one.digicert.com`. |
 
 ### Optional Secrets for Tauri Updater
 
@@ -110,23 +105,23 @@ The Windows signing configuration is already set up:
 
 ### 2. GitHub Workflow (.github/workflows/tauri-build.yml)
 
-The workflow includes three Windows signing steps:
+The workflow includes four Windows signing steps, all gated on `SM_API_KEY` being
+set and the ref being the release branch:
 
-1. **Import Certificate**: Decodes and imports the .pfx certificate into Windows certificate store
-2. **Build Tauri App**: Builds and signs the application using the imported certificate
-3. **Verify Signature**: Validates that both .exe and .msi files are properly signed
+1. **Setup DigiCert KeyLocker**: Installs the DigiCert signing tools via `digicert/ssm-code-signing`
+2. **Setup DigiCert KeyLocker Certificate**: Writes the client cert and exports the PKCS#11 config
+3. **Configure Windows code signing / Build Tauri app**: Signs through the PKCS#11 provider
+4. **Verify Windows Code Signature**: Validates that the .exe and .msi are properly signed
 
 ## Testing the Setup
 
 ### 1. Local Testing (Windows Only)
 
-Before pushing to GitHub, test locally:
+KeyLocker is CI-only. To check signing locally, install your own certificate into
+the Windows store and point Tauri at it; the build no longer reads any certificate
+from an environment variable.
 
 ```powershell
-# Set environment variables
-$env:WINDOWS_CERTIFICATE = [Convert]::ToBase64String([IO.File]::ReadAllBytes("certificate.pfx"))
-$env:WINDOWS_CERTIFICATE_PASSWORD = "your-certificate-password"
-
 # Build the application
 cd frontend
 npm run tauri build
@@ -191,9 +186,10 @@ Look for:
 - Consider EV certificate for immediate reputation
 
 ### Certificate Not Found During Build
-- Verify `WINDOWS_CERTIFICATE` secret is set
-- Check base64 encoding is correct (no extra whitespace)
-- Ensure password is correct
+- Verify `SM_API_KEY` is present in the `release-signing` environment. If it is empty
+  the signing steps skip silently and the build succeeds unsigned.
+- Check `SM_CLIENT_CERT_FILE_B64` base64 encoding is correct (no extra whitespace)
+- Ensure `SM_CLIENT_CERT_PASSWORD` and `SM_KEYPAIR_ALIAS` match the DigiCert keypair
 
 ## Security Best Practices
 
@@ -220,11 +216,10 @@ Look for:
 ## Certificate Lifecycle
 
 ### Before Expiration
-1. Obtain new certificate from CA (typically annual renewal)
-2. Convert to .pfx format if needed
-3. Update `WINDOWS_CERTIFICATE` secret with new base64-encoded certificate
-4. Update `WINDOWS_CERTIFICATE_PASSWORD` if password changed
-5. Test build to verify new certificate works
+1. Renew the certificate in the DigiCert ONE console (typically annual)
+2. If the keypair alias changed, update `SM_KEYPAIR_ALIAS` in the `release-signing` environment
+3. If the client authentication certificate was reissued, update `SM_CLIENT_CERT_FILE_B64` and `SM_CLIENT_CERT_PASSWORD`
+4. Test build to verify the new certificate works
 
 ### Expired Certificates
 - Signed binaries remain valid (timestamp proves signing time)
diff --git a/frontend/editor/src/core/services/updateService.ts b/frontend/editor/src/core/services/updateService.ts
index 043c53bc36..8b23d26deb 100644
--- a/frontend/editor/src/core/services/updateService.ts
+++ b/frontend/editor/src/core/services/updateService.ts
@@ -185,7 +185,7 @@ export class UpdateService {
    */
   async getCurrentVersionFromGitHub(): Promise {
     const url =
-      "https://raw.githubusercontent.com/Stirling-Tools/Stirling-PDF/V2-master/build.gradle";
+      "https://raw.githubusercontent.com/Stirling-Tools/Stirling-PDF/release/build.gradle";
 
     try {
       const response = await fetch(url);

From a14eec94ec13677d71541bcfd26c1397a776ade4 Mon Sep 17 00:00:00 2001
From: James Brunton 
Date: Tue, 18 Aug 2026 09:02:17 +0000
Subject: [PATCH 20/59] Fix corner radius on Mantine checkboxes in Processor
 (#7537)

# Description of Changes

## Before
image

## After
image
---
 frontend/editor/src/portal/theme/mantineTheme.ts | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/frontend/editor/src/portal/theme/mantineTheme.ts b/frontend/editor/src/portal/theme/mantineTheme.ts
index 7108a90073..e371b4aeb8 100644
--- a/frontend/editor/src/portal/theme/mantineTheme.ts
+++ b/frontend/editor/src/portal/theme/mantineTheme.ts
@@ -163,6 +163,10 @@ export const mantineTheme = createTheme({
     CloseButton: { defaultProps: { "aria-label": "Close" } },
     Modal: { defaultProps: { closeButtonProps: { "aria-label": "Close" } } },
     Drawer: { defaultProps: { closeButtonProps: { "aria-label": "Close" } } },
+    // The portal's md default radius (8px) is right for cards and buttons but
+    // rounds a 20px checkbox into a circle. Pin it to the smaller radius the
+    // editor's checkboxes use so the box reads as a checkbox.
+    Checkbox: { styles: { input: { borderRadius: "var(--radius-sm)" } } },
   },
   fontFamily: "var(--font-sans)",
   fontFamilyMonospace: "var(--font-mono)",

From fb70fc13da03e25ee535b74116de00a83648764e Mon Sep 17 00:00:00 2001
From: James Brunton 
Date: Tue, 18 Aug 2026 13:08:23 +0000
Subject: [PATCH 21/59] Fix tools which crash in the Pipelines page (#7538)

# Description of Changes
Overlay PDFs and Change Metadata both crashed in the Processor because
they required `FilesModalContext` and `ViewerContext` respectively.
Neither of those contexts make sense to provide in the Processor because
there are no files in context and there is no Viewer, so redesign both
tool settings to only optionally require these contexts. Their behaviour
is unchanged in the Editor but they now work in the Processor (just
without the extra info about the active files, since there are none).

Also hooks up the Reorganise Pages settings so that it can be used from
Automate. The component already existed but just wasn't being used,
which just looks like an oversight.
---
 .../ChangeMetadataSingleStep.tsx              | 153 ++++++++++------
 .../tools/overlayPdfs/OverlayPdfsSettings.tsx |  61 +++++--
 .../src/core/contexts/FilesModalContext.tsx   |   4 +-
 ...tomatableToolsHaveOperationConfig.test.tsx |  11 ++
 .../core/data/useTranslatedToolRegistry.tsx   |   5 +-
 .../pipelines/PipelineStepSettings.test.tsx   | 168 +++++++++++++++++-
 6 files changed, 323 insertions(+), 79 deletions(-)

diff --git a/frontend/editor/src/core/components/tools/changeMetadata/ChangeMetadataSingleStep.tsx b/frontend/editor/src/core/components/tools/changeMetadata/ChangeMetadataSingleStep.tsx
index 2eff20b23b..07ba4e7e05 100644
--- a/frontend/editor/src/core/components/tools/changeMetadata/ChangeMetadataSingleStep.tsx
+++ b/frontend/editor/src/core/components/tools/changeMetadata/ChangeMetadataSingleStep.tsx
@@ -1,5 +1,7 @@
+import { useContext, useEffect, useState } from "react";
 import { Stack, Divider, Text } from "@mantine/core";
 import { useTranslation } from "react-i18next";
+import { ViewerContext } from "@app/contexts/ViewerContext";
 import {
   ChangeMetadataParameters,
   createCustomMetadataFunctions,
@@ -19,6 +21,31 @@ interface ChangeMetadataSingleStepProps {
   disabled?: boolean;
 }
 
+/**
+ * Pre-fills the form from the currently open document's existing metadata.
+ * Isolated in its own component so it only mounts where a ViewerProvider exists
+ * (the editor and the in-editor Automate modal). The pipeline builder has no
+ * viewer and no single "current document", so it is skipped there rather than
+ * crashing on useViewer.
+ */
+const MetadataPrefill = ({
+  onParameterChange,
+  onExtractingChange,
+}: {
+  onParameterChange: ChangeMetadataSingleStepProps["onParameterChange"];
+  onExtractingChange: (extracting: boolean) => void;
+}) => {
+  const { isExtractingMetadata } = useMetadataExtraction({
+    updateParameter: onParameterChange,
+  });
+
+  useEffect(() => {
+    onExtractingChange(isExtractingMetadata);
+  }, [isExtractingMetadata, onExtractingChange]);
+
+  return null;
+};
+
 const ChangeMetadataSingleStep = ({
   parameters,
   onParameterChange,
@@ -26,77 +53,85 @@ const ChangeMetadataSingleStep = ({
 }: ChangeMetadataSingleStepProps) => {
   const { t } = useTranslation();
 
+  // Auto-prefill reads the viewer/file contexts, which only exist in the editor.
+  // Gate on the viewer so the pipeline builder renders the fields without it.
+  const hasViewerContext = useContext(ViewerContext) !== null;
+  const [isExtractingMetadata, setIsExtractingMetadata] = useState(false);
+
   // Get custom metadata functions using the utility
   const { addCustomMetadata, removeCustomMetadata, updateCustomMetadata } =
     createCustomMetadataFunctions(parameters, onParameterChange);
 
-  // Extract metadata from uploaded files
-  const { isExtractingMetadata } = useMetadataExtraction({
-    updateParameter: onParameterChange,
-  });
-
   const isDeleteAllEnabled = parameters.deleteAll;
   const fieldsDisabled = disabled || isDeleteAllEnabled || isExtractingMetadata;
 
   return (
-    
-      {/* Delete All */}
-      
-        
-          {t("changeMetadata.deleteAll.label", "Delete All Metadata")}
-        
-        
+      {hasViewerContext && (
+        
-      
-
-      
-
-      {/* Standard Metadata Fields */}
+      )}
       
-        
-          {t("changeMetadata.standardFields.title", "Standard Metadata")}
-        
-        
+        {/* Delete All */}
+        
+          
+            {t("changeMetadata.deleteAll.label", "Delete All Metadata")}
+          
+          
+        
+
+        
+
+        {/* Standard Metadata Fields */}
+        
+          
+            {t("changeMetadata.standardFields.title", "Standard Metadata")}
+          
+          
+        
+
+        
+
+        {/* Document Dates */}
+        
+          
+            {t("changeMetadata.dates.title", "Document Dates")}
+          
+          
+        
+
+        
+
+        {/* Advanced Options */}
+        
+          
+            {t("changeMetadata.advanced.title", "Advanced Options")}
+          
+          
+        
       
-
-      
-
-      {/* Document Dates */}
-      
-        
-          {t("changeMetadata.dates.title", "Document Dates")}
-        
-        
-      
-
-      
-
-      {/* Advanced Options */}
-      
-        
-          {t("changeMetadata.advanced.title", "Advanced Options")}
-        
-        
-      
-    
+    
   );
 };
 
diff --git a/frontend/editor/src/core/components/tools/overlayPdfs/OverlayPdfsSettings.tsx b/frontend/editor/src/core/components/tools/overlayPdfs/OverlayPdfsSettings.tsx
index 2648990248..e7246bb4e3 100644
--- a/frontend/editor/src/core/components/tools/overlayPdfs/OverlayPdfsSettings.tsx
+++ b/frontend/editor/src/core/components/tools/overlayPdfs/OverlayPdfsSettings.tsx
@@ -1,3 +1,4 @@
+import { useContext, useRef } from "react";
 import {
   Stack,
   Text,
@@ -7,6 +8,7 @@ import {
   Divider,
 } from "@mantine/core";
 import { Button } from "@app/ui/Button";
+import { FilePicker } from "@app/ui/FilePicker";
 import { ActionIcon } from "@app/ui/ActionIcon";
 import { SegmentedControl } from "@app/ui/SegmentedControl";
 import { useTranslation } from "react-i18next";
@@ -15,7 +17,7 @@ import {
   type OverlayMode,
 } from "@app/hooks/tools/overlayPdfs/useOverlayPdfsParameters";
 import LocalIcon from "@app/components/shared/LocalIcon";
-import { useFilesModalContext } from "@app/contexts/FilesModalContext";
+import { FilesModalContext } from "@app/contexts/FilesModalContext";
 import styles from "@app/components/tools/overlayPdfs/OverlayPdfsSettings.module.css";
 import { Z_INDEX_AUTOMATE_DROPDOWN } from "@app/styles/zIndex";
 
@@ -34,7 +36,12 @@ export default function OverlayPdfsSettings({
   disabled = false,
 }: OverlayPdfsSettingsProps) {
   const { t } = useTranslation();
-  const { openFilesModal } = useFilesModalContext();
+  // Read optionally: the portal pipeline builder mounts no FilesModalProvider.
+  // Present (editor tool + Automate modal) -> keep the workspace file picker;
+  // absent (portal) -> fall back to the plain file input below.
+  const filesModal = useContext(FilesModalContext);
+  // Clears the FilePicker so the same file can be re-selected (Mantine resetRef).
+  const resetOverlayPicker = useRef<() => void>(null);
 
   const handleOverlayFilesChange = (files: File[]) => {
     onParameterChange("overlayFiles", files);
@@ -66,8 +73,8 @@ export default function OverlayPdfsSettings({
   };
 
   const handleOpenOverlayFilesModal = () => {
-    if (disabled) return;
-    openFilesModal({
+    if (disabled || !filesModal) return;
+    filesModal.openFilesModal({
       customHandler: (files: File[]) => {
         handleOverlayFilesChange([
           ...(parameters.overlayFiles || []),
@@ -77,6 +84,17 @@ export default function OverlayPdfsSettings({
     });
   };
 
+  const appendOverlayFiles = (files: File[]) => {
+    if (files.length === 0) return;
+    handleOverlayFilesChange([...(parameters.overlayFiles || []), ...files]);
+    resetOverlayPicker.current?.();
+  };
+
+  const overlayFilesButtonLabel =
+    parameters.overlayFiles?.length > 0
+      ? t("overlay-pdfs.overlayFiles.addMore", "Add more PDFs...")
+      : t("overlay-pdfs.overlayFiles.placeholder", "Choose PDF(s)...");
+
   return (
     
       
@@ -183,17 +201,30 @@ export default function OverlayPdfsSettings({
         
           {t("overlay-pdfs.overlayFiles.label", "Overlay Files")}
         
-        
+        {filesModal ? (
+          
+        ) : (
+          }
+            fullWidth
+          >
+            {overlayFilesButtonLabel}
+          
+        )}
 
         {parameters.overlayFiles?.length > 0 &&
           (() => {
diff --git a/frontend/editor/src/core/contexts/FilesModalContext.tsx b/frontend/editor/src/core/contexts/FilesModalContext.tsx
index 73d1b0477f..17585ae7e8 100644
--- a/frontend/editor/src/core/contexts/FilesModalContext.tsx
+++ b/frontend/editor/src/core/contexts/FilesModalContext.tsx
@@ -41,7 +41,9 @@ interface FilesModalContextType {
   setOnModalClose: (callback: () => void) => void;
 }
 
-const FilesModalContext = createContext(null);
+export const FilesModalContext = createContext(
+  null,
+);
 
 export const FilesModalProvider: React.FC<{ children: React.ReactNode }> = ({
   children,
diff --git a/frontend/editor/src/core/data/automatableToolsHaveOperationConfig.test.tsx b/frontend/editor/src/core/data/automatableToolsHaveOperationConfig.test.tsx
index 461c07ebdf..c8942942df 100644
--- a/frontend/editor/src/core/data/automatableToolsHaveOperationConfig.test.tsx
+++ b/frontend/editor/src/core/data/automatableToolsHaveOperationConfig.test.tsx
@@ -30,4 +30,15 @@ describe("automatable tools", () => {
 
     expect(offeredWithoutConfig).toEqual([]);
   });
+
+  // Reorganize Pages has an automatable form (organization mode + page-order string) and a
+  // context-free settings component, but its registry entry once left automationSettings null,
+  // so both Automate and the pipeline builder showed "no configurable settings". Guard the wiring.
+  test("Reorganize Pages exposes automation settings so it is configurable, not no-settings", () => {
+    const { result } = renderHook(() => useTranslatedToolCatalog());
+
+    expect(
+      result.current.regularTools.reorganizePages?.automationSettings,
+    ).toBeTruthy();
+  });
 });
diff --git a/frontend/editor/src/core/data/useTranslatedToolRegistry.tsx b/frontend/editor/src/core/data/useTranslatedToolRegistry.tsx
index 0a39da2fb9..5ae8075d0f 100644
--- a/frontend/editor/src/core/data/useTranslatedToolRegistry.tsx
+++ b/frontend/editor/src/core/data/useTranslatedToolRegistry.tsx
@@ -700,7 +700,10 @@ export function useTranslatedToolCatalog(): TranslatedToolCatalog {
         endpoints: ["rearrange-pages"],
         operationConfig: asRegistryConfig(reorganizePagesOperationConfig),
         synonyms: getSynonyms(t, "reorganizePages"),
-        automationSettings: null,
+        automationSettings: lazySettings(
+          () =>
+            import("@app/components/tools/reorganizePages/ReorganizePagesSettings"),
+        ),
       },
       scalePages: {
         icon: (
diff --git a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx
index 946e24b810..55a52b496e 100644
--- a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx
+++ b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx
@@ -1,10 +1,23 @@
 import { describe, expect, it, vi } from "vitest";
-import { useEffect, useState } from "react";
-import { render, screen } from "@testing-library/react";
+import {
+  Component,
+  Suspense,
+  useEffect,
+  useState,
+  type ComponentType,
+  type ReactNode,
+} from "react";
+import { render, renderHook, screen, waitFor } from "@testing-library/react";
 import { PortalTestProviders } from "@portal/test/TestQueryProvider";
+import { useTranslatedToolCatalog } from "@app/data/useTranslatedToolRegistry";
+import { PreferencesProvider } from "@app/contexts/PreferencesContext";
+import { SidebarProvider } from "@app/contexts/SidebarContext";
 import { Tooltip } from "@app/components/shared/Tooltip";
 import type { ToolRegistry } from "@app/data/toolsTaxonomy";
-import type { WorkingToolStep } from "@app/hooks/tools/shared/toolAutomation";
+import {
+  getExecutableTools,
+  type WorkingToolStep,
+} from "@app/hooks/tools/shared/toolAutomation";
 import {
   asRegistryConfig,
   type ErasedToolParams,
@@ -13,6 +26,10 @@ import {
 import ConvertSettings from "@app/components/tools/convert/ConvertSettings";
 import { convertOperationConfig } from "@app/hooks/tools/convert/useConvertOperation";
 import { defaultParameters as convertDefaults } from "@app/hooks/tools/convert/useConvertParameters";
+import ChangeMetadataSingleStep from "@app/components/tools/changeMetadata/ChangeMetadataSingleStep";
+import { defaultParameters as changeMetadataDefaults } from "@app/hooks/tools/changeMetadata/useChangeMetadataParameters";
+import OverlayPdfsSettings from "@app/components/tools/overlayPdfs/OverlayPdfsSettings";
+import { defaultParameters as overlayDefaults } from "@app/hooks/tools/overlayPdfs/useOverlayPdfsParameters";
 import { PipelineStepSettings } from "@portal/components/pipelines/PipelineStepSettings";
 
 // Override only useTranslation; keep the rest of react-i18next (initReactI18next et al.) real, so
@@ -21,6 +38,7 @@ vi.mock("react-i18next", async (importOriginal) => ({
   ...(await importOriginal()),
   useTranslation: () => ({
     t: (key: string, fallback?: string) => fallback ?? key,
+    i18n: { language: "en-US", changeLanguage: vi.fn() },
   }),
 }));
 
@@ -63,6 +81,32 @@ const convertRegistry = {
   },
 } as unknown as Partial;
 
+// The real Change Metadata automation settings. Its editor variant auto-prefills the
+// form from the open document via useViewer; that path is now gated on a ViewerProvider
+// so it renders here (the portal mounts none) instead of crashing on useViewer.
+const changeMetadataStep = {
+  support: "editable",
+  toolId: "changeMetadata",
+  params: changeMetadataDefaults,
+} as unknown as WorkingToolStep;
+
+const changeMetadataRegistry = {
+  changeMetadata: { automationSettings: ChangeMetadataSingleStep },
+} as unknown as Partial;
+
+// The real Overlay PDFs automation settings. Its overlay-file picker uses the
+// editor FilesModal when present; that read is now optional so the portal (which
+// mounts no FilesModalProvider) renders a plain file input instead of crashing.
+const overlayStep = {
+  support: "editable",
+  toolId: "overlayPdfs",
+  params: overlayDefaults,
+} as unknown as WorkingToolStep;
+
+const overlayRegistry = {
+  overlayPdfs: { automationSettings: OverlayPdfsSettings },
+} as unknown as Partial;
+
 describe("PipelineStepSettings", () => {
   it("renders reused editor tool settings (which use the shared Tooltip) without app-wide Preferences/Sidebar providers", () => {
     expect(() =>
@@ -94,6 +138,36 @@ describe("PipelineStepSettings", () => {
     expect(screen.getByText(/Convert from/)).toBeInTheDocument();
   });
 
+  it("renders the Change Metadata tool's fields in the portal, with no ViewerProvider mounted", () => {
+    expect(() =>
+      render(
+        
+           {}}
+          />
+        ,
+      ),
+    ).not.toThrow();
+    expect(screen.getByText("Standard Metadata")).toBeInTheDocument();
+  });
+
+  it("renders the Overlay PDFs tool's fields in the portal, with no FilesModalProvider mounted", () => {
+    expect(() =>
+      render(
+        
+           {}}
+          />
+        ,
+      ),
+    ).not.toThrow();
+    expect(screen.getByText("Overlay Mode")).toBeInTheDocument();
+  });
+
   // Reproduces the convert-in-pipeline bug: picking a source format fires several onParameterChange
   // calls in one tick (set fromExtension, auto-target, reset options). If each rebuilt from the
   // step snapshot captured at render they'd clobber each other and the earlier field would be lost.
@@ -148,3 +222,91 @@ describe("PipelineStepSettings", () => {
     });
   });
 });
+
+// Records a render crash and swallows it (renders nothing), so one broken tool is attributed by id
+// instead of aborting the whole sweep - mirroring the portal's own ErrorBoundary around the builder.
+class CaptureBoundary extends Component<
+  { onError: (error: Error) => void; children: ReactNode },
+  { failed: boolean }
+> {
+  state = { failed: false };
+  static getDerivedStateFromError() {
+    return { failed: true };
+  }
+  componentDidCatch(error: Error) {
+    this.props.onError(error);
+  }
+  render() {
+    return this.state.failed ? null : this.props.children;
+  }
+}
+
+// Automated version of the manual "add every tool" sweep: render each tool's real automation
+// settings in a portal-only context (the same Preferences + Sidebar + Suspense wrappers
+// PipelineStepSettings uses, and NO editor providers) and fail listing any that throw. This is the
+// guard that would have caught Change Metadata (useViewer) and Overlay PDFs (useFilesModalContext).
+describe("PipelineStepSettings: every tool's settings render in the portal", () => {
+  it("renders each tool's automation settings without throwing", async () => {
+    const { result } = renderHook(() => useTranslatedToolCatalog());
+    const catalog = result.current.allTools;
+    // getExecutableTools is exactly what PipelineBuilder feeds its "Add a tool" picker, so this
+    // sweeps precisely the tools a user can add. Narrow to "editable" (renders a settings
+    // component); "noSettings"/"unsupported" steps show a Banner instead and can't crash.
+    const editableTools = getExecutableTools(catalog)
+      .filter((tool) => tool.support === "editable")
+      .map((tool) => [tool.toolId, catalog[tool.toolId]] as const)
+      .filter(([, entry]) => Boolean(entry?.automationSettings));
+    // Guard against the filter silently matching nothing (e.g. a registry-shape change).
+    expect(editableTools.length).toBeGreaterThan(10);
+
+    const failures: { toolId: string; message: string }[] = [];
+
+    for (const [toolId, entry] of editableTools) {
+      const Settings = entry.automationSettings as ComponentType<
+        ToolAutomationSettingsProps
+      >;
+      const params = (entry.operationConfig?.defaultParameters ??
+        {}) as ErasedToolParams;
+
+      const caught: { error: Error | null } = { error: null };
+      // The sentinel sibling commits only once the lazy Settings actually renders, so we wait for a
+      // real render (or a caught throw) - not just the providers' wrapper DOM.
+      const { unmount } = render(
+        
+          
+            
+               {
+                  caught.error = error;
+                }}
+              >
+                
+                   {}}
+                    disabled={false}
+                  />
+                  
+                
+              
+            
+          
+        ,
+      );
+
+      await waitFor(() =>
+        expect(
+          caught.error !== null ||
+            screen.queryByTestId(`rendered-${toolId}`) !== null,
+        ).toBe(true),
+      );
+
+      if (caught.error) {
+        failures.push({ toolId, message: caught.error.message });
+      }
+      unmount();
+    }
+
+    expect(failures).toEqual([]);
+  }, 30000);
+});

From cf49742d9774802c603b4d068c2f8ac9d3ffbfd1 Mon Sep 17 00:00:00 2001
From: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com>
Date: Tue, 18 Aug 2026 13:56:47 +0000
Subject: [PATCH 22/59] Fix the top bar styling (#7544)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Every top bar styled itself, so none of them matched the new UI. Also,
colors on the premium banner (and possibly others) clashed since the
theme changes.

## Before Example Issue

Screenshot 2026-08-17 at 11 47
20 PM


## After (all)

danger__dark
danger__light
default-app__dark
default-app__light
free-tier-limit__dark
free-tier-limit__light
server-attention__dark
server-attention__light
team-invitation__dark
team-invitation__light
upgrade-prompt__dark
upgrade-prompt__light


## What changed

- `InfoBanner` exposed 8 colour-override props (`background`,
`borderColor`, `textColor`, `iconColor`, `buttonColor`,
`buttonTextColor`, `closeIconColor`, `buttonVariant`), so every caller
invented its own look. Replaced with a closed tone set: `info` · `promo`
· `warning` · `danger`.
- Tone drives the whole bar — fill, border, icon and the button — so a
CTA can't drift from the bar it sits on. Text is neutral in every tone;
only the icon carries the tone colour.
- All colour comes from `--c-*` tokens mixed over `--c-surface`, so the
bars follow light and dark instead of ignoring them. The old bars were
hardcoded: in dark mode the two licence warnings stayed cream-on-white.
- `promo` keeps the gradient it was always meant to have, built from the
existing `--c-hue-indigo`/`--c-hue-purple` stops (documented in
`colors.css` as gradient hues, deliberately not accent-following), with
the existing `premium` button accent on it.
- Deleted the hardcoded colours from all four callers: the purple
gradient (`#667eea`→`#764ba2`), the orange soup (`#FFF4E6` / `#9A3412` /
`#EA580C`) duplicated across the urgent banner and the admin plan
section, and the fixed dark bar (`--mantine-color-dark-7`) on the team
invitation.
- `UpgradeBanner|AdminPlanSection` sat on the theme linter's exemption
list, which is how those colours survived the theme migration. Exemption
removed, so `code-colors` now guards them.
- The banner's class was colliding with `core/ui/Banner.css`'s
`.sui-banner` (16 live rules), which restyled it in the app but not in
Storybook — that's why the two disagreed on radius, border and tone.
Renamed to `.app-banner`; the two surfaces now render identically.
- Bar is square and full-bleed with a single hairline rule underneath;
button labels are optically centred.
- Added `--c-warning-subtle`, matching the existing `--c-danger-subtle`
/ `--c-success-subtle`.
- New `Shared → Top bars` story renders all six bars at once, so a
change to the shared component is visible against the whole set.
- Unrelated one-liner: `frontend/.prettierignore` now ignores the
gitignored `editor/screenshots/` capture artifacts, which were failing
`format:check` locally. Happy to drop it if you'd rather keep this PR to
the bars.

## Testing

- `task frontend:check` — typecheck, lint (oxlint + 4 theme-lint passes
+ stylelint), format, 244 files / 2119 tests.
- `frontend:storybook:a11y:changed` — clean in light and dark.
- The a11y gate caught a real defect mid-change: giving each banner
`role="region"` with the same label produced duplicate landmarks, which
the app hits for real whenever two banners show at once. Landmark
removed.
- All six bars captured in the running editor, light and dark, and
diffed against `origin/main`'s component rendered with each caller's
original props.
---
 .../public/locales/en-US/translation.toml     |   6 +-
 frontend/editor/scripts/lint/theme-lint.mjs   |   1 -
 .../shared/TeamInvitationBanner.tsx           |   8 +-
 .../src/core/components/AppLayout.stories.tsx |   4 +-
 .../src/core/components/shared/AppBanner.css  | 115 ++++++++
 .../components/shared/AppBanner.stories.tsx   | 194 +++++++++++++
 .../src/core/components/shared/AppBanner.tsx  | 124 +++++++++
 .../components/shared/InfoBanner.stories.tsx  |  38 ---
 .../src/core/components/shared/InfoBanner.tsx | 263 ------------------
 frontend/editor/src/core/theme/colors.css     |   5 +
 .../components/shared/DefaultAppBanner.tsx    |   4 +-
 .../components/shared/UpgradeBanner.tsx       |  22 +-
 .../configSections/AdminPlanSection.tsx       |  11 +-
 13 files changed, 453 insertions(+), 342 deletions(-)
 create mode 100644 frontend/editor/src/core/components/shared/AppBanner.css
 create mode 100644 frontend/editor/src/core/components/shared/AppBanner.stories.tsx
 create mode 100644 frontend/editor/src/core/components/shared/AppBanner.tsx
 delete mode 100644 frontend/editor/src/core/components/shared/InfoBanner.stories.tsx
 delete mode 100644 frontend/editor/src/core/components/shared/InfoBanner.tsx

diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml
index 70d3d03bf4..850f69ca02 100644
--- a/frontend/editor/public/locales/en-US/translation.toml
+++ b/frontend/editor/public/locales/en-US/translation.toml
@@ -1879,6 +1879,9 @@ width = "Width"
 [app]
 description = "The Free Adobe Acrobat alternative (10M+ Downloads)"
 
+[appBanner]
+dismiss = "Dismiss"
+
 [attachments]
 convertToPdfA3b = "Convert to PDF/A-3b"
 convertToPdfA3bDescription = "Creates an archival PDF with embedded attachments"
@@ -4827,9 +4830,6 @@ title = "Image to PDF"
 [imageToPdf]
 tags = "conversion,img,jpg,picture,photo"
 
-[infoBanner]
-dismiss = "Dismiss"
-
 [invite]
 acceptError = "Failed to create account"
 accountFor = "Creating account for"
diff --git a/frontend/editor/scripts/lint/theme-lint.mjs b/frontend/editor/scripts/lint/theme-lint.mjs
index 494fc03cca..97b32ef208 100644
--- a/frontend/editor/scripts/lint/theme-lint.mjs
+++ b/frontend/editor/scripts/lint/theme-lint.mjs
@@ -642,7 +642,6 @@ const CODE_EXEMPT_PATH = [
   /mantineTheme|\/theme\.ts$|toolsTaxonomy|LayoutPreview|PageNumberPreview|CloudStorageIcons|BrandMarks/,
   /\/onboarding\//,
   /addStamp|addWatermark|\/tooltips\//,
-  /UpgradeBanner|AdminPlanSection/,
   // Stories are checked like app code; colour-as-data lines opt out with
   // `theme-allow-color`.
   /\.test\.[jt]sx?$|\/types\//,
diff --git a/frontend/editor/src/cloud/components/shared/TeamInvitationBanner.tsx b/frontend/editor/src/cloud/components/shared/TeamInvitationBanner.tsx
index 3b373e9c3b..638c752f9e 100644
--- a/frontend/editor/src/cloud/components/shared/TeamInvitationBanner.tsx
+++ b/frontend/editor/src/cloud/components/shared/TeamInvitationBanner.tsx
@@ -3,7 +3,7 @@ import { Group, Text } from "@mantine/core";
 import { Button } from "@app/ui/Button";
 import { useTranslation } from "react-i18next";
 import LocalIcon from "@app/components/shared/LocalIcon";
-import { InfoBanner } from "@app/components/shared/InfoBanner";
+import { AppBanner } from "@app/components/shared/AppBanner";
 import { useSaaSTeam } from "@app/contexts/SaaSTeamContext";
 
 /**
@@ -105,7 +105,7 @@ export function TeamInvitationBanner() {
   );
 
   return (
-    
   );
 }
diff --git a/frontend/editor/src/core/components/AppLayout.stories.tsx b/frontend/editor/src/core/components/AppLayout.stories.tsx
index 4d7e6780cf..69aceef9d7 100644
--- a/frontend/editor/src/core/components/AppLayout.stories.tsx
+++ b/frontend/editor/src/core/components/AppLayout.stories.tsx
@@ -4,7 +4,7 @@ import { AppLayout } from "@app/components/AppLayout";
 import { BannerProvider, useBanner } from "@app/contexts/BannerContext";
 import { NavigationProvider } from "@app/contexts/NavigationContext";
 import { ToolRegistryProvider } from "@app/contexts/ToolRegistryProvider";
-import { InfoBanner } from "@app/components/shared/InfoBanner";
+import { AppBanner } from "@app/components/shared/AppBanner";
 
 const meta = {
   title: "Components/AppLayout",
@@ -49,7 +49,7 @@ function BannerSetter() {
   const { setBanner } = useBanner();
   useEffect(() => {
     setBanner(
-      ;
+export default meta;
+type Story = StoryObj;
+
+export const Info: Story = {
+  args: {
+    icon: "info-rounded",
+    title: "Heads up",
+    message: "This document contains form fields that will be flattened.",
+  },
+};
+
+export const Promo: Story = {
+  args: {
+    tone: "promo",
+    icon: "stars-rounded",
+    title: "Upgrade to Server Plan",
+    message:
+      "Get the most out of Stirling PDF with unlimited users and advanced features.",
+    buttonText: "Upgrade Now",
+    buttonIcon: "upgrade-rounded",
+    onButtonClick: () => {},
+    compact: true,
+  },
+};
+
+export const Warning: Story = {
+  args: {
+    tone: "warning",
+    icon: "warning-rounded",
+    title: "Action required",
+    message: "Some pages could not be processed and were skipped.",
+    buttonText: "Review",
+    onButtonClick: () => {},
+  },
+};
+
+export const Danger: Story = {
+  args: {
+    tone: "danger",
+    icon: "warning-rounded",
+    title: "This server needs admin attention",
+    message: "Review the license requirements to keep this server compliant.",
+    buttonText: "See info",
+    buttonIcon: "info-rounded",
+    onButtonClick: () => {},
+    dismissible: false,
+  },
+};
+
+export const Compact: Story = {
+  args: {
+    compact: true,
+    icon: "info-rounded",
+    message: "Autosave is enabled for this file.",
+    dismissible: false,
+  },
+};
+
+/** Message-only, no title: the message takes the title's weight so the bar still reads. */
+export const MessageOnly: Story = {
+  args: {
+    icon: "picture-as-pdf-rounded",
+    message:
+      "Make Stirling PDF your default application for opening PDF files.",
+    buttonText: "Set Default",
+    onButtonClick: () => {},
+    secondaryButtonText: "Don't remind me again",
+    onSecondaryButtonClick: () => {},
+  },
+};
+
+function Row({ caption, children }: { caption: string; children: ReactNode }) {
+  return (
+    
+ + {caption} + + {children} +
+ ); +} + +/** + * Every top bar the app can show, in one place: each entry mirrors a real caller, + * so a change to the component is visible against the whole set at once. Renders a + * composition rather than the component, so it takes no args of its own. + */ +export const AllTopBars: StoryObj = { + render: () => ( +
+ + {}} + /> + + + + {}} + dismissible={false} + /> + + + + {}} + dismissible={false} + /> + + + + {}} + secondaryButtonText="Decline" + onSecondaryButtonClick={() => {}} + dismissible={false} + /> + + + + {}} + secondaryButtonText="Don't remind me again" + onSecondaryButtonClick={() => {}} + /> + + + + {}} + dismissible={false} + /> + +
+ ), +}; diff --git a/frontend/editor/src/core/components/shared/AppBanner.tsx b/frontend/editor/src/core/components/shared/AppBanner.tsx new file mode 100644 index 0000000000..ee0ba03741 --- /dev/null +++ b/frontend/editor/src/core/components/shared/AppBanner.tsx @@ -0,0 +1,124 @@ +import React, { ReactNode } from "react"; +import { Button } from "@app/ui/Button"; +import { ActionIcon } from "@app/ui/ActionIcon"; +import { useTranslation } from "react-i18next"; +import LocalIcon from "@app/components/shared/LocalIcon"; +import "@app/components/shared/AppBanner.css"; + +/** Picks the whole look. Callers choose meaning, never colours. */ +export type AppBannerTone = "info" | "promo" | "warning" | "danger"; + +/** Tone decides the button too, so the CTA can't drift from the bar it sits on. */ +const TONE_BUTTON = { + info: { variant: "secondary", accent: "default" }, + promo: { variant: "primary", accent: "premium" }, + warning: { variant: "primary", accent: "warning" }, + danger: { variant: "primary", accent: "danger" }, +} as const; + +interface AppBannerProps { + /** A LocalIcon name, or a pre-rendered node (e.g. a logo) dropped in as-is. */ + icon?: string | ReactNode; + title?: ReactNode; + message: ReactNode; + buttonText?: string; + buttonIcon?: string; + onButtonClick?: () => void; + /** Muted secondary action, e.g. "Don't remind me again". */ + secondaryButtonText?: string; + onSecondaryButtonClick?: () => void; + onDismiss?: () => void; + dismissible?: boolean; + loading?: boolean; + show?: boolean; + tone?: AppBannerTone; + compact?: boolean; +} + +/** The app's top bar: dismissible messaging above the workspace. */ +export const AppBanner: React.FC = ({ + icon, + title, + message, + buttonText, + buttonIcon = "check-circle-rounded", + onButtonClick, + secondaryButtonText, + onSecondaryButtonClick, + onDismiss, + dismissible = true, + loading = false, + show = true, + tone = "info", + compact = false, +}) => { + const { t } = useTranslation(); + if (!show) return null; + + const iconSize = compact ? "1rem" : "1.25rem"; + + return ( +
+ {icon != null && ( + + {typeof icon === "string" ? ( + + ) : ( + icon + )} + + )} + +
+ {title && {title}} + {message} +
+ +
+ {buttonText && onButtonClick && ( + + )} + {secondaryButtonText && onSecondaryButtonClick && ( + + )} + {dismissible && ( + onDismiss?.()} + aria-label={t("appBanner.dismiss", "Dismiss")} + > + + + )} +
+
+ ); +}; diff --git a/frontend/editor/src/core/components/shared/InfoBanner.stories.tsx b/frontend/editor/src/core/components/shared/InfoBanner.stories.tsx deleted file mode 100644 index 5fad071d05..0000000000 --- a/frontend/editor/src/core/components/shared/InfoBanner.stories.tsx +++ /dev/null @@ -1,38 +0,0 @@ -import type { Meta, StoryObj } from "@storybook/react-vite"; -import { InfoBanner } from "@app/components/shared/InfoBanner"; - -const meta = { - title: "Shared/InfoBanner", - component: InfoBanner, - parameters: { layout: "padded" }, -} satisfies Meta; -export default meta; -type Story = StoryObj; - -export const Default: Story = { - args: { - icon: "info-rounded", - title: "Heads up", - message: "This document contains form fields that will be flattened.", - }, -}; - -export const Warning: Story = { - args: { - tone: "warning", - icon: "warning-rounded", - title: "Action required", - message: "Some pages could not be processed and were skipped.", - buttonText: "Review", - onButtonClick: () => {}, - }, -}; - -export const Compact: Story = { - args: { - compact: true, - icon: "info-rounded", - message: "Autosave is enabled for this file.", - dismissible: false, - }, -}; diff --git a/frontend/editor/src/core/components/shared/InfoBanner.tsx b/frontend/editor/src/core/components/shared/InfoBanner.tsx deleted file mode 100644 index 2056b6a92f..0000000000 --- a/frontend/editor/src/core/components/shared/InfoBanner.tsx +++ /dev/null @@ -1,263 +0,0 @@ -import React, { ReactNode } from "react"; -import { Paper, Group, Text, Stack } from "@mantine/core"; -import { Button, type ButtonVariant, type ButtonAccent } from "@app/ui/Button"; -import { ActionIcon } from "@app/ui/ActionIcon"; -import { useTranslation } from "react-i18next"; -import LocalIcon from "@app/components/shared/LocalIcon"; - -type InfoBannerTone = "info" | "warning"; - -const toneStyles: Record< - InfoBannerTone, - { - background: string; - border: string; - text: string; - icon: string; - buttonColor: string; - } -> = { - info: { - background: "var(--mantine-color-blue-0)", - border: "var(--mantine-color-blue-2)", - text: "var(--mantine-color-blue-9)", - icon: "var(--mantine-color-blue-6)", - buttonColor: "blue", - }, - warning: { - background: "var(--mantine-color-orange-0)", - border: "var(--mantine-color-orange-3)", - text: "var(--color-amber-dark)", - icon: "var(--mantine-color-orange-7)", - buttonColor: "orange", - }, -}; - -function toSharedButtonVariant( - variant: "light" | "filled" | "white" | "outline" | "subtle", -): ButtonVariant { - switch (variant) { - case "filled": - return "primary"; - case "outline": - return "secondary"; - case "subtle": - return "tertiary"; - case "light": - case "white": - default: - return "secondary"; - } -} - -function toSharedButtonAccent(color: string | undefined): ButtonAccent { - // Mantine colours may carry a shade suffix (e.g. "orange.7"); use the hue. - const hue = (color ?? "").split(".")[0]; - switch (hue) { - case "red": - return "danger"; - case "green": - return "success"; - case "yellow": - case "orange": - return "warning"; - case "blue": - default: - return "default"; - } -} - -interface InfoBannerProps { - /** - * Either a LocalIcon name (string) for the standard sized icon slot, or a - * pre-rendered ReactNode (e.g. a logo image) which is dropped in as-is. - */ - icon?: string | ReactNode; - title?: ReactNode; - message: ReactNode; - buttonText?: string; - buttonIcon?: string; - onButtonClick?: () => void; - /** Optional muted secondary action (e.g. "Don't remind me again"). */ - secondaryButtonText?: string; - onSecondaryButtonClick?: () => void; - onDismiss?: () => void; - dismissible?: boolean; - loading?: boolean; - show?: boolean; - tone?: InfoBannerTone; - background?: string; - borderColor?: string; - textColor?: string; - iconColor?: string; - buttonColor?: string; - buttonVariant?: "light" | "filled" | "white" | "outline" | "subtle"; - /** Override the button label colour (for dark/custom theme variants). */ - buttonTextColor?: string; - minHeight?: number | string; - closeIconColor?: string; - compact?: boolean; -} - -/** - * Generic info banner component for displaying dismissible messages at the top of the app - */ -export const InfoBanner: React.FC = ({ - icon, - title, - message, - buttonText, - buttonIcon = "check-circle-rounded", - onButtonClick, - secondaryButtonText, - onSecondaryButtonClick, - onDismiss, - dismissible = true, - loading = false, - show = true, - tone = "info", - background, - borderColor, - textColor, - iconColor, - buttonColor, - buttonVariant = "light", - buttonTextColor, - minHeight = 56, - closeIconColor, - compact = false, -}) => { - const { t } = useTranslation(); - if (!show) { - return null; - } - - const toneStyle = toneStyles[tone] ?? toneStyles.info; - const resolvedTextColor = textColor ?? toneStyle.text; - const handleDismiss = () => { - onDismiss?.(); - }; - - const iconSize = compact ? "1rem" : "1.2rem"; - const textSize = compact ? "xs" : "sm"; - - return ( - - - - {icon != null && - (typeof icon === "string" ? ( - - ) : ( -
- {icon} -
- ))} - - {title && ( - - {title} - - )} - - {message} - - -
- - {buttonText && onButtonClick && ( - - )} - {secondaryButtonText && onSecondaryButtonClick && ( - - )} - {dismissible && ( - - - - )} - -
-
- ); -}; diff --git a/frontend/editor/src/core/theme/colors.css b/frontend/editor/src/core/theme/colors.css index dffd6beaba..dd3c2aec7c 100644 --- a/frontend/editor/src/core/theme/colors.css +++ b/frontend/editor/src/core/theme/colors.css @@ -71,6 +71,11 @@ html[data-app-theme="light"] { var(--c-success) 10%, var(--c-surface) ); + --c-warning-subtle: color-mix( + in srgb, + var(--c-warning) 10%, + var(--c-surface) + ); /* ── Decorative / brand / categorical palette ────────────────────────── Fixed hues that intentionally do NOT follow the chosen accent: brand diff --git a/frontend/editor/src/desktop/components/shared/DefaultAppBanner.tsx b/frontend/editor/src/desktop/components/shared/DefaultAppBanner.tsx index 1b24cd4675..5ba92c2782 100644 --- a/frontend/editor/src/desktop/components/shared/DefaultAppBanner.tsx +++ b/frontend/editor/src/desktop/components/shared/DefaultAppBanner.tsx @@ -1,6 +1,6 @@ import React, { useState } from "react"; import { useTranslation } from "react-i18next"; -import { InfoBanner } from "@app/components/shared/InfoBanner"; +import { AppBanner } from "@app/components/shared/AppBanner"; import { useDefaultApp } from "@app/hooks/useDefaultApp"; export const DefaultAppBanner: React.FC = () => { @@ -15,7 +15,7 @@ export const DefaultAppBanner: React.FC = () => { const [sessionDismissed, setSessionDismissed] = useState(false); return ( - { ); return ( - { buttonIcon="info-rounded" onButtonClick={buttonText ? handleSeeInfo : undefined} dismissible={false} - minHeight={60} - background="#FFF4E6" - borderColor="var(--mantine-color-orange-7)" - textColor="#9A3412" - iconColor="#EA580C" - buttonVariant="filled" - buttonColor="orange.7" /> ); }; @@ -341,7 +334,7 @@ const UpgradeBanner: React.FC = () => { return ( <> {friendlyVisible && ( - { onButtonClick={handleUpgrade} onDismiss={handleFriendlyDismiss} show={friendlyVisible} - background="linear-gradient(135deg, #667eea 0%, #764ba2 100%)" - borderColor="transparent" - textColor="#fff" - iconColor="#fff" - closeIconColor="#fff" - buttonVariant="filled" - buttonColor="blue" - minHeight={48} + tone="promo" compact /> )} diff --git a/frontend/editor/src/proprietary/components/shared/config/configSections/AdminPlanSection.tsx b/frontend/editor/src/proprietary/components/shared/config/configSections/AdminPlanSection.tsx index f683861e8d..c69040f0e7 100644 --- a/frontend/editor/src/proprietary/components/shared/config/configSections/AdminPlanSection.tsx +++ b/frontend/editor/src/proprietary/components/shared/config/configSections/AdminPlanSection.tsx @@ -12,7 +12,7 @@ import AvailablePlansSection from "@app/components/shared/config/configSections/ import StaticPlanSection from "@app/components/shared/config/configSections/plan/StaticPlanSection"; import LicenseKeySection from "@app/components/shared/config/configSections/plan/LicenseKeySection"; import { alert } from "@app/components/toast"; -import { InfoBanner } from "@app/components/shared/InfoBanner"; +import { AppBanner } from "@app/components/shared/AppBanner"; import { useLicenseAlert } from "@app/hooks/useLicenseAlert"; import { getPreferredCurrency, @@ -200,7 +200,7 @@ const AdminPlanSection: React.FC = () => { {shouldShowLicenseWarning && ( - { buttonIcon="upgrade-rounded" onButtonClick={scrollToPlans} dismissible={false} - minHeight={68} - background="#FFF4E6" - borderColor="var(--mantine-color-orange-7)" - textColor="#9A3412" - iconColor="#EA580C" - buttonVariant="filled" - buttonColor="orange.7" /> )} From 913601ff0372d3fa1cadd11e48b1ebd2c921cdaa Mon Sep 17 00:00:00 2001 From: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com> Date: Tue, 18 Aug 2026 14:00:43 +0000 Subject: [PATCH 23/59] Consolidate the editor + processor sidebar footers into one component (#7539) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Both sidebars ended in a different bottom section. The editor showed an account row (avatar, name, settings); the processor showed a "Link Stirling account" CTA plus a `Settings` nav item and no identity at all. They are now **one shared ``** rendering the same rows in both apps, in this order: 1. the link-account CTA (self-hosted, when unlinked) 2. free credits remaining 3. **Open \** 4. the account row — avatar, name, settings It's a **single surface** with hairline dividers between rows, not stacked cards. Rows are assembled as a list, so a row this build doesn't show (no wallet, no processor access, nothing to link) takes its divider with it rather than leaving a stray line. This also fixes the profile-picture/initials desync between the sidebar and the account settings page. ## Screenshots Captured with the stubbed Playwright harness at 1600x900, scoped to the sidebar and auto-cropped to the region that actually changed. Base is `origin/main`; every state is driven by dummy backend stubs so all the nav-bar permutations are covered. montage_cloud-dark montage_cloud-light montage_editor-dark montage_editor-light montage_processor-dark montage_processor-light The free-credits meter is a cloud-build surface, so the self-hosted capture can't reach it. Those states come from the new Storybook stories with dummy wallet data (`Shared/NavFooter`), which is also where the credit tone bands and the collapsed rail are easiest to review. ## How it's wired `NavFooter` is purely presentational. Each app resolves its own data through three `@app/*` seams, so core carries no build-specific gating and any box whose data is absent is dropped rather than rendered empty. | Seam | core | cloud / proprietary / saas | |---|---|---| | `useFreeCreditsSummary` | `null` — self-hosted editor installs aren't metered | cloud reads `freeRemaining` / `freeAllowance` off the same `useWallet()` the Plan page's free meter uses, so the sidebar and Plan can't disagree | | `useOtherAppSwitch` | `null` — core ships no processor | gated on `portalAccess` (`/api/v1/auth/me` in SaaS, the Spring session flag self-hosted) | | Link-account CTA | n/a | unchanged conditions — passed in as `accountExtras`, still only when `linkState === "unlinked"`, still a no-op in SaaS | - The processor reads the meter through its own `@portal/hooks/useFreeCreditsSummary` rather than the editor's `@app` one. Self-hosted resolves `@app/*` as proprietary → core, where the cloud wallet hook isn't in the cascade, and the implementation can't live in `proprietary/` because core/desktop builds ship no portal and must never resolve `@portal`. Keeping it in `portal/` gets the figure to the linked self-hosted processor without weakening that rule; it reads the same `GET /api/v1/payg/wallet` the Usage page's trial meter already renders, gated on link state and behind the portal's query cache. `portal-saas/` just re-exports the cloud hook, so both footers share one fetch. The processor-access gate previously lived in two near-identical `AppSwitcher` copies. It moves into `useOtherAppSwitch`, `AppSwitcher` now reads it too, and the duplicate `saas/components/shared/AppSwitcher.tsx` is deleted — the logo switcher and the footer row can no longer disagree about access. ## Profile picture sync One `useAccountIdentity` hook now backs the editor footer, the processor footer and the account settings page. Previously settings derived its initial from `email[0]` while the sidebar used `displayName[0]`, and the two drew different blue discs. Alongside that, the shared `Avatar`: - falls back to initials when a picture URL fails to load, instead of leaving an empty disc - renders one letter for single-word names (`admin` → "A", not "AD") - gains an `xl` size so the settings hero disc is the same component ## Notes - Labelled **"Free credits"** rather than "free monthly credits": `freeAllowance` is documented as a one-time lifetime grant, not a monthly reset, so "monthly" would misdescribe the data. Happy to change if the backend semantics differ from the type comments. ## Testing - `task frontend:check` and `task frontend:typecheck:all` pass (all 9 build variants). - 9 new `Shared/NavFooter` stories pass the Chromium + axe story scan; `frontend:storybook:a11y:changed` reports no regressions. - Stubbed E2E suite passes, including the `config-button` tour/settings specs that target the account row. Two failures (`console-clean › landing`, `viewer-text-selection › Ctrl+C`) also fail on `origin/main` locally — they need a backend on :8080 and clipboard permissions. --- .../public/locales/en-US/translation.toml | 23 +- .../config/configSections/usageMeters.tsx | 22 +- .../src/cloud/hooks/useFreeCreditsSummary.ts | 50 ++++ .../editor/src/cloud/hooks/useOpenPlan.ts | 13 ++ frontend/editor/src/cloud/hooks/useWallet.ts | 90 +++++++- .../src/core/components/shared/BrandMark.css | 46 ++++ .../core/components/shared/FileSidebar.css | 92 +------- .../core/components/shared/FileSidebar.tsx | 130 ++--------- .../components/shared/navFooter/NavFooter.css | 156 +++++++++++++ .../shared/navFooter/NavFooter.stories.tsx | 119 ++++++++++ .../shared/navFooter/NavFooter.test.tsx | 58 +++++ .../components/shared/navFooter/NavFooter.tsx | 213 ++++++++++++++++++ .../shared/navFooter/NavFooterCreditsRow.css | 83 +++++++ .../shared/navFooter/NavFooterCreditsRow.tsx | 158 +++++++++++++ .../src/core/hooks/useAccountIdentity.ts | 64 ++++++ .../src/core/hooks/useFreeCreditsSummary.ts | 12 + frontend/editor/src/core/hooks/useOpenPlan.ts | 10 + .../src/core/hooks/useOtherAppSwitch.ts | 12 + frontend/editor/src/core/query/keys.ts | 3 + .../src/core/services/navFooterCache.ts | 73 ++++++ frontend/editor/src/core/ui/Avatar.css | 6 + frontend/editor/src/core/ui/Avatar.tsx | 27 ++- .../hooks/useFreeCreditsSummary.ts | 7 + .../src/portal-saas/hooks/useOpenPlan.ts | 11 + .../editor/src/portal/components/Sidebar.css | 8 +- .../editor/src/portal/components/Sidebar.tsx | 29 ++- .../billing/PrepaidCapacityCard.tsx | 9 +- .../portal/components/billing/WalletMeter.tsx | 26 ++- .../hooks/useFreeCreditsSummary.test.tsx | 110 +++++++++ .../src/portal/hooks/useFreeCreditsSummary.ts | 65 ++++++ .../editor/src/portal/hooks/useOpenPlan.ts | 13 ++ frontend/editor/src/portal/queries/keys.ts | 2 + .../editor/src/proprietary/billing/format.ts | 23 ++ .../editor/src/proprietary/billing/index.ts | 1 + .../components/shared/AppSwitcher.tsx | 20 +- .../proprietary/hooks/useOtherAppSwitch.ts | 15 ++ .../saas/components/shared/AppSwitcher.tsx | 41 ---- .../shared/config/configSections/Overview.tsx | 24 +- .../src/saas/hooks/useOtherAppSwitch.ts | 16 ++ .../src/saas/hooks/usePortalAccess.test.tsx | 43 +++- .../editor/src/saas/hooks/usePortalAccess.ts | 76 ++++--- .../src/saas/hooks/useWallet.poll.test.tsx | 158 +++++++++++++ 42 files changed, 1804 insertions(+), 353 deletions(-) create mode 100644 frontend/editor/src/cloud/hooks/useFreeCreditsSummary.ts create mode 100644 frontend/editor/src/cloud/hooks/useOpenPlan.ts create mode 100644 frontend/editor/src/core/components/shared/navFooter/NavFooter.css create mode 100644 frontend/editor/src/core/components/shared/navFooter/NavFooter.stories.tsx create mode 100644 frontend/editor/src/core/components/shared/navFooter/NavFooter.test.tsx create mode 100644 frontend/editor/src/core/components/shared/navFooter/NavFooter.tsx create mode 100644 frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.css create mode 100644 frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.tsx create mode 100644 frontend/editor/src/core/hooks/useAccountIdentity.ts create mode 100644 frontend/editor/src/core/hooks/useFreeCreditsSummary.ts create mode 100644 frontend/editor/src/core/hooks/useOpenPlan.ts create mode 100644 frontend/editor/src/core/hooks/useOtherAppSwitch.ts create mode 100644 frontend/editor/src/core/services/navFooterCache.ts create mode 100644 frontend/editor/src/portal-saas/hooks/useFreeCreditsSummary.ts create mode 100644 frontend/editor/src/portal-saas/hooks/useOpenPlan.ts create mode 100644 frontend/editor/src/portal/hooks/useFreeCreditsSummary.test.tsx create mode 100644 frontend/editor/src/portal/hooks/useFreeCreditsSummary.ts create mode 100644 frontend/editor/src/portal/hooks/useOpenPlan.ts create mode 100644 frontend/editor/src/proprietary/hooks/useOtherAppSwitch.ts delete mode 100644 frontend/editor/src/saas/components/shared/AppSwitcher.tsx create mode 100644 frontend/editor/src/saas/hooks/useOtherAppSwitch.ts create mode 100644 frontend/editor/src/saas/hooks/useWallet.poll.test.tsx diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml index 850f69ca02..12ad26ceec 100644 --- a/frontend/editor/public/locales/en-US/translation.toml +++ b/frontend/editor/public/locales/en-US/translation.toml @@ -5057,6 +5057,14 @@ title = "Upload from Mobile" tags = "Multi Tool,Multi operation,UI,click drag,front end,client side,interactive,intractable,move,delete,migrate,divide" title = "PDF Multi Tool" +[navFooter] +openEditor = "Open PDF Editor" +openProcessor = "Open PDF Processor" + +[navFooter.credits] +count = "{{remaining}} of {{total}}" +label = "Free credits" + [oauth.error] message = "Authentication was not successful. You can close this window and try again." title = "Authentication Failed" @@ -5621,8 +5629,8 @@ freeBody = "View, edit, merge, split, sign, watermark, compress, convert and man freeTitle = "Unlimited PDF editing" [payg.free.hero] -barAria = "Free PDFs used" -capSuffix = "/ {{limit}} free PDFs" +barAria = "Free PDFs remaining" +capSuffix = "of {{limit}} free PDFs left" metaCategories = "Automation · AI · API requests" [payg.free.member] @@ -6685,12 +6693,12 @@ reachedTitle = "Monthly spend limit reached" title = "Couldn't open Stripe portal" [portal.billing.walletMeter] -barAria = "Free PDFs used" -capSuffix_one = "of {{allowance}} free PDFs used" -capSuffix_other = "of {{allowance}} free PDFs used" +barAria = "Free PDFs remaining" +capSuffix_one = "of {{allowance}} free PDF left" +capSuffix_other = "of {{allowance}} free PDFs left" eyebrow = "Processor trial" -statusLabel_one = "{{remaining}} left" -statusLabel_other = "{{remaining}} left" +statusLabel_one = "{{used}} used" +statusLabel_other = "{{used}} used" sub = "Use the PDF Editor for free. Pay to process PDFs automatically." title_one = "Process {{allowance}} PDFs free" title_other = "Process {{allowance}} PDFs free" @@ -7665,7 +7673,6 @@ integrations = "Integrations" pipelines = "Pipelines" policies = "Policies" procurement = "Procurement" -settings = "Settings" sources = "Sources" usage = "Usage & Billing" users = "Users" diff --git a/frontend/editor/src/cloud/components/shared/config/configSections/usageMeters.tsx b/frontend/editor/src/cloud/components/shared/config/configSections/usageMeters.tsx index 8811e537b2..c4407cae7f 100644 --- a/frontend/editor/src/cloud/components/shared/config/configSections/usageMeters.tsx +++ b/frontend/editor/src/cloud/components/shared/config/configSections/usageMeters.tsx @@ -12,6 +12,7 @@ import { formatPeriodDate, MeterBar, meterState, + remainingMeter, } from "@app/billing"; import "@app/components/shared/config/configSections/Payg.css"; import "@app/components/shared/config/configSections/PaygFree.css"; @@ -48,7 +49,8 @@ export function useFreeSnapshot(): FreeSnapshot { export function FreeMeterPanel({ snap }: { snap: FreeSnapshot }) { const { t } = useTranslation(); - const { state, pct } = meterState(snap.billableUsed, snap.billableLimit); + const remaining = Math.max(0, snap.billableLimit - snap.billableUsed); + const { state, pct } = remainingMeter(remaining, snap.billableLimit); const stateLabel = state === "DEGRADED" ? t("payg.free.state.limitReached", "Limit reached") @@ -60,9 +62,9 @@ export function FreeMeterPanel({ snap }: { snap: FreeSnapshot }) { { + if (live !== undefined) writeCachedCredits(live); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [wallet]); + + return (live !== undefined ? live : seed) ?? null; +} diff --git a/frontend/editor/src/cloud/hooks/useOpenPlan.ts b/frontend/editor/src/cloud/hooks/useOpenPlan.ts new file mode 100644 index 0000000000..4d532319f6 --- /dev/null +++ b/frontend/editor/src/cloud/hooks/useOpenPlan.ts @@ -0,0 +1,13 @@ +import { useCallback } from "react"; +import { useNavigate } from "react-router-dom"; + +/** + * Cloud editor builds open the settings modal on its Plan section, which is + * where the free grant is explained and the Processor plan is switched on. + * Routed rather than called directly because the modal is URL-driven here + * (`/settings/*`), the same path the admin tour uses to open it. + */ +export function useOpenPlan(): (() => void) | null { + const navigate = useNavigate(); + return useCallback(() => navigate("/settings/plan"), [navigate]); +} diff --git a/frontend/editor/src/cloud/hooks/useWallet.ts b/frontend/editor/src/cloud/hooks/useWallet.ts index 0a3f78b3ce..ed3cb2ce6b 100644 --- a/frontend/editor/src/cloud/hooks/useWallet.ts +++ b/frontend/editor/src/cloud/hooks/useWallet.ts @@ -32,6 +32,14 @@ * promise see the UI flip exactly once the new state is visible — no * intermediate flash of the old value. * + *

Freshness

+ * + * The figures drain as metered work runs, so a mounted consumer re-reads the + * wallet every {@link WALLET_POLL_MS} and again whenever the tab regains + * visibility. Those refreshes are silent — they leave {@code loading} and + * {@code error} alone and only commit fresher data — so consumers that gate on + * those flags don't flicker on a background tick. + * *

Dev preview fallback

* * When the hook is rendered outside the saas app (e.g. on {@code @@ -178,6 +186,13 @@ function reuseIfEqual(prev: Wallet | null, next: Wallet): Wallet { return prev; } +/** + * How often a mounted consumer re-reads the wallet. Matches the app query + * client's staleTime, so the sidebar meter and anything cached elsewhere age + * out on the same clock. + */ +const WALLET_POLL_MS = 30_000; + export function useWallet(): UseWalletResult { // Resolved once: the dev-preview side-channel when rendered outside the real // app (saas /dev/payg-preview route), else null (every real build + desktop). @@ -201,13 +216,29 @@ export function useWallet(): UseWalletResult { // "the request fired." Cleared when no load is pending. const inFlight = useRef | null>(null); + // Set for refreshes the user didn't ask for (the poll below). Silence governs + // whether a load may RAISE `loading` / `error`, never whether it may clear + // them: consumers gate on both — the limit modals do + // `if (loading || !wallet) return null`, and Plan swaps in an error alert — + // so a background tick must not blink an open modal out or replace a working + // page over a transient failure. Clearing is always the latest request's job, + // silent or not; a silent load that skipped the clear would strand `loading` + // true after superseding a visible one, which suppresses those modals for the + // rest of the session. + const silentRefresh = useRef(false); + useEffect(() => { const reqId = ++latestReqId.current; let cancelled = false; + const silent = silentRefresh.current; + silentRefresh.current = false; + const promise = (async () => { - setLoading(true); - setError(null); + if (!silent) { + setLoading(true); + setError(null); + } if (devPreview) { const synth = devPreview.buildWallet(devPreview.role()); @@ -221,11 +252,22 @@ export function useWallet(): UseWalletResult { const res = await apiClient.get("/api/v1/payg/wallet"); if (cancelled || reqId !== latestReqId.current) return; setWallet((prev) => reuseIfEqual(prev, res.data)); + // Fresh data retires any earlier failure, including one a silent poll + // is recovering from — otherwise Plan keeps its alert over good data. + setError(null); } catch (e: unknown) { if (cancelled || reqId !== latestReqId.current) return; - console.warn("[useWallet] fetch failed", e); - setError(e instanceof Error ? e.message : "Failed to load wallet"); + if (!silent) { + console.warn("[useWallet] fetch failed", e); + setError(e instanceof Error ? e.message : "Failed to load wallet"); + } + // A failed background refresh is a non-event: the last good snapshot + // stands and the next tick self-heals, so it neither surfaces nor + // logs — otherwise an offline tab warns every WALLET_POLL_MS. } finally { + // Deliberately not gated on `silent`: whichever load is latest owns + // settling the flag, or a silent refresh that supersedes a visible one + // leaves it stuck true. if (!cancelled && reqId === latestReqId.current) { setLoading(false); } @@ -242,6 +284,46 @@ export function useWallet(): UseWalletResult { }; }, [devPreview, refetchTick]); + // The wallet drains as automation, AI and API work runs, so a figure fetched + // on mount goes stale while the user watches it. Refresh on a timer, and + // immediately on returning to the tab — coming back to a stale number is the + // case people actually notice. Hidden tabs don't poll, and the dev-preview + // wallet is synthesised locally so there is nothing to re-read. + useEffect(() => { + if (devPreview) return; + + let timer: ReturnType | undefined; + const refresh = () => { + silentRefresh.current = true; + setRefetchTick((t) => t + 1); + }; + const stop = () => { + if (timer !== undefined) { + clearInterval(timer); + timer = undefined; + } + }; + const start = () => { + stop(); + timer = setInterval(refresh, WALLET_POLL_MS); + }; + const onVisibilityChange = () => { + if (document.visibilityState === "visible") { + refresh(); + start(); + } else { + stop(); + } + }; + + if (document.visibilityState === "visible") start(); + document.addEventListener("visibilitychange", onVisibilityChange); + return () => { + stop(); + document.removeEventListener("visibilitychange", onVisibilityChange); + }; + }, [devPreview]); + const refetch = useCallback(async () => { setRefetchTick((t) => t + 1); // Snapshot the next-tick promise so the caller awaits this refetch diff --git a/frontend/editor/src/core/components/shared/BrandMark.css b/frontend/editor/src/core/components/shared/BrandMark.css index 7ddff9b4c7..df05ff1307 100644 --- a/frontend/editor/src/core/components/shared/BrandMark.css +++ b/frontend/editor/src/core/components/shared/BrandMark.css @@ -48,9 +48,55 @@ transform: matrix(0.483871, -0.017568, 0, 0.338028, 23.887097, 26.886428); } +/* One-shot "thinking" drift — the two parallelograms swap past each other and + settle back. Same motion the chat FAB loops while the agent works, but this + pair starts and ends at rest (translate 0, full opacity) so a single + iteration can end without snapping. Callers apply it for one beat; see + NavFooter.css for the hover use. */ +@keyframes sui-brandmark-drift-a { + 0%, + 100% { + transform: translate(0, 0); + opacity: 1; + } + 25% { + transform: translate(-1px, -5px); + opacity: 0.55; + } + 50% { + transform: translate(-6px, 0); + opacity: 0.9; + } + 75% { + transform: translate(-1px, 5px); + opacity: 0.6; + } +} + +@keyframes sui-brandmark-drift-b { + 0%, + 100% { + transform: translate(0, 0); + opacity: 1; + } + 25% { + transform: translate(1px, 5px); + opacity: 0.85; + } + 50% { + transform: translate(6px, 0); + opacity: 0.5; + } + 75% { + transform: translate(1px, -5px); + opacity: 0.85; + } +} + @media (prefers-reduced-motion: reduce) { .sui-brandmark__a, .sui-brandmark__b { transition: none; + animation: none; } } diff --git a/frontend/editor/src/core/components/shared/FileSidebar.css b/frontend/editor/src/core/components/shared/FileSidebar.css index 590f59fa7d..2347d9a2b2 100644 --- a/frontend/editor/src/core/components/shared/FileSidebar.css +++ b/frontend/editor/src/core/components/shared/FileSidebar.css @@ -75,16 +75,13 @@ padding: 0.25rem 0; overflow: hidden; } -.file-sidebar-footer-box { - padding: 0.25rem 0; - flex-shrink: 0; -} +/* The footer is the shared : it brings its own boxes and padding, + so this class only positions it in the column. */ /* Collapsed rail: the file tree isn't rendered, so hide its (empty) box and let the boxes stack at the top — controls, then the settings footer right after — instead of the files box stretching to fill. */ -.file-sidebar[data-collapsed="true"] .file-sidebar-controls, -.file-sidebar[data-collapsed="true"] .file-sidebar-footer-box { +.file-sidebar[data-collapsed="true"] .file-sidebar-controls { padding: 0.25rem; } .file-sidebar[data-collapsed="true"] .file-sidebar-files-box { @@ -538,86 +535,3 @@ pointer-events: none; animation: none; } - -/* ---- Bottom bar (user + settings) ---- */ -.file-sidebar-bottom-bar { - display: flex; - align-items: center; - gap: 8px; - padding: 4px 6px; - flex-shrink: 0; - min-height: 40px; -} - -/* Bottom bar settings icon tracks the right edge during collapse animation */ - -.file-sidebar-bottom-avatar { - width: 28px; - height: 28px; - border-radius: 50%; - background-color: var(--c-accent-text); - color: var(--c-text-on-primary); - font-size: 12px; - font-weight: 600; - display: flex; - align-items: center; - justify-content: center; - flex-shrink: 0; - user-select: none; - overflow: hidden; -} - -/* No colored disc behind an actual photo; keep it for the initials fallback. */ -.file-sidebar-bottom-avatar--picture { - background-color: transparent; -} - -.file-sidebar-bottom-avatar-img { - width: 100%; - height: 100%; - border-radius: 50%; - object-fit: cover; -} - -.file-sidebar-bottom-name { - flex: 1; - font-size: 13px; - font-weight: 500; - color: var(--c-text); - white-space: nowrap; - overflow: hidden; - text-overflow: ellipsis; - min-width: 0; -} - -.file-sidebar-bottom-bar[role="button"]:hover { - background-color: var(--c-hover); -} - -.file-sidebar-bottom-bar[role="button"]:focus-visible { - outline: 2px solid var(--c-primary); - outline-offset: -2px; -} - -.file-sidebar-bottom-settings { - display: flex; - align-items: center; - justify-content: center; - width: 28px; - height: 28px; - border-radius: 6px; - color: var(--c-text-subtle); - padding: 0; - flex-shrink: 0; - margin-left: auto; -} - -.file-sidebar[data-collapsed="true"] .file-sidebar-bottom-settings { - width: 32px; - height: 32px; -} - -.file-sidebar[data-collapsed="true"] .file-sidebar-bottom-bar { - justify-content: center; - padding: 8px 0; -} diff --git a/frontend/editor/src/core/components/shared/FileSidebar.tsx b/frontend/editor/src/core/components/shared/FileSidebar.tsx index 2514e7f556..1c06236027 100644 --- a/frontend/editor/src/core/components/shared/FileSidebar.tsx +++ b/frontend/editor/src/core/components/shared/FileSidebar.tsx @@ -22,13 +22,15 @@ import { } from "@app/contexts/NavigationContext"; import { useViewer } from "@app/contexts/ViewerContext"; import { useFileHandler } from "@app/hooks/useFileHandler"; -import { useAuth } from "@app/auth/UseSession"; -import { useProfilePictureUrl } from "@app/hooks/useProfilePictureUrl"; +import { useAccountIdentity } from "@app/hooks/useAccountIdentity"; +import { useFreeCreditsSummary } from "@app/hooks/useFreeCreditsSummary"; +import { useOtherAppSwitch } from "@app/hooks/useOtherAppSwitch"; +import { useOpenPlan } from "@app/hooks/useOpenPlan"; +import { NavFooter } from "@app/components/shared/navFooter/NavFooter"; import { useIndexedDB, useIndexedDBRevision, } from "@app/contexts/IndexedDBContext"; -import { accountService } from "@app/services/accountService"; import { GoogleDriveIcon } from "@app/components/shared/CloudStorageIcons"; import { AppSwitcher } from "@app/components/shared/AppSwitcher"; import { SidebarToggleIcon } from "@app/components/shared/SidebarToggleIcon"; @@ -37,8 +39,7 @@ import FolderOpenIcon from "@mui/icons-material/FolderOpen"; import FolderSpecialIcon from "@mui/icons-material/FolderSpecial"; import UploadFileIcon from "@mui/icons-material/UploadFile"; import AddIcon from "@mui/icons-material/Add"; -import OpenInNewIcon from "@mui/icons-material/OpenInNew"; -import SettingsIcon from "@mui/icons-material/Settings"; +import OpenInFullIcon from "@mui/icons-material/OpenInFull"; import type { FileId } from "@app/types/file"; import { FileItem } from "@app/components/shared/FileSidebarFileItem"; import { useLabelName } from "@app/data/labelDisplay"; @@ -241,43 +242,11 @@ const FileSidebar = forwardRef( const { addFiles } = useFileHandler(); const indexedDB = useIndexedDB(); - // Each auth layer derives its own displayName from its native user shape. - // Fall back to the proprietary REST endpoint only when the auth - // context yields nothing - then to "User" as a generic last resort. - const { displayName: authDisplayName, isAnonymous } = useAuth(); - const [accountUsername, setAccountUsername] = useState(null); - const displayName = - authDisplayName ?? accountUsername ?? t("auth.displayName.user", "User"); - - const profilePictureUrl = useProfilePictureUrl(); - const [pictureFailed, setPictureFailed] = useState(false); - useEffect(() => setPictureFailed(false), [profilePictureUrl]); - const showProfilePicture = !!profilePictureUrl && !pictureFailed; - - useEffect(() => { - if (!config?.enableLogin) { - setAccountUsername(null); - return; - } - if (authDisplayName) { - // The auth context has a name; don't bother hitting the REST - // endpoint, but clear any stale cached value from a prior call. - setAccountUsername(null); - return; - } - accountService - .getAccountData() - .then((data) => { - // Always reflect the latest result - including clearing it on - // sign-out, when the endpoint returns no username (or 401s into - // the catch branch below). Without this, signing out would leave - // the old username on screen. - setAccountUsername(data?.username ?? null); - }) - .catch(() => { - setAccountUsername(null); - }); - }, [config?.enableLogin, authDisplayName]); + const { displayName, profilePictureUrl, isAnonymous } = + useAccountIdentity(); + const credits = useFreeCreditsSummary(); + const otherApp = useOtherAppSwitch(); + const openPlan = useOpenPlan(); // Leaf files = user-visible files (excludes intermediate tool outputs) const [allFileStubs, setAllFileStubs] = useState([]); @@ -1115,7 +1084,7 @@ const FileSidebar = forwardRef( )} data-testid="open-files-page" > - + ( {/* Getting-started checklist, floating above the footer (SaaS only). */} - {/* Box 3 — account footer (avatar + name + settings). */} - - {/* Bottom bar: user name + settings */} - -
e.key === "Enter" && onOpenSettings() - : undefined - } - data-testid={onOpenSettings ? "config-button" : undefined} - data-tour={onOpenSettings ? "config-button" : undefined} - aria-label={ - onOpenSettings - ? t("fileSidebar.openSettings", "Open settings") - : displayName - } - style={onOpenSettings ? { cursor: "pointer" } : undefined} - > -
- {showProfilePicture ? ( - setPictureFailed(true)} - /> - ) : ( - displayName.charAt(0).toUpperCase() - )} -
- {!collapsed && ( - - {displayName} - - )} - {onOpenSettings && !collapsed && ( -
- -
- )} -
-
-
+ {/* Box 3 — the shared footer: credits, app switch, account row. */} +
); }, diff --git a/frontend/editor/src/core/components/shared/navFooter/NavFooter.css b/frontend/editor/src/core/components/shared/navFooter/NavFooter.css new file mode 100644 index 0000000000..aa2688cd35 --- /dev/null +++ b/frontend/editor/src/core/components/shared/navFooter/NavFooter.css @@ -0,0 +1,156 @@ +/* Shared sidebar footer: one surface holding the link-account CTA, the credits + meter, the other-app switch and the account row, hairline-separated. + Structural only — every colour comes from a --c-* semantic token. */ + +.nav-footer { + display: flex; + flex-direction: column; + flex-shrink: 0; + /* Vertical only: the slots carry the horizontal padding so their separator + runs the full width of the surface. */ + padding: 0.25rem 0; + overflow: hidden; +} + +.nav-footer__slot { + padding-inline: 0.375rem; +} + +/* Separators are drawn by the slots themselves, never as their own elements. + A slot whose contents render nothing (the link-account CTA returns null once + the org is linked, and an element is truthy even when it renders null) is + :empty, so it is skipped by both rules below — it can't leave a line behind, + and it can't push one to the top or bottom of the surface. A rule that only + ever matches a slot PRECEDED by another visible slot cannot draw a leading + separator, whatever the caller passes in. */ +.nav-footer__slot:empty { + display: none; +} + +.nav-footer__slot:not(:empty) ~ .nav-footer__slot:not(:empty) { + border-top: 1px solid var(--c-border-subtle); + margin-top: 0.25rem; + padding-top: 0.25rem; +} + +/* Fades the rows up on the first footer mount of a page session only. They are + seeded from cache, so they're already present at first paint; replaying this + on every later mount (switching apps, remounting a view) would animate + content that never changed and read as a twitch. */ +@keyframes nav-footer-row-in { + from { + opacity: 0; + transform: translateY(0.25rem); + } + to { + opacity: 1; + transform: none; + } +} + +.nav-footer[data-animate] .nav-footer__slot:not(:empty) { + animation: nav-footer-row-in var(--motion-enter) both; +} + +@media (prefers-reduced-motion: reduce) { + .nav-footer[data-animate] .nav-footer__slot:not(:empty) { + animation: none; + } +} + +/* ---- Rows (link-account, credits, switch, account) ---- */ + +.nav-footer__row { + display: flex; + align-items: center; + gap: 0.5rem; + width: 100%; + min-height: 2.25rem; + padding: 0.25rem 0.375rem; + border: 0; + border-radius: 0.5rem; + background: none; + color: var(--c-text); + font: inherit; + text-align: left; + cursor: pointer; +} + +.nav-footer__row:disabled { + cursor: default; +} + +.nav-footer__row:not(:disabled):hover { + background-color: var(--c-hover); +} + +.nav-footer__row:focus-visible { + outline: 2px solid var(--c-primary); + outline-offset: -2px; +} + +.nav-footer__row-icon { + display: flex; + align-items: center; + justify-content: center; + flex-shrink: 0; + width: 1.625rem; +} + +/* Hovering the switch row plays the mark's "thinking" drift once — the same + motion the chat FAB loops, for a single beat, as a hint that the row hands + off to the other app. One iteration only: it starts and ends at rest, so + nothing snaps when it finishes, and re-entering the row replays it. */ +.nav-footer__row:hover .sui-brandmark__a { + animation: sui-brandmark-drift-a 1.1s ease-in-out 1; +} +.nav-footer__row:hover .sui-brandmark__b { + animation: sui-brandmark-drift-b 1.1s ease-in-out 1; +} + +.nav-footer__row-label { + flex: 1; + min-width: 0; + font-size: 0.8125rem; + font-weight: 500; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +/* Trailing affordance on a row: the account row's gear, the switch row's + leaving-this-app arrow. */ +.nav-footer__trailing { + display: flex; + align-items: center; + justify-content: center; + flex-shrink: 0; + margin-left: auto; + color: var(--c-text-subtle); +} + +/* Rows contributed by a caller (the link-account NavItem) sit in the same + surface, so match this footer's row metrics rather than the nav rail's. */ +.nav-footer .sui-navitem { + min-height: 2.25rem; + padding: 0.25rem 0.375rem; + margin: 0; + border-radius: 0.5rem; + font-size: 0.8125rem; +} + +/* ---- Collapsed icon rail ---- */ + +.nav-footer[data-collapsed] .nav-footer__slot { + padding-inline: 0.25rem; +} + +.nav-footer[data-collapsed] .nav-footer__row { + justify-content: center; + padding-inline: 0; +} + +.nav-footer[data-collapsed] .sui-navitem { + justify-content: center; + padding-inline: 0; +} diff --git a/frontend/editor/src/core/components/shared/navFooter/NavFooter.stories.tsx b/frontend/editor/src/core/components/shared/navFooter/NavFooter.stories.tsx new file mode 100644 index 0000000000..31a04be596 --- /dev/null +++ b/frontend/editor/src/core/components/shared/navFooter/NavFooter.stories.tsx @@ -0,0 +1,119 @@ +import type { Meta, StoryObj } from "@storybook/react-vite"; +import LinkIcon from "@mui/icons-material/Link"; +import { NavItem } from "@app/ui/NavItem"; +import { NavFooter } from "@app/components/shared/navFooter/NavFooter"; + +/** Stands in for a CTA that has decided it has nothing to show. */ +function RendersNothing() { + return null; +} + +const meta: Meta = { + title: "Shared/NavFooter", + component: NavFooter, + parameters: { layout: "padded" }, + args: { + displayName: "admin", + onOpenSettings: () => {}, + credits: { remaining: 247, total: 500 }, + onOpenPlan: () => {}, + otherApp: { app: "processor", onOpen: () => {} }, + }, + decorators: [ + (S) => ( +
+ +
+ ), + ], +}; +export default meta; +type Story = StoryObj; + +/** The editor's footer: credits, "Open PDF Processor", the account row. */ +export const InEditor: Story = {}; + +/** The processor's footer. Same three boxes, opposite switch target. */ +export const InProcessor: Story = { + args: { otherApp: { app: "editor", onOpen: () => {} } }, +}; + +/** Self-hosted processor: no wallet, so no meter, and the link-account CTA + * rides along in the account box. */ +export const WithLinkAccountCta: Story = { + args: { + credits: null, + otherApp: { app: "editor", onOpen: () => {} }, + accountExtras: ( + } + /> + ), + }, +}; + +/** Regression guard: the processor always passes its link-account CTA, but that + * component renders null once the org is linked. An element is truthy even + * when it renders nothing, so this must not leave a separator above the first + * visible row. */ +export const ExtrasThatRenderNothing: Story = { + args: { accountExtras: }, +}; + +/** A real profile picture replaces the initials disc. */ +export const WithProfilePicture: Story = { + args: { + displayName: "Ada Lovelace", + profilePictureUrl: + "data:image/svg+xml;utf8," + + encodeURIComponent( + '', + ), + }, +}; + +/** Credits running low — the dot and bar shift to the warning tone at 20% left. */ +export const CreditsLow: Story = { + args: { credits: { remaining: 42, total: 500 } }, +}; + +/** Allowance exhausted. */ +export const CreditsExhausted: Story = { + args: { credits: { remaining: 0, total: 500 } }, +}; + +/** Core OSS: no wallet, no second app, settings only. */ +export const MinimalBuild: Story = { + args: { credits: null, otherApp: null }, +}; + +/** No settings handler — the account row is inert identity, not a button. */ +export const NoSettings: Story = { + args: { onOpenSettings: undefined }, +}; + +/** Collapsed icon rail: labels become tooltips. */ +export const Collapsed: Story = { + args: { collapsed: true }, + decorators: [ + (S) => ( +
+ +
+ ), + ], +}; diff --git a/frontend/editor/src/core/components/shared/navFooter/NavFooter.test.tsx b/frontend/editor/src/core/components/shared/navFooter/NavFooter.test.tsx new file mode 100644 index 0000000000..a32063e56f --- /dev/null +++ b/frontend/editor/src/core/components/shared/navFooter/NavFooter.test.tsx @@ -0,0 +1,58 @@ +import { describe, it, expect } from "vitest"; +import { cleanup, render } from "@testing-library/react"; +import { MantineProvider } from "@mantine/core"; +import { NavFooter } from "@app/components/shared/navFooter/NavFooter"; + +/** The footer's tooltips need Mantine's theme context. */ +function withProviders(ui: React.ReactNode) { + return {ui}; +} + +function renderFooter() { + const { container } = render( + withProviders( + {}} + credits={{ remaining: 247, total: 500 }} + otherApp={{ app: "processor", onOpen: () => {} }} + />, + ), + ); + return container.querySelector(".nav-footer") as HTMLElement; +} + +describe("NavFooter — enter animation", () => { + it("plays once per page session, not on every remount", () => { + // The rows are seeded from cache, so they're present at first paint. Every + // later mount — switching apps, remounting a view — would otherwise replay + // the fade on content that never changed, which reads as a twitch. + expect(renderFooter().dataset.animate).toBe("true"); + cleanup(); + expect(renderFooter().dataset.animate).toBeUndefined(); + cleanup(); + expect(renderFooter().dataset.animate).toBeUndefined(); + }); +}); + +describe("NavFooter — separators", () => { + it("never renders a divider beside a row that renders nothing", () => { + // Dividers are CSS between adjacent non-empty slots, so an extras element + // that returns null (the linked org's link-account CTA) can't leave a line. + const { container } = render( + withProviders( + {}} + credits={null} + otherApp={null} + accountExtras={<>{null}} + />, + ), + ); + const slots = container.querySelectorAll(".nav-footer__slot"); + const filled = [...slots].filter((s) => s.childElementCount > 0); + expect(filled).toHaveLength(1); + expect(container.querySelectorAll(".nav-footer__divider")).toHaveLength(0); + }); +}); diff --git a/frontend/editor/src/core/components/shared/navFooter/NavFooter.tsx b/frontend/editor/src/core/components/shared/navFooter/NavFooter.tsx new file mode 100644 index 0000000000..373c91bccf --- /dev/null +++ b/frontend/editor/src/core/components/shared/navFooter/NavFooter.tsx @@ -0,0 +1,213 @@ +import { useState, type ReactNode } from "react"; +import { useTranslation } from "react-i18next"; +import { Tooltip } from "@mantine/core"; +import ArrowForwardIcon from "@mui/icons-material/ArrowForward"; +import SettingsIcon from "@mui/icons-material/Settings"; +import { Avatar, NavSurface } from "@app/ui"; +import { BrandMark } from "@app/components/shared/BrandMark"; +import { type AppSwitchTarget } from "@app/components/shared/AppSwitch"; +import { + NavFooterCreditsRow, + type NavFooterCredits, +} from "@app/components/shared/navFooter/NavFooterCreditsRow"; +import "@app/components/shared/navFooter/NavFooter.css"; + +export interface NavFooterAppLink { + /** The app this footer is NOT in — the one the row opens. */ + app: AppSwitchTarget; + onOpen: () => void; +} + +export interface NavFooterProps { + /** Name shown next to the avatar, and the source of its initials fallback. */ + displayName: string; + /** Profile picture; initials are drawn when absent or the URL fails to load. */ + profilePictureUrl?: string | null; + /** Omit to render the account row as static text (no settings affordance). */ + onOpenSettings?: () => void; + /** Null/undefined hides the meter — builds with no wallet never show it. */ + credits?: NavFooterCredits | null; + /** Opens the plan surface from the credits row; omit to leave it inert. */ + onOpenPlan?: () => void; + /** Null/undefined hides the switch row — e.g. no access to the other app. */ + otherApp?: NavFooterAppLink | null; + /** Extra rows above the account row (the self-hosted link-account CTA). */ + accountExtras?: ReactNode; + /** Icon-rail state: labels collapse to tooltips. */ + collapsed?: boolean; + className?: string; +} + +/** + * Whether the enter animation has already played this page session. The rows + * are seeded from cache now, so they're present from first paint and every + * later mount — switching apps, remounting a view — would otherwise replay the + * animation on content that never changed, which reads as the UI twitching. + */ +let hasPlayedEnter = false; + +/** + * The bottom section every sidebar ends with, shared by the editor and the + * processor so both present the same rows. ONE surface, hairline-separated, in + * this order: + * + * 1. caller-contributed rows (the self-hosted link-account CTA) + * 2. free credits remaining + * 3. "Open " + * 4. the account row — avatar, name, settings + * + * Purely presentational: each app resolves its own identity, wallet and + * app-switch access and passes them in, so this file carries no build-specific + * gating. A row whose data is absent is dropped, and so is the separator that + * would have sat beside it. + */ +export function NavFooter({ + displayName, + profilePictureUrl, + onOpenSettings, + credits, + onOpenPlan, + otherApp, + accountExtras, + collapsed = false, + className, +}: NavFooterProps) { + const { t } = useTranslation(); + const [animate] = useState(() => { + if (hasPlayedEnter) return false; + hasPlayedEnter = true; + return true; + }); + + const settingsLabel = t("fileSidebar.openSettings", "Open settings"); + const accountLabel = onOpenSettings + ? `${displayName} - ${settingsLabel}` + : displayName; + + // One surface, hairline-separated rows. Each row gets a slot; the separators + // are drawn by CSS between adjacent NON-EMPTY slots (see NavFooter.css), so a + // row that renders nothing — the link-account CTA returns null once the org is + // linked, and an element is truthy even then — can't leave a line behind. + const rows: Array<{ key: string; node: ReactNode }> = []; + + if (accountExtras) rows.push({ key: "extras", node: accountExtras }); + + if (credits) { + rows.push({ + key: "credits", + node: ( + + ), + }); + } + + if (otherApp) { + rows.push({ + key: "switch", + node: ( + + + + ), + }); + } + + rows.push({ + key: "account", + node: ( + + + + ), + }); + + return ( + + {rows.map((row) => ( +
+ {row.node} +
+ ))} +
+ ); +} + +function openAppLabel( + app: AppSwitchTarget, + t: (key: string, fallback: string) => string, +): string { + return app === "editor" + ? t("navFooter.openEditor", "Open PDF Editor") + : t("navFooter.openProcessor", "Open PDF Processor"); +} diff --git a/frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.css b/frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.css new file mode 100644 index 0000000000..b37def6b2d --- /dev/null +++ b/frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.css @@ -0,0 +1,83 @@ +/* Free-credits meter inside the sidebar footer. The row base (padding, hover, + focus) comes from NavFooter.css; these rules are the meter itself. */ + +.nav-footer__credits { + flex-direction: column; + align-items: stretch; + gap: 0.375rem; + cursor: default; +} + +/* Inert by default, so it must not read as hoverable; the actionable variant + opts back into the shared row hover. */ +.nav-footer__credits:hover { + background: none; +} + +.nav-footer__credits--actionable { + cursor: pointer; +} +.nav-footer__credits--actionable:hover { + background-color: var(--c-hover); +} + +.nav-footer__credits-head { + display: flex; + align-items: center; + gap: 0.5rem; + font-size: 0.8125rem; +} + +.nav-footer__dot { + width: 0.4375rem; + height: 0.4375rem; + border-radius: 50%; + flex-shrink: 0; + background-color: var(--c-success); +} +.nav-footer__dot[data-tone="warning"] { + background-color: var(--c-warning); +} +.nav-footer__dot[data-tone="danger"] { + background-color: var(--c-danger); +} + +.nav-footer__credits-label { + flex: 1; + min-width: 0; + font-weight: 500; + color: var(--c-text); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.nav-footer__credits-count { + flex-shrink: 0; + color: var(--c-text-muted); + font-variant-numeric: tabular-nums; +} + +/* ---- Collapsed rail ---- */ + +/* Rotated so the fill starts at 12 o'clock and runs clockwise. */ +.nav-footer__credits-ring { + width: 1.25rem; + height: 1.25rem; + margin-inline: auto; + transform: rotate(-90deg); +} + +.nav-footer__credits-ring-track, +.nav-footer__credits-ring-fill { + fill: none; + stroke-width: 3; +} + +.nav-footer__credits-ring-track { + stroke: var(--c-surface-sunken); +} + +.nav-footer__credits-ring-fill { + stroke-linecap: round; +} diff --git a/frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.tsx b/frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.tsx new file mode 100644 index 0000000000..94e941d71c --- /dev/null +++ b/frontend/editor/src/core/components/shared/navFooter/NavFooterCreditsRow.tsx @@ -0,0 +1,158 @@ +import type { ReactNode } from "react"; +import { useTranslation } from "react-i18next"; +import { Tooltip } from "@mantine/core"; +import { ProgressBar } from "@app/ui"; +import "@app/components/shared/navFooter/NavFooterCreditsRow.css"; + +export interface NavFooterCredits { + /** Free credits still available to spend. */ + remaining: number; + /** Size of the free allowance — the "of N" denominator. */ + total: number; +} + +/** Remaining-credit bands, mirroring the usage meters' 80% / 100% thresholds. */ +function creditsTone(remaining: number, total: number): string { + if (remaining <= 0) return "danger"; + return total > 0 && remaining / total <= 0.2 ? "warning" : "success"; +} + +interface NavFooterCreditsRowProps { + credits: NavFooterCredits; + /** Icon rail: the figures drop and the bar alone carries the state. */ + collapsed: boolean; + /** Row label, passed in so the meter owns no copy of its own. */ + label: string; + /** Opens the plan surface. Omit to render the meter as inert text. */ + onOpen?: () => void; +} + +/** + * The free-credits meter as it appears in the sidebar footer: a state dot, the + * label, "X of Y" remaining, and a fill bar underneath. Figures are clamped + * here so a wallet that reports more remaining than the allowance (or negative) + * can't overflow the bar. + * + * Rendered as a {@code nav-footer__row}, so it inherits that row's metrics + * from NavFooter.css and only brings its own meter styling. + */ +export function NavFooterCreditsRow({ + credits, + collapsed, + label, + onOpen, +}: NavFooterCreditsRowProps) { + const { t } = useTranslation(); + + const total = Math.max(0, credits.total); + const remaining = Math.min(Math.max(0, credits.remaining), total); + const tone = creditsTone(remaining, total); + const count = t("navFooter.credits.count", "{{remaining}} of {{total}}", { + remaining: remaining.toLocaleString(), + total: total.toLocaleString(), + }); + + return ( + + + {collapsed ? ( + // The rail is one icon wide, so a full-width bar would read as a + // stray line; a ring carries the same fraction at icon size. + 0 ? remaining / total : 0} + tone={tone} + label={`${label}: ${count}`} + /> + ) : ( + <> +
+ + {label} + {count} +
+ 0 ? remaining / total : 0} + height={6} + color={`var(--c-${tone})`} + label={`${label}: ${count}`} + /> + + )} +
+
+ ); +} + +/** Icon-sized donut carrying the same remaining fraction as the expanded bar. */ +function CreditsRing({ + fraction, + tone, + label, +}: { + fraction: number; + tone: string; + label: string; +}) { + const RADIUS = 8; + const circumference = 2 * Math.PI * RADIUS; + const filled = Math.min(1, Math.max(0, fraction)) * circumference; + + return ( + + + + + ); +} + +/** + * The meter is a button only where there is a plan surface to open — otherwise + * it stays a plain div, so a build with nowhere to go doesn't advertise a + * click that does nothing. + */ +function Row({ + onOpen, + label, + children, +}: { + onOpen?: () => void; + label: string; + children: ReactNode; +}) { + const className = `nav-footer__row nav-footer__credits${ + onOpen ? " nav-footer__credits--actionable" : "" + }`; + if (!onOpen) return
{children}
; + return ( + + ); +} diff --git a/frontend/editor/src/core/hooks/useAccountIdentity.ts b/frontend/editor/src/core/hooks/useAccountIdentity.ts new file mode 100644 index 0000000000..026ac08dab --- /dev/null +++ b/frontend/editor/src/core/hooks/useAccountIdentity.ts @@ -0,0 +1,64 @@ +import { useEffect, useState } from "react"; +import { useTranslation } from "react-i18next"; +import { useAuth } from "@app/auth/UseSession"; +import { useProfilePictureUrl } from "@app/hooks/useProfilePictureUrl"; +import { useAppConfig } from "@app/contexts/AppConfigContext"; +import { accountService } from "@app/services/accountService"; + +export interface AccountIdentity { + /** Never empty — falls back to a generic "User" so a row is never blank. */ + displayName: string; + profilePictureUrl: string | null; + isAnonymous: boolean; +} + +/** + * The signed-in identity as the UI should draw it: one name and one picture, + * resolved the same way everywhere. Every surface that shows "who am I" (the + * editor and processor sidebar footers, the account settings page) reads this, + * so a user can't see one initial in the sidebar and a different one in + * settings. + * + * Resolution order for the name: the auth layer's own displayName (each layer + * derives it from its native user shape), then the proprietary REST endpoint, + * then a generic last resort. + */ +export function useAccountIdentity(): AccountIdentity { + const { t } = useTranslation(); + const { config } = useAppConfig(); + const { displayName: authDisplayName, isAnonymous } = useAuth(); + const profilePictureUrl = useProfilePictureUrl(); + const [accountUsername, setAccountUsername] = useState(null); + + useEffect(() => { + if (!config?.enableLogin) { + setAccountUsername(null); + return; + } + if (authDisplayName) { + // The auth context has a name; don't bother hitting the REST + // endpoint, but clear any stale cached value from a prior call. + setAccountUsername(null); + return; + } + accountService + .getAccountData() + .then((data) => { + // Always reflect the latest result - including clearing it on + // sign-out, when the endpoint returns no username (or 401s into + // the catch branch below). Without this, signing out would leave + // the old username on screen. + setAccountUsername(data?.username ?? null); + }) + .catch(() => { + setAccountUsername(null); + }); + }, [config?.enableLogin, authDisplayName]); + + return { + displayName: + authDisplayName ?? accountUsername ?? t("auth.displayName.user", "User"), + profilePictureUrl, + isAnonymous, + }; +} diff --git a/frontend/editor/src/core/hooks/useFreeCreditsSummary.ts b/frontend/editor/src/core/hooks/useFreeCreditsSummary.ts new file mode 100644 index 0000000000..52702a3f74 --- /dev/null +++ b/frontend/editor/src/core/hooks/useFreeCreditsSummary.ts @@ -0,0 +1,12 @@ +import { type NavFooterCredits } from "@app/components/shared/navFooter/NavFooterCreditsRow"; + +/** + * Free credits left on this team's allowance, for the sidebar footer meter. + * Null hides the meter entirely. + * + * Core has no wallet — self-hosted installs aren't metered — so there is + * nothing to show. Cloud builds override this with the live wallet figure. + */ +export function useFreeCreditsSummary(): NavFooterCredits | null { + return null; +} diff --git a/frontend/editor/src/core/hooks/useOpenPlan.ts b/frontend/editor/src/core/hooks/useOpenPlan.ts new file mode 100644 index 0000000000..da6fe2207e --- /dev/null +++ b/frontend/editor/src/core/hooks/useOpenPlan.ts @@ -0,0 +1,10 @@ +/** + * Opens the plan surface behind the sidebar footer's free-credits row, or null + * when this build has none (the row is then inert text rather than a button). + * + * Core ships no wallet and no plan section, so there is nothing to open. Builds + * that meter usage override this with their own surface. + */ +export function useOpenPlan(): (() => void) | null { + return null; +} diff --git a/frontend/editor/src/core/hooks/useOtherAppSwitch.ts b/frontend/editor/src/core/hooks/useOtherAppSwitch.ts new file mode 100644 index 0000000000..612589899b --- /dev/null +++ b/frontend/editor/src/core/hooks/useOtherAppSwitch.ts @@ -0,0 +1,12 @@ +import { type NavFooterAppLink } from "@app/components/shared/navFooter/NavFooter"; + +/** + * The sibling app this build can switch to (editor ⇄ processor), or null when + * there is none. The single gate behind both the brand switcher and the + * sidebar footer's "Open ..." row, so the two can never disagree about access. + * + * Core ships no processor, so there is nothing to switch to. + */ +export function useOtherAppSwitch(): NavFooterAppLink | null { + return null; +} diff --git a/frontend/editor/src/core/query/keys.ts b/frontend/editor/src/core/query/keys.ts index a7a68ea256..5354b56b63 100644 --- a/frontend/editor/src/core/query/keys.ts +++ b/frontend/editor/src/core/query/keys.ts @@ -6,5 +6,8 @@ export const qk = { ["editor", "endpointEnabled", endpoint] as const, footerInfo: () => ["editor", "footerInfo"] as const, groupEnabled: (group: string) => ["editor", "groupEnabled", group] as const, + /** Keyed on the asking identity: two users must never share one answer. */ + portalAccess: (userId: string | null) => + ["editor", "portalAccess", userId] as const, users: () => ["editor", "users"] as const, } as const; diff --git a/frontend/editor/src/core/services/navFooterCache.ts b/frontend/editor/src/core/services/navFooterCache.ts new file mode 100644 index 0000000000..cbde941c81 --- /dev/null +++ b/frontend/editor/src/core/services/navFooterCache.ts @@ -0,0 +1,73 @@ +/** + * Last-known sidebar-footer state, so the rows are correct at first paint + * instead of arriving a request later. + * + * The footer is mounted by both apps, and the editor and processor are separate + * React trees with separate query caches — so without this, every navigation + * between them (and every remount inside them) re-ran the fetches and the rows + * visibly popped in and shoved each other around. Persisting to storage rather + * than to an in-memory cache is what makes it survive that boundary, and a + * reload. + * + * Deliberately stale-then-revalidate: what's stored is only ever what the + * backend last said, every reader refetches immediately and overwrites, and + * nothing is gated on it — the processor enforces its own access server-side, + * and a stale credit figure is replaced within a second of the wallet landing. + */ +const CREDITS_KEY = "stirling.navFooter.credits"; +const OTHER_APP_KEY = "stirling.navFooter.otherApp"; + +/** Figures, or null for a team that sees no meter at all (a paying one). */ +export type CachedCredits = { remaining: number; total: number } | null; + +function read(key: string): string | null { + try { + return localStorage.getItem(key); + } catch { + // Private mode / storage disabled — behave as a first-ever load. + return null; + } +} + +function write(key: string, value: string): void { + try { + localStorage.setItem(key, value); + } catch { + // Nothing to do: the cache is an optimisation, never a correctness input. + } +} + +/** `undefined` when this browser has never seen an answer. */ +export function readCachedCredits(): CachedCredits | undefined { + const raw = read(CREDITS_KEY); + if (raw === null) return undefined; + if (raw === "none") return null; + try { + const parsed: unknown = JSON.parse(raw); + if ( + typeof parsed === "object" && + parsed !== null && + typeof (parsed as CachedCredits & object).remaining === "number" && + typeof (parsed as CachedCredits & object).total === "number" + ) { + return parsed as CachedCredits; + } + } catch { + // Corrupt entry — fall through and treat it as never-seen. + } + return undefined; +} + +export function writeCachedCredits(credits: CachedCredits): void { + write(CREDITS_KEY, credits === null ? "none" : JSON.stringify(credits)); +} + +/** `undefined` when this browser has never seen an answer. */ +export function readCachedOtherApp(): boolean | undefined { + const raw = read(OTHER_APP_KEY); + return raw === null ? undefined : raw === "true"; +} + +export function writeCachedOtherApp(canOpen: boolean): void { + write(OTHER_APP_KEY, String(canOpen)); +} diff --git a/frontend/editor/src/core/ui/Avatar.css b/frontend/editor/src/core/ui/Avatar.css index e2e8dc63bd..360c15c566 100644 --- a/frontend/editor/src/core/ui/Avatar.css +++ b/frontend/editor/src/core/ui/Avatar.css @@ -46,6 +46,12 @@ height: 2.5rem; font-size: 1rem; } +/* Account-settings hero disc. */ +.sui-avatar--xl { + width: 4.5rem; + height: 4.5rem; + font-size: 1.75rem; +} .sui-avatar__img { width: 100%; diff --git a/frontend/editor/src/core/ui/Avatar.tsx b/frontend/editor/src/core/ui/Avatar.tsx index c7cfac501b..42e7aca11f 100644 --- a/frontend/editor/src/core/ui/Avatar.tsx +++ b/frontend/editor/src/core/ui/Avatar.tsx @@ -1,6 +1,7 @@ +import { useEffect, useState } from "react"; import "@app/ui/Avatar.css"; -export type AvatarSize = "xs" | "sm" | "md" | "lg"; +export type AvatarSize = "xs" | "sm" | "md" | "lg" | "xl"; export type AvatarTone = | "blue" | "purple" @@ -23,10 +24,12 @@ export interface AvatarProps { className?: string; } -function initialsOf(name: string): string { +function avatarInitials(name: string): string { const parts = name.trim().split(/\s+/).filter(Boolean); if (parts.length === 0) return "?"; - if (parts.length === 1) return parts[0].slice(0, 2).toUpperCase(); + // Single word (a username or an email) reads as one letter — two letters of + // "admin" ("AD") looks like a different person's initials, not a truncation. + if (parts.length === 1) return parts[0].slice(0, 1).toUpperCase(); return (parts[0][0] + parts[parts.length - 1][0]).toUpperCase(); } @@ -43,6 +46,13 @@ export function Avatar({ ariaLabel, className, }: AvatarProps) { + // A picture URL that 404s (expired signed URL, deleted upload) must not leave + // an empty disc — fall back to the same initials the no-picture case shows, so + // every surface rendering this identity agrees on what it draws. + const [srcFailed, setSrcFailed] = useState(false); + useEffect(() => setSrcFailed(false), [src]); + const showImage = Boolean(src) && !srcFailed; + const classes = [ "sui-avatar", `sui-avatar--${size}`, @@ -53,11 +63,16 @@ export function Avatar({ .filter(Boolean) .join(" "); - const content = src ? ( - {ariaLabel + const content = showImage ? ( + {ariaLabel setSrcFailed(true)} + /> ) : ( - {initialsOf(name)} + {avatarInitials(name)} ); diff --git a/frontend/editor/src/portal-saas/hooks/useFreeCreditsSummary.ts b/frontend/editor/src/portal-saas/hooks/useFreeCreditsSummary.ts new file mode 100644 index 0000000000..8f7a16ca90 --- /dev/null +++ b/frontend/editor/src/portal-saas/hooks/useFreeCreditsSummary.ts @@ -0,0 +1,7 @@ +/** + * SaaS has no link concept — the signed-in account IS the SaaS account, and the + * editor's cloud wallet hook is already in this build's {@code @app/*} cascade. + * Delegating to it means the processor footer and the editor footer share one + * wallet fetch and can't disagree, so there is nothing portal-specific to do. + */ +export { useFreeCreditsSummary } from "@app/hooks/useFreeCreditsSummary"; diff --git a/frontend/editor/src/portal-saas/hooks/useOpenPlan.ts b/frontend/editor/src/portal-saas/hooks/useOpenPlan.ts new file mode 100644 index 0000000000..ce12e843b3 --- /dev/null +++ b/frontend/editor/src/portal-saas/hooks/useOpenPlan.ts @@ -0,0 +1,11 @@ +import { useCallback } from "react"; +import { useUI } from "@portal/contexts/UIContext"; + +/** + * SaaS processor: the settings modal it hosts carries the same Plan section the + * editor opens, so the footer's credits row lands both apps in one place. + */ +export function useOpenPlan(): (() => void) | null { + const { openSettings } = useUI(); + return useCallback(() => openSettings("plan"), [openSettings]); +} diff --git a/frontend/editor/src/portal/components/Sidebar.css b/frontend/editor/src/portal/components/Sidebar.css index dfb533f8ab..48b54b177d 100644 --- a/frontend/editor/src/portal/components/Sidebar.css +++ b/frontend/editor/src/portal/components/Sidebar.css @@ -123,8 +123,6 @@ } .portal-sidebar[data-collapsed] .portal-sidebar__footer { margin-inline: 0.375rem; - padding-inline: 0; - align-items: center; } .portal-sidebar__logo { @@ -179,10 +177,8 @@ gap: 0.125rem; } +/* The shared brings its own boxes, padding and gap; the sidebar + only positions it. */ .portal-sidebar__footer { margin: 0 0.625rem 0.75rem; - padding: 0.5rem 0.375rem; - display: flex; - flex-direction: column; - gap: 0.5rem; } diff --git a/frontend/editor/src/portal/components/Sidebar.tsx b/frontend/editor/src/portal/components/Sidebar.tsx index ebf91c636e..8ce7008d67 100644 --- a/frontend/editor/src/portal/components/Sidebar.tsx +++ b/frontend/editor/src/portal/components/Sidebar.tsx @@ -2,6 +2,10 @@ import { useMediaQuery } from "@mantine/hooks"; import { Tooltip } from "@mantine/core"; import { ActionIcon, NavItem, NavSurface } from "@app/ui"; import { BrandSwitcher } from "@app/components/shared/BrandSwitcher"; +import { NavFooter } from "@app/components/shared/navFooter/NavFooter"; +import { useAccountIdentity } from "@app/hooks/useAccountIdentity"; +import { useFreeCreditsSummary } from "@portal/hooks/useFreeCreditsSummary"; +import { useOpenPlan } from "@portal/hooks/useOpenPlan"; import { SidebarToggleIcon } from "@app/components/shared/SidebarToggleIcon"; import { useTranslation } from "react-i18next"; import { useNavigate } from "react-router-dom"; @@ -10,7 +14,7 @@ import { useUI } from "@portal/contexts/UIContext"; import { LinkAccountFooterItem } from "@portal/components/LinkAccountFooterItem"; import { EDITOR_URL, EDITOR_IS_SAME_APP } from "@portal/auth/editorUrl"; import { EDITOR_BASENAME } from "@app/routes/editorBasename"; -import { CloseIcon, SettingsIcon } from "@portal/components/icons"; +import { CloseIcon } from "@portal/components/icons"; import { GROUP_PROCESSOR, GROUP_PLATFORM, @@ -41,6 +45,9 @@ export function Sidebar() { const isMobile = useMediaQuery(MOBILE_QUERY, false, { getInitialValueInEffect: false, }); + const { displayName, profilePictureUrl } = useAccountIdentity(); + const credits = useFreeCreditsSummary(); + const openPlan = useOpenPlan(); // Collapse is a desktop-only affordance: on mobile the sidebar is an // off-canvas drawer, so the icon-rail state never applies there. @@ -146,15 +153,17 @@ export function Sidebar() { ))} - - - } - onClick={() => openSettings()} - /> - + } + collapsed={collapsed} + /> ); } diff --git a/frontend/editor/src/portal/components/billing/PrepaidCapacityCard.tsx b/frontend/editor/src/portal/components/billing/PrepaidCapacityCard.tsx index c7ab0e8704..6ea4d8d336 100644 --- a/frontend/editor/src/portal/components/billing/PrepaidCapacityCard.tsx +++ b/frontend/editor/src/portal/components/billing/PrepaidCapacityCard.tsx @@ -1,6 +1,6 @@ import { useTranslation } from "react-i18next"; import { Button, Card } from "@app/ui"; -import { formatPeriodDate, MeterBar, meterState } from "@app/billing"; +import { formatPeriodDate, MeterBar, remainingMeter } from "@app/billing"; import type { Wallet } from "@portal/api/billing"; /** @@ -10,8 +10,8 @@ import type { Wallet } from "@portal/api/billing"; * - No bundle → a slim "Get 12 months for the price of 10" offer nudge with a * "Review offer" CTA (the demo's commit-nudge card), shown only when a buyer * ({@code onBuy}, leader) is present. - * - Bundle held → the capacity meter (fills as the pool is drawn down, so it - * warns as capacity runs low) plus a "Top up" action for the leader. + * - Bundle held → the capacity meter (drains towards empty as the pool is drawn + * down, so it warns as capacity runs low) plus a "Top up" action for the leader. * * Prepaid is consumed before metered billing and sits outside the spend limit, so * it reads as its own dimension. Buying/topping up opens {@code BundleCheckoutModal} @@ -55,8 +55,7 @@ export function PrepaidCapacityCard({ const remaining = wallet.prepaidUnitsRemaining; const total = wallet.prepaidUnitsTotal; - const used = Math.max(0, total - remaining); - const { state, pct } = meterState(used, total); + const { state, pct } = remainingMeter(remaining, total); const stateLabel = state === "DEGRADED" ? t("portal.billing.prepaid.state.exhausted", "Used up") diff --git a/frontend/editor/src/portal/components/billing/WalletMeter.tsx b/frontend/editor/src/portal/components/billing/WalletMeter.tsx index c8be188390..9558960e89 100644 --- a/frontend/editor/src/portal/components/billing/WalletMeter.tsx +++ b/frontend/editor/src/portal/components/billing/WalletMeter.tsx @@ -1,7 +1,7 @@ import type { ReactNode } from "react"; import { useTranslation } from "react-i18next"; import { Card } from "@app/ui"; -import { formatMinor, MeterBar, meterState } from "@app/billing"; +import { formatMinor, MeterBar, remainingMeter } from "@app/billing"; import type { Wallet } from "@portal/api/billing"; import type { LocalUsage } from "@portal/api/link"; @@ -15,8 +15,10 @@ interface Props { } /** - * The free Processor-trial meter — "X / N free PDFs used" against the one-time - * grant. Uses the shared {@link MeterBar} (same `paygf-meter` structure as the + * The free Processor-trial meter — "X of N free PDFs left" against the one-time + * grant, with what has been used alongside as the status badge. The bar shows what + * is left, so it drains towards empty as the grant is spent. + * Uses the shared {@link MeterBar} (same `paygf-meter` structure as the * cloud plan page). The subscribed spend-vs-cap meter is a separate surface * ({@code SpendLimitCard}); this card is only the free face. * @@ -30,7 +32,7 @@ export function WalletMeter({ wallet, unsynced, action }: Props) { const pending = unsynced?.totalUnsyncedUnits ?? 0; const used = wallet.billableUsed + pending; const remaining = Math.max(0, wallet.freeRemaining - pending); - const { state, pct } = meterState(used, wallet.freeAllowance); + const { state, pct } = remainingMeter(remaining, wallet.freeAllowance); const rate = wallet.pricePerDocMinor != null && wallet.pricePerDocMinor > 0 ? wallet.pricePerDocMinor @@ -76,11 +78,14 @@ export function WalletMeter({ wallet, unsynced, action }: Props) {
diff --git a/frontend/editor/src/portal/hooks/useFreeCreditsSummary.test.tsx b/frontend/editor/src/portal/hooks/useFreeCreditsSummary.test.tsx new file mode 100644 index 0000000000..fb48250fed --- /dev/null +++ b/frontend/editor/src/portal/hooks/useFreeCreditsSummary.test.tsx @@ -0,0 +1,110 @@ +import { describe, expect, it, vi, beforeEach } from "vitest"; +import { cleanup, render, waitFor } from "@testing-library/react"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { LinkProvider, type LinkState } from "@portal/contexts/LinkContext"; +import { useFreeCreditsSummary } from "@portal/hooks/useFreeCreditsSummary"; + +const fetchWallet = vi.fn(); +vi.mock("@portal/api/billing", () => ({ + fetchWallet: () => fetchWallet(), +})); + +function Probe() { + const credits = useFreeCreditsSummary(); + return ( + + {credits ? `${credits.remaining}/${credits.total}` : "none"} + + ); +} + +function renderFor(initialState: LinkState) { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }); + return render( + + + + + , + ).getByTestId("credits"); +} + +describe("useFreeCreditsSummary (self-hosted) — wallet behind the link gate", () => { + beforeEach(() => { + // The figures persist across mounts now, so isolate the suite from itself. + localStorage.clear(); + fetchWallet.mockReset(); + fetchWallet.mockResolvedValue({ + status: "free", + freeRemaining: 247, + freeAllowance: 500, + }); + }); + + it("unlinked reads no wallet at all", async () => { + const el = renderFor("unlinked"); + await waitFor(() => expect(el.textContent).toBe("none")); + expect(fetchWallet).not.toHaveBeenCalled(); + }); + + it("linked surfaces the free grant", async () => { + const el = renderFor("linked-free"); + await waitFor(() => expect(el.textContent).toBe("247/500")); + }); + + it("hides the meter once the team subscribes", async () => { + // The grant is a lifetime pool that survives subscribing, so a paying team + // would otherwise sit on a spent meter forever. + fetchWallet.mockResolvedValue({ + status: "subscribed", + freeRemaining: 0, + freeAllowance: 500, + }); + const el = renderFor("linked-subscribed"); + // The row holds its space while the wallet loads, then drops once the + // answer says this team is paying. + await waitFor(() => expect(el.textContent).toBe("none")); + }); + + it("hides the meter when the wallet read fails", async () => { + fetchWallet.mockRejectedValue(new Error("saas unreachable")); + const el = renderFor("linked-subscribed"); + await waitFor(() => expect(el.textContent).toBe("none")); + }); + + it("ignores cached figures once the instance is unlinked", async () => { + // The cache survives an unlink and nothing rewrites it afterwards, so the + // linkage gate has to cover the seed too, not just the fetch. + const linked = renderFor("linked-free"); + await waitFor(() => expect(linked.textContent).toBe("247/500")); + cleanup(); + + fetchWallet.mockClear(); + const unlinked = renderFor("unlinked"); + expect(unlinked.textContent).toBe("none"); + expect(fetchWallet).not.toHaveBeenCalled(); + }); + + it("shows the last known figures while the wallet reloads", async () => { + // What stops the row popping in — and resizing the footer — every time the + // processor mounts. + const el = renderFor("linked-free"); + await waitFor(() => expect(el.textContent).toBe("247/500")); + cleanup(); + + let release: (v: unknown) => void = () => {}; + fetchWallet.mockReturnValue( + new Promise((resolve) => { + release = resolve; + }), + ); + const second = renderFor("linked-free"); + // Seeded before the refetch lands... + expect(second.textContent).toBe("247/500"); + release({ status: "free", freeRemaining: 12, freeAllowance: 500 }); + // ...then updated in place, without the row ever being absent. + await waitFor(() => expect(second.textContent).toBe("12/500")); + }); +}); diff --git a/frontend/editor/src/portal/hooks/useFreeCreditsSummary.ts b/frontend/editor/src/portal/hooks/useFreeCreditsSummary.ts new file mode 100644 index 0000000000..3fd814ebb1 --- /dev/null +++ b/frontend/editor/src/portal/hooks/useFreeCreditsSummary.ts @@ -0,0 +1,65 @@ +import { useEffect, useState } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { useLink } from "@portal/contexts/LinkContext"; +import { fetchWallet } from "@portal/api/billing"; +import { qk } from "@portal/queries/keys"; +import { + readCachedCredits, + writeCachedCredits, + type CachedCredits, +} from "@app/services/navFooterCache"; +import { type NavFooterCredits } from "@app/components/shared/navFooter/NavFooterCreditsRow"; + +/** + * Free credits left on this team's allowance, for the processor's sidebar + * footer meter. Null hides the meter. + * + * This is the portal's own seam rather than the editor's {@code + * @app/hooks/useFreeCreditsSummary}, because self-hosted resolves {@code @app/*} + * as proprietary → core: the cloud wallet hook isn't in that cascade, and the + * implementation can't move down into proprietary either, since core/desktop + * builds ship no portal and must never resolve {@code @portal}. Keeping it here + * means only builds that actually have a processor pull in the wallet read. + * + * Self-hosted reads the same {@code GET /api/v1/payg/wallet} the Usage page's + * trial meter renders — {@code apiClient.saas} with the admin's Supabase JWT, + * since the wallet lives in the cloud even when the instance doesn't. Gated on + * linkage: an unlinked instance has no wallet to read. + * + * Free teams only, matching the editor and the Plan page. The grant is a + * lifetime pool that survives subscribing, so a paying team would otherwise sit + * on a permanent "0 of 500" in red; their usage lives on Usage & Billing. + */ +export function useFreeCreditsSummary(): NavFooterCredits | null { + const { isLinked } = useLink(); + // Shared query key, so the footer rides the same cached snapshot as any other + // wallet reader rather than adding a fetch per mount. + const { data: wallet } = useQuery({ + queryKey: qk.wallet(isLinked), + queryFn: fetchWallet, + enabled: isLinked, + }); + // Shared with the editor's seam, so crossing between the two apps shows the + // figures the other one last saw rather than re-fetching into an empty row. + const [seed] = useState(readCachedCredits); + + const live: CachedCredits | undefined = !wallet + ? undefined + : wallet.status === "subscribed" + ? null + : { remaining: wallet.freeRemaining, total: wallet.freeAllowance }; + + useEffect(() => { + // Only once linked: an unlinked instance never asks, so it has no answer of + // its own and must not overwrite what the editor recorded. + if (isLinked && live !== undefined) writeCachedCredits(live); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [wallet, isLinked]); + + // Linkage gates the seed as well as the fetch. The cache outlives an unlink + // — nothing refetches or rewrites it once the instance stops asking — so + // without this an unlinked instance would keep showing the figures from when + // it was linked, indefinitely. + if (!isLinked) return null; + return (live !== undefined ? live : seed) ?? null; +} diff --git a/frontend/editor/src/portal/hooks/useOpenPlan.ts b/frontend/editor/src/portal/hooks/useOpenPlan.ts new file mode 100644 index 0000000000..57b30c91d0 --- /dev/null +++ b/frontend/editor/src/portal/hooks/useOpenPlan.ts @@ -0,0 +1,13 @@ +import { useCallback } from "react"; +import { useView } from "@portal/contexts/ViewContext"; + +/** + * Self-hosted processor: settings carries no Plan section (it is a cloud + * surface, and this build's registry has none), so the footer's credits row + * opens the portal's own Usage & Billing view instead — the same figures, on + * the surface this flavor actually owns. + */ +export function useOpenPlan(): (() => void) | null { + const { setActiveView } = useView(); + return useCallback(() => setActiveView("usage"), [setActiveView]); +} diff --git a/frontend/editor/src/portal/queries/keys.ts b/frontend/editor/src/portal/queries/keys.ts index 6c29430cb4..e1c46a59c0 100644 --- a/frontend/editor/src/portal/queries/keys.ts +++ b/frontend/editor/src/portal/queries/keys.ts @@ -20,6 +20,8 @@ export const qk = { // Keyed on linkage: an unlinked account has no deal to read, so linking must not // serve the unlinked (null) snapshot back from cache. procurement: (linked: boolean) => ["portal", "procurement", linked] as const, + // Same reasoning: an unlinked instance has no wallet in the cloud. + wallet: (linked: boolean) => ["portal", "wallet", linked] as const, // Tier-dependent documents: (tier: Tier) => ["portal", "documents", tier] as const, diff --git a/frontend/editor/src/proprietary/billing/format.ts b/frontend/editor/src/proprietary/billing/format.ts index f91f98af44..ff44e69882 100644 --- a/frontend/editor/src/proprietary/billing/format.ts +++ b/frontend/editor/src/proprietary/billing/format.ts @@ -293,6 +293,29 @@ export function computeBundleQuote( export type MeterState = "FULL" | "WARNED" | "DEGRADED"; +/** + * Meter for a balance that is spent DOWN — a free grant, a prepaid pool. The + * bar shows what is LEFT, so full reads as "plenty" and empty as "none", which + * is how the sidebar footer's credits row reads and the only direction that + * matches a figure quoting the remainder. + * + * The state bands still key on consumption, so the tone is unchanged: amber + * once 80% is gone, red once it's exhausted. Meters for money SPENT against a + * cap keep using {@link meterState} directly — there a full bar correctly means + * "at your ceiling". + */ +export function remainingMeter( + remaining: number, + total: number, +): { state: MeterState; pct: number } { + const { state } = meterState(Math.max(0, total - remaining), total); + const pct = + total > 0 + ? Math.min(100, Math.max(0, (Math.max(0, remaining) / total) * 100)) + : 0; + return { state, pct }; +} + /** Warn (≥80%) / degrade (≥100%) band for a usage meter; mirrors the BE thresholds. */ export function meterState( used: number, diff --git a/frontend/editor/src/proprietary/billing/index.ts b/frontend/editor/src/proprietary/billing/index.ts index 687adec541..2fe9dffd93 100644 --- a/frontend/editor/src/proprietary/billing/index.ts +++ b/frontend/editor/src/proprietary/billing/index.ts @@ -14,6 +14,7 @@ export { docCapForMoney, formatPeriodDate, meterState, + remainingMeter, PREPAID_MONTHS_GRANTED, PREPAID_MONTHS_PAID, PDFS_PER_USER_MONTH, diff --git a/frontend/editor/src/proprietary/components/shared/AppSwitcher.tsx b/frontend/editor/src/proprietary/components/shared/AppSwitcher.tsx index 2e02db3068..9ba0b6438d 100644 --- a/frontend/editor/src/proprietary/components/shared/AppSwitcher.tsx +++ b/frontend/editor/src/proprietary/components/shared/AppSwitcher.tsx @@ -1,15 +1,21 @@ -import { useNavigate } from "react-router-dom"; -import { useAuth } from "@app/auth/context"; import { Logo } from "@app/ui/Logo"; import { BrandSwitcher } from "@app/components/shared/BrandSwitcher"; import { type AppSwitcherProps } from "@core/components/shared/AppSwitcher"; -import { PORTAL_BASENAME } from "@app/routes/portalBasename"; +import { useOtherAppSwitch } from "@app/hooks/useOtherAppSwitch"; +/** + * Sidebar brand header for builds that ship the processor. When this user can + * open it, the Stirling logo doubles as the editor⇄processor switcher: the mark + * morphs into a chevron and opens the switch menu (the same BrandSwitcher the + * processor sidebar uses). Users without access get a plain logo. + * + * The access gate lives in {@link useOtherAppSwitch} so this header and the + * sidebar footer's "Open PDF Processor" row are driven by one answer. + */ export function AppSwitcher({ collapsed }: AppSwitcherProps) { - const { portalAccess } = useAuth(); - const navigate = useNavigate(); + const otherApp = useOtherAppSwitch(); - if (!portalAccess) { + if (!otherApp) { return ( navigate(PORTAL_BASENAME)} + onSwitch={otherApp.onOpen} collapsed={collapsed} /> ); diff --git a/frontend/editor/src/proprietary/hooks/useOtherAppSwitch.ts b/frontend/editor/src/proprietary/hooks/useOtherAppSwitch.ts new file mode 100644 index 0000000000..8bf07b5c2f --- /dev/null +++ b/frontend/editor/src/proprietary/hooks/useOtherAppSwitch.ts @@ -0,0 +1,15 @@ +import { useNavigate } from "react-router-dom"; +import { useAuth } from "@app/auth/context"; +import { PORTAL_BASENAME } from "@app/routes/portalBasename"; +import { type NavFooterAppLink } from "@app/components/shared/navFooter/NavFooter"; + +/** + * Self-hosted: the Spring session carries `portalAccess`, so the switch to the + * processor is offered exactly when that flag is set. + */ +export function useOtherAppSwitch(): NavFooterAppLink | null { + const { portalAccess } = useAuth(); + const navigate = useNavigate(); + if (!portalAccess) return null; + return { app: "processor", onOpen: () => navigate(PORTAL_BASENAME) }; +} diff --git a/frontend/editor/src/saas/components/shared/AppSwitcher.tsx b/frontend/editor/src/saas/components/shared/AppSwitcher.tsx deleted file mode 100644 index 364f094478..0000000000 --- a/frontend/editor/src/saas/components/shared/AppSwitcher.tsx +++ /dev/null @@ -1,41 +0,0 @@ -import { useNavigate } from "react-router-dom"; -import { Logo } from "@app/ui/Logo"; -import { BrandSwitcher } from "@app/components/shared/BrandSwitcher"; -import { type AppSwitcherProps } from "@core/components/shared/AppSwitcher"; -import { usePortalAccess } from "@app/hooks/usePortalAccess"; -import { PORTAL_BASENAME } from "@app/routes/portalBasename"; - -/** - * SaaS sidebar brand header. When the backend says this user can open the - * processor (`/api/v1/auth/me` → `portalAccess` — the exact signal the - * processor's own gate uses), the Stirling logo doubles as the - * editor⇄processor switcher: the mark morphs into a chevron and opens the - * switch menu (same BrandSwitcher the processor sidebar uses). Users without - * access get a plain logo. - * - * Deliberately NOT gated on the editor's Supabase auth context: that context - * never fetches /me, so it can't know about portal access (and its session - * state doesn't always mirror the backend login that actually grants it). - */ -export function AppSwitcher({ collapsed }: AppSwitcherProps) { - const portalAccess = usePortalAccess(); - const navigate = useNavigate(); - - if (!portalAccess) { - return ( - - ); - } - - return ( - navigate(PORTAL_BASENAME)} - collapsed={collapsed} - /> - ); -} diff --git a/frontend/editor/src/saas/components/shared/config/configSections/Overview.tsx b/frontend/editor/src/saas/components/shared/config/configSections/Overview.tsx index fac87148fe..3a28152425 100644 --- a/frontend/editor/src/saas/components/shared/config/configSections/Overview.tsx +++ b/frontend/editor/src/saas/components/shared/config/configSections/Overview.tsx @@ -1,7 +1,6 @@ import React, { useState } from "react"; import { Alert, - Avatar, Divider, Group, Image, @@ -11,10 +10,12 @@ import { TextInput, Modal, } from "@mantine/core"; +import { Avatar } from "@app/ui/Avatar"; import { Button as DSButton } from "@app/ui/Button"; import { FilePicker } from "@app/ui/FilePicker"; import { useTranslation } from "react-i18next"; import { useAuth } from "@app/auth/UseSession"; +import { useAccountIdentity } from "@app/hooks/useAccountIdentity"; import { isUserAnonymous, linkEmailIdentity, @@ -46,6 +47,8 @@ const Overview: React.FC = ({ onLogoutClick }) => { refreshProfilePicture, refreshProfilePictureMetadata, } = useAuth(); + // Same name + initials the sidebar footer draws, so the two discs agree. + const { displayName } = useAccountIdentity(); const PROFILE_BUCKET = "profile-pictures"; @@ -67,7 +70,6 @@ const Overview: React.FC = ({ onLogoutClick }) => { const provider = profilePictureMetadata?.provider; const profilePath = user ? `${user.id}/avatar` : null; - const profileInitial = user?.email?.trim()?.charAt(0)?.toUpperCase() || "U"; const handleProfileUpload = async (file: File | null) => { if (!file || !user || !profilePath) { @@ -410,12 +412,9 @@ const Overview: React.FC = ({ onLogoutClick }) => { - {profileInitial} - + name={displayName} + size="xl" + />
= ({ onLogoutClick }) => { - {profileInitial} - + name={displayName} + size="xl" + />
navigate(PORTAL_BASENAME) }; +} diff --git a/frontend/editor/src/saas/hooks/usePortalAccess.test.tsx b/frontend/editor/src/saas/hooks/usePortalAccess.test.tsx index a0e8ba618d..809138850a 100644 --- a/frontend/editor/src/saas/hooks/usePortalAccess.test.tsx +++ b/frontend/editor/src/saas/hooks/usePortalAccess.test.tsx @@ -1,5 +1,7 @@ import { describe, it, expect, vi, beforeEach } from "vitest"; -import { renderHook, waitFor } from "@testing-library/react"; +import { renderHook as baseRenderHook, waitFor } from "@testing-library/react"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import type { ReactNode } from "react"; const get = vi.fn(); let currentUserId: string | null = null; @@ -20,10 +22,28 @@ function meReturning(portalAccess: boolean) { return { data: { user: { portalAccess } } }; } +// A fresh client per render, so one test's cached answer can't satisfy the +// next — each case exercises a cold cache unless it deliberately shares one. +let client: QueryClient; + +function renderHook(cb: () => T) { + return baseRenderHook(cb, { + wrapper: ({ children }: { children: ReactNode }) => ( + {children} + ), + }); +} + describe("usePortalAccess", () => { beforeEach(() => { + // The hook now remembers the last answer across mounts, so without this a + // prior test's result seeds the next one. + localStorage.clear(); get.mockReset(); currentUserId = null; + client = new QueryClient({ + defaultOptions: { queries: { retry: false, gcTime: 0, staleTime: 0 } }, + }); }); it("reports the backend's answer for the signed-in user", async () => { @@ -82,12 +102,31 @@ describe("usePortalAccess", () => { expect(first.result.current).toBe(false); first.unmount(); - // The failure isn't sticky. + // The failure isn't sticky — a cold cache asks again. + client.clear(); get.mockResolvedValue(meReturning(true)); const second = renderHook(() => usePortalAccess()); await waitFor(() => expect(second.result.current).toBe(true)); }); + it("shows the last known answer at first paint, then revalidates", async () => { + // What stops the switcher and the footer's "Open ..." row popping in a + // request late on every mount. + currentUserId = "admin-1"; + get.mockResolvedValue(meReturning(true)); + const first = renderHook(() => usePortalAccess()); + await waitFor(() => expect(first.result.current).toBe(true)); + first.unmount(); + + client.clear(); + get.mockResolvedValue(meReturning(false)); + const second = renderHook(() => usePortalAccess()); + // Seeded from the remembered answer before the request lands... + expect(second.result.current).toBe(true); + // ...and corrected once the backend disagrees. + await waitFor(() => expect(second.result.current).toBe(false)); + }); + it("ignores a response that lands after unmount", async () => { currentUserId = "admin-1"; let resolveMe: (v: unknown) => void = () => {}; diff --git a/frontend/editor/src/saas/hooks/usePortalAccess.ts b/frontend/editor/src/saas/hooks/usePortalAccess.ts index 442061cbe1..6e91f0864c 100644 --- a/frontend/editor/src/saas/hooks/usePortalAccess.ts +++ b/frontend/editor/src/saas/hooks/usePortalAccess.ts @@ -1,52 +1,64 @@ import { useEffect, useState } from "react"; +import { useQuery } from "@tanstack/react-query"; import apiClient from "@app/services/apiClient"; import { useAuth } from "@app/auth/UseSession"; +import { + readCachedOtherApp, + writeCachedOtherApp, +} from "@app/services/navFooterCache"; +import { qk } from "@app/query/keys"; + +async function fetchPortalAccess(): Promise { + const res = await apiClient.get<{ user?: { portalAccess?: boolean } }>( + "/api/v1/auth/me", + ); + return res.data.user?.portalAccess === true; +} /** * Whether the current user can open the processor (admin portal), straight * from the backend (`/api/v1/auth/me` → `portalAccess`) — the same signal the * processor's own SaasPortalGate uses. Components that must mirror processor - * access (e.g. the sidebar's editor⇄processor switcher) ask here. + * access (the sidebar's editor⇄processor switcher and its footer row) ask here. * * The editor's Supabase auth context can't *answer* this — it never fetches - * /me — so it is used only to identify who is asking. Keying the effect on - * that identity is what keeps the answer per-user: the SPA can swap users + * /me — so it is used only to identify who is asking. That identity is the + * cache key, which is what keeps the answer per-user: the SPA can swap users * without a reload (Supabase fires SIGNED_OUT/SIGNED_IN in place; only the - * settings Logout button hard-navigates), so any answer held beyond the - * current identity would leak to whoever signs in next. + * settings Logout button hard-navigates), and a keyed cache addresses each + * identity separately rather than holding one answer that would have to be + * invalidated on the swap — the bug class this hook once had. * - * Deliberately unmemoised beyond the mount: the one consumer (the sidebar - * switcher) mounts once, so a cross-mount cache would only add user-scoped - * state that has to be invalidated on identity change — the bug class this - * hook already had once. Guests skip the request entirely. + * Cached through the app query client, so leaving the editor for the processor + * and coming back resolves from cache: the switcher is there on first paint + * instead of appearing a request later. Guests skip the request entirely. */ export function usePortalAccess(): boolean { const { user } = useAuth(); const userId = user?.id ?? null; - const [access, setAccess] = useState(false); + // The query cache is per-tree and per-load, so it can't help a cold start or + // the hop into the processor, which mounts its own client. Seed from the last + // answer this browser saw so the switcher and the footer's "Open ..." row are + // there at first paint. Marked ancient so it still revalidates immediately. + const [seed] = useState(readCachedOtherApp); + + const { data, isSuccess } = useQuery({ + queryKey: qk.portalAccess(userId), + queryFn: fetchPortalAccess, + // Signed out: nothing to ask, and any previous answer is void. + enabled: userId !== null, + // Backend unreachable or guest (401) means no access now; a later refetch + // asks again rather than trusting the failure. + retry: false, + initialData: seed, + initialDataUpdatedAt: 0, + }); useEffect(() => { - // Signed out: nothing to ask, and any previous answer is void. - if (userId === null) { - setAccess(false); - return; - } + // Only a real answer is recorded — a failed probe is not one, so the next + // mount trusts the last backend response rather than a network blip. + if (isSuccess && data !== undefined) writeCachedOtherApp(data); + }, [isSuccess, data]); - let cancelled = false; - apiClient - .get<{ user?: { portalAccess?: boolean } }>("/api/v1/auth/me") - .then((res) => { - if (!cancelled) setAccess(res.data.user?.portalAccess === true); - }) - .catch(() => { - // Backend unreachable or guest (401): no access now; a remount or - // identity change asks again rather than trusting a failure. - if (!cancelled) setAccess(false); - }); - return () => { - cancelled = true; - }; - }, [userId]); - - return access; + return data === true; } diff --git a/frontend/editor/src/saas/hooks/useWallet.poll.test.tsx b/frontend/editor/src/saas/hooks/useWallet.poll.test.tsx new file mode 100644 index 0000000000..b72eac67f8 --- /dev/null +++ b/frontend/editor/src/saas/hooks/useWallet.poll.test.tsx @@ -0,0 +1,158 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { renderHook, act, waitFor } from "@testing-library/react"; +import { expectConsole } from "@app/tests/failOnConsole"; + +const get = vi.fn(); +vi.mock("@app/services/apiClient", () => ({ + default: { get: (...args: unknown[]) => get(...args) }, +})); +vi.mock("@app/hooks/walletDevPreview", () => ({ + getWalletDevPreview: () => null, +})); +vi.mock("@app/services/billing", () => ({ createPortalSession: vi.fn() })); +vi.mock("@app/platform/openExternal", () => ({ openExternal: vi.fn() })); + +const { useWallet } = await import("@app/hooks/useWallet"); + +/** Full enough for the hook's deep-compare, which reads every field. */ +function walletWith(freeRemaining: number) { + return { + data: { + teamId: 1, + status: "free", + role: "leader", + billingPeriodStart: "2026-08-01", + billingPeriodEnd: "2026-08-31", + billableUsed: 500 - freeRemaining, + billableLimit: 500, + freeAllowance: 500, + freeRemaining, + pricePerDocMinor: 2, + bundleRatePerCreditMinor: null, + currency: "usd", + estimatedBillMinor: 0, + capUsd: null, + noCap: false, + stripeSubscriptionId: null, + spendUnitsThisPeriod: 0, + docsProcessedThisPeriod: 0, + uniquePdfsThisPeriod: 0, + sizeMultiplierPdfsThisPeriod: 0, + billingMode: "metered", + prepaidUnitsRemaining: 0, + prepaidUnitsTotal: 0, + prepaidExpiresAt: null, + recent: [], + members: [], + categoryBreakdown: { api: 0, ai: 0, automation: 0 }, + categoryDocs: { api: 0, ai: 0, automation: 0 }, + }, + }; +} + +describe("useWallet — keeping the figures fresh", () => { + beforeEach(() => { + vi.useFakeTimers({ shouldAdvanceTime: true }); + get.mockReset(); + get.mockResolvedValue(walletWith(500)); + }); + afterEach(() => vi.useRealTimers()); + + it("re-reads the wallet on the poll interval", async () => { + const { result } = renderHook(() => useWallet()); + await waitFor(() => expect(result.current.wallet).not.toBeNull()); + expect(get).toHaveBeenCalledTimes(1); + + get.mockResolvedValue(walletWith(480)); + await act(async () => { + vi.advanceTimersByTime(30_000); + }); + + await waitFor(() => expect(result.current.wallet?.freeRemaining).toBe(480)); + }); + + it("polls silently, so consumers gating on loading/error don't flicker", async () => { + const { result } = renderHook(() => useWallet()); + await waitFor(() => expect(result.current.wallet).not.toBeNull()); + + // A poll that fails must leave the last good snapshot, and must not raise + // `error` — Plan swaps a working page for an alert on that. + get.mockRejectedValue(new Error("network blip")); + await act(async () => { + vi.advanceTimersByTime(30_000); + }); + + expect(result.current.loading).toBe(false); + expect(result.current.error).toBeNull(); + expect(result.current.wallet?.freeRemaining).toBe(500); + }); + + it("settles loading when a silent poll supersedes an in-flight visible load", async () => { + // The mount load raises `loading`; a poll firing before it lands cancels it. + // If clearing the flag were the silent load's to skip, both would decline + // and `loading` would stay true forever — which permanently suppresses the + // limit modals, since they do `if (loading || !wallet) return null`. + const visibility = vi.spyOn(document, "visibilityState", "get"); + visibility.mockReturnValue("visible"); + + let landMount: (v: unknown) => void = () => {}; + get.mockReturnValueOnce( + new Promise((resolve) => { + landMount = resolve; + }), + ); + const { result } = renderHook(() => useWallet()); + expect(result.current.loading).toBe(true); + + get.mockResolvedValue(walletWith(470)); + await act(async () => { + document.dispatchEvent(new Event("visibilitychange")); + }); + await act(async () => { + landMount(walletWith(500)); + }); + + await waitFor(() => expect(result.current.wallet?.freeRemaining).toBe(470)); + expect(result.current.loading).toBe(false); + visibility.mockRestore(); + }); + + it("clears a stale error once a silent poll succeeds", async () => { + // The visible mount load failing is meant to be logged; only the silent + // retries stay quiet. + expectConsole.warn(/\[useWallet\] fetch failed/); + get.mockRejectedValueOnce(new Error("network blip")); + const { result } = renderHook(() => useWallet()); + await waitFor(() => expect(result.current.error).not.toBeNull()); + + get.mockResolvedValue(walletWith(500)); + await act(async () => { + vi.advanceTimersByTime(30_000); + }); + + await waitFor(() => expect(result.current.error).toBeNull()); + expect(result.current.wallet?.freeRemaining).toBe(500); + }); + + it("stops polling while the tab is hidden and re-reads on return", async () => { + const visibility = vi.spyOn(document, "visibilityState", "get"); + visibility.mockReturnValue("visible"); + const { result } = renderHook(() => useWallet()); + await waitFor(() => expect(result.current.wallet).not.toBeNull()); + const afterMount = get.mock.calls.length; + + visibility.mockReturnValue("hidden"); + await act(async () => { + document.dispatchEvent(new Event("visibilitychange")); + vi.advanceTimersByTime(120_000); + }); + expect(get).toHaveBeenCalledTimes(afterMount); + + visibility.mockReturnValue("visible"); + await act(async () => { + document.dispatchEvent(new Event("visibilitychange")); + }); + await waitFor(() => expect(get.mock.calls.length).toBe(afterMount + 1)); + visibility.mockRestore(); + }); +}); From ec3de16c0862c01190bf45896bae87e9f0e10ca7 Mon Sep 17 00:00:00 2001 From: Ludy Date: Tue, 18 Aug 2026 20:25:55 +0200 Subject: [PATCH 24/59] ci: centralize Gradle caching across GitHub Actions workflows (#7546) ## Summary This pull request restructures Gradle dependency caching across the GitHub Actions workflows. The central `gradle-cache-prime` job is responsible for preparing the shared backend Gradle cache. Reusable workflows restore that shared cache without writing to the same key, while independently triggered workflows use isolated cache namespaces. ## What changed ### Shared Gradle cache - Added a stable `gradle-v1-` cache namespace for the shared backend cache. - The cache key includes the runner OS, runner architecture, JDK version, and the relevant Gradle configuration files. - The cache key is calculated before Gradle runs and reused for the later save step. - The prime job performs a lookup first and resolves backend dependencies only when the exact cache is missing. - This prevents Gradle or Spotless changes during the prime step from producing a different save key from the key used by downstream jobs. ### Reusable workflows - Backend, OpenAPI, license, Docker, E2E, and migration workflows restore the shared cache instead of writing to the shared key. - The backend build matrix includes `matrix.jdk-version` in its cache key. - Enterprise, Tauri, and generated-model workflows support the `use_shared_cache` boolean input. - When `use_shared_cache` is enabled, those workflows restore the shared cache. - When it is disabled, they use workflow-specific cache namespaces. ### Independent workflows Independent workflows now use separate cache prefixes, including: - `gradle-license-report-v1-` - `gradle-swagger-v1-` - `gradle-push-docker-v1-` - `gradle-tauri-releases-v1-` - `gradle-deploy-pr-v1-` - `gradle-playwright-e2e-v1-` - `gradle-generated-models-v1-` This prevents them from creating or affecting the shared backend cache before the prime job. ### Build and E2E flow - Removed the `-PnoSpotless` option from the central Gradle dependency-resolution command. - Removed the separate Gradle dependency prime/retry logic from the live E2E workflow. - Connected the Tauri build and generated-models check to the central cache-prime job. ## Motivation Previously, multiple workflows could use and save the same Gradle cache key independently. The first workflow to save the cache could therefore determine its contents, even if it had resolved a different or incomplete set of dependencies. The cache key was also evaluated after some Gradle tasks had run. If Gradle or Spotless modified a file covered by `hashFiles(...)`, the save key could differ from the restore key used by downstream jobs. This change gives the shared cache a single owner, isolates workflow-specific caches, and makes cache usage deterministic across the CI pipeline. ## Expected result - `gradle-cache-prime` is the single writer for the shared backend Gradle cache. - Downstream jobs restore the same cache without competing cache writes. - Independently triggered workflows remain isolated through their own cache namespaces. - Changes to the monitored Gradle configuration files produce a new cache key. - The normal Gradle/Spotless path is included when the shared cache is populated. ## Validation - Compared the cache key expressions and `hashFiles(...)` inputs across the affected workflows. - Verified that the central restore and save steps use the same precomputed key. - CI should confirm that the prime job populates the shared cache and downstream workflows only restore it. ## Checklist - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have performed a self-review of my changes - [ ] I have run the relevant CI checks - [ ] I have tested the workflow changes --- .../workflows/PR-Demo-Comment-with-react.yml | 19 +++--- .github/workflows/backend-build.yml | 21 +++---- .github/workflows/build-enterprise.yml | 33 ++++++++--- .github/workflows/build.yml | 59 ++++++++++++++----- .github/workflows/check-generated-models.yml | 30 +++++++--- .github/workflows/check-licence.yml | 19 +++--- .github/workflows/check-openapi.yml | 19 +++--- .github/workflows/coverage-aggregate.yml | 19 +++--- .github/workflows/db-migration-test.yml | 19 +++--- .github/workflows/docker-compose-tests.yml | 19 +++--- .github/workflows/e2e-live.yml | 38 ++++-------- .../frontend-backend-licenses-update.yml | 19 +++--- .github/workflows/multiOSReleases.yml | 57 ++++++++---------- .github/workflows/push-docker.yml | 19 +++--- .github/workflows/swagger.yml | 19 +++--- .github/workflows/tauri-build.yml | 33 +++++++---- .github/workflows/test-build-docker.yml | 19 +++--- 17 files changed, 235 insertions(+), 226 deletions(-) diff --git a/.github/workflows/PR-Demo-Comment-with-react.yml b/.github/workflows/PR-Demo-Comment-with-react.yml index fe3f28a637..410aa82dc9 100644 --- a/.github/workflows/PR-Demo-Comment-with-react.yml +++ b/.github/workflows/PR-Demo-Comment-with-react.yml @@ -191,22 +191,19 @@ jobs: # untrusted tree gets built below - never leave credentials in .git/config persist-credentials: false - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - - name: Cache Gradle User Home + - name: Cache Gradle uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-deploy-pr-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 diff --git a/.github/workflows/backend-build.yml b/.github/workflows/backend-build.yml index 54bd4cb907..6623940bce 100644 --- a/.github/workflows/backend-build.yml +++ b/.github/workflows/backend-build.yml @@ -35,23 +35,20 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-${{ matrix.jdk-version }}-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK ${{ matrix.jdk-version }} uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: ${{ matrix.jdk-version }} distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-${{ matrix.jdk-version }}-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-${{ matrix.jdk-version }}- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 - name: Check Java formatting (Spotless) @@ -156,7 +153,7 @@ jobs: STIRLING_FLAVOR: ${{ matrix.flavor }} # Configure the Gradle daemon explicitly; GRADLE_OPTS alone only # configures the Gradle client JVM. - GRADLE_OPTS: '-Dorg.gradle.jvmargs=-Xmx4g -XX:+UseG1GC' + GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx4g -XX:+UseG1GC" - name: Check Test Reports Exist if: always() diff --git a/.github/workflows/build-enterprise.yml b/.github/workflows/build-enterprise.yml index 0604f7176f..b4a8373ccc 100644 --- a/.github/workflows/build-enterprise.yml +++ b/.github/workflows/build-enterprise.yml @@ -15,6 +15,11 @@ name: Enterprise E2E (Playwright) on: workflow_call: + inputs: + use_shared_cache: + required: false + type: boolean + default: false push: branches: ["main"] schedule: @@ -56,21 +61,31 @@ jobs: egress-policy: audit - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + + - name: Restore cache Gradle User Home + if: inputs.use_shared_cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: - java-version: "25" - distribution: "temurin" - - name: Cache Gradle User Home + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Restore cache Gradle + if: inputs.use_shared_cache == false uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-playwright-e2e-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 2f50249099..566262d24f 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -73,29 +73,48 @@ jobs: egress-policy: audit - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + + - name: Calculate Gradle cache key + id: gradle-cache-key + shell: bash + run: | + echo "key=gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }}" >> "$GITHUB_OUTPUT" + + - name: Cache Gradle (lookup-only) + id: cache-gradle-restore + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: ${{ steps.gradle-cache-key.outputs.key }} + lookup-only: true + + - name: Set up JDK 25 + if: steps.cache-gradle-restore.outputs.cache-hit != 'true' + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" + - name: Resolve backend dependencies - run: ./gradlew :stirling-pdf:classes -PnoSpotless --no-daemon + if: steps.cache-gradle-restore.outputs.cache-hit != 'true' + run: ./gradlew :stirling-pdf:classes --no-daemon env: STIRLING_FLAVOR: saas MAVEN_USER: ${{ secrets.MAVEN_USER }} MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }} MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }} + - name: Save cache Gradle User Home + if: steps.cache-gradle-restore.outputs.cache-hit != 'true' + uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: ${{ steps.gradle-cache-key.outputs.key }} + build: needs: [files-changed, gradle-cache-prime] permissions: @@ -170,6 +189,8 @@ jobs: contents: read uses: ./.github/workflows/build-enterprise.yml secrets: inherit + with: + use_shared_cache: true check-licence: if: needs.files-changed.outputs.build == 'true' @@ -193,7 +214,14 @@ jobs: test-build-docker-images: if: github.event_name == 'pull_request' && needs.files-changed.outputs.project == 'true' - needs: [files-changed, build, check-generateOpenApiDocs, check-licence, gradle-cache-prime] + needs: + [ + files-changed, + build, + check-generateOpenApiDocs, + check-licence, + gradle-cache-prime, + ] permissions: contents: read packages: read @@ -205,7 +233,7 @@ jobs: tauri-build: if: needs.files-changed.outputs.tauri == 'true' - needs: [files-changed] + needs: [files-changed, gradle-cache-prime] permissions: contents: read pull-requests: write @@ -219,6 +247,7 @@ jobs: with: platform: windows-macos sign: true + use_shared_cache: true ai-engine: if: needs.files-changed.outputs.engine == 'true' @@ -242,6 +271,8 @@ jobs: pull-requests: write uses: ./.github/workflows/check-generated-models.yml secrets: inherit + with: + use_shared_cache: true pre-commit: needs: [files-changed] diff --git a/.github/workflows/check-generated-models.yml b/.github/workflows/check-generated-models.yml index 39c6467889..fafffcc241 100644 --- a/.github/workflows/check-generated-models.yml +++ b/.github/workflows/check-generated-models.yml @@ -9,6 +9,11 @@ name: Check generated models # post-merge safety net. on: workflow_call: + inputs: + use_shared_cache: + required: false + type: boolean + default: false push: branches: [main] @@ -39,22 +44,29 @@ jobs: engine/uv.lock cache-suffix: generated-models - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + - name: Restore cache Gradle User Home + if: inputs.use_shared_cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: - java-version: "25" - distribution: "temurin" + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - - name: Cache Gradle User Home + - name: Restore cache Gradle + if: inputs.use_shared_cache == false uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-generated-models-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Set up Node uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 diff --git a/.github/workflows/check-licence.yml b/.github/workflows/check-licence.yml index 2eec970b8f..17c64d5c64 100644 --- a/.github/workflows/check-licence.yml +++ b/.github/workflows/check-licence.yml @@ -21,23 +21,20 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: "25" distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 - name: Check licenses for compatibility diff --git a/.github/workflows/check-openapi.yml b/.github/workflows/check-openapi.yml index f224ce18cf..ed83447335 100644 --- a/.github/workflows/check-openapi.yml +++ b/.github/workflows/check-openapi.yml @@ -22,23 +22,20 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: "25" distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 - name: Generate OpenAPI documentation diff --git a/.github/workflows/coverage-aggregate.yml b/.github/workflows/coverage-aggregate.yml index a97b579f15..61ef8793c4 100644 --- a/.github/workflows/coverage-aggregate.yml +++ b/.github/workflows/coverage-aggregate.yml @@ -40,23 +40,20 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: "25" distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Install uv uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 with: diff --git a/.github/workflows/db-migration-test.yml b/.github/workflows/db-migration-test.yml index d6a61b45c4..ccb46d3988 100644 --- a/.github/workflows/db-migration-test.yml +++ b/.github/workflows/db-migration-test.yml @@ -25,23 +25,20 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: 25 distribution: temurin - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - # Keep the normal formatting path here so this smoke test exercises the # same Gradle configuration as the backend build. - name: Build Stirling-PDF JAR diff --git a/.github/workflows/docker-compose-tests.yml b/.github/workflows/docker-compose-tests.yml index 9d5911404f..039c73e5db 100644 --- a/.github/workflows/docker-compose-tests.yml +++ b/.github/workflows/docker-compose-tests.yml @@ -33,23 +33,20 @@ jobs: - name: Checkout Repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: "25" distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - # When the PR changes the base image, test.sh builds it locally # (stirling-pdf-base:local) into the daemon image store. A buildx # container builder can't see that store, so skip it here and let diff --git a/.github/workflows/e2e-live.yml b/.github/workflows/e2e-live.yml index 7bc95df05e..43d66dd1cf 100644 --- a/.github/workflows/e2e-live.yml +++ b/.github/workflows/e2e-live.yml @@ -21,39 +21,21 @@ jobs: egress-policy: audit - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: "25" distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - # Gradle does not retry 429s, and a cold cache resolving the buildscript - # classpath is exactly where Maven Central rate-limits us. Retry it here, - # where a failure is cheap, instead of inside the backgrounded bootRun. - - name: Prime Gradle dependencies - env: - MAVEN_USER: ${{ secrets.MAVEN_USER }} - MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }} - MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }} - run: | - for attempt in 1 2 3; do - if ./gradlew --quiet -PnoSpotless :stirling-pdf:classes; then - exit 0 - fi - echo "::warning::Gradle dependency resolution failed (attempt $attempt of 3)" - sleep $((attempt * 30)) - done - echo "::error::Gradle could not resolve dependencies after 3 attempts" - exit 1 + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: diff --git a/.github/workflows/frontend-backend-licenses-update.yml b/.github/workflows/frontend-backend-licenses-update.yml index 458766660f..7fd0136718 100644 --- a/.github/workflows/frontend-backend-licenses-update.yml +++ b/.github/workflows/frontend-backend-licenses-update.yml @@ -348,22 +348,19 @@ jobs: app-id: ${{ secrets.GH_APP_ID }} private-key: ${{ secrets.GH_APP_PRIVATE_KEY }} - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - - name: Cache Gradle User Home + - name: Cache Gradle uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-license-report-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 diff --git a/.github/workflows/multiOSReleases.yml b/.github/workflows/multiOSReleases.yml index d9477f722f..0f0b2d3585 100644 --- a/.github/workflows/multiOSReleases.yml +++ b/.github/workflows/multiOSReleases.yml @@ -52,22 +52,19 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - - name: Cache Gradle User Home + - name: Cache Gradle uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-tauri-releases-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 @@ -145,22 +142,19 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - - name: Cache Gradle User Home + - name: Cache Gradle uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-tauri-releases-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Setup Node.js if: matrix.variant.build_frontend == true @@ -238,6 +232,14 @@ jobs: toolchain: stable targets: ${{ matrix.platform == 'macos-15' && 'aarch64-apple-darwin,x86_64-apple-darwin' || '' }} + - name: Cache Gradle + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-tauri-releases-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + # x86_64 JDK is set up first so the aarch64 step below can leave its # JAVA_HOME as the active one. The macOS universal JRE build needs # jmods from both arches; the x64 path is captured into the env @@ -261,17 +263,6 @@ jobs: java-version: "25" distribution: ${{ matrix.platform == 'windows-11-arm' && 'microsoft' || 'temurin' }} - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 diff --git a/.github/workflows/push-docker.yml b/.github/workflows/push-docker.yml index ec9d14822c..ea379cf7c6 100644 --- a/.github/workflows/push-docker.yml +++ b/.github/workflows/push-docker.yml @@ -58,22 +58,19 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - - name: Cache Gradle User Home + - name: Cache Gradle uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-push-docker-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Set up Docker Buildx id: buildx diff --git a/.github/workflows/swagger.yml b/.github/workflows/swagger.yml index 115de87d4e..1bfc94be5b 100644 --- a/.github/workflows/swagger.yml +++ b/.github/workflows/swagger.yml @@ -36,22 +36,19 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 - - name: Set up JDK 25 - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 - with: - java-version: "25" - distribution: "temurin" - - - name: Cache Gradle User Home + - name: Cache Gradle uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | ~/.gradle/caches ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- + key: gradle-swagger-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Set up JDK 25 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + java-version: "25" + distribution: "temurin" - name: Generate Swagger documentation run: ./gradlew :stirling-pdf:generateOpenApiDocs diff --git a/.github/workflows/tauri-build.yml b/.github/workflows/tauri-build.yml index ddf1104bac..e3f3122773 100644 --- a/.github/workflows/tauri-build.yml +++ b/.github/workflows/tauri-build.yml @@ -26,6 +26,10 @@ on: required: false type: boolean default: false + use_shared_cache: + required: false + type: boolean + default: false workflow_dispatch: inputs: platform: @@ -168,6 +172,24 @@ jobs: # Save the dependency cache even if a later step fails cache-on-failure: true + - name: Restore cache Gradle User Home + if: inputs.use_shared_cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + + - name: Restore cache Gradle + if: inputs.use_shared_cache == false + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-tauri-build-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up x86_64 JDK 25 (macOS universal JRE) if: matrix.platform == 'macos-15' uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 @@ -187,17 +209,6 @@ jobs: java-version: "25" distribution: ${{ matrix.platform == 'windows-11-arm' && 'microsoft' || 'temurin' }} - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Setup Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 diff --git a/.github/workflows/test-build-docker.yml b/.github/workflows/test-build-docker.yml index 4a79cb3733..12d5a35a1f 100644 --- a/.github/workflows/test-build-docker.yml +++ b/.github/workflows/test-build-docker.yml @@ -79,23 +79,20 @@ jobs: docker system prune -af || true echo "Disk space after cleanup:" && df -h + - name: Restore cache Gradle User Home + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: gradle-v1-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} + - name: Set up JDK 25 uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: java-version: "25" distribution: "temurin" - - name: Cache Gradle User Home - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: | - ~/.gradle/caches - ~/.gradle/wrapper - key: gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25-${{ hashFiles('gradle/wrapper/gradle-wrapper.properties', 'gradle/libs.versions.toml', 'buildSrc/**', 'settings.gradle', 'build.gradle', 'app/**/build.gradle', 'gradle/**/*.gradle') }} - restore-keys: | - gradle-${{ runner.os }}-${{ runner.arch }}-jdk-25- - gradle-${{ runner.os }}-${{ runner.arch }}- - - name: Install Task uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0 - name: Build application From 6f7f28946c6643aecd96da043e9a1d1549193437 Mon Sep 17 00:00:00 2001 From: Anthony Stirling <77850077+Frooodle@users.noreply.github.com> Date: Wed, 19 Aug 2026 14:19:26 +0000 Subject: [PATCH 25/59] Set deployment: false on environment jobs that do not deploy (#7562) # Description of Changes thanks ludy for the tip :P --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details. --- .github/workflows/PR-Auto-Deploy-V2.yml | 5 ++++- .github/workflows/PR-Demo-cleanup.yml | 5 ++++- .github/workflows/backend-build.yml | 4 +++- .github/workflows/build-enterprise.yml | 8 ++++++-- .github/workflows/build.yml | 4 +++- .github/workflows/check-licence.yml | 4 +++- .github/workflows/check-openapi.yml | 4 +++- .github/workflows/db-migration-test.yml | 4 +++- .github/workflows/docker-compose-tests.yml | 4 +++- .github/workflows/e2e-live.yml | 4 +++- .github/workflows/frontend-backend-licenses-update.yml | 8 ++++++-- .github/workflows/multiOSReleases.yml | 8 ++++++-- .github/workflows/nightly.yml | 4 +++- .github/workflows/tauri-build.yml | 4 +++- .github/workflows/test-build-docker.yml | 4 +++- 15 files changed, 56 insertions(+), 18 deletions(-) diff --git a/.github/workflows/PR-Auto-Deploy-V2.yml b/.github/workflows/PR-Auto-Deploy-V2.yml index 0f07aabbe5..4375b1b8b0 100644 --- a/.github/workflows/PR-Auto-Deploy-V2.yml +++ b/.github/workflows/PR-Auto-Deploy-V2.yml @@ -462,7 +462,10 @@ jobs: }); cleanup-v2-deployment: - environment: pr-preview + # Tearing a preview down is not a deployment - no deployment object. + environment: + name: pr-preview + deployment: false if: github.event.action == 'closed' runs-on: ubuntu-latest permissions: diff --git a/.github/workflows/PR-Demo-cleanup.yml b/.github/workflows/PR-Demo-cleanup.yml index 1407939994..098f8d7803 100644 --- a/.github/workflows/PR-Demo-cleanup.yml +++ b/.github/workflows/PR-Demo-cleanup.yml @@ -9,7 +9,10 @@ permissions: jobs: cleanup: - environment: pr-preview + # Tearing a preview down is not a deployment - no deployment object. + environment: + name: pr-preview + deployment: false if: github.event.action == 'closed' runs-on: ubuntu-latest permissions: diff --git a/.github/workflows/backend-build.yml b/.github/workflows/backend-build.yml index 6623940bce..596be96e8c 100644 --- a/.github/workflows/backend-build.yml +++ b/.github/workflows/backend-build.yml @@ -20,7 +20,9 @@ permissions: jobs: build: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest strategy: fail-fast: false diff --git a/.github/workflows/build-enterprise.yml b/.github/workflows/build-enterprise.yml index b4a8373ccc..194d86d9da 100644 --- a/.github/workflows/build-enterprise.yml +++ b/.github/workflows/build-enterprise.yml @@ -42,7 +42,9 @@ jobs: uses: ./.github/workflows/_runner-pick.yml playwright-e2e-enterprise: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false needs: pick # Skip on fork PRs / untrusted authors: they have no PREMIUM_KEY_ENTERPRISE, # so the suite can't boot premium and would fail. See the header comment. @@ -325,7 +327,9 @@ jobs: # Multi-node regression: builds + seeds the clustered stack (testing/compose/docker-compose-multinode.yml) # and runs behave features/multinode. Licence-gated, so it runs after the Playwright job (not in parallel). multinode-e2e: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false needs: [pick, playwright-e2e-enterprise] # Nightly cron + manual dispatch only (heavy build), fork-gated for the licence secret. if: >- diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 566262d24f..1feaff2560 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -61,7 +61,9 @@ jobs: filters: .github/config/.files.yaml gradle-cache-prime: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false name: Prime shared Gradle cache needs: [files-changed] runs-on: ubuntu-latest diff --git a/.github/workflows/check-licence.yml b/.github/workflows/check-licence.yml index 17c64d5c64..4e04a83656 100644 --- a/.github/workflows/check-licence.yml +++ b/.github/workflows/check-licence.yml @@ -10,7 +10,9 @@ permissions: jobs: check-licence: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest steps: - name: Harden Runner diff --git a/.github/workflows/check-openapi.yml b/.github/workflows/check-openapi.yml index ed83447335..bc9b302857 100644 --- a/.github/workflows/check-openapi.yml +++ b/.github/workflows/check-openapi.yml @@ -11,7 +11,9 @@ permissions: jobs: check-generate-openapi-docs: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest steps: - name: Harden Runner diff --git a/.github/workflows/db-migration-test.yml b/.github/workflows/db-migration-test.yml index ccb46d3988..785073944e 100644 --- a/.github/workflows/db-migration-test.yml +++ b/.github/workflows/db-migration-test.yml @@ -13,7 +13,9 @@ permissions: jobs: migration-test: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest timeout-minutes: 30 steps: diff --git a/.github/workflows/docker-compose-tests.yml b/.github/workflows/docker-compose-tests.yml index 039c73e5db..439d4240b2 100644 --- a/.github/workflows/docker-compose-tests.yml +++ b/.github/workflows/docker-compose-tests.yml @@ -17,7 +17,9 @@ permissions: jobs: docker-compose-tests: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest permissions: actions: write diff --git a/.github/workflows/e2e-live.yml b/.github/workflows/e2e-live.yml index 43d66dd1cf..844d26a3d1 100644 --- a/.github/workflows/e2e-live.yml +++ b/.github/workflows/e2e-live.yml @@ -11,7 +11,9 @@ permissions: jobs: playwright-e2e-live: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest timeout-minutes: 30 steps: diff --git a/.github/workflows/frontend-backend-licenses-update.yml b/.github/workflows/frontend-backend-licenses-update.yml index 7fd0136718..9aef2316d2 100644 --- a/.github/workflows/frontend-backend-licenses-update.yml +++ b/.github/workflows/frontend-backend-licenses-update.yml @@ -43,7 +43,9 @@ jobs: generate-frontend-license-report: # ci-bot, not bot-identity: this job runs on PRs too, and bot-identity is main-only. - environment: ci-bot + environment: + name: ci-bot + deployment: false if: needs.files-changed.outputs.licenses-frontend == 'true' name: Generate Frontend License Report needs: files-changed @@ -319,7 +321,9 @@ jobs: generate-backend-license-report: # ci-bot, not bot-identity: this job runs on PRs too, and bot-identity is main-only. - environment: ci-bot + environment: + name: ci-bot + deployment: false if: needs.files-changed.outputs.licenses-backend == 'true' needs: files-changed name: Generate Backend License Report diff --git a/.github/workflows/multiOSReleases.yml b/.github/workflows/multiOSReleases.yml index 0f0b2d3585..0ac94ffe68 100644 --- a/.github/workflows/multiOSReleases.yml +++ b/.github/workflows/multiOSReleases.yml @@ -38,7 +38,9 @@ permissions: jobs: determine-matrix: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false if: ${{ vars.CI_PROFILE != 'lite' }} runs-on: ubuntu-latest outputs: @@ -116,7 +118,9 @@ jobs: env: INPUT_PLATFORM: ${{ github.event.inputs.platform }} build-jars: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false needs: determine-matrix runs-on: ubuntu-latest strategy: diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index c92d17f027..65c25b7b66 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -127,7 +127,9 @@ jobs: # Runs the @nightly tag (conversion scenarios) plus a 10-shard concurrency run # of every other feature. cucumber-nightly: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false name: Cucumber (nightly scenarios + full concurrency) runs-on: ubuntu-latest # Fork pull requests get no MAVEN_* secrets, so the image build cannot work. diff --git a/.github/workflows/tauri-build.yml b/.github/workflows/tauri-build.yml index e3f3122773..0a82647690 100644 --- a/.github/workflows/tauri-build.yml +++ b/.github/workflows/tauri-build.yml @@ -63,7 +63,9 @@ jobs: determine-matrix: # Only probes APPLE_CERTIFICATE for presence, so it stays on the unrestricted # signing environment - release-signing would block every PR run. - environment: ci-signing + environment: + name: ci-signing + deployment: false if: ${{ vars.CI_PROFILE != 'lite' }} runs-on: ubuntu-latest outputs: diff --git a/.github/workflows/test-build-docker.yml b/.github/workflows/test-build-docker.yml index 12d5a35a1f..37cb7cb546 100644 --- a/.github/workflows/test-build-docker.yml +++ b/.github/workflows/test-build-docker.yml @@ -37,7 +37,9 @@ jobs: # spring-security=true matrix entry if `task backend:build` and # `task backend:build:ci` produce equivalent JARs (verify before wiring). test-build-docker-images: - environment: ci-unsigned + environment: + name: ci-unsigned + deployment: false runs-on: ubuntu-latest strategy: fail-fast: false From 0f8803f35f14cf8e9c191cfc5b6101a33586674c Mon Sep 17 00:00:00 2001 From: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com> Date: Wed, 19 Aug 2026 14:27:39 +0000 Subject: [PATCH 26/59] Require the policy-management role to run a policy against its sources (#7565) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Running a stored policy against its **configured sources** (`POST /api/v1/policies/{id}/trigger`, the manual "run now") now requires the policy-management role — global admin self-hosted, team leader on SaaS — alongside the existing team scoping. ## Why A source sweep operates on the team's configured sources using the server's stored connection credentials, so it belongs with the other policy-management capabilities rather than with ordinary use. Team scoping on its own didn't express that distinction. ## Not changed - `POST /{id}/run` — running a policy over documents the **caller supplied** stays open to every team member. That's ordinary editor enforcement on upload and export, and gating it would break it. - Ad-hoc pipelines (`/run`, `/run/stream`). - The scheduled, folder-watch and webhook triggers. - Single-user deployments (login disabled), which have no roles. ## Implementation `PolicyManagementAuthority` gains `canTriggerPolicies()`, kept separate from `canEditPolicies()` so the two capabilities can diverge later. Both current implementations grant it to the same principals that may edit policies. ## Tests - role absent → 403, rejected before any run starts - role present → 202 - login disabled → check skipped entirely - `/{id}/run` asserted to consult neither authority method, so the gate can't quietly extend to the editor path later --- .../AdminPolicyManagementAuthority.java | 5 ++ .../config/PolicyManagementAuthority.java | 11 +++ .../policy/controller/PolicyController.java | 30 +++++++- .../AdminPolicyManagementAuthorityTest.java | 12 +++ .../controller/PolicyControllerTest.java | 74 +++++++++++++++++++ .../TeamLeaderPolicyManagementAuthority.java | 5 ++ ...amLeaderPolicyManagementAuthorityTest.java | 12 +++ 7 files changed, 145 insertions(+), 4 deletions(-) diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthority.java b/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthority.java index a49c8e5aa9..6d8229aa26 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthority.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthority.java @@ -26,6 +26,11 @@ public class AdminPolicyManagementAuthority implements PolicyManagementAuthority return userService.isCurrentUserAdmin(); } + @Override + public boolean canTriggerPolicies() { + return userService.isCurrentUserAdmin(); + } + @Override public Long currentUserTeamId() { String username = userService.getCurrentUsername(); diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/PolicyManagementAuthority.java b/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/PolicyManagementAuthority.java index 0ea3c298ad..d7e4f50ad1 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/PolicyManagementAuthority.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/policy/config/PolicyManagementAuthority.java @@ -12,6 +12,17 @@ public interface PolicyManagementAuthority { /** Whether the current user may create, edit, or delete policies (for their own team). */ boolean canEditPolicies(); + /** + * Whether the current user may run a policy against its configured sources (the manual + * "run now" sweep). Kept separate from {@link #canEditPolicies()} because the two are distinct + * capabilities, even where a deployment grants both to the same people: a sweep operates on the + * team's configured sources using the server's stored connection credentials, which makes it a + * policy-management capability rather than ordinary use. Running a policy over the caller's + * own uploaded files is not covered by this and stays open to every team member — that + * is ordinary editor enforcement. + */ + boolean canTriggerPolicies(); + /** * The team that scopes the current user's policies — the team a new policy is stamped with and * the only team whose policies the user may see/run/edit. {@code null} when it can't be diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java b/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java index 506bb75578..778a04e169 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java @@ -432,9 +432,10 @@ public class PolicyController { * admin gets no say on SaaS. Team scoping (which team's policies) is enforced separately by * {@link PolicyAccessGuard}. Every mutation routes through {@link #savePolicy} (pause/resume * re-save with a flipped {@code enabled} flag) or {@link #deletePolicy}, so gating those two - * covers them all; runs ({@code /run}) stay open to the team. Single-user deployments (login - * disabled) have no such role, so they trust the local operator. The path allowlist for folder - * sources/outputs is enforced separately by {@link PolicyValidator} at validation time. + * covers them all; runs over the caller's own files ({@code /{id}/run}) stay open to the team, + * while source sweeps are gated by {@link #requirePolicySweepAllowed}. Single-user deployments + * (login disabled) have no such role, so they trust the local operator. The path allowlist for + * folder sources/outputs is enforced separately by {@link PolicyValidator} at validation time. */ private void requirePolicyEditingAllowed() { if (!applicationProperties.getSecurity().isEnableLogin()) { @@ -447,6 +448,25 @@ public class PolicyController { } } + /** + * Sweeping a policy's configured sources requires the same role as managing policies: the sweep + * operates on the team's configured sources using the server's stored connection credentials, + * which makes it a policy-management capability rather than ordinary use, and team scoping on + * its own does not express that. Deliberately narrower than it looks: it gates only the sweep, + * not {@link #runStoredPolicy}, because running a policy over documents the caller supplied is + * ordinary editor enforcement that every member performs on upload and export. + */ + private void requirePolicySweepAllowed() { + if (!applicationProperties.getSecurity().isEnableLogin()) { + return; + } + if (!policyManagementAuthority.canTriggerPolicies()) { + throw new ResponseStatusException( + HttpStatus.FORBIDDEN, + "Not permitted to run this policy against its configured sources"); + } + } + @GetMapping @Operation( summary = "List policies", @@ -571,8 +591,10 @@ public class PolicyController { + " the enabled flag (which only gates automatic triggering). Returns" + " the ids of the runs started (poll the run-status endpoint for each)" + " plus what the sweep skipped - already-processed, parked-by-failure," - + " and in-flight counts - so an empty result explains itself.") + + " and in-flight counts - so an empty result explains itself. Requires" + + " the policy-management role.") public ResponseEntity trigger(@PathVariable String policyId) { + requirePolicySweepAllowed(); Policy policy = policyStore .get(policyId) diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthorityTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthorityTest.java index 811aae6b4f..0f97fcaf62 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthorityTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/policy/config/AdminPolicyManagementAuthorityTest.java @@ -39,6 +39,18 @@ class AdminPolicyManagementAuthorityTest { assertFalse(authority().canEditPolicies()); } + @Test + void adminMayTriggerPolicies() { + when(userService.isCurrentUserAdmin()).thenReturn(true); + assertTrue(authority().canTriggerPolicies()); + } + + @Test + void nonAdminMayNotTriggerPolicies() { + when(userService.isCurrentUserAdmin()).thenReturn(false); + assertFalse(authority().canTriggerPolicies()); + } + @Test void currentUserTeamIdResolvesFromTheCurrentUsersTeam() { Team team = new Team(); diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java index 84e9998b90..2fa675597c 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java @@ -2,6 +2,7 @@ package stirling.software.proprietary.policy.controller; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -738,5 +739,78 @@ class PolicyControllerTest { assertThat(((ResponseStatusException) e).getStatusCode()) .isEqualTo(HttpStatus.NOT_FOUND)); } + + @Test + @DisplayName("trigger is forbidden for a team member who cannot manage policies") + void triggerForbiddenForMember() { + // Sweeping a policy's configured sources is a policy-management capability, so being + // in the policy's team is not on its own enough to perform it. + applicationProperties.getSecurity().setEnableLogin(true); + when(policyManagementAuthority.canTriggerPolicies()).thenReturn(false); + + assertThatThrownBy(() -> controller.trigger("a")) + .isInstanceOf(ResponseStatusException.class) + .satisfies( + e -> + assertThat(((ResponseStatusException) e).getStatusCode()) + .isEqualTo(HttpStatus.FORBIDDEN)); + // Rejected before the policy is looked up, so no run starts. + verify(policyRunner, never()).run(any()); + verify(policyStore, never()).get(any()); + } + + @Test + @DisplayName("trigger runs for a caller who may manage policies") + void triggerAllowedForLeader() { + applicationProperties.getSecurity().setEnableLogin(true); + when(policyManagementAuthority.canTriggerPolicies()).thenReturn(true); + Policy p = policy("a", 1L); + when(policyStore.get("a")).thenReturn(Optional.of(p)); + when(policyAccessGuard.canAccess(p)).thenReturn(true); + SweepOutcome outcome = new SweepOutcome(List.of("run-a"), 1, 0, 0, 0); + when(policyRunner.run(p)).thenReturn(outcome); + + ResponseEntity response = controller.trigger("a"); + + assertThat(response.getStatusCode()).isEqualTo(HttpStatus.ACCEPTED); + assertThat(response.getBody()).isEqualTo(outcome); + } + + @Test + @DisplayName("trigger skips the role check when login is disabled") + void triggerTrustsTheLocalOperator() { + // Single-user deployments have no roles at all; the gate must not lock them out of + // their + // own sweeps. + applicationProperties.getSecurity().setEnableLogin(false); + Policy p = policy("a", null); + when(policyStore.get("a")).thenReturn(Optional.of(p)); + when(policyAccessGuard.canAccess(p)).thenReturn(true); + SweepOutcome outcome = new SweepOutcome(List.of("run-a"), 1, 0, 0, 0); + when(policyRunner.run(p)).thenReturn(outcome); + + assertThat(controller.trigger("a").getStatusCode()).isEqualTo(HttpStatus.ACCEPTED); + verify(policyManagementAuthority, never()).canTriggerPolicies(); + } + + @Test + @DisplayName("running a policy over the caller's own files stays open to any member") + void storedRunIsNotGatedByRole() { + // Editor enforcement: every member's upload/export runs the team's stored policies on + // their own documents. Gating this the way the sweep is gated would break the editor. + applicationProperties.getSecurity().setEnableLogin(true); + Policy p = policy("a", 1L); + when(policyStore.get("a")).thenReturn(Optional.of(p)); + when(policyAccessGuard.canAccess(p)).thenReturn(true); + when(policyRunner.runWith(eq(p), any(), eq(PolicyProgressListener.NOOP))) + .thenReturn(handle("run-9")); + + ResponseEntity> response = + assertDoesNotThrow(() -> controller.runStoredPolicy("a", new PolicyRunFiles())); + + assertThat(response.getStatusCode()).isEqualTo(HttpStatus.ACCEPTED); + verify(policyManagementAuthority, never()).canTriggerPolicies(); + verify(policyManagementAuthority, never()).canEditPolicies(); + } } } diff --git a/app/saas/src/main/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthority.java b/app/saas/src/main/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthority.java index e2f5b65b47..0ce5e8e308 100644 --- a/app/saas/src/main/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthority.java +++ b/app/saas/src/main/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthority.java @@ -25,6 +25,11 @@ public class TeamLeaderPolicyManagementAuthority implements PolicyManagementAuth return teamSecurity.isCurrentUserTeamLeader(); } + @Override + public boolean canTriggerPolicies() { + return teamSecurity.isCurrentUserTeamLeader(); + } + @Override public Long currentUserTeamId() { return teamSecurity.currentUserTeamId(); diff --git a/app/saas/src/test/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthorityTest.java b/app/saas/src/test/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthorityTest.java index 70cd360d7c..2c37980a5c 100644 --- a/app/saas/src/test/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthorityTest.java +++ b/app/saas/src/test/java/stirling/software/saas/security/TeamLeaderPolicyManagementAuthorityTest.java @@ -32,6 +32,18 @@ class TeamLeaderPolicyManagementAuthorityTest { assertFalse(authority().canEditPolicies()); } + @Test + void teamLeaderMayTriggerPolicies() { + when(teamSecurity.isCurrentUserTeamLeader()).thenReturn(true); + assertTrue(authority().canTriggerPolicies()); + } + + @Test + void nonLeaderMayNotTriggerPolicies() { + when(teamSecurity.isCurrentUserTeamLeader()).thenReturn(false); + assertFalse(authority().canTriggerPolicies()); + } + @Test void currentUserTeamIdDelegatesToTeamSecurity() { when(teamSecurity.currentUserTeamId()).thenReturn(9L); From 6bae9d516dc029869fee80ec5c2e92da89a5d541 Mon Sep 17 00:00:00 2001 From: ConnorYoh <40631091+ConnorYoh@users.noreply.github.com> Date: Wed, 19 Aug 2026 14:46:00 +0000 Subject: [PATCH 27/59] chore(saas): one task per environment, and make the frontend follow it (#7483) ## The problem The `dev` profile hardcoded one project ref (`qacaivhsjtftfwtgjvva`) in five places: the ref, the Supabase URL, the publishable key, the datasource host and the meter endpoint. That made it both the shared environment everyone relies on *and* the only thing you could point the backend at. Testing an open SaaS PR meant hand-overriding all five via env just to reach that PR's Supabase preview branch, which is the only place the PR's migrations have actually been applied. Get it wrong and you see `relation "stirling_pdf." does not exist` for a table the PR added, which is what happened on [#7414](https://github.com/Stirling-Tools/Stirling-PDF/pull/7414). ## One task per environment ```bash task dev:saas # backend + frontend + engine, against this PR's preview branch task staging:saas # backend + frontend + engine, against the shared v3 project task backend:dev:saas # backend only, preview branch task backend:staging:saas # backend only, v3 ``` | | how | vars | project | |---|---|---|---| | prod | `PROFILES=none` | `SAAS_DB_*` | the live one | | staging | `PROFILES=staging` | `SAAS_STAGING_*` | pinned to v3, always there | | dev | `PROFILES=dev` | `SAAS_DEV_*` | follows a SaaS PR's preview branch | `PROFILES` is still the underlying switch, so the old spelling keeps working. Production deliberately has no named task: reaching it should take a conscious `PROFILES=none`, not a tab-complete. **staging** is the old `dev` configuration, moved and kept pinned. The value of a shared environment is that it is still there tomorrow: reproduce a bug, paste a link to a colleague, share data. **dev** is parameterised by `SAAS_DEV_PROJECT_REF` and derives the Supabase URL, JWT issuer, JWKS, meter endpoint and (unless overridden) the database host from it. Switching which PR you are testing is one variable instead of five. With no ref set, `task backend:dev:saas` stops and says what to set rather than falling back. ## The frontend was the real gap `frontend/editor/.env` is committed and pins the **production** Supabase project, and nothing in the frontend knew about dev or staging. So `task dev:saas` gave you a backend on a preview branch and a login against prod, unless you happened to have hand-written `frontend/editor/.env.saas.local`. The dev tasks now read the backend's env files and derive `VITE_SUPABASE_URL` and `VITE_SUPABASE_PUBLISHABLE_DEFAULT_KEY` from the same project ref the backend resolved, so the two halves cannot point at different projects. Nothing to keep in sync by hand, no new vite mode, and `SAAS_ENV=prod` opts back out to the committed values. ## Where to put your local values Two files, both gitignored, neither ever committed: **`app/.env.saas.local`** is the only one you normally need. The tasks load it for the backend *and* the frontend. ```bash # staging: everything else is already defaulted, so this is all it takes SAAS_STAGING_DB_PASSWORD=... # dev: the preview branch of the PR you are testing, from its "Supabase Preview" check. # A branch has its OWN password and API keys; the parent project's will not authenticate. SAAS_DEV_PROJECT_REF=... SAAS_DEV_DB_PASSWORD=... SAAS_DEV_PUBLISHABLE_KEY=... # prod, if you ever need it SAAS_DB_PROJECT_REF=... SAAS_DB_URL=... SAAS_DB_PASSWORD=... SUPABASE_EDGE_FUNCTION_SECRET=... ``` **`frontend/editor/.env.saas.local`** is no longer needed for choosing a Supabase project, and is best left empty or deleted. If you have one from before this PR, note that the task-supplied values now win, which is the point: the frontend follows the backend. **A blank is not the same as absent.** A dotenv line with an empty value still *sets* the variable, and Spring's `${VAR:default}` only falls back when a variable is absent. So `.env.saas` lists what you must set as blanks, and leaves out the two `*_DB_URL` overrides, which have real defaults to fall back to. This is not theoretical, see below. Committed `app/.env.saas` holds non-secret defaults only. Real secrets are passwords, the edge-function secret and service-role keys. Project refs and publishable keys are neither: a ref is the public `.supabase.co` subdomain and a publishable key ships in the browser bundle by design, which is why `frontend/editor/.env` has always carried prod's. ## Three bugs found while building the tasks All three were in this PR's own earlier commits, and all three were caught by actually booting things rather than by reading the config. **staging could not boot at all.** A blank `SAAS_STAGING_DB_URL=` in `.env.saas` set the variable to empty, so `${SAAS_STAGING_DB_URL:jdbc:...}` resolved to `""` and startup failed with `spring.datasource.url is required when the saas profile is active`. The file already carried a comment warning about exactly this; it had only been applied to the dev block. The original verification for this PR was "placeholders resolve" and "the task parses", neither of which boots anything. **The dev to staging fallback ran `ddl-auto=update` against shared v3.** The dev profile sets `update`, which is right for a disposable preview branch, and separately fell back to staging's project ref. Together that meant Hibernate was free to reconcile tables that RLS policies depend on. `application-staging.properties` pins `none`, but that only applies when the staging profile is the active one, which it was not on the fallback path. There is no fallback now: with no ref the task stops before gradle, and the frontend fails the same way, both naming the variable. **`PROFILES=` never selected production.** Go template `default` treats `""` as absent, so it silently resolved back to `dev`. It is `PROFILES=none` now. ## Two choices worth reviewing **Staging keeps its committed project ref**, now as a `${SAAS_STAGING_PROJECT_REF:...}` default in one place, with the URL, database host and meter endpoint all derived from it. So staging still works with zero setup, and repointing it is one variable. Nothing in CI referenced the ref or the profile. Its publishable key default carries no inline `gitleaks:allow`: a trailing comment in a `.properties` file is part of the value, so the pragma ended up inside the key. It is in `.gitleaksignore` instead. **`SAAS_DEV_DB_URL` still overrides the whole URL**, so a branch needing the pooler host rather than the direct one is reachable without touching committed config. ## Verification - `task backend:staging:saas` boots against v3 and serves `200`. It could not boot before this commit. - `task backend:dev:saas` with no ref stops before gradle naming the variable, and `PROFILES=none` still reaches production. `task frontend:dev:saas` fails the same way; `SAAS_ENV=staging` still resolves with no local config. - Frontend routing picks the SaaS runner for dev/staging and the plain runner for prod; the derivation returns the right URL and key for each. - Vite's `process.env` precedence and Task's dotenv/env semantics were measured, not assumed. That is how one trap surfaced: Task sets an `env:` key even when its value resolves to empty, and Vite treats an empty `process.env` `VITE_*` as authoritative over a committed `.env`. Putting the Supabase vars on the shared `dev:_run` would have blanked Supabase config for the core, proprietary and desktop dev servers, so the SaaS path has its own runner. - `:saas:spotlessApply` and `:saas:compileJava` green. `DevProfileProjectNotice` becomes `SaasProjectNotice` and covers both profiles, stating the project ref and `ddl-auto` at startup so which environment you are on is never a guess. No behaviour change for prod: the `saas` profile is untouched. --- .gitleaksignore | 5 ++ .taskfiles/backend.yml | 55 ++++++++++++-- .taskfiles/frontend.yml | 74 ++++++++++++++++--- Taskfile.yml | 21 +++++- app/.env.saas | 52 ++++++++----- .../saas/config/SaasProjectNotice.java | 53 +++++++++++++ .../main/resources/application-dev.properties | 38 ++++++---- .../resources/application-staging.properties | 39 ++++++++++ 8 files changed, 286 insertions(+), 51 deletions(-) create mode 100644 app/saas/src/main/java/stirling/software/saas/config/SaasProjectNotice.java create mode 100644 app/saas/src/main/resources/application-staging.properties diff --git a/.gitleaksignore b/.gitleaksignore index 12d98aebeb..c3917e985f 100644 --- a/.gitleaksignore +++ b/.gitleaksignore @@ -27,3 +27,8 @@ app/core/src/main/java/stirling/software/SPDF/pdf/signature/CreateSignatureBase. # Supabase publishable key (public by design, RLS-protected) used as a CI fallback # default in the tauri-build workflow when the GitHub secret is unset - not a real secret. .github/workflows/tauri-build.yml:generic-api-key:402 + +# Staging Supabase publishable key (public by design). Ignored here rather than with an +# inline gitleaks:allow because a trailing comment in a .properties file is part of the +# value, so the pragma would end up inside the key. +app/saas/src/main/resources/application-staging.properties:generic-api-key:16 diff --git a/.taskfiles/backend.yml b/.taskfiles/backend.yml index 63773f61fc..08a12b9535 100644 --- a/.taskfiles/backend.yml +++ b/.taskfiles/backend.yml @@ -57,16 +57,57 @@ tasks: - cmd: ./gradlew clean bootRun -PbuildWithFrontend=true platforms: [linux, darwin] + # SaaS backend. dev:saas -> the PR's preview branch, staging:saas -> shared v3, + # PROFILES=none -> production against your own SAAS_DB_*. Production has no named + # task on purpose. Use `none`, not an empty value: Go template `default` treats "" + # as absent and would resolve back to dev. + dev:saas: - desc: "Start backend in SaaS flavor against Supabase" - # `dotenv:` reads from the root Taskfile's directory (".") because this - # subtaskfile is included with `dir: .`. + desc: "Start SaaS backend against the current PR's Supabase preview branch" + dotenv: ['app/.env.saas.local', 'app/.env.saas'] + vars: + PROFILES: '{{.PROFILES | default "dev"}}' + cmds: + # Don't move this check into a `sh:` var: dotenv is visible in cmds but not + # during var evaluation, so the test would always see an empty value. + - cmd: | + if [ "{{.PROFILES}}" = "dev" ] && [ -z "${SAAS_DEV_PROJECT_REF:-}" ]; then + echo ">> SAAS_DEV_PROJECT_REF is not set." + echo ">> Testing a SaaS PR? Put its ref, DB password and publishable key in app/.env.saas.local." + echo ">> Wanted the shared v3 project? Use 'task backend:staging:saas' instead." + exit 1 + fi + - task: _run:saas + vars: + PORT: '{{.PORT}}' + PROFILES: '{{.PROFILES}}' + AIENGINE_URL: '{{.AIENGINE_URL}}' + AIENGINE_ENABLED: '{{.AIENGINE_ENABLED}}' + AIENGINE_TIMEOUTSECONDS: '{{.AIENGINE_TIMEOUTSECONDS}}' + + staging:saas: + desc: "Start SaaS backend against the shared v3 staging project" + cmds: + - task: _run:saas + vars: + PORT: '{{.PORT}}' + PROFILES: staging + AIENGINE_URL: '{{.AIENGINE_URL}}' + AIENGINE_ENABLED: '{{.AIENGINE_ENABLED}}' + AIENGINE_TIMEOUTSECONDS: '{{.AIENGINE_TIMEOUTSECONDS}}' + + _run:saas: + internal: true dotenv: ['app/.env.saas.local', 'app/.env.saas'] ignore_error: true vars: PORT: '{{.PORT | default "8080"}}' - # Override to "" to run the pure `saas` profile against your own SAAS_DB_*. PROFILES: '{{.PROFILES | default "dev"}}' + # Built here rather than inline in the cmds below: the Windows line is an + # unquoted YAML scalar wrapping a cmd.exe string, so a nested {{if ne .X + # "none"}} needs escaped quotes that reach the Go template as literal + # backslashes and fail with `unexpected "\" in operand`. + PROFILE_ARGS: '{{if ne .PROFILES "none"}}--spring.profiles.include={{.PROFILES}}{{end}}' AIENGINE_URL: '{{.AIENGINE_URL | default ""}}' AIENGINE_ENABLED: '{{.AIENGINE_ENABLED | default "false"}}' AIENGINE_TIMEOUTSECONDS: '{{.AIENGINE_TIMEOUTSECONDS | default "120"}}' @@ -77,9 +118,11 @@ tasks: AIENGINE_ENABLED: '{{.AIENGINE_ENABLED}}' AIENGINE_TIMEOUTSECONDS: '{{.AIENGINE_TIMEOUTSECONDS}}' cmds: - - cmd: cmd /c ".\gradlew.bat :stirling-pdf:bootRun {{if .PROFILES}}--args=\"--spring.profiles.include={{.PROFILES}}\"{{end}}" + # PROFILE_ARGS is empty when PROFILES=none, i.e. the bare `saas` profile + # against SAAS_DB_* (production). + - cmd: cmd /c ".\gradlew.bat :stirling-pdf:bootRun {{if .PROFILE_ARGS}}--args=\"{{.PROFILE_ARGS}}\"{{end}}" platforms: [windows] - - cmd: ./gradlew :stirling-pdf:bootRun {{if .PROFILES}}--args='--spring.profiles.include={{.PROFILES}}'{{end}} + - cmd: ./gradlew :stirling-pdf:bootRun {{if .PROFILE_ARGS}}--args='{{.PROFILE_ARGS}}'{{end}} platforms: [linux, darwin] build: diff --git a/.taskfiles/frontend.yml b/.taskfiles/frontend.yml index 481a225ce1..f5325c9ed7 100644 --- a/.taskfiles/frontend.yml +++ b/.taskfiles/frontend.yml @@ -5,6 +5,14 @@ version: '3' # mode flag) or use `--project editor/...` for tsc — so the editor lives # under frontend/editor/ without each task needing a cd. +vars: + # Dev-only browser-tab label so concurrent worktrees are distinguishable. Only + # the worktree folder basename (e.g. "wt1") is exposed — never the full path, + # hostname, or user. Dropped from production builds. + DEV_LABEL: + sh: >- + {{if eq OS "windows"}}powershell -NoProfile -Command '$root = git rev-parse --show-toplevel 2>$null; if (-not $root) { $root = (Get-Location).Path }; Split-Path -Leaf $root'{{else}}basename "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"{{end}} + tasks: install: desc: "Install dependencies" @@ -80,16 +88,52 @@ tasks: OPEN: '{{.OPEN | default ""}}' env: BACKEND_URL: '{{.BACKEND_URL}}' - # Dev-only browser-tab label so concurrent worktrees are distinguishable. - # Only the worktree folder basename (e.g. "wt1") is exposed — never the - # full path, hostname, or user. Consumed at dev-serve time by vite.config - # and dropped from production builds. - STIRLING_DEV_LABEL: - sh: >- - {{if eq OS "windows"}}powershell -NoProfile -Command '$root = git rev-parse --show-toplevel 2>$null; if (-not $root) { $root = (Get-Location).Path }; Split-Path -Leaf $root'{{else}}basename "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"{{end}} + STIRLING_DEV_LABEL: '{{.DEV_LABEL}}' cmds: - npx vite editor --mode {{.MODE}} --port {{.PORT}}{{if .OPEN}} --open{{end}} + # Separate from dev:_run rather than a flag on it: Task sets an `env:` key even + # when its value resolves to empty, and Vite treats an empty process.env VITE_* as + # authoritative over the committed editor/.env, so folding these in blanks Supabase + # config for the core, proprietary and desktop dev servers. + dev:_run:saas: + internal: true + ignore_error: true + # The backend's own env files, so both halves target one project. Paths are + # relative to this taskfile's dir, `frontend`. + dotenv: ['../app/.env.saas.local', '../app/.env.saas'] + vars: + PORT: '{{.PORT | default "5173"}}' + BACKEND_URL: '{{.BACKEND_URL | default "http://localhost:8080"}}' + OPEN: '{{.OPEN | default ""}}' + SAAS_ENV: '{{.SAAS_ENV | default "dev"}}' + env: + BACKEND_URL: '{{.BACKEND_URL}}' + STIRLING_DEV_LABEL: '{{.DEV_LABEL}}' + SAAS_ENV: '{{.SAAS_ENV}}' + # A real process.env VITE_* beats a committed .env in Vite (loadEnv applies + # process.env last), which is what lets this override editor/.env. + # + # These must stay `sh:`, not Go templates: dotenv values are visible to Task's + # embedded shell but not to templates, where {{.SAAS_DEV_PROJECT_REF}} is + # always empty. + VITE_SUPABASE_URL: + sh: | + case "${SAAS_ENV:-dev}" in + staging) ref="${SAAS_STAGING_PROJECT_REF:?set it in app/.env.saas.local}" ;; + *) ref="${SAAS_DEV_PROJECT_REF:?set it in app/.env.saas.local, or run task staging:saas}" ;; + esac + echo "https://${ref}.supabase.co" + VITE_SUPABASE_PUBLISHABLE_DEFAULT_KEY: + sh: | + case "${SAAS_ENV:-dev}" in + staging) echo "${SAAS_STAGING_PUBLISHABLE_KEY:?set it in app/.env.saas.local}" ;; + *) echo "${SAAS_DEV_PUBLISHABLE_KEY:?set it in app/.env.saas.local}" ;; + esac + cmds: + - 'echo ">> frontend Supabase target: $VITE_SUPABASE_URL"' + - npx vite editor --mode saas --port {{.PORT}}{{if .OPEN}} --open{{end}} + dev: desc: "Start frontend dev server" cmds: @@ -111,13 +155,23 @@ tasks: vars: { MODE: proprietary, PORT: '{{.PORT}}', BACKEND_URL: '{{.BACKEND_URL}}', OPEN: '{{.OPEN}}' } dev:saas: - desc: "Start frontend dev server in SaaS mode" + desc: "Start frontend dev server in SaaS mode (SAAS_ENV=dev|staging|prod)" deps: - task: prepare vars: { MODE: saas } + vars: + SAAS_ENV: '{{.SAAS_ENV | default "dev"}}' + # prod routes to the plain runner, which sets no VITE_SUPABASE_* and so leaves + # the committed editor/.env alone. + RUNNER: '{{if eq .SAAS_ENV "prod"}}dev:_run{{else}}dev:_run:saas{{end}}' cmds: - - task: dev:_run - vars: { MODE: saas, PORT: '{{.PORT}}', BACKEND_URL: '{{.BACKEND_URL}}', OPEN: '{{.OPEN}}' } + - task: '{{.RUNNER}}' + vars: + MODE: saas + PORT: '{{.PORT}}' + BACKEND_URL: '{{.BACKEND_URL}}' + OPEN: '{{.OPEN}}' + SAAS_ENV: '{{.SAAS_ENV}}' dev:desktop: desc: "Start frontend dev server in desktop mode" diff --git a/Taskfile.yml b/Taskfile.yml index fc7a564032..92dcdcc742 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -99,11 +99,22 @@ tasks: BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}' OPEN: "true" + # Set SAAS_DEV_PROJECT_REF in app/.env.saas.local to pick the PR. dev:saas: - desc: "Start SaaS backend + frontend concurrently on free ports" + desc: "Start SaaS backend + frontend + engine against the current PR's preview branch" cmds: - task: dev:_all - vars: { FRONTEND: saas, BACKEND: saas } + vars: { FRONTEND: saas, BACKEND: saas, SAAS_ENV: dev } + + staging:saas: + desc: "Start SaaS backend + frontend + engine against the shared v3 staging project" + cmds: + - task: dev:_all + vars: + FRONTEND: saas + BACKEND: saas + BACKEND_TASK: backend:staging:saas + SAAS_ENV: staging dev:all: desc: "Start backend + frontend + engine concurrently on free ports" @@ -115,6 +126,9 @@ tasks: vars: FRONTEND: '{{.FRONTEND | default "proprietary"}}' BACKEND: '{{.BACKEND | default "proprietary"}}' + BACKEND_TASK: '{{.BACKEND_TASK | default (printf "backend:dev:%s" .BACKEND)}}' + # Only meaningful to the saas frontend; every other flavor ignores it. + SAAS_ENV: '{{.SAAS_ENV | default ""}}' PORTS: sh: '{{if eq OS "windows"}}{{.FIND_FREE_PORT_PS}} 8080 5173 5001{{else}}{{.FIND_FREE_PORT_SH}} 8080 5173 5001{{end}}' BACKEND_PORT: '{{index (splitList "\n" .PORTS) 0}}' @@ -124,7 +138,7 @@ tasks: - task: engine:dev vars: PORT: '{{.ENGINE_PORT}}' - - task: 'backend:dev:{{.BACKEND}}' + - task: '{{.BACKEND_TASK}}' vars: PORT: '{{.BACKEND_PORT}}' AIENGINE_URL: 'http://localhost:{{.ENGINE_PORT}}' @@ -134,6 +148,7 @@ tasks: PORT: '{{.FRONTEND_PORT}}' BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}' OPEN: "true" + SAAS_ENV: '{{.SAAS_ENV}}' # ============================================================ # Build diff --git a/app/.env.saas b/app/.env.saas index fb5feec559..25eefb84c5 100644 --- a/app/.env.saas +++ b/app/.env.saas @@ -1,15 +1,16 @@ -############################################################################### -# Stirling-PDF SaaS environment defaults. +# Stirling-PDF SaaS environment defaults. Committed, non-secret. Real values for secrets go in +# .env.saas.local, which is loaded first and wins. Do not commit that file. # -# This file is committed and provides non-secret defaults loaded by -# `task backend:dev:saas`. Put real values for secrets (passwords, project -# refs, edge function secrets) in `.env.saas.local` - any variable set there -# takes precedence over what's defined here. +# Three environments, each deriving its Supabase URLs, JWT issuer and JWKS from one project ref: # -# DO NOT commit `.env.saas.local`. Only `.env.saas` is checked in. -############################################################################### +# prod PROFILES=none SAAS_DB_* the live project +# staging PROFILES=staging SAAS_STAGING_* pinned to v3, always there +# dev PROFILES=dev SAAS_DEV_* follows a SaaS PR's preview branch +# +# dev is the default for `task backend:dev:saas`. Use staging for somewhere stable; use dev when +# testing an open SaaS PR, since its preview branch is the only place those migrations are applied. -# ---------- Supabase project ---------- +# ---------- Supabase project (prod / no-profile) ---------- # Project reference (the subdomain part of .supabase.co). Required. # Set in .env.saas.local. SAAS_DB_PROJECT_REF= @@ -17,18 +18,35 @@ SAAS_DB_PROJECT_REF= # Edge function secret used by billing/license rollup calls. Set in .env.saas.local. SUPABASE_EDGE_FUNCTION_SECRET= -# ---------- Database (saas profile) ---------- -# Direct JDBC URL to the Supabase Postgres. Required when running the plain -# `saas` profile (i.e. without `--spring.profiles.include=dev`). +# ---------- Database (no profile) ---------- +# Direct JDBC URL to the Supabase Postgres. Required when running without +# `--spring.profiles.include=...`. # Example: jdbc:postgresql://db..supabase.co:5432/postgres SAAS_DB_URL= SAAS_DB_USERNAME=postgres SAAS_DB_PASSWORD= -# ---------- Database (dev profile overrides) ---------- -# Used when `--spring.profiles.include=dev` is active. The dev profile -# defaults the URL/username to the shared dev Supabase project, but the -# password must still be provided in .env.saas.local. -SAAS_DEV_DB_URL= +# ---------- staging profile ---------- +# The shared long-lived v3 project. application-staging.properties defaults the ref, +# URL, database host and meter endpoint, so staging needs only the password, in +# .env.saas.local. Set SAAS_STAGING_PROJECT_REF to repoint it; everything derives. +# +# The ref and publishable key are duplicated here because the task derives the +# frontend's VITE_SUPABASE_* from them and a shell cannot read a Spring default. +# Neither is secret: the ref is a public subdomain, the key ships in the bundle. +SAAS_STAGING_PROJECT_REF=qacaivhsjtftfwtgjvva +SAAS_STAGING_PUBLISHABLE_KEY=sb_publishable_nIM8y-9ARPE7EzQwAQHKMg_40fCN6kY # gitleaks:allow +SAAS_STAGING_DB_USERNAME=postgres +SAAS_STAGING_DB_PASSWORD= + +# ---------- dev profile ---------- +# The SaaS PR's Supabase preview branch. Take the ref from that PR's "Supabase +# Preview" check; the profile derives URL, JWT issuer, JWKS, meter endpoint and +# database host from it, so this one value follows a different PR. +# +# A preview branch has its own password and keys; the parent project's will not +# authenticate. Both go in .env.saas.local, along with the ref. +SAAS_DEV_PROJECT_REF= +SAAS_DEV_PUBLISHABLE_KEY= SAAS_DEV_DB_USERNAME=postgres SAAS_DEV_DB_PASSWORD= diff --git a/app/saas/src/main/java/stirling/software/saas/config/SaasProjectNotice.java b/app/saas/src/main/java/stirling/software/saas/config/SaasProjectNotice.java new file mode 100644 index 0000000000..39305fcbcd --- /dev/null +++ b/app/saas/src/main/java/stirling/software/saas/config/SaasProjectNotice.java @@ -0,0 +1,53 @@ +package stirling.software.saas.config; + +import org.springframework.beans.factory.annotation.Value; +import org.springframework.boot.context.event.ApplicationReadyEvent; +import org.springframework.context.annotation.Profile; +import org.springframework.context.event.EventListener; +import org.springframework.core.env.Environment; +import org.springframework.stereotype.Component; + +import lombok.extern.slf4j.Slf4j; + +/** Logs which Supabase project this backend is talking to, and its schema policy. */ +@Slf4j +@Component +@Profile({"dev", "staging"}) +public class SaasProjectNotice { + + private final Environment environment; + private final String projectRef; + private final String ddlAuto; + + public SaasProjectNotice( + Environment environment, + @Value("${app.supabase.project-ref:unknown}") String projectRef, + @Value("${spring.jpa.hibernate.ddl-auto:none}") String ddlAuto) { + this.environment = environment; + this.projectRef = projectRef; + this.ddlAuto = ddlAuto; + } + + @EventListener(ApplicationReadyEvent.class) + public void announceProject() { + boolean staging = environment.matchesProfiles("staging"); + if (staging) { + log.info( + """ + SaaS staging profile: Supabase project {}, ddl-auto={}. This is the SHARED \ + long-lived environment, so its data and schema are not yours alone. Testing an \ + open SaaS PR? Use that PR's preview branch instead \ + (SAAS_DEV_PROJECT_REF in app/.env.saas.local); staging will not have its \ + migrations.\ + """, + projectRef, + ddlAuto); + return; + } + log.info( + "SaaS dev profile: Supabase preview branch {}, ddl-auto={}. Disposable, so Hibernate" + + " is allowed to add the inherited tables the migrations do not create.", + projectRef, + ddlAuto); + } +} diff --git a/app/saas/src/main/resources/application-dev.properties b/app/saas/src/main/resources/application-dev.properties index ee8bf80ff2..b289fc95cc 100644 --- a/app/saas/src/main/resources/application-dev.properties +++ b/app/saas/src/main/resources/application-dev.properties @@ -1,32 +1,40 @@ -# SaaS dev profile. Points at the dev Supabase project. -# Boot: java -jar stirling-pdf.jar --spring.profiles.include=dev +# SaaS dev profile: follows the Supabase preview branch of the SaaS PR under test. +# One variable switches PR, SAAS_DEV_PROJECT_REF; everything else derives from it. +# Want a stable shared environment instead? Use the staging profile. + spring.config.import=optional:classpath:application-dev-local.properties -app.supabase.project-ref=qacaivhsjtftfwtgjvva +# Let Hibernate reconcile the entity tables so a fresh preview branch heals itself. A branch is built +# from the Supabase migrations, which cover the SaaS-owned tables but not the ~28 inherited from the +# self-hosted app -- those have only ever been created by ddl-auto. Safe here because a preview branch +# is disposable and `update` only ever adds; staging pins `none`, so keep this profile-scoped. +spring.jpa.hibernate.ddl-auto=update -stirling.supabase.url=https://qacaivhsjtftfwtgjvva.supabase.co -stirling.supabase.publishable-key=sb_publishable_nIM8y-9ARPE7EzQwAQHKMg_40fCN6kY # gitleaks:allow +# From the PR's "Supabase Preview" check. Required with no fallback: ddl-auto=update above must never +# be aimed at the shared project. +app.supabase.project-ref=${SAAS_DEV_PROJECT_REF} -spring.datasource.url=${SAAS_DEV_DB_URL:jdbc:postgresql://db.qacaivhsjtftfwtgjvva.supabase.co:5432/postgres?ApplicationName=stirling-consolidation-${user.name}} +stirling.supabase.url=https://${app.supabase.project-ref}.supabase.co +# Per-branch, not derivable. Dashboard > Settings > API. +stirling.supabase.publishable-key=${SAAS_DEV_PUBLISHABLE_KEY} + +# Override the whole URL if the branch needs the pooler host rather than the direct one. +spring.datasource.url=${SAAS_DEV_DB_URL:jdbc:postgresql://db.${app.supabase.project-ref}.supabase.co:5432/postgres?ApplicationName=stirling-dev-${user.name}} spring.datasource.username=${SAAS_DEV_DB_USERNAME:postgres} -# Password not committed; export SAAS_DEV_DB_PASSWORD or pass --spring.datasource.password=... +# A preview branch has its own password; the parent project's will not authenticate. spring.datasource.password=${SAAS_DEV_DB_PASSWORD:} -# Conservative dev pool sizing. spring.datasource.hikari.maximum-pool-size=2 spring.datasource.hikari.minimum-idle=1 spring.datasource.hikari.idle-timeout=60000 spring.datasource.hikari.max-lifetime=1800000 spring.datasource.hikari.keepalive-time=300000 -spring.datasource.hikari.data-source-properties.ApplicationName=stirling-consolidation-${user.name} +spring.datasource.hikari.data-source-properties.ApplicationName=stirling-dev-${user.name} logging.level.stirling.software.saas=DEBUG logging.level.org.springframework.security.oauth2.jwt=WARN logging.level.org.springframework.security.oauth2.server.resource=WARN -# Supabase meter edge fn the Java backend calls (server-to-server, on job close). -# URL is not a secret; auth rides the existing SUPABASE_EDGE_FUNCTION_SECRET (same -# shared secret the team-invitation flow uses — no service-role key in the Java env). -# Blank secret → the meter service no-ops with a WARN, so the app still boots. -# The billing portal is NOT here — the FE calls create-customer-portal-session directly. -payg.meter.endpoint=https://qacaivhsjtftfwtgjvva.supabase.co/functions/v1/meter-payg-units +# Server-to-server meter call. Auth rides SUPABASE_EDGE_FUNCTION_SECRET; blank secret means the meter +# service no-ops with a WARN rather than failing the boot. +payg.meter.endpoint=https://${app.supabase.project-ref}.supabase.co/functions/v1/meter-payg-units diff --git a/app/saas/src/main/resources/application-staging.properties b/app/saas/src/main/resources/application-staging.properties new file mode 100644 index 0000000000..4874ed3fb5 --- /dev/null +++ b/app/saas/src/main/resources/application-staging.properties @@ -0,0 +1,39 @@ +# SaaS staging profile: the long-lived shared v3 project, pinned so it is still there tomorrow. +# For work on an open SaaS PR use the dev profile, which follows that PR's preview branch. + +spring.config.import=optional:classpath:application-staging-local.properties + +# Stated rather than inherited: application-saas.properties defaults to `update`, and staging's +# schema is shared and RLS-dependent, so it must not be reconciled by Hibernate. +spring.jpa.hibernate.ddl-auto=none + +# Committed as a default rather than a literal, so staging needs no setup but stays repointable. +# Neither the ref nor the publishable key is secret: the ref is a public subdomain, the key ships in +# the browser bundle. Everything below derives from the ref, so an override follows through. +app.supabase.project-ref=${SAAS_STAGING_PROJECT_REF:qacaivhsjtftfwtgjvva} + +stirling.supabase.url=https://${app.supabase.project-ref}.supabase.co +stirling.supabase.publishable-key=${SAAS_STAGING_PUBLISHABLE_KEY:sb_publishable_nIM8y-9ARPE7EzQwAQHKMg_40fCN6kY} + +spring.datasource.url=${SAAS_STAGING_DB_URL:jdbc:postgresql://db.${app.supabase.project-ref}.supabase.co:5432/postgres?ApplicationName=stirling-staging-${user.name}} +spring.datasource.username=${SAAS_STAGING_DB_USERNAME:postgres} +# Password not committed; export SAAS_STAGING_DB_PASSWORD or pass --spring.datasource.password=... +spring.datasource.password=${SAAS_STAGING_DB_PASSWORD:} + +# Conservative pool sizing: this is a shared project, so don't hold connections others need. +spring.datasource.hikari.maximum-pool-size=2 +spring.datasource.hikari.minimum-idle=1 +spring.datasource.hikari.idle-timeout=60000 +spring.datasource.hikari.max-lifetime=1800000 +spring.datasource.hikari.keepalive-time=300000 +spring.datasource.hikari.data-source-properties.ApplicationName=stirling-staging-${user.name} + +logging.level.stirling.software.saas=DEBUG +logging.level.org.springframework.security.oauth2.jwt=WARN +logging.level.org.springframework.security.oauth2.server.resource=WARN + +# Supabase meter edge fn the Java backend calls (server-to-server, on job close). +# URL is not a secret; auth rides the existing SUPABASE_EDGE_FUNCTION_SECRET (same +# shared secret the team-invitation flow uses — no service-role key in the Java env). +# Blank secret → the meter service no-ops with a WARN, so the app still boots. +payg.meter.endpoint=https://${app.supabase.project-ref}.supabase.co/functions/v1/meter-payg-units From 088e0ef4e25e06ae5f911f8be74912fef8673240 Mon Sep 17 00:00:00 2001 From: Ludy Date: Wed, 19 Aug 2026 18:29:02 +0000 Subject: [PATCH 28/59] deps(frontend): upgrade Cantoo PDF library to 2.8.2 (#7493) # Description of Changes This pull request upgrades the frontend PDF dependency from `@cantoo/pdf-lib` 2.6.5 to 2.8.2. - Updated `frontend/package.json` to require `@cantoo/pdf-lib` `^2.8.2`. - Regenerated `frontend/package-lock.json` with `@cantoo/pdf-lib@2.8.2`, `pako@2.2.0`, and `node-html-better-parser@1.5.9`. - Added the root npm `pako` override recommended by the upstream release. - The upgrade brings upstream parser, object-stream, encryption, form, PNG, and PDF serialization fixes into the frontend dependency. - No application API migration was required because the project does not use the newly added PDF/A, XFA, Factur-X, incremental-update, fontkit, or page-content-extraction APIs. The main challenge was validating the broad upstream change set against the project's actual usage. The frontend typecheck and a direct PDF create/save/load smoke test passed. The complete `frontend:check` and `frontend:test` tasks exceeded the available execution timeout without reporting a test failure. No related issue. --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/HowToAddNewLanguage.md) (if applicable) - [x] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [x] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#6-testing) for more details. --- frontend/package-lock.json | 38 ++++++++++++++++++++++++++------------ frontend/package.json | 3 ++- 2 files changed, 28 insertions(+), 13 deletions(-) diff --git a/frontend/package-lock.json b/frontend/package-lock.json index d50868bd60..37bca97b95 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -10,7 +10,7 @@ "license": "SEE LICENSE IN https://raw.githubusercontent.com/Stirling-Tools/Stirling-PDF/refs/heads/main/proprietary/LICENSE", "dependencies": { "@atlaskit/pragmatic-drag-and-drop": "^1.7.7", - "@cantoo/pdf-lib": "^2.5.3", + "@cantoo/pdf-lib": "^2.8.2", "@dnd-kit/core": "^6.3.1", "@embedpdf/core": "^2.14.4", "@embedpdf/engines": "^2.14.4", @@ -606,18 +606,22 @@ } }, "node_modules/@cantoo/pdf-lib": { - "version": "2.6.5", - "resolved": "https://registry.npmjs.org/@cantoo/pdf-lib/-/pdf-lib-2.6.5.tgz", - "integrity": "sha512-3eMHEaqKHt/G/q+6QjT06A3lz0S/a8x3+myiSN7FNeL3uWcedO0lpfs6TWofa4C03Z1wz3tWeHoa4CsI7DrTSA==", + "version": "2.8.2", + "resolved": "https://registry.npmjs.org/@cantoo/pdf-lib/-/pdf-lib-2.8.2.tgz", + "integrity": "sha512-f0BJM3uPOjbPR3YriSEUIaTM0qnqthjFmTZX9NGI0NDM2Tj4a8xv7Z5Hb6jzUrhfb3/9Y77+xxoOln0IIiYq+w==", "license": "MIT", "dependencies": { "@pdf-lib/standard-fonts": "^1.0.0", "@pdf-lib/upng": "^1.0.1", "color": "^4.2.3", "crypto-js": "^4.2.0", - "node-html-better-parser": ">=1.4.0", - "pako": "^1.0.11", + "html-entities": "^2.3.2", + "node-html-better-parser": ">=1.5.9", + "pako": "^2.2.0", "tslib": ">=2" + }, + "peerDependencies": { + "html-entities": "^2.3.2" } }, "node_modules/@csstools/color-helpers": { @@ -12459,9 +12463,9 @@ } }, "node_modules/node-html-better-parser": { - "version": "1.5.8", - "resolved": "https://registry.npmjs.org/node-html-better-parser/-/node-html-better-parser-1.5.8.tgz", - "integrity": "sha512-t/wAKvaTSKco43X+yf9+76RiMt18MtMmzd4wc7rKj+fWav6DV4ajDEKdWlLzSE8USDF5zr/06uGj0Wr/dGAFtw==", + "version": "1.5.9", + "resolved": "https://registry.npmjs.org/node-html-better-parser/-/node-html-better-parser-1.5.9.tgz", + "integrity": "sha512-z1I5UINMezJXYL9cH3h0a9KBth2G978gSLlfkpQ+CQzzVHVQy9gpARgm9eDsz1O4gn1HtgUqjdAIYxKFZm6uHQ==", "license": "MIT", "dependencies": { "html-entities": "^2.3.2" @@ -12751,9 +12755,19 @@ "license": "MIT" }, "node_modules/pako": { - "version": "1.0.11", - "resolved": "https://registry.npmjs.org/pako/-/pako-1.0.11.tgz", - "integrity": "sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw==", + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/pako/-/pako-2.2.0.tgz", + "integrity": "sha512-zJq6RP/5q+TO2OpFV3FHzlPnFjmkb7Nc99a5SNjJE+uu/PkpChs+NIZSSzbBoD+6kjiISXjfYdwj1ZRQ81dz/w==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/puzrin" + }, + { + "type": "github", + "url": "https://github.com/sponsors/nodeca" + } + ], "license": "(MIT AND Zlib)" }, "node_modules/parent-module": { diff --git a/frontend/package.json b/frontend/package.json index 90a0e10b05..f874dedfa3 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -7,7 +7,7 @@ "proxy": "http://localhost:8080", "dependencies": { "@atlaskit/pragmatic-drag-and-drop": "^1.7.7", - "@cantoo/pdf-lib": "^2.5.3", + "@cantoo/pdf-lib": "^2.8.2", "@dnd-kit/core": "^6.3.1", "@embedpdf/core": "^2.14.4", "@embedpdf/engines": "^2.14.4", @@ -171,6 +171,7 @@ }, "overrides": { "devalue": "^5.8.1", + "pako": "^2.2.0", "tsconfck": { "typescript": "$typescript" } From 1690cc25ccbf100f3be1699ac098bca419f1ca6a Mon Sep 17 00:00:00 2001 From: "stirlingbot[bot]" <195170888+stirlingbot[bot]@users.noreply.github.com> Date: Thu, 20 Aug 2026 05:15:37 +0000 Subject: [PATCH 29/59] Update Frontend 3rd Party Licenses (#7573) Auto-generated by stirlingbot[bot] This PR updates the frontend license report based on changes to package.json dependencies. Signed-off-by: stirlingbot[bot] Co-authored-by: stirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com> --- frontend/editor/src/assets/3rdPartyLicenses.json | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/frontend/editor/src/assets/3rdPartyLicenses.json b/frontend/editor/src/assets/3rdPartyLicenses.json index 110a6c88a5..e17cef8f8b 100644 --- a/frontend/editor/src/assets/3rdPartyLicenses.json +++ b/frontend/editor/src/assets/3rdPartyLicenses.json @@ -10,7 +10,7 @@ { "moduleName": "@cantoo/pdf-lib", "moduleUrl": "https://github.com/cantoo-scribe/pdf-lib", - "moduleVersion": "2.6.5", + "moduleVersion": "2.8.2", "moduleLicense": "MIT", "moduleLicenseUrl": "https://opensource.org/licenses/MIT" }, @@ -255,14 +255,14 @@ { "moduleName": "@stripe/react-stripe-js", "moduleUrl": "https://github.com/stripe/react-stripe-js", - "moduleVersion": "4.0.2", + "moduleVersion": "6.8.0", "moduleLicense": "MIT", "moduleLicenseUrl": "https://opensource.org/licenses/MIT" }, { "moduleName": "@stripe/stripe-js", "moduleUrl": "https://github.com/stripe/stripe-js", - "moduleVersion": "7.9.0", + "moduleVersion": "9.10.0", "moduleLicense": "MIT", "moduleLicenseUrl": "https://opensource.org/licenses/MIT" }, @@ -287,6 +287,13 @@ "moduleLicense": "MIT", "moduleLicenseUrl": "https://opensource.org/licenses/MIT" }, + { + "moduleName": "@tanstack/react-table", + "moduleUrl": "https://github.com/TanStack/table", + "moduleVersion": "9.1.2", + "moduleLicense": "MIT", + "moduleLicenseUrl": "https://opensource.org/licenses/MIT" + }, { "moduleName": "@tanstack/react-virtual", "moduleUrl": "https://github.com/TanStack/virtual", From a744102cb68441a5dcf9b436a77919cf73c7eaab Mon Sep 17 00:00:00 2001 From: James Brunton Date: Thu, 20 Aug 2026 08:23:20 +0000 Subject: [PATCH 30/59] Support Supporting Files in Pipelines (#7547) # Description of Changes Currently in the Processor's Pipelines page, none of the tools which require supporting files are usable because it's never been hooked up to the new API to upload supporting files. This PR hooks it up to that so all tools using supporting files work in the processor. I had to tweak the type generation a little for this so we have a static map of which params are for supporting files so we know to handle them differently. The `Test with a file` button has to work a little differently than the main run since it's running an ad-hoc pipeline so the files haven't necessarily been saved to the server yet. In this case, it'll use whatever local changes the user has made for those pipeline steps, and for all other steps, it'll just use what's saved in the server. --- .../policy/controller/PolicyController.java | 42 +++- .../controller/PolicyControllerTest.java | 48 +++- .../public/locales/en-US/translation.toml | 4 +- .../scripts/generate-tool-api-types.mts | 64 ++++- .../hooks/tools/shared/toolApiMapping.test.ts | 19 +- .../core/hooks/tools/shared/toolApiMapping.ts | 22 +- .../hooks/tools/shared/toolAutomation.test.ts | 178 +++++++++++++- .../core/hooks/tools/shared/toolAutomation.ts | 223 +++++++++++++++++- .../hooks/tools/shared/toolOperationTypes.ts | 47 +++- .../editor/src/core/types/toolApiTypes.ts | 46 ++-- .../editor/src/portal/api/pipelineAssets.ts | 41 ++++ frontend/editor/src/portal/api/pipelines.ts | 28 ++- .../pipelines/PipelineStepSettings.css | 17 ++ .../PipelineStepSettings.stories.tsx | 2 + .../pipelines/PipelineStepSettings.test.tsx | 10 + .../pipelines/PipelineStepSettings.tsx | 146 +++++++++--- .../src/portal/mocks/handlers/pipelines.ts | 42 ++++ .../src/portal/views/PipelineBuilder.test.tsx | 122 +++++++++- .../src/portal/views/PipelineBuilder.tsx | 201 +++++++++++++--- 19 files changed, 1148 insertions(+), 154 deletions(-) create mode 100644 frontend/editor/src/portal/api/pipelineAssets.ts create mode 100644 frontend/editor/src/portal/components/pipelines/PipelineStepSettings.css diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java b/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java index 778a04e169..9b9fca4133 100644 --- a/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java +++ b/app/proprietary/src/main/java/stirling/software/proprietary/policy/controller/PolicyController.java @@ -21,6 +21,7 @@ import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.PutMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RequestPart; import org.springframework.web.bind.annotation.RestController; import org.springframework.web.context.request.RequestContextHolder; @@ -51,6 +52,7 @@ import stirling.software.common.util.TempFile; import stirling.software.common.util.TempFileManager; import stirling.software.proprietary.audit.AuditContext; import stirling.software.proprietary.policy.asset.PolicyAssetCleaner; +import stirling.software.proprietary.policy.asset.PolicyAssetResolver; import stirling.software.proprietary.policy.config.PolicyAccessGuard; import stirling.software.proprietary.policy.config.PolicyManagementAuthority; import stirling.software.proprietary.policy.engine.PolicyRunHandle; @@ -106,6 +108,7 @@ public class PolicyController { private final PolicyTriggerManager policyTriggerManager; private final PolicyOverviewService policyOverviewService; private final PolicyAssetCleaner assetCleaner; + private final PolicyAssetResolver assetResolver; private final ProcessedLedger processedLedger; private final List policyTriggers; private final ApplicationProperties applicationProperties; @@ -125,12 +128,13 @@ public class PolicyController { + " endpoint and download outputs via /api/v1/general/files/{id}.") public ResponseEntity> run( @RequestPart("json") PipelineDefinition definition, + @RequestParam(value = "policyId", required = false) String policyId, @Valid @ModelAttribute PolicyRunFiles files) throws IOException { stampPolicyAudit(definition); requireRunnable(definition); validateAdHocRun(definition); - PolicyInputs inputs = toInputs(files); + PolicyInputs inputs = resolveStoredAssets(policyId, toInputs(files)); PolicyRunHandle handle = policyRunner.runAdHoc(definition, inputs, PolicyProgressListener.NOOP); recordEditorDocs(inputs); @@ -146,12 +150,13 @@ public class PolicyController { + " 'cancelled', or 'waiting' event carrying the final run view.") public SseEmitter runStream( @RequestPart("json") PipelineDefinition definition, + @RequestParam(value = "policyId", required = false) String policyId, @Valid @ModelAttribute PolicyRunFiles files) throws IOException { stampPolicyAudit(definition); requireRunnable(definition); validateAdHocRun(definition); - PolicyInputs inputs = toInputs(files); + PolicyInputs inputs = resolveStoredAssets(policyId, toInputs(files)); SseEmitter emitter = new SseEmitter(applicationProperties.getPolicies().getStreamTimeoutMs()); @@ -438,10 +443,7 @@ public class PolicyController { * folder sources/outputs is enforced separately by {@link PolicyValidator} at validation time. */ private void requirePolicyEditingAllowed() { - if (!applicationProperties.getSecurity().isEnableLogin()) { - return; - } - if (!policyManagementAuthority.canEditPolicies()) { + if (!policyEditingAllowed()) { throw new ResponseStatusException( HttpStatus.FORBIDDEN, "Policies may only be created or modified by a team leader"); @@ -467,6 +469,15 @@ public class PolicyController { } } + /** + * Whether the caller may create/modify policies (a team leader, or any operator when login is + * off). + */ + private boolean policyEditingAllowed() { + return !applicationProperties.getSecurity().isEnableLogin() + || policyManagementAuthority.canEditPolicies(); + } + @GetMapping @Operation( summary = "List policies", @@ -672,6 +683,25 @@ public class PolicyController { inputs.primary().size()); } + /** + * Resolve a test run's stored {@code asset:} bindings from the saved policy the builder is + * editing, so their bytes need not be re-uploaded. Scoped to that policy (the resolver loads + * only the assets it references, in its own team) and gated to policy editors - the same + * authority that can read asset bytes - so a member can't rebind a policy's stored asset into + * an ad-hoc step to read it back. A blank id (an unsaved pipeline has no stored bindings) or an + * inaccessible policy leaves the run-supplied inputs untouched. + */ + private PolicyInputs resolveStoredAssets(String policyId, PolicyInputs inputs) { + if (policyId == null || policyId.isBlank() || !policyEditingAllowed()) { + return inputs; + } + return policyStore + .get(policyId) + .filter(policyAccessGuard::canAccess) + .map(policy -> assetResolver.resolve(policy, inputs)) + .orElse(inputs); + } + /** * Turn the typed run files into engine {@link PolicyInputs}: the primary documents plus the * named supporting-file store, where each asset's {@code key} is the name a step references diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java index 2fa675597c..36f5cc221b 100644 --- a/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java +++ b/app/proprietary/src/test/java/stirling/software/proprietary/policy/controller/PolicyControllerTest.java @@ -83,6 +83,8 @@ class PolicyControllerTest { @Mock private stirling.software.proprietary.policy.asset.PolicyAssetCleaner assetCleaner; + @Mock private stirling.software.proprietary.policy.asset.PolicyAssetResolver assetResolver; + @Mock private ProcessedLedger processedLedger; @Mock private TempFileManager tempFileManager; @@ -115,6 +117,7 @@ class PolicyControllerTest { policyTriggerManager, policyOverviewService, assetCleaner, + assetResolver, processedLedger, policyTriggers, applicationProperties, @@ -232,7 +235,7 @@ class PolicyControllerTest { .thenReturn(handle("run-1")); ResponseEntity> response = - controller.run(definitionWithStep(), new PolicyRunFiles()); + controller.run(definitionWithStep(), null, new PolicyRunFiles()); assertThat(response.getStatusCode()).isEqualTo(HttpStatus.ACCEPTED); assertThat(response.getBody().getJobId()).isEqualTo("run-1"); @@ -245,7 +248,7 @@ class PolicyControllerTest { .thenReturn(handle("run-1")); when(sourceAccessGuard.currentTeamId()).thenReturn(3L); - controller.run(definitionWithStep(), new PolicyRunFiles()); + controller.run(definitionWithStep(), null, new PolicyRunFiles()); verify(docCounter).record(EditorSource.counterKey(3L), 0L); } @@ -255,7 +258,7 @@ class PolicyControllerTest { void rejectsEmptyPipeline() { PipelineDefinition empty = new PipelineDefinition("pipe", List.of(), List.of()); - assertThatThrownBy(() -> controller.run(empty, new PolicyRunFiles())) + assertThatThrownBy(() -> controller.run(empty, null, new PolicyRunFiles())) .isInstanceOf(ResponseStatusException.class) .satisfies( e -> @@ -277,7 +280,7 @@ class PolicyControllerTest { .when(policyValidator) .validateOutput(any()); - assertThatThrownBy(() -> controller.run(definition, new PolicyRunFiles())) + assertThatThrownBy(() -> controller.run(definition, null, new PolicyRunFiles())) .isInstanceOf(ResponseStatusException.class) .satisfies( e -> @@ -285,6 +288,38 @@ class PolicyControllerTest { .isEqualTo(HttpStatus.BAD_REQUEST)); verify(policyRunner, never()).runAdHoc(any(), any(), any()); } + + @Test + @DisplayName("resolves stored assets from the supplied policy when the caller may edit it") + void resolvesStoredAssetsForEditor() throws Exception { + applicationProperties.getSecurity().setEnableLogin(false); // editing allowed + Policy p = policy("pol-1", 1L); + when(policyStore.get("pol-1")).thenReturn(Optional.of(p)); + when(policyAccessGuard.canAccess(p)).thenReturn(true); + when(assetResolver.resolve(eq(p), any())).thenAnswer(inv -> inv.getArgument(1)); + when(policyRunner.runAdHoc(any(), any(), eq(PolicyProgressListener.NOOP))) + .thenReturn(handle("run-1")); + + controller.run(definitionWithStep(), "pol-1", new PolicyRunFiles()); + + verify(assetResolver).resolve(eq(p), any()); + } + + @Test + @DisplayName("does not resolve a policy's stored assets for a caller who cannot edit it") + void skipsStoredAssetsForNonEditor() throws Exception { + // Gating asset resolution to editors keeps a member from rebinding a policy's stored + // asset into an ad-hoc step to read it back. + applicationProperties.getSecurity().setEnableLogin(true); + when(policyManagementAuthority.canEditPolicies()).thenReturn(false); + when(policyRunner.runAdHoc(any(), any(), eq(PolicyProgressListener.NOOP))) + .thenReturn(handle("run-1")); + + controller.run(definitionWithStep(), "pol-1", new PolicyRunFiles()); + + verify(assetResolver, never()).resolve(any(), any()); + verify(policyStore, never()).get(any()); + } } @Nested @@ -296,7 +331,8 @@ class PolicyControllerTest { void returnsEmitter() throws Exception { when(policyRunner.runAdHoc(any(), any(), any())).thenReturn(handle("run-2")); - SseEmitter emitter = controller.runStream(definitionWithStep(), new PolicyRunFiles()); + SseEmitter emitter = + controller.runStream(definitionWithStep(), null, new PolicyRunFiles()); assertThat(emitter).isNotNull(); } @@ -306,7 +342,7 @@ class PolicyControllerTest { void rejectsEmpty() { PipelineDefinition empty = new PipelineDefinition("pipe", List.of(), List.of()); - assertThatThrownBy(() -> controller.runStream(empty, new PolicyRunFiles())) + assertThatThrownBy(() -> controller.runStream(empty, null, new PolicyRunFiles())) .isInstanceOf(ResponseStatusException.class); } } diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml index 12ad26ceec..01db314031 100644 --- a/frontend/editor/public/locales/en-US/translation.toml +++ b/frontend/editor/public/locales/en-US/translation.toml @@ -7705,7 +7705,6 @@ moreActions = "More actions" needsConfiguring = "Needs setting up" needsDestination = "No destination chosen" needsSource = "No source chosen" -needsUpload = "Needs an uploaded file" noToolMatches = "No tools match your search." pause = "Pause" rename = "Rename pipeline" @@ -7713,11 +7712,11 @@ searchTools = "Search tools" sendToSystem = "Send to another system" stepsIncompatible = "These steps can't run on what their prior step produces: {{tools}}." stepsNeedSetup = "These steps still need setting up before saving: {{tools}}." +supportingFiles = "Supporting files" testRun = "Test with a file" unknownStep = "Unrecognized operation, kept as-is." unsavedBody = "You have unsaved changes. Save them before leaving, or discard them?" unsavedTitle = "Unsaved changes" -uploadUnsupported = "Uploaded files aren't supported in pipelines yet, so these steps can't be saved: {{tools}}." usesDefaults = "Runs with default settings" viewDefinition = "View definition" @@ -7730,7 +7729,6 @@ saveHeading = "To save your changes:" schedule = "Set how often it runs" setup = "Finish setting up: {{tools}}" source = "Choose an input source" -upload = "Remove steps that need an uploaded file: {{tools}}" [portal.pipelines.builder.diagnostic] fan-in = "Combines every incoming file" diff --git a/frontend/editor/scripts/generate-tool-api-types.mts b/frontend/editor/scripts/generate-tool-api-types.mts index d5990096af..38eb7fe7b6 100644 --- a/frontend/editor/scripts/generate-tool-api-types.mts +++ b/frontend/editor/scripts/generate-tool-api-types.mts @@ -28,10 +28,11 @@ const ALLOWED_PATH_PREFIXES = [ "/api/v1/integration/", ]; -// File plumbing, not user parameters: `fileInput` is the uploaded document and -// `fileId` a server-side handle. Stripped from every generated request model. -// Named file fields (stampImage, attachments, ...) are real parameters and kept. -const BASE_FILE_FIELDS = new Set(["fileInput", "fileId"]); +// File plumbing, not user parameters: `fileInput` and `file` are the uploaded primary document +// (endpoints use one name or the other - `file` is never a second, supporting upload) and `fileId` +// a server-side handle. Stripped from every generated request model. Named supporting-file fields +// (stampImage, attachments, ...) are real parameters and kept. +const BASE_FILE_FIELDS = new Set(["fileInput", "file", "fileId"]); // The shared "upload a file or provide a file ID" wrapper schema and its two // branches. An endpoint whose body is exactly this has no parameters, so it must @@ -73,6 +74,19 @@ function isObject(value: unknown): value is Json { return typeof value === "object" && value !== null && !Array.isArray(value); } +/** A single file upload: `type: string, format: binary` (a Java MultipartFile param). */ +function isBinaryField(schema: unknown): schema is Json { + return ( + isObject(schema) && schema.type === "string" && schema.format === "binary" + ); +} + +/** A multi file upload: an array of binary items (some specs also flag the array itself binary). */ +function isBinaryArrayField(schema: unknown): schema is Json { + if (!isObject(schema) || schema.type !== "array") return false; + return schema.format === "binary" || isBinaryField(schema.items); +} + /** * Recursively sort object keys so the output is byte-stable regardless of the * key ordering springdoc happens to emit. @@ -358,6 +372,9 @@ async function main(): Promise { const usedClassNames = new Set(); const pendingComponents = new Set(); const skipped: string[] = []; + // Named file fields (as File uploads) per model, so a caller can tell a file param from a scalar + // string param - which `format: binary` -> `string` would otherwise erase. + const fileFieldsByClass: Record = {}; for (const path of Object.keys(paths).sort()) { if ( @@ -408,7 +425,32 @@ async function main(): Promise { const query = queryParameters(pathItem); // Body wins over query on a name collision. const properties: Json = { ...query.props, ...bodyProps }; + // `file` is stripped as a primary-document alias (see BASE_FILE_FIELDS). That only holds while + // no endpoint uses `file` as a *supporting* upload beside a primary `fileInput`; if one ever + // does, blanket-stripping would silently drop it. Fail generation so the assumption is fixed + // here rather than shipping a lost file. + if ("file" in properties && "fileInput" in properties) { + throw new Error( + `${path} has both 'fileInput' and 'file' uploads. 'file' is stripped as a primary-document` + + " alias, which would drop it as a supporting file. Rename the supporting param or revise" + + " BASE_FILE_FIELDS handling in this generator.", + ); + } for (const field of BASE_FILE_FIELDS) delete properties[field]; + // Type each named file upload as File/File[] (not the `string` a binary format yields) via + // json-schema-to-typescript's `tsType` override, and record it. Base file fields are already + // stripped, so what remains is the real supporting-file params. + const fileFields: string[] = []; + for (const [name, prop] of Object.entries(properties)) { + if (isBinaryField(prop)) { + prop.tsType = "File"; + fileFields.push(name); + } else if (isBinaryArrayField(prop)) { + prop.tsType = "File[]"; + fileFields.push(name); + } + } + fileFieldsByClass[className] = fileFields; modelSchema.properties = properties; const required = new Set(computeRequired(modelSchema, properties)); for (const name of query.required) { @@ -464,6 +506,7 @@ async function main(): Promise { await compileAndWrite( tools, definitions, + fileFieldsByClass, outputPath, values.check ?? false, skipped, @@ -473,6 +516,7 @@ async function main(): Promise { async function compileAndWrite( tools: DiscoveredTool[], definitions: Record, + fileFieldsByClass: Record, outputPath: string, check: boolean, skipped: string[], @@ -525,6 +569,15 @@ async function compileAndWrite( const endpointList = tools .map((t) => ` ${JSON.stringify(t.path)},`) .join("\n"); + // Endpoints that take supporting files, mapped to those file params' names. Only endpoints with at + // least one are listed, so membership answers "does this tool take extra files". + const fileFieldEntries = tools + .filter((t) => (fileFieldsByClass[t.className] ?? []).length > 0) + .map( + (t) => + ` ${JSON.stringify(t.path)}: ${JSON.stringify(fileFieldsByClass[t.className])},`, + ) + .join("\n"); const footer = [ "/** Endpoint path for a generated tool operation (the operation identity across languages). */", @@ -536,6 +589,9 @@ async function compileAndWrite( "/** Every generated tool endpoint, for iteration. */", `export const TOOL_ENDPOINTS = [\n${endpointList}\n] as const satisfies readonly ToolEndpoint[];`, "", + "/** The supporting-file parameters each endpoint accepts beyond its primary fileInput, by name. */", + `export const TOOL_FILE_FIELDS = {\n${fileFieldEntries}\n} as const satisfies Partial<\n Record\n>;`, + "", "/** Union of every generated tool request model. */", `export type ToolApiRequest = ToolApiParams[ToolEndpoint];`, ].join("\n"); diff --git a/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.test.ts b/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.test.ts index ba8c0bc2be..f3443dfb51 100644 --- a/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.test.ts +++ b/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.test.ts @@ -41,12 +41,13 @@ describe("objectToFormData", () => { }); test("expands arrays into repeated fields", () => { - const request: ToolApiParams["/api/v1/misc/add-attachments"] = { - attachments: ["a.png", "b.png", "c.png"], + const request: ToolApiParams["/api/v1/misc/ocr-pdf"] = { + ocrType: "Normal", + languages: ["eng", "fra", "deu"], }; const formData = objectToFormData(request); - expect(formData.getAll("attachments")).toEqual(["a.png", "b.png", "c.png"]); + expect(formData.getAll("languages")).toEqual(["eng", "fra", "deu"]); }); test("throws on a non-primitive field value rather than dropping it", () => { @@ -70,6 +71,18 @@ describe("objectToFormData", () => { expect(formData.get("optimizeLevel")).toBe("5"); }); + test("sends a File-valued model field as a file part, not stringified", () => { + const stamp = new File(["s"], "stamp.png", { type: "image/png" }); + const request: ToolApiParams["/api/v1/misc/add-stamp"] = { + stampType: "image", + stampImage: stamp, + }; + const formData = objectToFormData(request); + + expect(formData.get("stampImage")).toBe(stamp); + expect(formData.get("stampType")).toBe("image"); + }); + test("appends multiple files under the same field name", () => { const files = [ new File(["1"], "a.pdf", { type: "application/pdf" }), diff --git a/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.ts b/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.ts index 9c6f5d9e4b..81663fbf69 100644 --- a/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.ts +++ b/frontend/editor/src/core/hooks/tools/shared/toolApiMapping.ts @@ -47,26 +47,28 @@ function appendPrimitive( formData.append(key, value); } else if (typeof value === "number" || typeof value === "boolean") { formData.append(key, `${value}`); + } else if (typeof Blob !== "undefined" && value instanceof Blob) { + // A File upload (models type binary params as File): send it as the file part, not stringified. + formData.append(key, value); } else { - // A non-primitive here means a mapper produced a value the backend cannot - // receive as a form field. Fail loudly rather than silently drop it: - // structured fields must be JSON-encoded in the mapper, and Files passed via - // the `files` argument. + // Any other non-primitive means a mapper produced a value the backend cannot receive as a form + // field. Fail loudly rather than silently drop it: structured fields must be JSON-encoded first. throw new Error( `objectToFormData: field "${key}" has an unsupported value of type ` + - `"${typeof value}"; expected a string, number, or boolean.`, + `"${typeof value}"; expected a string, number, boolean, or File.`, ); } } /** * Serialize a backend request model (the output of a `toApiParams` function) - * into multipart FormData: primitives become string fields, arrays become - * repeated fields, and `undefined`/`null` are omitted. Files are appended - * separately via `files`, keeping file plumbing out of the parameter mapper. + * into multipart FormData: primitives become string fields, `File` values become + * file parts, arrays become repeated fields, and `undefined`/`null` are omitted. + * Extra files may still be passed via `files` (the primary `fileInput`, or a + * field the mapper doesn't carry). * - * Throws if a field holds a non-primitive value, since that cannot be sent as a - * form field: structured fields must be JSON-encoded by the mapper. + * Throws if a field holds any other non-primitive value, since that cannot be + * sent as a form field: structured fields must be JSON-encoded by the mapper. */ export function objectToFormData( params: ToolApiRequest, diff --git a/frontend/editor/src/core/hooks/tools/shared/toolAutomation.test.ts b/frontend/editor/src/core/hooks/tools/shared/toolAutomation.test.ts index 5be4831907..0f7184f848 100644 --- a/frontend/editor/src/core/hooks/tools/shared/toolAutomation.test.ts +++ b/frontend/editor/src/core/hooks/tools/shared/toolAutomation.test.ts @@ -10,11 +10,14 @@ import { asRegistryConfig, ToolType, } from "@app/hooks/tools/shared/toolOperationTypes"; +import { objectToFormData } from "@app/hooks/tools/shared/toolApiMapping"; import { + activeFileFields, deserializeToolStep, + extractStepFiles, getExecutableTools, serializeToolStep, - stepRequiresUpload, + stepNeedsConfiguring, type WorkingToolStep, } from "@app/hooks/tools/shared/toolAutomation"; import { compressOperationConfig } from "@app/hooks/tools/compress/useCompressOperation"; @@ -28,6 +31,10 @@ import { addPasswordOperationConfig } from "@app/hooks/tools/addPassword/useAddP import { changePermissionsOperationConfig } from "@app/hooks/tools/changePermissions/useChangePermissionsOperation"; import { convertOperationConfig } from "@app/hooks/tools/convert/useConvertOperation"; import { defaultParameters as convertDefaults } from "@app/hooks/tools/convert/useConvertParameters"; +import { overlayPdfsOperationConfig } from "@app/hooks/tools/overlayPdfs/useOverlayPdfsOperation"; +import { defaultParameters as overlayDefaults } from "@app/hooks/tools/overlayPdfs/useOverlayPdfsParameters"; +import { certSignOperationConfig } from "@app/hooks/tools/certSign/useCertSignOperation"; +import { defaultParameters as certSignDefaults } from "@app/hooks/tools/certSign/useCertSignParameters"; function entry(over: Partial): ToolRegistryEntry { return { @@ -419,18 +426,167 @@ describe("convert (format-routed custom tool)", () => { }); }); -describe("stepRequiresUpload", () => { - const step = (params: Record): WorkingToolStep => ({ - toolId: "compress" as ToolId, - operation: "/api/v1/misc/compress-pdf", - params, +describe("supporting files", () => { + const fileRegistry: Partial = { + overlayPdfs: entry({ + name: "Overlay", + automationSettings: NoopSettings, + operationConfig: asRegistryConfig(overlayPdfsOperationConfig), + }), + certSign: entry({ + name: "Cert sign", + automationSettings: NoopSettings, + operationConfig: asRegistryConfig(certSignOperationConfig), + }), + }; + + const overlayStep = ( + params: Record, + fileParameters?: Record, + ): WorkingToolStep => ({ + toolId: "overlayPdfs" as ToolId, + operation: "/api/v1/general/overlay-pdfs", + params: { ...overlayDefaults, ...params }, support: "editable", + fileParameters, }); - test("detects a File (or list of Files) among the parameters", () => { - const image = new File(["x"], "logo.png", { type: "image/png" }); - expect(stepRequiresUpload(step({ level: 5 }))).toBe(false); - expect(stepRequiresUpload(step({ watermarkImage: image }))).toBe(true); - expect(stepRequiresUpload(step({ attachments: [image] }))).toBe(true); + const certStep = ( + params: Record, + fileParameters?: Record, + ): WorkingToolStep => ({ + toolId: "certSign" as ToolId, + operation: "/api/v1/security/cert-sign", + params: { ...certSignDefaults, signMode: "MANUAL", ...params }, + support: "editable", + fileParameters, + }); + + test("extractStepFiles groups fresh picks by their backend file field", () => { + const a = new File(["1"], "a.pdf", { type: "application/pdf" }); + const b = new File(["2"], "b.pdf", { type: "application/pdf" }); + expect( + extractStepFiles(overlayStep({ overlayFiles: [a, b] }), fileRegistry), + ).toEqual({ overlayFiles: [a, b] }); + }); + + test("extractStepFiles respects a tool's file selection (certSign by certType)", () => { + const p12 = new File(["k"], "key.p12"); + expect( + extractStepFiles( + certStep({ certType: "PKCS12", p12File: p12 }), + fileRegistry, + ), + ).toEqual({ p12File: [p12] }); + }); + + test("serialize/deserialize round-trips fileParameters", () => { + const step = certStep({ certType: "PKCS12" }, { p12File: "asset:abc" }); + const api = serializeToolStep(step, fileRegistry); + expect(api.fileParameters).toEqual({ p12File: "asset:abc" }); + expect(deserializeToolStep(api, fileRegistry).fileParameters).toEqual({ + p12File: "asset:abc", + }); + }); + + test("stepNeedsConfiguring: a stored binding satisfies the file requirement", () => { + expect( + stepNeedsConfiguring( + certStep({ certType: "PKCS12" }, { p12File: "asset:abc" }), + fileRegistry, + ), + ).toBe(false); + // Without the binding the keystore is still owed. + expect( + stepNeedsConfiguring(certStep({ certType: "PKCS12" }), fileRegistry), + ).toBe(true); + }); + + test("activeFileFields drops a stored binding the tool no longer emits", () => { + // Still PKCS12: the p12File binding is what the tool sends. + expect( + activeFileFields( + certStep({ certType: "PKCS12" }, { p12File: "asset:abc" }), + fileRegistry, + ), + ).toEqual(["p12File"]); + // Switched to PEM: certSign wants privateKeyFile/certFile, so the p12File binding is stale. + expect( + activeFileFields( + certStep({ certType: "PEM" }, { p12File: "asset:abc" }), + fileRegistry, + ), + ).toEqual([]); + }); + + test("activeFileFields is null (not empty) when the tool can't be probed", () => { + // A buildFormData that throws can't be probed; returning null (vs []) tells callers to keep the + // step's stored bindings rather than drop them and let the server GC the assets. + const config = asRegistryConfig<{ signingCert?: File }>({ + toolType: ToolType.singleFile, + operationType: "certSign", + endpoint: "/api/v1/security/cert-sign", + defaultParameters: {}, + buildFormData: () => { + throw new Error("cannot build"); + }, + }); + const registry: Partial = { + certSign: entry({ name: "Boom", operationConfig: config }), + }; + const step: WorkingToolStep = { + toolId: "certSign" as ToolId, + operation: "/api/v1/security/cert-sign", + params: {}, + support: "editable", + fileParameters: { certFile: "asset:x" }, + }; + expect(activeFileFields(step, registry)).toBeNull(); + }); + + test("the overlay sentinel is sized to the binding's asset count", () => { + // Two ids -> two files, matching two counts, so FixedRepeat validation passes. + const step = overlayStep( + { overlayMode: "FixedRepeatOverlay", counts: [1, 2] }, + { overlayFiles: "asset:one,two" }, + ); + expect(activeFileFields(step, fileRegistry)).toEqual(["overlayFiles"]); + expect(stepNeedsConfiguring(step, fileRegistry)).toBe(false); + }); + + test("a rename override binds a backend field to a differently-named param", () => { + // The cert-sign endpoint's `certFile` is held by a frontend param named `signingCert`. + const config = asRegistryConfig<{ signingCert?: File }>({ + toolType: ToolType.singleFile, + operationType: "certSign", + endpoint: "/api/v1/security/cert-sign", + defaultParameters: {}, + validateParams: (p) => p.signingCert !== undefined, + // Sends the File under the backend field `certFile`, like real tools do via objectToFormData + // (which sends a param's File or File[] under a named field, iterating arrays). + buildFormData: (p, file) => + objectToFormData({}, { fileInput: file, certFile: p.signingCert }), + fileParamOverrides: [{ field: "certFile", param: "signingCert" }], + }); + const registry: Partial = { + certSign: entry({ name: "Sign", operationConfig: config }), + }; + const step = ( + fileParameters?: Record, + ): WorkingToolStep => ({ + toolId: "certSign" as ToolId, + operation: "/api/v1/security/cert-sign", + params: {}, + support: "editable", + fileParameters, + }); + // The stored binding is keyed by the backend field, but satisfies the frontend param on reload. + expect(stepNeedsConfiguring(step({ certFile: "asset:x" }), registry)).toBe( + false, + ); + expect(stepNeedsConfiguring(step(), registry)).toBe(true); + expect(activeFileFields(step({ certFile: "asset:x" }), registry)).toEqual([ + "certFile", + ]); }); }); diff --git a/frontend/editor/src/core/hooks/tools/shared/toolAutomation.ts b/frontend/editor/src/core/hooks/tools/shared/toolAutomation.ts index 0485791f15..a9368ccdb7 100644 --- a/frontend/editor/src/core/hooks/tools/shared/toolAutomation.ts +++ b/frontend/editor/src/core/hooks/tools/shared/toolAutomation.ts @@ -18,11 +18,13 @@ import { type ToolRegistryEntry, } from "@app/data/toolsTaxonomy"; import { type ToolId } from "@app/types/toolId"; +import { TOOL_FILE_FIELDS } from "@app/types/toolApiTypes"; import { isToolEndpoint, type ToolEndpoint, } from "@app/hooks/tools/shared/toolApiMapping"; import { + ToolType, type ErasedToolParams, type RegistryToolOperationConfig, } from "@app/hooks/tools/shared/toolOperationTypes"; @@ -62,6 +64,12 @@ export interface ExecutableTool { export interface ToolApiStep { operation: string; parameters: Record; + /** + * Supporting-file bindings: a backend file field (e.g. `stampImage`, `overlayFiles`) mapped to + * `asset:[,]` (stored supporting files) or a run-supplied key. Absent when the step needs + * no supporting file. Mirrors the wire {@code PipelineStep.fileParameters}. + */ + fileParameters?: SupportingFileBindings; } /** A step being edited in a UI that maps to a known tool: parameters are in the tool's frontend shape. */ @@ -70,6 +78,12 @@ export interface KnownToolStep { operation: ToolEndpoint; params: ErasedToolParams; support: ToolStepSupport; + /** + * Stored supporting-file bindings carried from a saved step (field -> `asset:`), so an edit + * round-trips them without the user re-picking. A field the user re-picks lands in `params` as a + * File and takes precedence on save. + */ + fileParameters?: SupportingFileBindings; } /** A stored step whose endpoint maps to no known tool: preserved verbatim, not editable. */ @@ -78,6 +92,8 @@ export interface UnknownToolStep { operation: string; params: ErasedToolParams; support: "unknown"; + /** Supporting-file bindings preserved verbatim, so an unknown step's files round-trip untouched. */ + fileParameters?: SupportingFileBindings; } /** A step being edited in a UI, discriminated by whether its endpoint maps to a known tool. */ @@ -135,12 +151,176 @@ function isFileValue(value: unknown): boolean { } /** - * True if any of a step's parameters is an uploaded file (or list of files). Such a step cannot be - * saved into a stored pipeline yet: the file bytes are not persisted with the policy, so a later - * (e.g. scheduled) run would have nothing to send for that named file field. + * A stored supporting-file id, as returned by the asset store. */ -export function stepRequiresUpload(step: WorkingToolStep): boolean { - return Object.values(step.params).some(isFileValue); +declare const ASSET_ID_BRAND: unique symbol; +export type AssetId = string & { readonly [ASSET_ID_BRAND]: never }; + +/** + * A step's supporting-file bindings: each backend file field (e.g. `stampImage`) mapped to its file. + * A value of `asset:[,]` names stored assets loaded at run time; any other value is a key for + * a file supplied with the run itself. + */ +export type SupportingFileBindings = Record; + +/** + * The `fileParameters` binding format shared with the backend (see PolicyAssetRefs). This module owns + * the frontend side of the step contract, so the format lives here and the builder/settings reuse it. + */ +export const ASSET_REF_PREFIX = "asset:"; + +/** A `fileParameters` value binding one tool file field to the given stored asset ids. */ +export function assetRef(ids: readonly AssetId[]): string { + return ASSET_REF_PREFIX + ids.join(","); +} + +/** The stored asset ids inside a binding value, or none when it isn't an `asset:` ref. */ +export function assetRefIds(binding: string): AssetId[] { + if (!binding.startsWith(ASSET_REF_PREFIX)) return []; + return binding + .slice(ASSET_REF_PREFIX.length) + .split(",") + .map((id) => id.trim()) + .filter(Boolean) as AssetId[]; +} + +/** A throwaway primary document for probing a tool's buildFormData; never sent anywhere. */ +function dummyPrimaryFile(): File { + return new File([], "input.pdf", { type: "application/pdf" }); +} + +/** + * Run a tool's buildFormData so we can read the request it would produce. + * Returns null when File is unavailable or buildFormData throws. + */ +function probeFormData( + config: RegistryToolOperationConfig, + params: ErasedToolParams, +): FormData | null { + if (typeof File === "undefined") return null; + const dummy = dummyPrimaryFile(); + try { + switch (config.toolType) { + case ToolType.singleFile: + return config.buildFormData(params, dummy); + case ToolType.multiFile: + return config.buildFormData(params, [dummy]); + default: + return null; + } + } catch { + return null; + } +} + +/** Defaults merged under the step's params - the shape a tool's mappers and buildFormData expect. */ +function mergedStepParams( + step: WorkingToolStep, + config: RegistryToolOperationConfig, +): ErasedToolParams { + return { ...(config.defaultParameters ?? {}), ...step.params }; +} + +/** The backend file fields an endpoint accepts, from the generated spec-sourced table. */ +function backendFileFields(operation: string): readonly string[] { + return ( + (TOOL_FILE_FIELDS as Partial>)[ + operation + ] ?? [] + ); +} + +/** + * Each backend file field the step's endpoint accepts (from {@link TOOL_FILE_FIELDS}), mapped to the + * tool param that holds it - the same name unless the tool declared a rename override. + */ +function fileFieldMappings( + operation: string, + config: RegistryToolOperationConfig, +): { field: string; param: string }[] { + // The override's erased type collapses `param` to `never`; restore the real runtime shape. + const overrides = (config.fileParamOverrides ?? []) as readonly { + field: string; + param: string; + }[]; + const paramByField = new Map(overrides.map((o) => [o.field, o.param])); + return backendFileFields(operation).map((field) => ({ + field, + param: paramByField.get(field) ?? field, + })); +} + +/** + * The step's params with a stand-in File array injected for each stored binding whose param has no + * fresh pick, so a tool's buildFormData/validateParams sees the supporting file as present. Stored + * bindings are keyed by the backend field (from {@link TOOL_FILE_FIELDS}), so each field finds its + * binding and the sentinel lands on its param - the two coincide unless the tool declared a rename + * override. The array is sized to the binding's asset count (overlay validates count == file count). + * Sentinels are empty and live only in this local object - never written back to step.params, so they + * can never be uploaded. + */ +function withStoredFileSentinels( + step: WorkingToolStep, + config: RegistryToolOperationConfig, +): ErasedToolParams { + const merged = mergedStepParams(step, config); + const bindings = step.fileParameters; + if (!bindings || typeof File === "undefined") return merged; + for (const { param, field } of fileFieldMappings(step.operation, config)) { + const binding = bindings[field]; + if (binding == null || isFileValue(merged[param])) continue; // unbound, or a fresh pick stands in + const count = Math.max(1, assetRefIds(binding).length); + merged[param] = Array.from({ length: count }, () => new File([], "stored")); + } + return merged; +} + +/** + * The fresh File picks on a step, grouped by the backend file field its buildFormData sends them + * under (excluding the primary `fileInput`). buildFormData is the source of truth for the field name + * and for tool-specific selection (certSign picks files by certType), so probing it - rather than + * scanning params - keeps the field mapping correct. These are the files to upload on save. + */ +export function extractStepFiles( + step: WorkingToolStep, + registry: Partial, +): Record { + if (step.toolId === null) return {}; + const config = registry[step.toolId]?.operationConfig; + if (!config) return {}; + const formData = probeFormData(config, mergedStepParams(step, config)); + if (!formData) return {}; + const files: Record = {}; + formData.forEach((value, key) => { + if (key !== "fileInput" && value instanceof File) { + (files[key] ??= []).push(value); + } + }); + return files; +} + +/** + * The backend file fields this step actually uses right now, per its own buildFormData: fresh picks + * plus any stored binding the tool still emits (a stale one - e.g. a PKCS12 keystore after switching + * to PEM - is dropped, because buildFormData no longer sends it). Drives the stored-file chips, the + * save-time binding set, and the test run. + */ +export function activeFileFields( + step: WorkingToolStep, + registry: Partial, +): string[] | null { + if (step.toolId === null) { + return step.fileParameters ? Object.keys(step.fileParameters) : []; + } + const config = registry[step.toolId]?.operationConfig; + if (!config) return null; + const formData = probeFormData(config, withStoredFileSentinels(step, config)); + if (!formData) return null; + const fields = new Set(); + formData.forEach((value, key) => { + if (key !== "fileInput" && value instanceof File) fields.add(key); + }); + return [...fields]; } /** @@ -158,9 +338,10 @@ export function stepNeedsConfiguring( ): boolean { if (step.toolId === null) return false; const config = registry[step.toolId]?.operationConfig; - if (!config?.validateParams) return false; - const merged = { ...(config.defaultParameters ?? {}), ...step.params }; - return !config.validateParams(merged); + if (!config || !config.validateParams) return false; + // Stored supporting files satisfy their field just as a fresh pick would, so validate against the + // sentinel-injected params rather than the bare ones (which drop the file on reload). + return !config.validateParams(withStoredFileSentinels(step, config)); } /** @@ -240,14 +421,27 @@ export function serializeToolStep( step.toolId !== null ? registry[step.toolId]?.operationConfig : undefined; if (!config) { // Unmapped step (unknown endpoint on edit): round-trip it unchanged. - return { operation: step.operation, parameters: step.params }; + return withFileParameters( + { operation: step.operation, parameters: step.params }, + step, + ); } const merged = { ...(config.defaultParameters ?? {}), ...step.params }; const operation = resolveEndpoint(config, merged) ?? step.operation; const parameters = config.toApiParams ? (config.toApiParams(merged) as Record) : {}; - return { operation, parameters }; + return withFileParameters({ operation, parameters }, step); +} + +/** Attach the step's supporting-file bindings to a serialized step, omitting the field when empty. */ +function withFileParameters( + serialized: ToolApiStep, + step: WorkingToolStep, +): ToolApiStep { + const bindings = step.fileParameters; + if (!bindings || Object.keys(bindings).length === 0) return serialized; + return { ...serialized, fileParameters: bindings }; } /** @@ -308,6 +502,7 @@ function unmappedStep(step: ToolApiStep): UnknownToolStep { operation: step.operation, params: { ...step.parameters }, support: "unknown", + fileParameters: step.fileParameters, }; } @@ -345,5 +540,11 @@ export function deserializeToolStep( resolveEndpoint(config, params) ?? (isToolEndpoint(step.operation) ? step.operation : undefined); if (operation === undefined) return unmappedStep(step); - return { toolId, operation, params, support: classifyToolStepSupport(entry) }; + return { + toolId, + operation, + params, + support: classifyToolStepSupport(entry), + fileParameters: step.fileParameters, + }; } diff --git a/frontend/editor/src/core/hooks/tools/shared/toolOperationTypes.ts b/frontend/editor/src/core/hooks/tools/shared/toolOperationTypes.ts index d6dbf3b355..d01d897002 100644 --- a/frontend/editor/src/core/hooks/tools/shared/toolOperationTypes.ts +++ b/frontend/editor/src/core/hooks/tools/shared/toolOperationTypes.ts @@ -3,7 +3,11 @@ import { StirlingFile } from "@app/types/fileContext"; import type { ResponseHandler } from "@app/utils/toolResponseProcessor"; import { ToolId } from "@app/types/toolId"; import type { ProcessingProgress } from "@app/hooks/tools/shared/useToolState"; -import type { ToolApiParams, ToolEndpoint } from "@app/types/toolApiTypes"; +import { + TOOL_FILE_FIELDS, + type ToolApiParams, + type ToolEndpoint, +} from "@app/types/toolApiTypes"; export type { ProcessingProgress, ResponseHandler }; @@ -45,6 +49,39 @@ export interface CustomProcessorResult { consumedAllInputs?: boolean; } +/** + * The parameter keys that carry a supporting file - a `File` or `File[]` value the tool sends + * beyond its primary document. Derived from the tool's own parameter type, so a file field can only + * ever be declared against a param that genuinely holds a file. + */ +export type FileParamKey = { + [K in keyof TParams]-?: NonNullable extends File | File[] + ? K + : never; +}[keyof TParams] & + string; + +/** + * The backend multipart file fields an endpoint accepts, from the generated {@link TOOL_FILE_FIELDS} + * (which the spec derives from the Java MultipartFile params). `never` for an endpoint that takes no + * supporting files. This is what makes a rename override's `field` a checked name, not a free string. + */ +export type BackendFileField = + TEndpoint extends keyof typeof TOOL_FILE_FIELDS + ? (typeof TOOL_FILE_FIELDS)[TEndpoint][number] + : never; + +/** + * A remap for the rare case where a tool's frontend file param has a different name from the backend + * field it is sent under. Both sides are checked: `field` must be one of the endpoint's generated + * backend file fields, and `param` a real file param of the tool. Same-name fields need no entry - + * they are derived from {@link TOOL_FILE_FIELDS} directly. + */ +export interface FileParamOverride { + field: BackendFileField; + param: FileParamKey; +} + /** * Configuration for tool operations defining processing behavior and API integration. * @@ -79,6 +116,14 @@ interface BaseToolOperationConfig { /** Default parameter values for automation */ defaultParameters?: TParams; + /** + * Rename overrides for supporting-file params. The set of a tool's file fields is derived from the + * generated {@link TOOL_FILE_FIELDS} (spec-sourced), keyed by the backend field name; declare an + * override only when a backend field maps to a differently-named frontend param, so a step composer + * can bind the stored file to the right param. Omitted by the common case where field == param. + */ + fileParamOverrides?: readonly FileParamOverride[]; + /** * Whether these parameters are complete enough to run. The same predicate a tool gives * `useBaseParameters` as its `validateFn`, so the Run button in the editor and anything composing diff --git a/frontend/editor/src/core/types/toolApiTypes.ts b/frontend/editor/src/core/types/toolApiTypes.ts index bf498eb450..b2bdf7fb4b 100644 --- a/frontend/editor/src/core/types/toolApiTypes.ts +++ b/frontend/editor/src/core/types/toolApiTypes.ts @@ -7,7 +7,7 @@ export interface AddAttachmentRequest { /** * The image file to be overlaid onto the PDF. */ - attachments: string[]; + attachments: File[]; /** * Convert the resulting PDF to PDF/A-3b format after adding attachments */ @@ -148,7 +148,7 @@ export interface AddStampRequest { * The rotation of the stamp in degrees */ rotation?: number; - stampImage?: string; + stampImage?: File; /** * The stamp text */ @@ -187,7 +187,7 @@ export interface AddWatermarkRequest { * The rotation of the watermark in degrees */ rotation?: number; - watermarkImage?: string; + watermarkImage?: File; /** * The watermark text */ @@ -525,9 +525,7 @@ export interface FlattenRequest { */ renderDpi?: number; } -export interface GeneralExtractBookmarksRequest { - file: string; -} +export type GeneralExtractBookmarksRequest = Record; export type GeneralFile = Record; export type GeneralPdfToSinglePageRequest = Record; export type GeneralRemoveImagePdfRequest = Record; @@ -788,7 +786,7 @@ export interface OverlayImageRequest { * Whether to overlay the image onto every page of the PDF. */ everyPage?: boolean; - imageFile: string; + imageFile: File; /** * The x-coordinate at which to place the top-left corner of the image. */ @@ -806,7 +804,7 @@ export interface OverlayPdfsRequest { /** * An array of PDF files to be used as overlays on the base PDF. The order in these files is applied based on the selected mode. */ - overlayFiles: string[]; + overlayFiles: File[]; /** * The mode of overlaying: 'SequentialOverlay' for sequential application, 'InterleavedOverlay' for round-robin application, 'FixedRepeatOverlay' for fixed repetition based on provided counts */ @@ -1276,7 +1274,6 @@ export interface ScannerEffectRequest { yellowish?: boolean; } export interface SecurityCertSignSessionsRequest { - file: string; request?: WorkflowCreationRequest; } export interface WorkflowCreationRequest { @@ -1291,8 +1288,8 @@ export interface WorkflowCreationRequest { } export interface SecurityCertSignValidateCertificateRequest { certType: string; - jksFile?: string; - p12File?: string; + jksFile?: File; + p12File?: File; password?: string; } export type SecurityGetInfoOnPdfRequest = Record; @@ -1302,7 +1299,7 @@ export interface SignPDFWithCertRequest { * The alias of the certificate to sign with. Required for WINDOWS_STORE and recommended for PKCS11 tokens holding multiple certificates. */ alias?: string; - certFile?: string; + certFile?: File; /** * The type of the digital certificate. WINDOWS_STORE and PKCS11 are hardware-backed and only available in the desktop app. */ @@ -1314,7 +1311,7 @@ export interface SignPDFWithCertRequest { | "SERVER" | "WINDOWS_STORE" | "PKCS11"; - jksFile?: string; + jksFile?: File; /** * The location where the PDF is signed */ @@ -1323,7 +1320,7 @@ export interface SignPDFWithCertRequest { * The name of the signer */ name?: string; - p12File?: string; + p12File?: File; /** * The page number where the signature should be visible. This is required if showSignature is set to true */ @@ -1340,7 +1337,7 @@ export interface SignPDFWithCertRequest { * Optional PKCS#11 slot index. When omitted the first slot with a token is used. */ pkcs11Slot?: number; - privateKeyFile?: string; + privateKeyFile?: File; /** * The reason for signing the PDF */ @@ -1355,7 +1352,7 @@ export interface SignPDFWithCertRequest { showSignature?: boolean; } export interface SignatureValidationRequest { - certFile?: string; + certFile?: File; } export interface SplitPagesRequest { /** @@ -1741,5 +1738,22 @@ export const TOOL_ENDPOINTS = [ "/api/v1/security/verify-pdf", ] as const satisfies readonly ToolEndpoint[]; +/** The supporting-file parameters each endpoint accepts beyond its primary fileInput, by name. */ +export const TOOL_FILE_FIELDS = { + "/api/v1/general/overlay-pdfs": ["overlayFiles"], + "/api/v1/misc/add-attachments": ["attachments"], + "/api/v1/misc/add-image": ["imageFile"], + "/api/v1/misc/add-stamp": ["stampImage"], + "/api/v1/security/add-watermark": ["watermarkImage"], + "/api/v1/security/cert-sign": [ + "privateKeyFile", + "certFile", + "p12File", + "jksFile", + ], + "/api/v1/security/cert-sign/validate-certificate": ["p12File", "jksFile"], + "/api/v1/security/validate-signature": ["certFile"], +} as const satisfies Partial>; + /** Union of every generated tool request model. */ export type ToolApiRequest = ToolApiParams[ToolEndpoint]; diff --git a/frontend/editor/src/portal/api/pipelineAssets.ts b/frontend/editor/src/portal/api/pipelineAssets.ts new file mode 100644 index 0000000000..b68c95fda5 --- /dev/null +++ b/frontend/editor/src/portal/api/pipelineAssets.ts @@ -0,0 +1,41 @@ +import { apiClient } from "@portal/api/http"; +import { type AssetId } from "@app/hooks/tools/shared/toolAutomation"; + +export { type AssetId }; + +/** + * Stored supporting files for pipeline steps (backend PolicyAssetController). + * + * A pipeline step that needs more than the document stream - a signing + * certificate, a watermark/stamp image, overlay PDFs, attachments - references + * its file by id from the step's `fileParameters` as `asset:`. The bytes are + * uploaded here first (the save-time validator rejects a policy that binds an + * asset id that doesn't yet exist), then a triggered or scheduled run loads the + * file server-side without anyone re-supplying it. Assets are team-scoped exactly + * like the policies that reference them, and unreferenced uploads are cleaned up + * server-side, so the builder never has to delete what a cancelled edit left. + */ + +/** Metadata for one stored supporting file. Mirrors the Java `PolicyAsset` record. */ +export interface PolicyAsset { + id: AssetId; + fileName: string; + contentType: string | null; + size: number; + createdAt: number; +} + +/** POST /api/v1/policies/assets: store a supporting file, returning its metadata (with the id). */ +export async function uploadPipelineAsset(file: File): Promise { + const form = new FormData(); + form.append("file", file); + return apiClient.local.multipart( + "/api/v1/policies/assets", + form, + ); +} + +/** GET /api/v1/policies/assets: the team's stored supporting files (metadata only). */ +export async function listPipelineAssets(): Promise { + return apiClient.local.json("/api/v1/policies/assets"); +} diff --git a/frontend/editor/src/portal/api/pipelines.ts b/frontend/editor/src/portal/api/pipelines.ts index 50bdc173c4..e441fcdac8 100644 --- a/frontend/editor/src/portal/api/pipelines.ts +++ b/frontend/editor/src/portal/api/pipelines.ts @@ -1,5 +1,8 @@ import { apiClient } from "@portal/api/http"; -import { type ToolApiStep } from "@app/hooks/tools/shared/toolAutomation"; +import { + type SupportingFileBindings, + type ToolApiStep, +} from "@app/hooks/tools/shared/toolAutomation"; /** * Pipelines service layer: the backend contract. @@ -15,7 +18,7 @@ import { type ToolApiStep } from "@app/hooks/tools/shared/toolAutomation"; export interface PipelineStep { operation: string; parameters: Record; - fileParameters?: Record; + fileParameters?: SupportingFileBindings; } /** When a policy input fires automatically. `type` keys a trigger bean (e.g. "schedule"). */ @@ -217,14 +220,28 @@ export interface TestRunDefinition { output: OutputSpec; } +/** + * A fresh, in-memory supporting file sent inline with a test run, bound to the run key a test step's + * `fileParameters` references. Only unsaved picks ride along here; a stored file keeps its + * `asset:` binding, which the backend resolves from the saved policy (see `runPipelineTest`). + */ +export interface TestRunAsset { + key: string; + file: File; +} + /** * POST /api/v1/policies/run: run a definition against one uploaded file now. The builder's test * path - callers force an inline output so nothing reaches the pipeline's real destination, and - * the pipeline need not be saved first. + * the pipeline need not be saved first. Fresh supporting files travel as keyed `assets[i]` parts; + * a stored file keeps its `asset:` binding, and `policyId` lets the backend resolve it from that + * saved policy (so its bytes need not be re-sent). */ export async function runPipelineTest( definition: TestRunDefinition, file: File, + assets: TestRunAsset[] = [], + policyId?: string, ): Promise<{ runId: string }> { const form = new FormData(); form.append( @@ -232,6 +249,11 @@ export async function runPipelineTest( new Blob([JSON.stringify(definition)], { type: "application/json" }), ); form.append("fileInput", file); + if (policyId) form.append("policyId", policyId); + assets.forEach((asset, i) => { + form.append(`assets[${i}].key`, asset.key); + form.append(`assets[${i}].file`, asset.file); + }); // The POST returns the identifier as `jobId`, but it is the same run id every other endpoint // (fetchRun, fetchRunOutput) calls `runId`; normalise to that here so callers see one name. const res = await apiClient.local.multipart<{ jobId: string }>( diff --git a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.css b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.css new file mode 100644 index 0000000000..46bb72a736 --- /dev/null +++ b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.css @@ -0,0 +1,17 @@ +.portal-step-settings__files { + display: flex; + flex-direction: column; + gap: 0.375rem; + margin-bottom: 0.75rem; +} + +.portal-step-settings__files-label { + font-size: 0.75rem; + color: var(--c-text-muted); +} + +.portal-step-settings__files-chips { + display: flex; + flex-wrap: wrap; + gap: 0.375rem; +} diff --git a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.stories.tsx b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.stories.tsx index d2bd6bc6f2..823d0ae4fe 100644 --- a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.stories.tsx +++ b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.stories.tsx @@ -52,6 +52,8 @@ const meta = { step: editableStep, registry, onChange: () => {}, + assetNames: {}, + onClearBinding: () => {}, }, } satisfies Meta; export default meta; diff --git a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx index 55a52b496e..0b15b7233a 100644 --- a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx +++ b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.test.tsx @@ -116,6 +116,8 @@ describe("PipelineStepSettings", () => { step={step} registry={registry} onChange={() => {}} + assetNames={{}} + onClearBinding={() => {}} /> , ), @@ -131,6 +133,8 @@ describe("PipelineStepSettings", () => { step={convertStep} registry={convertRegistry} onChange={() => {}} + assetNames={{}} + onClearBinding={() => {}} /> , ), @@ -146,6 +150,8 @@ describe("PipelineStepSettings", () => { step={changeMetadataStep} registry={changeMetadataRegistry} onChange={() => {}} + assetNames={{}} + onClearBinding={() => {}} /> , ), @@ -161,6 +167,8 @@ describe("PipelineStepSettings", () => { step={overlayStep} registry={overlayRegistry} onChange={() => {}} + assetNames={{}} + onClearBinding={() => {}} /> , ), @@ -205,6 +213,8 @@ describe("PipelineStepSettings", () => { typeof update === "function" ? update(prev) : update, ) } + assetNames={{}} + onClearBinding={() => {}} /> {JSON.stringify(params)} diff --git a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.tsx b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.tsx index 58fb2a96fa..3d17dbf652 100644 --- a/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.tsx +++ b/frontend/editor/src/portal/components/pipelines/PipelineStepSettings.tsx @@ -1,15 +1,22 @@ import { Suspense } from "react"; import { useTranslation } from "react-i18next"; -import { Banner } from "@app/ui"; +import InsertDriveFileOutlinedIcon from "@mui/icons-material/InsertDriveFileOutlined"; +import { Banner, Chip } from "@app/ui"; import { PreferencesProvider } from "@app/contexts/PreferencesContext"; import { SidebarProvider } from "@app/contexts/SidebarContext"; import { type ToolRegistry } from "@app/data/toolsTaxonomy"; import { type ErasedToolParams } from "@app/hooks/tools/shared/toolOperationTypes"; -import { type WorkingToolStep } from "@app/hooks/tools/shared/toolAutomation"; +import { + activeFileFields, + assetRefIds, + extractStepFiles, + type WorkingToolStep, +} from "@app/hooks/tools/shared/toolAutomation"; import { PolicyExternalApiConfig } from "@portal/components/policies/PolicyExternalApiConfig"; import { isIntegrationStep } from "@portal/components/pipelines/integrationStep"; import type { ExternalApiStepParams } from "@portal/components/policies/stepOperations"; +import "@portal/components/pipelines/PipelineStepSettings.css"; /** * A params update: the next params outright, or a merge from the latest params. Settings UIs fire @@ -25,17 +32,61 @@ interface PipelineStepSettingsProps { step: WorkingToolStep; registry: Partial; onChange: (update: ParamsUpdate) => void; + /** Stored asset id -> file name, for labelling the supporting-file chips on a reopened pipeline. */ + assetNames: Record; + /** Drop a field's stored supporting-file binding (the user re-picks a file if the step still needs one). */ + onClearBinding: (field: string) => void; +} + +/** One reopened supporting file shown as a chip: the field it binds and the stored file name(s). */ +interface StoredFileChip { + field: string; + label: string; +} + +/** + * The supporting files this step is reusing from a previous save: an active binding whose field has + * no fresh pick (a fresh pick shows in the tool's own file picker instead). Labelled by the resolved + * asset name so the user sees "using cert.pfx" rather than an empty picker. + */ +function storedFileChips( + step: WorkingToolStep, + registry: Partial, + assetNames: Record, +): StoredFileChip[] { + const bindings = step.fileParameters; + if (!bindings) return []; + // A null active set means the tool couldn't be probed; show every stored binding rather than hide + // the user's files (mirrors the save path, which keeps them too). + const active = activeFileFields(step, registry); + const activeSet = active === null ? null : new Set(active); + const fresh = extractStepFiles(step, registry); + return Object.entries(bindings) + .filter( + ([field]) => + (activeSet === null || activeSet.has(field)) && !fresh[field], + ) + .map(([field, binding]) => ({ + field, + label: + assetRefIds(binding) + .map((id) => assetNames[id] ?? id) + .join(", ") || binding, + })); } /** * Renders the parameter editor for one pipeline step, chosen by the tool's capability: * the tool's own settings UI when editable, an explanatory note when it has no parameters, - * or a "not supported yet" fallback for tools not yet migrated to the mapper seam. + * or a "not supported yet" fallback for tools not yet migrated to the mapper seam. Reopened + * supporting files appear as removable chips above the tool's own settings. */ export function PipelineStepSettings({ step, registry, onChange, + assetNames, + onClearBinding, }: PipelineStepSettingsProps) { // Hooks first: selecting a different step re-renders this same instance, so an early return // above useTranslation would change the hook count between renders and crash. @@ -52,41 +103,70 @@ export function PipelineStepSettings({ ); } - if (step.support === "noSettings") { - return ( - - ); - } + const chips = storedFileChips(step, registry, assetNames); - const entry = step.toolId ? registry[step.toolId] : undefined; - const Settings = - step.support === "editable" ? entry?.automationSettings : null; - - if (!Settings) { + function toolBody() { + if (step.support === "noSettings") { + return ( + + ); + } + const entry = step.toolId ? registry[step.toolId] : undefined; + const Settings = + step.support === "editable" ? entry?.automationSettings : null; + if (!Settings) { + return ( + + ); + } return ( - + + + + + onChange((prev) => ({ ...prev, [key]: value })) + } + disabled={false} + /> + + + ); } return ( - - - - - onChange((prev) => ({ ...prev, [key]: value })) - } - disabled={false} - /> - - - + <> + {chips.length > 0 && ( +
+ + {t("portal.pipelines.builder.supportingFiles")} + +
+ {chips.map((chip) => ( + + } + onRemove={() => onClearBinding(chip.field)} + > + {chip.label} + + ))} +
+
+ )} + {toolBody()} + ); } diff --git a/frontend/editor/src/portal/mocks/handlers/pipelines.ts b/frontend/editor/src/portal/mocks/handlers/pipelines.ts index ed7cec1626..c5505f38ff 100644 --- a/frontend/editor/src/portal/mocks/handlers/pipelines.ts +++ b/frontend/editor/src/portal/mocks/handlers/pipelines.ts @@ -116,6 +116,21 @@ function nextId(): string { return `plc_${Date.now().toString(36)}_${idCounter}`; } +/** Stored supporting files a step binds as `asset:` (PolicyAssetController), for mock mode. */ +interface StoredAsset { + id: string; + fileName: string; + contentType: string | null; + size: number; + createdAt: number; +} +let assetStore: StoredAsset[] = []; +let assetCounter = 0; +function nextAssetId(): string { + assetCounter += 1; + return `ast_${Date.now().toString(36)}_${assetCounter}`; +} + function deriveStatus(policy: StoredPolicy): PipelineStatus { return policy.enabled ? "active" : "paused"; } @@ -197,6 +212,33 @@ export const pipelinesHandlers = [ ]); }), + // Supporting files. Registered before the `/policies/:id` matcher so "assets" isn't read as an id. + http.get("/api/v1/policies/assets", async () => { + await delay(80); + return HttpResponse.json(assetStore); + }), + + http.post("/api/v1/policies/assets", async ({ request }) => { + const form = await request.formData(); + const file = form.get("file"); + if (!(file instanceof File)) { + return HttpResponse.json( + { detail: "Uploaded file is empty" }, + { status: 400 }, + ); + } + await delay(120); + const asset: StoredAsset = { + id: nextAssetId(), + fileName: file.name || "asset", + contentType: file.type || null, + size: file.size, + createdAt: Date.now(), + }; + assetStore = [...assetStore, asset]; + return HttpResponse.json(asset); + }), + // Run status: the mock completes runs immediately, so polling resolves at once. http.get("/api/v1/policies/run/:runId", async ({ params }) => { await delay(120); diff --git a/frontend/editor/src/portal/views/PipelineBuilder.test.tsx b/frontend/editor/src/portal/views/PipelineBuilder.test.tsx index 52fb5d516c..2737411c74 100644 --- a/frontend/editor/src/portal/views/PipelineBuilder.test.tsx +++ b/frontend/editor/src/portal/views/PipelineBuilder.test.tsx @@ -43,6 +43,13 @@ vi.mock("@portal/api/pipelines", () => ({ fetchRun: (runId: string) => fetchRun(runId), })); +const uploadPipelineAsset = vi.fn(); +const listPipelineAssets = vi.fn(); +vi.mock("@portal/api/pipelineAssets", () => ({ + uploadPipelineAsset: (file: File) => uploadPipelineAsset(file), + listPipelineAssets: () => listPipelineAssets(), +})); + const fetchSources = vi.fn(); vi.mock("@portal/api/sources", () => ({ fetchSources: () => fetchSources(), @@ -149,8 +156,21 @@ vi.mock("@app/contexts/ToolRegistryContext", () => { toolType: 0, endpoint: "/api/v1/misc/compress-pdf", defaultParameters: {}, - buildFormData: () => new FormData(), - toApiParams: (params: Record) => ({ ...params }), + // Sends the supporting file under a named field, like a real file tool, so the upload path + // has a field to bind. The scalar mapper drops the File (files never ride in parameters). + buildFormData: (params: Record, file: File | File[]) => { + const fd = new FormData(); + fd.append("fileInput", Array.isArray(file) ? file[0] : file); + if (params.watermarkImage instanceof File) { + fd.append("watermarkImage", params.watermarkImage); + } + return fd; + }, + toApiParams: (params: Record) => { + const scalars = { ...params }; + delete scalars.watermarkImage; + return scalars; + }, fromApiParams: (params: Record) => ({ ...params }), }, } as unknown as ToolRegistryEntry; @@ -203,10 +223,32 @@ vi.mock("@app/contexts/ToolRegistryContext", () => { fromApiParams: (params: Record) => ({ ...params }), }, } as unknown as ToolRegistryEntry; + // A tool whose buildFormData throws, so it can't be probed: exercises the "activeFileFields is + // null" path where a reopened step's stored binding must be kept, not dropped. + const sign = { + name: "Sign", + icon: null, + component: null, + description: "", + categoryId: "recommendedTools", + subcategoryId: "general", + operationConfig: { + operationType: "certSign", + toolType: 0, + endpoint: "/api/v1/security/cert-sign", + defaultParameters: {}, + buildFormData: () => { + throw new Error("cannot build"); + }, + toApiParams: (params: Record) => ({ ...params }), + fromApiParams: (params: Record) => ({ ...params }), + }, + } as unknown as ToolRegistryEntry; const allTools = { compress, extractImages, ocr, + sign, } as unknown as ToolRegistryCatalog["allTools"]; const catalog: ToolRegistryCatalog = { regularTools: allTools, @@ -288,6 +330,16 @@ describe("PipelineBuilder", () => { fetchS3Connections.mockReset(); fetchS3Connections.mockResolvedValue([]); createIntegration.mockReset(); + uploadPipelineAsset.mockReset(); + uploadPipelineAsset.mockResolvedValue({ + id: "ast-1", + fileName: "logo.png", + contentType: "image/png", + size: 1, + createdAt: 0, + }); + listPipelineAssets.mockReset(); + listPipelineAssets.mockResolvedValue([]); }); // The settings of a node are reached by selecting it in the graph, so every helper below opens @@ -798,7 +850,7 @@ describe("PipelineBuilder", () => { ).toBeInTheDocument(); }); - it("blocks saving a step that needs an uploaded file", async () => { + it("uploads a step's supporting file and saves it as an asset binding", async () => { renderBuilder("/processor/pipelines/new"); fireEvent.change( @@ -810,15 +862,65 @@ describe("PipelineBuilder", () => { }, ); await addTool("Compress"); - // The tool's settings upload a file, which a stored pipeline can't persist yet. + // The tool's settings attach a supporting file. fireEvent.click(await screen.findByText("upload logo")); - expect( - await screen.findByText("portal.pipelines.builder.uploadUnsupported"), - ).toBeInTheDocument(); - expect( - screen.getByText("portal.pipelines.composer.create").closest("button"), - ).toBeDisabled(); + await pickInputSource("Claims intake"); + await pickDestination(); + + fireEvent.click(screen.getByText("portal.pipelines.composer.create")); + + // The file is uploaded to the asset store first, then the policy is saved binding that asset. + await waitFor(() => expect(uploadPipelineAsset).toHaveBeenCalledTimes(1)); + expect(uploadPipelineAsset.mock.calls[0][0]).toBeInstanceOf(File); + await waitFor(() => expect(savePipeline).toHaveBeenCalledTimes(1)); + expect(savePipeline).toHaveBeenCalledWith( + expect.objectContaining({ + steps: [ + expect.objectContaining({ + operation: "/api/v1/misc/compress-pdf", + fileParameters: { watermarkImage: "asset:ast-1" }, + }), + ], + }), + ); + }); + + it("keeps a step's stored file binding on save when the tool can't be probed", async () => { + // buildFormData throws for `sign`, so activeFileFields is null. The stored binding must survive + // the save unchanged - dropping it would let the server GC the user's uploaded file - and no + // re-upload should happen. + fetchPipeline.mockResolvedValue({ + id: "plc-sign", + name: "Signed", + enabled: true, + inputs: [{ sourceId: "src-in", trigger: null }], + steps: [ + { + operation: "/api/v1/security/cert-sign", + parameters: {}, + fileParameters: { certFile: "asset:x" }, + }, + ], + output: { type: "inline", options: {} }, + outputIds: ["src-1"], + }); + renderBuilder("/processor/pipelines/plc-sign"); + + fireEvent.click(await screen.findByText("portal.pipelines.composer.save")); + + await waitFor(() => expect(savePipeline).toHaveBeenCalledTimes(1)); + expect(savePipeline).toHaveBeenCalledWith( + expect.objectContaining({ + steps: [ + expect.objectContaining({ + operation: "/api/v1/security/cert-sign", + fileParameters: { certFile: "asset:x" }, + }), + ], + }), + ); + expect(uploadPipelineAsset).not.toHaveBeenCalled(); }); it("blocks saving an integration step with no account chosen", async () => { diff --git a/frontend/editor/src/portal/views/PipelineBuilder.tsx b/frontend/editor/src/portal/views/PipelineBuilder.tsx index d5092f5c88..975a8e5cfe 100644 --- a/frontend/editor/src/portal/views/PipelineBuilder.tsx +++ b/frontend/editor/src/portal/views/PipelineBuilder.tsx @@ -17,14 +17,17 @@ import { } from "@app/ui"; import { useToolRegistry } from "@app/contexts/ToolRegistryContext"; import { + activeFileFields, + assetRef, deserializeToolStep, + extractStepFiles, getExecutableTools, newWorkingToolStep, serializeToolStep, stepNeedsConfiguring, - stepRequiresUpload, updateWorkingStepParams, type ExecutableTool, + type SupportingFileBindings, type WorkingToolStep, } from "@app/hooks/tools/shared/toolAutomation"; import { @@ -48,13 +51,20 @@ import { runPipelineTest, savePipeline, triggerPipeline, + type PipelineStep, type Policy, type PolicyRunView, type RunOutputFile, + type TestRunAsset, type TriggerConfig, type TriggerInfo, type TriggerOutcome, } from "@portal/api/pipelines"; +import { + listPipelineAssets, + uploadPipelineAsset, + type PolicyAsset, +} from "@portal/api/pipelineAssets"; import { clearProcessedHistory } from "@portal/api/policies"; import { DestinationPicker } from "@portal/components/pipelines/DestinationPicker"; import { availableOutputModes } from "@portal/components/pipelines/outputModes"; @@ -207,6 +217,17 @@ export function PipelineBuilder() { [allTools], ); + // Stored supporting files from earlier saves, so a reopened step can label its bindings by name. + const assetsState = useAsync( + async () => await listPipelineAssets(), + [], + ); + const assetNames = useMemo(() => { + const map: Record = {}; + for (const asset of assetsState.data ?? []) map[asset.id] = asset.fileName; + return map; + }, [assetsState.data]); + const policyState = useAsync( async () => (id ? await fetchPipeline(id) : null), [id], @@ -486,6 +507,21 @@ export function PipelineBuilder() { ); } + /** Drop a step's stored supporting-file binding for one field (the chip's remove action). */ + function clearStepBinding(index: number, field: string) { + setSteps((current) => + current.map((step, i) => { + if (i !== index || !step.fileParameters) return step; + const next = { ...step.fileParameters }; + delete next[field]; + return { + ...step, + fileParameters: Object.keys(next).length > 0 ? next : undefined, + }; + }), + ); + } + function stepLabel(step: WorkingToolStep): string { // An integration step's endpoint is the same for every vendor, so the raw path would read // "External api call" for all of them. Name it by the operation instead. @@ -512,11 +548,6 @@ export function PipelineBuilder() { return step.toolId ? allTools[step.toolId]?.icon : undefined; } - // Steps whose params carry an uploaded file can't be saved: the bytes aren't persisted with the - // policy, so a later run would send null for that field (see stepRequiresUpload). - const uploadStepLabels = steps.filter(stepRequiresUpload).map(stepLabel); - const hasUploadSteps = uploadStepLabels.length > 0; - // A step still missing a choice - an integration with no operation or account, a tool whose // mandatory parameters are unset - would fail at run time with a raw backend rejection, so block // saving on it here where the fix is one click away. @@ -601,11 +632,30 @@ export function PipelineBuilder() { // seeding, so leaving the builder can prompt to save or discard. `enabled` is deliberately left // out: in edit it is toggled and persisted at once (never an unsaved edit), and in create it is // chosen at submit - so it can never be the thing that makes the form dirty. + // Per-step dirty signature: the serialized step plus a stable identity (name/size/mtime) of its + // fresh file picks - a raw File JSON-stringifies to `{}`, so serializeToolStep (which excludes + // Files) can't see a file added or swapped. Memoized on the steps because it probes each tool's + // buildFormData; without this it would re-run for every step on any render (e.g. each keystroke in + // the name field). Stored bindings are covered by the serialized step. + const stepSnapshot = useMemo( + () => + steps.map((step) => { + const files: Record = {}; + for (const [field, picks] of Object.entries( + extractStepFiles(step, allTools), + )) { + files[field] = picks.map( + (file) => `${file.name}:${file.size}:${file.lastModified}`, + ); + } + return { step: serializeToolStep(step, allTools), files }; + }), + [steps, allTools], + ); const snapshot = JSON.stringify({ name: name.trim(), input, - steps: steps.map((step) => serializeToolStep(step, allTools)), - uploads: steps.map(stepRequiresUpload), + steps: stepSnapshot, outputIds: [...outputIds].sort(), }); const baseline = useRef(null); @@ -639,12 +689,6 @@ export function PipelineBuilder() { tools: unconfiguredStepLabels.join(", "), }), ); - if (hasUploadSteps) - blockers.push( - t("portal.pipelines.builder.blocker.upload", { - tools: uploadStepLabels.join(", "), - }), - ); if (hasIncompatibleSteps) blockers.push( t("portal.pipelines.builder.blocker.incompatible", { @@ -667,23 +711,84 @@ export function PipelineBuilder() { else navigate(destination); } + /** + * The active supporting-file fields of a step, each paired with its fresh in-memory pick(s) and its + * stored `asset:` binding (either may be absent). The single source both saving and test-running + * read, so the two agree on which fields are active and how a binding is chosen; they differ only in + * how a fresh pick is emitted - uploaded as an asset vs. sent inline. + */ + function stepFileFields( + step: WorkingToolStep, + ): { field: string; fresh: File[] | null; stored: string | null }[] { + const fresh = extractStepFiles(step, allTools); + const stored = step.fileParameters ?? {}; + const fields = activeFileFields(step, allTools) ?? Object.keys(stored); + return fields.map((field) => ({ + field, + fresh: fresh[field] ?? null, + stored: stored[field] ?? null, + })); + } + + /** A wire step, attaching fileParameters only when it has any. */ + function toWireStep( + operation: string, + parameters: Record, + bindings: SupportingFileBindings, + ): PipelineStep { + return Object.keys(bindings).length > 0 + ? { operation, parameters, fileParameters: bindings } + : { operation, parameters }; + } + + /** + * The wire steps for saving: scalar params from serialization, plus supporting-file bindings. A + * fresh pick is uploaded to the asset store (the save-time validator rejects a policy that binds an + * asset id that doesn't yet exist); a stored binding the tool still uses is kept when the user + * didn't replace it. Uploads run in parallel; any abandoned by a later failure are GC'd server-side. + */ + async function serializeStepsForSave(): Promise { + return Promise.all( + steps.map(async (step) => { + const { operation, parameters } = serializeToolStep(step, allTools); + const entries = await Promise.all( + stepFileFields(step).map(async ({ field, fresh, stored }) => { + if (fresh?.length) { + const ids = await Promise.all( + fresh.map((file) => + uploadPipelineAsset(file).then((a) => a.id), + ), + ); + return [field, assetRef(ids)] as const; + } + return stored ? ([field, stored] as const) : null; + }), + ); + const bindings: SupportingFileBindings = Object.fromEntries( + entries.filter((e): e is readonly [string, string] => e !== null), + ); + return toWireStep(operation, parameters, bindings); + }), + ); + } + async function save(destination: string, enabledOverride?: boolean) { if (!canSave) return; setSubmitting(true); setError(null); - const policy: Policy = { - id: policyState.data?.id ?? undefined, - name: name.trim(), - enabled: enabledOverride ?? enabled, - // The wire shape stays a list; canSave guarantees the one input has a source. - inputs: [{ sourceId: input.sourceId, trigger: buildTriggerFor(input) }], - steps: steps.map((step) => serializeToolStep(step, allTools)), - // Destinations are the referenced saved sources; the inline output field is - // preserved as-is (e.g. an editor policy's membership metadata) or defaults to inline. - output: policyState.data?.output ?? { type: "inline", options: {} }, - outputIds, - }; try { + const policy: Policy = { + id: policyState.data?.id ?? undefined, + name: name.trim(), + enabled: enabledOverride ?? enabled, + // The wire shape stays a list; canSave guarantees the one input has a source. + inputs: [{ sourceId: input.sourceId, trigger: buildTriggerFor(input) }], + steps: await serializeStepsForSave(), + // Destinations are the referenced saved sources; the inline output field is + // preserved as-is (e.g. an editor policy's membership metadata) or defaults to inline. + output: policyState.data?.output ?? { type: "inline", options: {} }, + outputIds, + }; await savePipeline(policy); await invalidatePipelines(); navigate(destination); @@ -741,6 +846,32 @@ export function PipelineBuilder() { return null; } + /** + * The steps + inline supporting files for a test run. A fresh (in-memory) pick rides along as a + * keyed `assets[i]` under a per-step run key; a stored file keeps its `asset:` binding, which + * the backend resolves from the pipeline's saved policy (passed as policyId) - no re-fetch needed. + */ + function buildTestSteps(): { steps: PipelineStep[]; assets: TestRunAsset[] } { + const assets: TestRunAsset[] = []; + const outSteps = steps.map((step, i) => { + const { operation, parameters } = serializeToolStep(step, allTools); + const bindings: SupportingFileBindings = {}; + for (const { field, fresh, stored } of stepFileFields(step)) { + if (fresh?.length) { + // In-memory pick: inline the bytes under a run key. + const key = `s${i}_${field}`; + bindings[field] = key; + for (const file of fresh) assets.push({ key, file }); + } else if (stored) { + // Already an asset: keep its ref for the backend to resolve from the saved policy. + bindings[field] = stored; + } + } + return toWireStep(operation, parameters, bindings); + }); + return { steps: outSteps, assets }; + } + /** * Run the steps as they stand against one uploaded file. Output is forced inline so nothing * reaches the pipeline's real destination, and the pipeline need not be saved first - this is @@ -752,13 +883,17 @@ export function PipelineBuilder() { setTestRun(null); setRunResult(null); try { + const { steps: testSteps, assets } = buildTestSteps(); const { runId } = await runPipelineTest( { name: name.trim() || t("portal.pipelines.builder.testRun"), - steps: steps.map((step) => serializeToolStep(step, allTools)), + steps: testSteps, output: { type: "inline", options: {} }, }, file, + assets, + // Lets the backend resolve any stored `asset:` refs from this saved policy. + policyState.data?.id, ); const final = await awaitRun(runId, (view) => { if (mounted.current) setTestRun(view); @@ -934,8 +1069,6 @@ export function PipelineBuilder() { return t("portal.pipelines.builder.chooseAccount"); return undefined; } - if (stepRequiresUpload(step)) - return t("portal.pipelines.builder.needsUpload"); if (stepNeedsConfiguring(step, allTools)) return t("portal.pipelines.builder.needsConfiguring"); return undefined; @@ -1120,6 +1253,8 @@ export function PipelineBuilder() { step={selectedStep} registry={allTools} onChange={(params) => updateStepParams(chosenSteps[0], params)} + assetNames={assetNames} + onClearBinding={(field) => clearStepBinding(chosenSteps[0], field)} /> ); } @@ -1165,14 +1300,6 @@ export function PipelineBuilder() { {runResult && ( )} - {hasUploadSteps && ( - - )} {hasUnconfiguredSteps && ( Date: Thu, 20 Aug 2026 09:33:26 +0000 Subject: [PATCH 31/59] Float the editor search when no file is open (#7575) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What The super-search work pinned the editor's `WorkbenchBar` visible on every view except My Files, even with no file open. That left an empty workbench showing a fully painted bar whose only live control was the search — download / close / print / save were all disabled, because those actions only make sense with a file open. This stops forcing the bar. When nothing is open, only the global search floats (unpainted, centered), mirroring how the Processor already works. When a file **is** open, the `WorkbenchBar` renders exactly as before. Also fixes a smaller Processor issue: its floating search strip was shorter than the sidebar logo row, so the search sat higher than the brand. Its height now matches the logo row (51px) so they line up. ## Changes - **`Workbench.tsx`** — render the `WorkbenchBar` only when a file is open (or a custom view supplies content); otherwise render the new floating search. My Files and `hideTopControls` custom views are unchanged. - **`WorkbenchFloatingSearch.tsx` / `.css`** (new) — the editor's `SuperSearch` floated in an unpainted strip, mirroring `PortalSearchBar`. Its vertical band matches the bar's so opening a file swaps in the bar without a shift. - **`PortalSearchBar.css`** — strip height matched to `.portal-sidebar__logo` (51px) so the Processor search aligns with the logo. The notification bell is intentionally out of scope — it ships in a separate PR. ## Before / after (ignore the bell icon in the after that’s not live yet) Screenshot 2026-08-20 at 2 28
17 AM Screenshot 2026-08-20 at 2 28
31 AM - **Editor, no file:** painted bar with disabled buttons → just a floating search. - **Editor, file open:** unchanged. - **Processor:** search now vertically aligned with the logo. ## Testing - `task frontend:check` — lint (incl. colour linters) + typecheck + tests (247 files / 2137 tests) all pass. - Processor alignment verified in Storybook (`Portal/Shell/AppShell`): logo row, search strip, and search pill share the same vertical center. - Editor float not verified in-browser (local backend is behind a login gate); covered by types/tests and reuses the verified Processor pattern. --- .../src/core/components/layout/Workbench.tsx | 77 +++++++++++-------- .../shared/WorkbenchFloatingSearch.css | 31 ++++++++ .../shared/WorkbenchFloatingSearch.tsx | 15 ++++ .../src/portal/components/PortalSearchBar.css | 7 +- 4 files changed, 94 insertions(+), 36 deletions(-) create mode 100644 frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.css create mode 100644 frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.tsx diff --git a/frontend/editor/src/core/components/layout/Workbench.tsx b/frontend/editor/src/core/components/layout/Workbench.tsx index 7dbc6f75be..8bc0794140 100644 --- a/frontend/editor/src/core/components/layout/Workbench.tsx +++ b/frontend/editor/src/core/components/layout/Workbench.tsx @@ -18,6 +18,7 @@ import { useCookieConsent } from "@app/hooks/useCookieConsent"; import styles from "@app/components/layout/Workbench.module.css"; import WorkbenchBar from "@app/components/shared/WorkbenchBar"; +import WorkbenchFloatingSearch from "@app/components/shared/WorkbenchFloatingSearch"; import LandingPage from "@app/components/shared/LandingPage"; import DismissAllErrorsButton from "@app/components/shared/DismissAllErrorsButton"; import { ChatFAB } from "@app/components/chat/ChatFAB"; @@ -77,6 +78,22 @@ export default function Workbench() { const [viewerToolbarCollapsed, setViewerToolbarCollapsed] = useState(false); const showReopenTab = currentView === "viewer" && viewerToolbarCollapsed; + // The WorkbenchBar carries file-scoped actions, so it only shows once a file + // is open or a custom view supplies content; otherwise the search floats. + const activeCustomView = customWorkbenchViews.find( + (v) => v.workbenchId === currentView, + ); + const topControlsAvailable = + currentView !== "myFiles" && !activeCustomView?.hideTopControls; + const hasWorkbenchContent = + hasFiles || + fileIds.length > 0 || + !isBaseWorkbench(currentView) || + // Shared signing drives the viewer from the sidebar with no file in context. + (currentView === "viewer" && !!signingOverlay?.file); + const showWorkbenchBar = topControlsAvailable && hasWorkbenchContent; + const showFloatingSearch = topControlsAvailable && !hasWorkbenchContent; + const handlePreviewClose = () => { setPreviewFile(null); const previousMode = sessionStorage.getItem("previousMode"); @@ -231,41 +248,35 @@ export default function Workbench() { data-tour="workbench" style={{ backgroundColor: "var(--c-bg)", minWidth: 0 }} > - {/* Workbench Bar — always visible outside My Files (it hosts the - global search), even with no files loaded. */} - {currentView !== "myFiles" && - !customWorkbenchViews.find((v) => v.workbenchId === currentView) - ?.hideTopControls && ( -
-
-
- -
-
- {/* Reopen tab: a little handle hanging off the bar's bottom-right - while the viewer tool row is retracted. */} - {showReopenTab && ( -
+ )} + {showFloatingSearch && } {/* Dismiss All Errors Button */} diff --git a/frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.css b/frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.css new file mode 100644 index 0000000000..47c7e9eed4 --- /dev/null +++ b/frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.css @@ -0,0 +1,31 @@ +/* Unpainted floating search for the empty workbench. Height + top margin match + the WorkbenchBar's band so opening a file swaps it in without a shift. */ +.workbench-floating-search { + display: flex; + align-items: center; + justify-content: center; + min-height: 38px; + margin-top: var(--nav-gutter); + padding: 0 1rem; + flex-shrink: 0; +} + +.workbench-floating-search .super-search { + flex: 0 1 24rem; + width: min(100%, 24rem); + max-width: 24rem; +} + +.workbench-floating-search .super-search input { + background-color: transparent; + padding-top: 4px; + padding-bottom: 4px; + font-size: 12.5px; +} + +[data-mantine-color-scheme="dark"] + .workbench-floating-search + .super-search + input { + background-color: transparent; +} diff --git a/frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.tsx b/frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.tsx new file mode 100644 index 0000000000..37e858adff --- /dev/null +++ b/frontend/editor/src/core/components/shared/WorkbenchFloatingSearch.tsx @@ -0,0 +1,15 @@ +import SuperSearch from "@app/components/shared/superSearch/SuperSearch"; +import { useEditorSearchScopes } from "@app/hooks/useSuperSearch"; +import "@app/components/shared/WorkbenchFloatingSearch.css"; + +// The editor's global search, floated while no file is open (mirrors the +// processor's PortalSearchBar). Renders only when the WorkbenchBar doesn't, so +// reusing the default input id is safe. +export default function WorkbenchFloatingSearch() { + const scopes = useEditorSearchScopes(); + return ( +
+ +
+ ); +} diff --git a/frontend/editor/src/portal/components/PortalSearchBar.css b/frontend/editor/src/portal/components/PortalSearchBar.css index cf1d9971c7..8e253b6136 100644 --- a/frontend/editor/src/portal/components/PortalSearchBar.css +++ b/frontend/editor/src/portal/components/PortalSearchBar.css @@ -1,10 +1,11 @@ -/* Slim strip at the top of the main column hosting the shared search bar. - Deliberately unpainted: only the input itself shows, on the page ground. */ +/* Unpainted strip at the top of the main column. Height matches the sidebar's + logo row (.portal-sidebar__logo, 51px) so the search lines up with the brand. */ .portal-searchbar { display: flex; align-items: center; justify-content: center; - padding: 0.22rem 1rem; + min-height: 3.1875rem; + padding: 0 1rem; flex-shrink: 0; } From 91fc26f10c21690af6d7c78c7f247ea1d5acfcaa Mon Sep 17 00:00:00 2001 From: Ludy Date: Thu, 20 Aug 2026 09:53:28 +0000 Subject: [PATCH 32/59] chore: Bump version to 2.14.3 (#7554) # Description of Changes This PR bumps the Stirling PDF application version from `2.14.2` to `2.14.3` across the project. Changes include: - Updated the Gradle project version in `build.gradle` to `2.14.3`. - Updated the Tauri desktop application version in `frontend/editor/src-tauri/tauri.conf.json`. - Updated the AUR package version for `stirling-pdf-desktop`. - Updated the AUR package version for `stirling-pdf-server-bin`. - Updated the mocked `appVersion` used by the core frontend server experience simulations. - Updated the mocked `appVersion` used by the proprietary frontend server experience simulations. - Kept all application, desktop, packaging, and test/simulation version references synchronized for the `2.14.3` release. The change prepares the project metadata and packaging configuration for the `2.14.3` release and prevents different components from reporting or packaging the previous `2.14.2` version. --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details. --- .github/aur/stirling-pdf-desktop/PKGBUILD | 2 +- .github/aur/stirling-pdf-server-bin/PKGBUILD | 2 +- build.gradle | 2 +- frontend/editor/src-tauri/tauri.conf.json | 2 +- frontend/editor/src/core/testing/serverExperienceSimulations.ts | 2 +- .../src/proprietary/testing/serverExperienceSimulations.ts | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/aur/stirling-pdf-desktop/PKGBUILD b/.github/aur/stirling-pdf-desktop/PKGBUILD index fb6a99cfca..c47c8e3ee7 100644 --- a/.github/aur/stirling-pdf-desktop/PKGBUILD +++ b/.github/aur/stirling-pdf-desktop/PKGBUILD @@ -1,6 +1,6 @@ # Maintainer: Stirling PDF Inc pkgname=stirling-pdf-desktop -pkgver=2.14.2 +pkgver=2.14.3 pkgrel=1 pkgdesc="Locally hosted, web-based PDF manipulation tool (Tauri desktop app, official Stirling PDF Inc build)" arch=('x86_64') diff --git a/.github/aur/stirling-pdf-server-bin/PKGBUILD b/.github/aur/stirling-pdf-server-bin/PKGBUILD index 70bcee0423..d6159ddce3 100644 --- a/.github/aur/stirling-pdf-server-bin/PKGBUILD +++ b/.github/aur/stirling-pdf-server-bin/PKGBUILD @@ -1,6 +1,6 @@ # Maintainer: Stirling PDF Inc pkgname=stirling-pdf-server-bin -pkgver=2.14.2 +pkgver=2.14.3 pkgrel=1 pkgdesc="Locally hosted, web-based PDF manipulation tool (server JAR, prebuilt)" arch=('any') diff --git a/build.gradle b/build.gradle index cadf4bafa6..30a9123ebb 100644 --- a/build.gradle +++ b/build.gradle @@ -108,7 +108,7 @@ springBoot { allprojects { group = 'stirling.software' - version = '2.14.2' + version = '2.14.3' configurations.configureEach { exclude group: "org.springframework.boot", module: "spring-boot-starter-tomcat" diff --git a/frontend/editor/src-tauri/tauri.conf.json b/frontend/editor/src-tauri/tauri.conf.json index dee2cc7023..c308e3ad4e 100644 --- a/frontend/editor/src-tauri/tauri.conf.json +++ b/frontend/editor/src-tauri/tauri.conf.json @@ -2,7 +2,7 @@ "$schema": "../node_modules/@tauri-apps/cli/config.schema.json", "productName": "Stirling PDF", "mainBinaryName": "Stirling-PDF", - "version": "2.14.2", + "version": "2.14.3", "identifier": "stirling.pdf.dev", "build": { "frontendDist": "../dist", diff --git a/frontend/editor/src/core/testing/serverExperienceSimulations.ts b/frontend/editor/src/core/testing/serverExperienceSimulations.ts index 97da95c730..55ab79c111 100644 --- a/frontend/editor/src/core/testing/serverExperienceSimulations.ts +++ b/frontend/editor/src/core/testing/serverExperienceSimulations.ts @@ -38,7 +38,7 @@ const FREE_LICENSE_INFO: LicenseInfo = { const BASE_NO_LOGIN_CONFIG: AppConfig = { enableAnalytics: true, - appVersion: "2.14.2", + appVersion: "2.14.3", serverCertificateEnabled: false, enableAlphaFunctionality: false, serverPort: 8080, diff --git a/frontend/editor/src/proprietary/testing/serverExperienceSimulations.ts b/frontend/editor/src/proprietary/testing/serverExperienceSimulations.ts index d92cf8fdf9..0519b9afb8 100644 --- a/frontend/editor/src/proprietary/testing/serverExperienceSimulations.ts +++ b/frontend/editor/src/proprietary/testing/serverExperienceSimulations.ts @@ -48,7 +48,7 @@ const FREE_LICENSE_INFO: LicenseInfo = { const BASE_NO_LOGIN_CONFIG: AppConfig = { enableAnalytics: true, - appVersion: "2.14.2", + appVersion: "2.14.3", serverCertificateEnabled: false, enableAlphaFunctionality: false, enableDesktopInstallSlide: true, From 4791d558c515c08bfbb954bb2af5315a68953b93 Mon Sep 17 00:00:00 2001 From: "stirlingbot[bot]" <195170888+stirlingbot[bot]@users.noreply.github.com> Date: Thu, 20 Aug 2026 11:16:18 +0000 Subject: [PATCH 33/59] Update Backend 3rd Party Licenses (#7579) Auto-generated by stirlingbot[bot] This PR updates the backend license report based on dependency changes. Signed-off-by: stirlingbot[bot] Co-authored-by: stirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com> --- .../resources/static/3rdPartyLicenses.json | 226 ++++++++++++++---- 1 file changed, 185 insertions(+), 41 deletions(-) diff --git a/app/core/src/main/resources/static/3rdPartyLicenses.json b/app/core/src/main/resources/static/3rdPartyLicenses.json index fa852846b3..fbdce0a158 100644 --- a/app/core/src/main/resources/static/3rdPartyLicenses.json +++ b/app/core/src/main/resources/static/3rdPartyLicenses.json @@ -14,6 +14,13 @@ "moduleLicense": "GNU Lesser General Public License", "moduleLicenseUrl": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html" }, + { + "moduleName": "ch.qos.logback:logback-classic", + "moduleUrl": "http://www.qos.ch", + "moduleVersion": "1.6.1", + "moduleLicense": "LGPL-2.1-only", + "moduleLicenseUrl": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html" + }, { "moduleName": "ch.qos.logback:logback-core", "moduleUrl": "http://www.qos.ch", @@ -21,6 +28,13 @@ "moduleLicense": "GNU Lesser General Public License", "moduleLicenseUrl": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html" }, + { + "moduleName": "ch.qos.logback:logback-core", + "moduleUrl": "http://www.qos.ch", + "moduleVersion": "1.6.1", + "moduleLicense": "LGPL-2.1-only", + "moduleLicenseUrl": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html" + }, { "moduleName": "com.adobe.xmp:xmpcore", "moduleUrl": "https://www.adobe.com/devnet/xmp/library/eula-xmp-library-java.html", @@ -182,7 +196,7 @@ { "moduleName": "com.github.mwiede:jsch", "moduleUrl": "https://github.com/mwiede/jsch", - "moduleVersion": "0.2.23", + "moduleVersion": "2.28.6", "moduleLicense": "Revised BSD", "moduleLicenseUrl": "https://github.com/mwiede/jsch/blob/master/LICENSE.txt" }, @@ -758,7 +772,7 @@ { "moduleName": "commons-net:commons-net", "moduleUrl": "https://commons.apache.org/proper/commons-net/", - "moduleVersion": "3.11.1", + "moduleVersion": "3.13.0", "moduleLicense": "Apache-2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt" }, @@ -1213,24 +1227,48 @@ "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "net.shibboleth:shib-networking", + "moduleVersion": "9.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "net.shibboleth:shib-security", "moduleVersion": "9.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "net.shibboleth:shib-security", + "moduleVersion": "9.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "net.shibboleth:shib-support", "moduleVersion": "9.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "net.shibboleth:shib-support", + "moduleVersion": "9.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "net.shibboleth:shib-velocity", "moduleVersion": "9.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "net.shibboleth:shib-velocity", + "moduleVersion": "9.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.antlr:antlr4-runtime", "moduleUrl": "https://www.antlr.org/", @@ -1325,13 +1363,6 @@ "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt" }, - { - "moduleName": "org.apache.httpcomponents:httpclient", - "moduleUrl": "http://hc.apache.org/httpcomponents-client", - "moduleVersion": "4.5.13", - "moduleLicense": "Apache License, Version 2.0", - "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" - }, { "moduleName": "org.apache.httpcomponents:httpclient", "moduleUrl": "http://hc.apache.org/httpcomponents-client-ga", @@ -1456,6 +1487,13 @@ "moduleLicense": "Apache-2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.apache.santuario:xmlsec", + "moduleUrl": "https://www.apache.org/", + "moduleVersion": "3.0.6", + "moduleLicense": "Apache-2.0", + "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.apache.tomcat.embed:tomcat-embed-el", "moduleUrl": "https://tomcat.apache.org/", @@ -1470,6 +1508,13 @@ "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.apache.velocity:velocity-engine-core", + "moduleUrl": "https://www.apache.org/", + "moduleVersion": "2.4.1", + "moduleLicense": "Apache-2.0", + "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.apache.xmlbeans:xmlbeans", "moduleUrl": "https://xmlbeans.apache.org/", @@ -1647,6 +1692,13 @@ "moduleLicense": "GNU Lesser General Public License", "moduleLicenseUrl": "http://www.gnu.org/licenses/lgpl-3.0.txt" }, + { + "moduleName": "org.cryptacular:cryptacular", + "moduleUrl": "https://www.cryptacular.org", + "moduleVersion": "1.3.0", + "moduleLicense": "GNU Lesser General Public License", + "moduleLicenseUrl": "https://www.gnu.org/licenses/lgpl-3.0.txt" + }, { "moduleName": "org.eclipse.angus:angus-activation", "moduleUrl": "https://www.eclipse.org", @@ -2016,78 +2068,156 @@ "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-core-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-core-impl", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-core-impl", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-messaging-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-messaging-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-profile-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-profile-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-saml-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-saml-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-saml-impl", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-saml-impl", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-security-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-security-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-security-impl", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-security-impl", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-soap-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-soap-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-soap-impl", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-soap-impl", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-storage-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-storage-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-xmlsec-api", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-xmlsec-api", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.opensaml:opensaml-xmlsec-impl", "moduleVersion": "5.1.6", "moduleLicense": "The Apache Software License, Version 2.0", "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" }, + { + "moduleName": "org.opensaml:opensaml-xmlsec-impl", + "moduleVersion": "5.2.2", + "moduleLicense": "The Apache Software License, Version 2.0", + "moduleLicenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.txt" + }, { "moduleName": "org.ow2.asm:asm", "moduleUrl": "http://asm.ow2.org", @@ -2564,6 +2694,13 @@ "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0" }, + { + "moduleName": "org.springframework.security:spring-security-core", + "moduleUrl": "https://spring.io/projects/spring-security", + "moduleVersion": "7.1.0", + "moduleLicense": "Apache License, Version 2.0", + "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0" + }, { "moduleName": "org.springframework.security:spring-security-crypto", "moduleUrl": "https://spring.io/projects/spring-security", @@ -2606,6 +2743,13 @@ "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0" }, + { + "moduleName": "org.springframework.security:spring-security-saml2-service-provider", + "moduleUrl": "https://spring.io/projects/spring-security", + "moduleVersion": "7.1.0", + "moduleLicense": "Apache License, Version 2.0", + "moduleLicenseUrl": "https://www.apache.org/licenses/LICENSE-2.0" + }, { "moduleName": "org.springframework.security:spring-security-web", "moduleUrl": "https://spring.io/projects/spring-security", @@ -2805,207 +2949,207 @@ }, { "moduleName": "software.amazon.awssdk:annotations", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { - "moduleName": "software.amazon.awssdk:apache-client", - "moduleVersion": "2.44.12", + "moduleName": "software.amazon.awssdk:apache5-client", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:arns", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:auth", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:aws-core", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:aws-query-protocol", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:aws-xml-protocol", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:checksums", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:checksums-spi", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:crt-core", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:endpoints-spi", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:http-auth", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:http-auth-aws", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:http-auth-aws-eventstream", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:http-auth-spi", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:http-client-spi", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:identity-spi", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:json-utils", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:metrics-spi", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:netty-nio-client", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:profiles", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:protocol-core", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:regions", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:retries", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:retries-spi", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:s3", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:sdk-core", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:third-party-jackson-core", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:url-connection-client", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:utils", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, { "moduleName": "software.amazon.awssdk:utils-lite", "moduleUrl": "https://aws.amazon.com/sdkforjava", - "moduleVersion": "2.44.12", + "moduleVersion": "2.51.3", "moduleLicense": "Apache License, Version 2.0", "moduleLicenseUrl": "https://aws.amazon.com/apache2.0" }, From 50d34fcca5e778ec9dd66ec5c6d2048a23a35a2e Mon Sep 17 00:00:00 2001 From: Anthony Stirling <77850077+Frooodle@users.noreply.github.com> Date: Thu, 20 Aug 2026 11:47:05 +0000 Subject: [PATCH 34/59] Add download, rename and duplicate to the file actions menu (#7536) # Description of Changes Adds expanded dropdown menu for download, rename and duplicate image image --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details. --- .../public/locales/en-US/translation.toml | 15 + .../core/components/filesPage/FileGrid.tsx | 552 +++++++++--------- .../components/filesPage/FileManagerView.tsx | 101 ++++ .../core/components/shared/FileSidebar.tsx | 137 +++++ .../components/shared/FileSidebarFileItem.css | 22 + .../components/shared/FileSidebarFileItem.tsx | 351 +++++++---- .../shared/RenameFileDialog.stories.tsx | 39 ++ .../components/shared/RenameFileDialog.tsx | 140 +++++ .../core/components/shared/WorkbenchBar.tsx | 6 +- .../editor/src/core/hooks/useFileHandler.ts | 4 + .../tests/stubbed/file-actions-menu.spec.ts | 167 ++++++ .../src/core/utils/duplicateFile.test.ts | 130 +++++ .../editor/src/core/utils/duplicateFile.ts | 85 +++ frontend/editor/src/core/utils/fileUtils.ts | 11 + 14 files changed, 1373 insertions(+), 387 deletions(-) create mode 100644 frontend/editor/src/core/components/shared/RenameFileDialog.stories.tsx create mode 100644 frontend/editor/src/core/components/shared/RenameFileDialog.tsx create mode 100644 frontend/editor/src/core/tests/stubbed/file-actions-menu.spec.ts create mode 100644 frontend/editor/src/core/utils/duplicateFile.test.ts create mode 100644 frontend/editor/src/core/utils/duplicateFile.ts diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml index 01db314031..c2287ace9c 100644 --- a/frontend/editor/public/locales/en-US/translation.toml +++ b/frontend/editor/public/locales/en-US/translation.toml @@ -3897,8 +3897,10 @@ collapse = "Collapse sidebar" customizeGroups = "Customize groups" dataLostBody = "This browser lost this file's contents. Upload it again to keep working with it." dataLostTitle = "File data is unavailable" +downloadFailed = "Download failed" dropHint = "Open files to get started" dropToAdd = "Drop files to add" +duplicateFailed = "Could not duplicate file" expand = "Expand sidebar" googleDrive = "Google Drive" googleDriveDisabled = "Google Drive is not configured" @@ -3918,8 +3920,10 @@ closeViewer = "Close viewer" dataLost = "Data lost" dataLostTooltip = "This browser lost this file's contents. Upload it again to keep working with it." delete = "Delete" +duplicate = "Duplicate" moreActions = "More actions" openInViewer = "Open in viewer" +rename = "Rename" savedToServer = "Saved to server" updateOnServer = "Update on server" uploadToServer = "Upload to server" @@ -3931,6 +3935,14 @@ reset = "Show all" subtitle = "Show or hide categories in the files sidebar." title = "Sidebar categories" +[fileSidebar.rename] +cancel = "Cancel" +error = "Could not rename the file." +illegalCharacters = "A file name can't contain \\ / : * ? \" < > |" +label = "File name" +save = "Rename" +title = "Rename file" + [filesPage] addToWorkspace = "Add to workspace" addToWorkspaceCount = "Add {{count}} to workspace" @@ -3972,6 +3984,7 @@ downloadAll = "Download all" downloadVersion = "Download this version" dropOverlay = "Drop files to upload" dropOverlaySub = "Files start in Local. Use 'Move to' or 'Save to cloud' to organize them into a folder." +duplicate = "Duplicate" file = "File" fileInfo = "File info" fileMenu = "File actions" @@ -4087,6 +4100,8 @@ cloudDeleteFailed_one = "Couldn't delete 1 file from the cloud." cloudDeleteFailed_other = "Couldn't delete {{count}} files from the cloud." deleteFolderFailed = "Could not delete folder." deleteFolderFailedDetail = "Could not delete folder: {{message}}" +downloadFailed = "Could not download the file." +duplicateFailed = "Could not duplicate the file." folderAppearanceFailed = "Could not update folder appearance." folderAppearanceFailedDetail = "Could not update folder appearance: {{message}}" moveFilesFailed = "Could not move files." diff --git a/frontend/editor/src/core/components/filesPage/FileGrid.tsx b/frontend/editor/src/core/components/filesPage/FileGrid.tsx index e54964c592..7fd2eda133 100644 --- a/frontend/editor/src/core/components/filesPage/FileGrid.tsx +++ b/frontend/editor/src/core/components/filesPage/FileGrid.tsx @@ -13,6 +13,7 @@ import DeleteIcon from "@mui/icons-material/Delete"; import HistoryIcon from "@mui/icons-material/History"; import OpenInNewIcon from "@mui/icons-material/OpenInNew"; import DriveFileRenameOutlineIcon from "@mui/icons-material/DriveFileRenameOutline"; +import ContentCopyOutlinedIcon from "@mui/icons-material/ContentCopyOutlined"; import CloudUploadIcon from "@mui/icons-material/CloudUpload"; import UploadFileIcon from "@mui/icons-material/UploadFile"; import CreateNewFolderIcon from "@mui/icons-material/CreateNewFolder"; @@ -36,6 +37,8 @@ import { FolderThumbnail } from "@app/components/filesPage/FolderThumbnail"; import { findFolderIcon } from "@app/components/filesPage/folderIcons"; import { FolderAppearancePicker } from "@app/components/filesPage/FolderAppearancePicker"; import { useLazyThumbnail } from "@app/hooks/useLazyThumbnail"; +import { useFileActionIcons } from "@app/hooks/useFileActionIcons"; +import { useFileActionTerminology } from "@app/hooks/useFileActionTerminology"; import type { FilesPageSortMode } from "@app/contexts/FilesPageContext"; import { OpenInNewWindowMenuItem } from "@app/components/filesPage/OpenInNewWindowMenuItem"; @@ -83,6 +86,12 @@ interface FileGridProps { onSaveToServer?: (file: StirlingFileStub) => void; /** Open the version-history modal for a file (only when it has >1 version). */ onVersionHistory?: (file: StirlingFileStub) => void; + /** Download a copy (desktop: save a copy). */ + onDownloadFile?: (file: StirlingFileStub) => void; + /** Open the rename dialog for a file. */ + onRenameFile?: (file: StirlingFileStub) => void; + /** Save a second copy of the file into the library. */ + onDuplicateFile?: (file: StirlingFileStub) => void; /** When set, the Save to server item renders disabled with this tooltip. */ saveToServerDisabledReason?: string | null; /** When supplied the list-view column headers become sortable. */ @@ -366,24 +375,21 @@ function EmptyState({ ); } -function GridView({ - entries, - selectedFileIds, - activeWorkspaceFileIds, - onSelectFile, - onOpenFolder, - onOpenFile, - onMoveFiles, - onMoveFolder, - onRenameFolder, - onDeleteFolder, - onChangeFolderAppearance, - onRemoveFiles, - onPromptMoveFiles, - onSaveToServer, - onVersionHistory, - saveToServerDisabledReason, -}: FileGridProps) { +function GridView(props: FileGridProps) { + const { + entries, + selectedFileIds, + activeWorkspaceFileIds, + onSelectFile, + onOpenFolder, + onOpenFile, + onMoveFiles, + onMoveFolder, + onRenameFolder, + onDeleteFolder, + onChangeFolderAppearance, + } = props; + const menuHandlersFor = useFileMenuHandlers(props); return (
{entries.map((entry) => { @@ -424,22 +430,7 @@ function GridView({ onSelectFile(entry.file!.id, e.shiftKey, e.metaKey || e.ctrlKey) } onDoubleClick={() => onOpenFile(entry.file!)} - onRemove={() => onRemoveFiles([entry.file!.id])} - onMove={() => { - const target = selectedFileIds.has(entry.file!.id) - ? Array.from(selectedFileIds) - : [entry.file!.id]; - onPromptMoveFiles(target); - }} - onSaveToServer={ - onSaveToServer ? () => onSaveToServer(entry.file!) : undefined - } - onVersionHistory={ - onVersionHistory - ? () => onVersionHistory(entry.file!) - : undefined - } - saveToServerDisabledReason={saveToServerDisabledReason} + {...menuHandlersFor(entry.file)} /> ); } @@ -626,7 +617,222 @@ function PolicyBadges({ fileId }: { fileId: string }) { return ; } -interface FileCardProps { +/** Per-file actions. Shared verbatim by the grid card and the list row, and + * kept in step with the file sidebar's kebab so both surfaces offer the same. */ +interface FileActionsMenuProps { + file: StirlingFileStub; + triggerRef: React.RefObject; + onOpen: () => void; + onMove: () => void; + onRemove: () => void; + onDownload?: () => void; + onRename?: () => void; + onDuplicate?: () => void; + onSaveToServer?: () => void; + onVersionHistory?: () => void; + saveToServerDisabledReason?: string | null; +} + +function FileActionsMenu({ + file, + triggerRef, + onOpen, + onMove, + onRemove, + onDownload, + onRename, + onDuplicate, + onSaveToServer, + onVersionHistory, + saveToServerDisabledReason, +}: FileActionsMenuProps) { + const { t } = useTranslation(); + const terminology = useFileActionTerminology(); + const DownloadIcon = useFileActionIcons().download; + const showSaveToServer = + Boolean(onSaveToServer) && file.remoteStorageId == null; + const showVersionHistory = + Boolean(onVersionHistory) && (file.versionNumber ?? 1) > 1; + return ( + + + e.stopPropagation()} + aria-label={t("filesPage.fileMenu", "File actions")} + data-testid="file-card-actions" + > + + + + + } + onClick={(e) => { + e.stopPropagation(); + onOpen(); + }} + > + {t("filesPage.addToWorkspace", "Add to workspace")} + + + } + onClick={(e) => { + e.stopPropagation(); + onMove(); + }} + data-testid="file-menu-move-to" + > + {t("filesPage.moveTo", "Move to…")} + + + {(onDownload || onRename || onDuplicate) && } + {onDownload && ( + } + onClick={(e) => { + e.stopPropagation(); + onDownload(); + }} + data-testid="file-menu-download" + > + {terminology.download} + + )} + {onRename && ( + } + onClick={(e) => { + e.stopPropagation(); + onRename(); + }} + data-testid="file-menu-rename" + > + {t("filesPage.rename", "Rename")} + + )} + {onDuplicate && ( + } + onClick={(e) => { + e.stopPropagation(); + onDuplicate(); + }} + data-testid="file-menu-duplicate" + > + {t("filesPage.duplicate", "Duplicate")} + + )} + + {(showSaveToServer || showVersionHistory) && } + {/* Per-file Save to server; shown for local-only files. When + storage is off it stays visible but disabled with a tooltip. */} + {showSaveToServer && onSaveToServer && ( + + } + disabled={Boolean(saveToServerDisabledReason)} + onClick={(e) => { + e.stopPropagation(); + onSaveToServer(); + }} + style={ + saveToServerDisabledReason + ? { pointerEvents: "auto" } + : undefined + } + > + {t("filesPage.saveToServer", "Save to server")} + + + )} + {showVersionHistory && onVersionHistory && ( + } + onClick={(e) => { + e.stopPropagation(); + onVersionHistory(); + }} + > + {t("filesPage.versionHistory", "Version history")} + + )} + + + } + onClick={(e) => { + e.stopPropagation(); + onRemove(); + }} + > + {t("filesPage.remove", "Delete")} + + + + ); +} + +/** Binds one file's kebab handlers, so grid and list wire them identically. */ +function useFileMenuHandlers( + props: FileGridProps, +): (file: StirlingFileStub) => FileMenuHandlers { + const { + selectedFileIds, + onRemoveFiles, + onPromptMoveFiles, + onSaveToServer, + onVersionHistory, + onDownloadFile, + onRenameFile, + onDuplicateFile, + saveToServerDisabledReason, + } = props; + return (file: StirlingFileStub) => ({ + onRemove: () => onRemoveFiles([file.id]), + // A move acts on the whole selection when this file is part of it. + onMove: () => + onPromptMoveFiles( + selectedFileIds.has(file.id) ? Array.from(selectedFileIds) : [file.id], + ), + onDownload: onDownloadFile ? () => onDownloadFile(file) : undefined, + onRename: onRenameFile ? () => onRenameFile(file) : undefined, + onDuplicate: onDuplicateFile ? () => onDuplicateFile(file) : undefined, + onSaveToServer: onSaveToServer ? () => onSaveToServer(file) : undefined, + onVersionHistory: onVersionHistory + ? () => onVersionHistory(file) + : undefined, + saveToServerDisabledReason, + }); +} + +/** Per-file kebab handlers, shared by the card and row wrappers. */ +interface FileMenuHandlers { + onRemove: () => void; + onMove: () => void; + onDownload?: () => void; + onRename?: () => void; + onDuplicate?: () => void; + /** Kebab Save to server; only fires when file is local-only. */ + onSaveToServer?: () => void; + /** Open the version-history modal; shown only when file has >1 version. */ + onVersionHistory?: () => void; + /** When set, the kebab Save to server is disabled with this tooltip. */ + saveToServerDisabledReason?: string | null; +} + +interface FileCardProps extends FileMenuHandlers { file: StirlingFileStub; isSelected: boolean; isInWorkspace: boolean; @@ -637,14 +843,6 @@ interface FileCardProps { multiSelectActive: boolean; onClick: (e: React.MouseEvent) => void; onDoubleClick: () => void; - onRemove: () => void; - onMove: () => void; - /** Kebab Save to server; only fires when file is local-only. */ - onSaveToServer?: () => void; - /** Open the version-history modal; shown only when file has >1 version. */ - onVersionHistory?: () => void; - /** When set, the kebab Save to server is disabled with this tooltip. */ - saveToServerDisabledReason?: string | null; } function FileCard({ @@ -656,11 +854,7 @@ function FileCard({ multiSelectActive, onClick, onDoubleClick, - onRemove, - onMove, - onSaveToServer, - onVersionHistory, - saveToServerDisabledReason, + ...menuHandlers }: FileCardProps) { const { t } = useTranslation(); const cardRef = useRef(null); @@ -790,120 +984,40 @@ function FileCard({
- - - e.stopPropagation()} - aria-label={t("filesPage.fileMenu", "File actions")} - data-testid="file-card-actions" - > - - - - - } - onClick={(e) => { - e.stopPropagation(); - onDoubleClick(); - }} - > - {t("filesPage.addToWorkspace", "Add to workspace")} - - - } - onClick={(e) => { - e.stopPropagation(); - onMove(); - }} - data-testid="file-menu-move-to" - > - {t("filesPage.moveTo", "Move to…")} - - {/* Per-file Save to server; shown for local-only files. When - storage is off it stays visible but disabled with a tooltip. */} - {onSaveToServer && file.remoteStorageId == null && ( - - } - disabled={Boolean(saveToServerDisabledReason)} - onClick={(e) => { - e.stopPropagation(); - onSaveToServer(); - }} - style={ - saveToServerDisabledReason - ? { pointerEvents: "auto" } - : undefined - } - > - {t("filesPage.saveToServer", "Save to server")} - - - )} - {onVersionHistory && (file.versionNumber ?? 1) > 1 && ( - } - onClick={(e) => { - e.stopPropagation(); - onVersionHistory(); - }} - > - {t("filesPage.versionHistory", "Version history")} - - )} - - } - onClick={(e) => { - e.stopPropagation(); - onRemove(); - }} - > - {t("filesPage.remove", "Delete")} - - - +
); } -function ListView({ - entries, - selectedFileIds, - activeWorkspaceFileIds, - onSelectFile, - onSetSelection, - onOpenFolder, - onOpenFile, - onMoveFiles, - onMoveFolder, - onRenameFolder, - onDeleteFolder, - onSaveToServer, - onVersionHistory, - saveToServerDisabledReason, - onChangeFolderAppearance, - onRemoveFiles, - onPromptMoveFiles, - sortMode, - onChangeSortMode, -}: FileGridProps & { - sortMode?: FilesPageSortMode; - onChangeSortMode?: (next: FilesPageSortMode) => void; -}) { +function ListView( + props: FileGridProps & { + sortMode?: FilesPageSortMode; + onChangeSortMode?: (next: FilesPageSortMode) => void; + }, +) { + const { + entries, + selectedFileIds, + activeWorkspaceFileIds, + onSelectFile, + onSetSelection, + onOpenFolder, + onOpenFile, + onMoveFiles, + onMoveFolder, + onRenameFolder, + onDeleteFolder, + onChangeFolderAppearance, + sortMode, + onChangeSortMode, + } = props; + const menuHandlersFor = useFileMenuHandlers(props); const { t } = useTranslation(); // Tri-state header checkbox state - computed from current entries. @@ -1029,22 +1143,7 @@ function ListView({ onSelectFile(entry.file!.id, e.shiftKey, e.metaKey || e.ctrlKey) } onOpen={() => onOpenFile(entry.file!)} - onRemove={() => onRemoveFiles([entry.file!.id])} - onMove={() => { - const target = selectedFileIds.has(entry.file!.id) - ? Array.from(selectedFileIds) - : [entry.file!.id]; - onPromptMoveFiles(target); - }} - onSaveToServer={ - onSaveToServer ? () => onSaveToServer(entry.file!) : undefined - } - onVersionHistory={ - onVersionHistory - ? () => onVersionHistory(entry.file!) - : undefined - } - saveToServerDisabledReason={saveToServerDisabledReason} + {...menuHandlersFor(entry.file)} /> ); } @@ -1241,7 +1340,7 @@ function FolderRow({ ); } -interface FileRowProps { +interface FileRowProps extends FileMenuHandlers { file: StirlingFileStub; isSelected: boolean; isInWorkspace: boolean; @@ -1251,14 +1350,6 @@ interface FileRowProps { multiSelectActive: boolean; onClick: (e: React.MouseEvent) => void; onOpen: () => void; - onRemove: () => void; - onMove: () => void; - /** Kebab Save to server; only fires when file is local-only. */ - onSaveToServer?: () => void; - /** Open the version-history modal; shown only when file has >1 version. */ - onVersionHistory?: () => void; - /** When set, the kebab Save to server is disabled with this tooltip. */ - saveToServerDisabledReason?: string | null; } function FileRow({ @@ -1270,11 +1361,7 @@ function FileRow({ multiSelectActive, onClick, onOpen, - onRemove, - onMove, - onSaveToServer, - onVersionHistory, - saveToServerDisabledReason, + ...menuHandlers }: FileRowProps) { const { t } = useTranslation(); const kebabRef = useRef(null); @@ -1414,91 +1501,12 @@ function FileRow({ {fileSize} {fileDate} - - - e.stopPropagation()} - aria-label={t("filesPage.fileMenu", "File actions")} - data-testid="file-card-actions" - > - - - - - } - onClick={(e) => { - e.stopPropagation(); - onOpen(); - }} - > - {t("filesPage.addToWorkspace", "Add to workspace")} - - - } - onClick={(e) => { - e.stopPropagation(); - onMove(); - }} - > - {t("filesPage.moveTo", "Move to…")} - - {/* Per-file Save to server; shown for local-only files. When - storage is off it stays visible but disabled with a tooltip. */} - {onSaveToServer && file.remoteStorageId == null && ( - - } - disabled={Boolean(saveToServerDisabledReason)} - onClick={(e) => { - e.stopPropagation(); - onSaveToServer(); - }} - style={ - saveToServerDisabledReason - ? { pointerEvents: "auto" } - : undefined - } - > - {t("filesPage.saveToServer", "Save to server")} - - - )} - {onVersionHistory && (file.versionNumber ?? 1) > 1 && ( - } - onClick={(e) => { - e.stopPropagation(); - onVersionHistory(); - }} - > - {t("filesPage.versionHistory", "Version history")} - - )} - - } - onClick={(e) => { - e.stopPropagation(); - onRemove(); - }} - > - {t("filesPage.remove", "Delete")} - - - + ); diff --git a/frontend/editor/src/core/components/filesPage/FileManagerView.tsx b/frontend/editor/src/core/components/filesPage/FileManagerView.tsx index a0b0c6cfa1..e87b348780 100644 --- a/frontend/editor/src/core/components/filesPage/FileManagerView.tsx +++ b/frontend/editor/src/core/components/filesPage/FileManagerView.tsx @@ -71,6 +71,10 @@ import { FolderNameDialog } from "@app/components/filesPage/FolderNameDialog"; import { DeleteFolderDialog } from "@app/components/filesPage/DeleteFolderDialog"; import { DeleteFilesDialog } from "@app/components/filesPage/DeleteFilesDialog"; import { VersionHistoryModal } from "@app/components/filesPage/VersionHistoryModal"; +import { RenameFileDialog } from "@app/components/shared/RenameFileDialog"; +import { duplicateStoredFile } from "@app/utils/duplicateFile"; +import { downloadFileFromStorage } from "@app/utils/downloadUtils"; +import { fileStorage } from "@app/services/fileStorage"; import { materializeServerStubs } from "@app/services/fileSyncService"; import { FILES_PAGE_DRAG_TYPE, @@ -794,6 +798,92 @@ export default function FileManagerView() { [removeFiles], ); + // ─── per-file kebab: download / rename / duplicate ─────────────────────── + // Same actions the file sidebar's kebab offers, so both surfaces match. + + /** Cloud-only rows hold no bytes; pull them local before acting on them. */ + const localCopyOf = useCallback( + async (file: StirlingFileStub): Promise => { + const [materialized] = await materializeServerStubs([file], { + addFiles: fileActions.addFilesWithOptions, + updateStub: fileActions.updateStirlingFileStub, + }); + return materialized ?? null; + }, + [fileActions], + ); + + const handleDownloadFile = useCallback( + async (file: StirlingFileStub) => { + try { + const local = await localCopyOf(file); + if (!local) return; + await downloadFileFromStorage(local); + } catch (err) { + console.error("[FilesPage] Download failed", err); + folders.setError( + t("filesPage.error.downloadFailed", "Could not download the file."), + ); + } + }, + [localCopyOf, folders, t], + ); + + const handleDuplicateFile = useCallback( + async (file: StirlingFileStub) => { + try { + const local = await localCopyOf(file); + if (!local) return; + const copyId = await duplicateStoredFile( + local, + allFiles.map((f) => f.name), + addFiles, + ); + if (!copyId) { + throw new Error(`File "${local.name}" not found in storage`); + } + await refresh(); + } catch (err) { + console.error("[FilesPage] Duplicate failed", err); + folders.setError( + t("filesPage.error.duplicateFailed", "Could not duplicate the file."), + ); + } + }, + [localCopyOf, allFiles, addFiles, refresh, folders, t], + ); + + const [renameTarget, setRenameTarget] = useState( + null, + ); + + // The stub name is what the UI and exports read, so a rename is a metadata + // write; the workbench copy (if any) is updated in the same breath. + const handleConfirmRename = useCallback( + async (name: string) => { + const file = renameTarget; + if (!file) return; + const local = await localCopyOf(file); + if (!local) return; + // quickKey is name|size|lastModified; a stale one would make a re-upload + // of the original look like a duplicate of the renamed file. + const quickKey = `${name}|${local.size}|${local.lastModified}`; + const saved = await fileStorage.updateFileMetadata(local.id, { + name, + quickKey, + }); + if (!saved) { + throw new Error( + t("fileSidebar.rename.error", "Could not rename the file."), + ); + } + fileActions.updateStirlingFileStub(local.id, { name, quickKey }); + setRenameTarget(null); + await refresh(); + }, + [renameTarget, localCopyOf, fileActions, refresh, t], + ); + // ─── derived UI bits ──────────────────────────────────────────────────── const currentFolderRecord = currentFolderId ? (foldersById.get(currentFolderId) ?? null) @@ -1503,6 +1593,9 @@ export default function FileManagerView() { onPromptMoveFiles={promptMoveFiles} onSaveToServer={(file) => setSaveToServerTarget([file])} onVersionHistory={(file) => setVersionHistoryFile(file)} + onDownloadFile={handleDownloadFile} + onRenameFile={setRenameTarget} + onDuplicateFile={handleDuplicateFile} saveToServerDisabledReason={saveToServerDisabledReason} // Center-of-grid CTAs when the empty state shows - same // handlers the corner header buttons use so behaviour @@ -1662,6 +1755,14 @@ export default function FileManagerView() { onConfirm={confirmRemoveFiles} /> + {/* Rename (opened from the card kebab). */} + setRenameTarget(null)} + onSubmit={handleConfirmRename} + /> + {/* Version journey in a modal (opened from the card kebab). */} ( const [deleteTarget, setDeleteTarget] = useState( null, ); + // Kebab "Rename" target; drives RenameFileDialog. + const [renameTarget, setRenameTarget] = useState( + null, + ); // Storage gate: only offer Save-to-cloud when the server allows it and // the user is signed in (guests have no cloud library). const storageEnabled = config?.storageEnabled === true && !isAnonymous; @@ -417,6 +425,121 @@ const FileSidebar = forwardRef( [allFileStubs], ); + const warnDataUnavailable = useCallback(() => { + alert({ + alertType: "warning", + title: t("fileSidebar.dataLostTitle", "File data is unavailable"), + body: t( + "fileSidebar.dataLostBody", + "This browser lost this file's contents. Upload it again to keep working with it.", + ), + expandable: false, + durationMs: 6000, + }); + }, [t]); + + // Kebab: download a copy (desktop saves via the native dialog). Routed + // through the policy wrapper so export policies enforce here too. + const handleDownload = useCallback( + async (fileId: FileId) => { + const stub = allFileStubs.find((s) => s.id === fileId); + const file = await fileStorage.getStirlingFile(fileId); + if (!file) { + warnDataUnavailable(); + return; + } + try { + await downloadFileWithPolicy({ + data: file, + filename: stub?.name ?? file.name, + fileId: fileId as string, + }); + } catch (error) { + console.error("[FileSidebar] Download failed:", error); + alert({ + alertType: "error", + title: t("fileSidebar.downloadFailed", "Download failed"), + body: error instanceof Error ? error.message : String(error), + expandable: false, + }); + } + }, + [allFileStubs, warnDataUnavailable, t], + ); + + // Kebab: copy the file into the library under a free "(copy)" name. + const handleDuplicate = useCallback( + async (fileId: FileId) => { + const stub = allFileStubs.find((s) => s.id === fileId); + if (!stub) return; + try { + const copyId = await duplicateStoredFile( + stub, + allFileStubs.map((s) => s.name), + addFiles, + ); + if (!copyId) { + warnDataUnavailable(); + return; + } + await refreshStubs(); + } catch (error) { + console.error("[FileSidebar] Duplicate failed:", error); + alert({ + alertType: "error", + title: t("fileSidebar.duplicateFailed", "Could not duplicate file"), + body: error instanceof Error ? error.message : String(error), + expandable: false, + }); + } + }, + [allFileStubs, addFiles, refreshStubs, warnDataUnavailable, t], + ); + + // Kebab: open the rename dialog for this one file. + const handleRename = useCallback( + (fileId: FileId) => { + const stub = allFileStubs.find((s) => s.id === fileId); + if (stub) setRenameTarget(stub); + }, + [allFileStubs], + ); + + // The stub name is what the UI and exports read, so a rename is a metadata + // write - storage first, then the workbench copy if the file is open. + const handleConfirmRename = useCallback( + async (name: string) => { + const stub = renameTarget; + if (!stub) return; + // quickKey is name|size|lastModified; a stale one would make a re-upload + // of the original look like a duplicate of the renamed file. + const quickKey = `${name}|${stub.size}|${stub.lastModified}`; + const saved = await fileStorage.updateFileMetadata(stub.id, { + name, + quickKey, + }); + if (!saved) { + throw new Error( + t("fileSidebar.rename.error", "Could not rename the file."), + ); + } + fileActions.updateStirlingFileStub(stub.id, { name, quickKey }); + setRenameTarget(null); + await refreshStubs(); + }, + [renameTarget, fileActions, refreshStubs, t], + ); + + // Desktop-only; a no-op stub on web, where this stays hidden. + const { canOpenInNewWindow, openInNewWindow } = useOpenInNewWindow(); + const handleOpenInNewWindow = useCallback( + (fileId: FileId) => { + const stub = allFileStubs.find((s) => s.id === fileId); + if (stub) openInNewWindow(stub); + }, + [allFileStubs, openInNewWindow], + ); + // Once a pending file lands in state, open it in the viewer. useEffect(() => { if (!pendingViewFileId) return; @@ -773,6 +896,12 @@ const FileSidebar = forwardRef( onFolderClick={openWatchedFolder} policies={policyFileBadges.get(stub.id as string) ?? NO_POLICIES} onDelete={isWatchedFoldersActive ? undefined : handleSidebarDelete} + onDownload={handleDownload} + onRename={isWatchedFoldersActive ? undefined : handleRename} + onDuplicate={isWatchedFoldersActive ? undefined : handleDuplicate} + onOpenInNewWindow={ + canOpenInNewWindow(stub) ? handleOpenInNewWindow : undefined + } onSaveToCloud={isWatchedFoldersActive ? undefined : handleSaveToCloud} canSaveToCloud={storageEnabled && fileOrigin !== "shared-with-me"} isUploadedToCloud={fileOrigin === "cloud"} @@ -1222,6 +1351,14 @@ const FileSidebar = forwardRef( onChanged={refreshStubs} /> + {/* Kebab "Rename" dialog. */} + setRenameTarget(null)} + onSubmit={handleConfirmRename} + /> + {/* Cloud-aware delete choice (only opened for cloud-uploaded files). */} void; + /** Download a copy (desktop: save a copy) from the kebab menu. */ + onDownload?: (fileId: FileId) => void; + /** Rename the file from the kebab menu. */ + onRename?: (fileId: FileId) => void; + /** Save a second copy of the file into the library. */ + onDuplicate?: (fileId: FileId) => void; + /** Desktop only: open the file in its own window. Omit where unsupported. */ + onOpenInNewWindow?: (fileId: FileId) => void; /** Save to cloud from the kebab menu. */ onSaveToCloud?: (fileId: FileId) => void; /** Whether the upload-to-server menu item is offered (storage on, signed in). */ @@ -171,6 +184,46 @@ export interface FileItemProps { const MAX_VISIBLE_FOLDER_TAGS = 2; +/** One kebab row. `disabledReason`, when set, greys the row out and says why. */ +function FileMenuItem({ + disabledReason, + icon, + color, + onClick, + children, +}: { + disabledReason?: React.ReactNode; + icon: React.ReactNode; + color?: string; + onClick: (e: React.MouseEvent) => void; + children: React.ReactNode; +}) { + return ( + + {/* Disabled items swallow pointer events, so the tooltip needs a live wrapper. */} +
+ { + e.stopPropagation(); + onClick(e); + }} + > + {children} + +
+
+ ); +} + // Memoized: sidebar rows bail out unless THEIR props change, so one file's // update (e.g. a new version landing) re-renders one row, not the whole list. export const FileItem = React.memo(function FileItem({ @@ -192,6 +245,10 @@ export const FileItem = React.memo(function FileItem({ policies = [], primaryLabel, onDelete, + onDownload, + onRename, + onDuplicate, + onOpenInNewWindow, onSaveToCloud, canSaveToCloud = false, isUploadedToCloud = false, @@ -199,9 +256,14 @@ export const FileItem = React.memo(function FileItem({ hasVersionHistory = false, }: FileItemProps) { const { t } = useTranslation(); + const terminology = useFileActionTerminology(); + const DownloadIcon = useFileActionIcons().download; const ext = getFileExtension(name); const dateLabel = lastModified ? formatFileDate(lastModified) : ""; const typeLabel = ext ? ext.toUpperCase() : "File"; + const metaLine = [typeLabel, size ? formatFileSize(size) : null, dateLabel] + .filter(Boolean) + .join(" · "); const policyEnforcing = policies.some((p) => p.enforcing); const enforcingTooltip = (action: string): React.ReactNode => ( @@ -220,6 +282,25 @@ export const FileItem = React.memo(function FileItem({ ); + // Why an action can't run right now: a policy is rewriting the file, or its + // bytes are gone. `needsBytes` actions are the ones that read the file. + const blockedReason = ( + action: string, + needsBytes = true, + ): React.ReactNode | null => { + if (policyEnforcing) return enforcingTooltip(action); + if (needsBytes && dataUnavailable) + return t( + "fileSidebar.fileItem.dataLostTooltip", + "This browser lost this file's contents. Upload it again to keep working with it.", + ); + return null; + }; + + const viewerLabel = isViewedInViewer + ? t("fileSidebar.fileItem.closeViewer", "Close viewer") + : t("fileSidebar.fileItem.openInViewer", "Open in viewer"); + const visibleFolders = folders.slice(0, MAX_VISIBLE_FOLDER_TAGS); const overflowFolders = folders.slice(MAX_VISIBLE_FOLDER_TAGS); @@ -233,6 +314,7 @@ export const FileItem = React.memo(function FileItem({ const itemRef = useRef(null); const [hoverRect, setHoverRect] = useState(null); + const [menuOpened, setMenuOpened] = useState(false); const handleMouseEnter = useCallback(() => { setHoverRect(itemRef.current?.getBoundingClientRect() ?? null); @@ -240,9 +322,10 @@ export const FileItem = React.memo(function FileItem({ const handleMouseLeave = useCallback(() => setHoverRect(null), []); - // Reactive: tooltip appears as soon as both hover rect and thumbnail are ready + // Reactive: tooltip appears as soon as both hover rect and thumbnail are ready. + // The kebab suppresses it - two cards floating off one row read as a glitch. const thumbPos = - hoverRect && resolvedThumbnail + hoverRect && resolvedThumbnail && !menuOpened ? { top: hoverRect.top + hoverRect.height / 2, left: hoverRect.right + 10, @@ -389,11 +472,7 @@ export const FileItem = React.memo(function FileItem({ onEyeClick(fileId, e); }} tabIndex={-1} - aria-label={ - isViewedInViewer - ? t("fileSidebar.fileItem.closeViewer", "Close viewer") - : t("fileSidebar.fileItem.openInViewer", "Open in viewer") - } + aria-label={viewerLabel} > - {(onDelete || - (canSaveToCloud && onSaveToCloud) || - (hasVersionHistory && onVersionHistory)) && ( - - - e.stopPropagation()} - tabIndex={-1} - aria-label={t( - "fileSidebar.fileItem.moreActions", - "More actions", - )} - > - - - - e.stopPropagation()}> - {hasVersionHistory && onVersionHistory && ( - } - onClick={(e) => { - e.stopPropagation(); - onVersionHistory(fileId); - }} - > - {t( - "fileSidebar.fileItem.versionHistory", - "Version history", - )} - + + + e.stopPropagation()} + tabIndex={-1} + aria-label={t( + "fileSidebar.fileItem.moreActions", + "More actions", )} - {canSaveToCloud && - onSaveToCloud && - (() => { - const uploadLabel = isUploadedToCloud - ? t( - "fileSidebar.fileItem.updateOnServer", - "Update on server", - ) - : t( - "fileSidebar.fileItem.uploadToServer", - "Upload to server", - ); - return ( - -
- - } - onClick={(e) => { - e.stopPropagation(); - onSaveToCloud(fileId); - }} - > - {uploadLabel} - -
-
- ); - })()} - {onDelete && - (() => { - const deleteLabel = t( - "fileSidebar.fileItem.delete", - "Delete", - ); - return ( - -
- - } - onClick={(e) => { - e.stopPropagation(); - onDelete(fileId); - }} - > - {deleteLabel} - -
-
- ); - })()} - -
- )} + > + + + + e.stopPropagation()}> + {/* Rows truncate long names; the menu header is where the whole + name (and the size the row has no space for) is readable. */} + + {name} + + {metaLine} + + + + + ) : ( + + ) + } + onClick={(e) => onEyeClick(fileId, e)} + > + {viewerLabel} + + + {onOpenInNewWindow && ( + } + onClick={() => onOpenInNewWindow(fileId)} + > + {t("openInNewWindow", "Open in new window")} + + )} + + {(onDownload || onRename || onDuplicate) && } + + {onDownload && ( + } + onClick={() => onDownload(fileId)} + > + {terminology.download} + + )} + + {onRename && ( + } + onClick={() => onRename(fileId)} + > + {t("fileSidebar.fileItem.rename", "Rename")} + + )} + + {onDuplicate && ( + } + onClick={() => onDuplicate(fileId)} + > + {t("fileSidebar.fileItem.duplicate", "Duplicate")} + + )} + + {((canSaveToCloud && onSaveToCloud) || + (hasVersionHistory && onVersionHistory)) && } + + {canSaveToCloud && + onSaveToCloud && + (() => { + const uploadLabel = isUploadedToCloud + ? t( + "fileSidebar.fileItem.updateOnServer", + "Update on server", + ) + : t( + "fileSidebar.fileItem.uploadToServer", + "Upload to server", + ); + return ( + } + onClick={() => onSaveToCloud(fileId)} + > + {uploadLabel} + + ); + })()} + + {hasVersionHistory && onVersionHistory && ( + } + onClick={() => onVersionHistory(fileId)} + > + {t("fileSidebar.fileItem.versionHistory", "Version history")} + + )} + + {onDelete && ( + <> + + } + onClick={() => onDelete(fileId)} + > + {t("fileSidebar.fileItem.delete", "Delete")} + + + )} + +
diff --git a/frontend/editor/src/core/components/shared/RenameFileDialog.stories.tsx b/frontend/editor/src/core/components/shared/RenameFileDialog.stories.tsx new file mode 100644 index 0000000000..9413db41b0 --- /dev/null +++ b/frontend/editor/src/core/components/shared/RenameFileDialog.stories.tsx @@ -0,0 +1,39 @@ +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { RenameFileDialog } from "@app/components/shared/RenameFileDialog"; + +const meta = { + title: "Shared/RenameFileDialog", + component: RenameFileDialog, +} satisfies Meta; +export default meta; + +type Story = StoryObj; + +export const Default: Story = { + args: { + opened: true, + fileName: "Q3 invoice bundle.pdf", + onClose: () => {}, + onSubmit: () => {}, + }, +}; + +export const NoExtension: Story = { + args: { + opened: true, + fileName: "scanned-document", + onClose: () => {}, + onSubmit: () => {}, + }, +}; + +export const SaveFails: Story = { + args: { + opened: true, + fileName: "contract.pdf", + onClose: () => {}, + onSubmit: () => { + throw new Error("Could not rename the file."); + }, + }, +}; diff --git a/frontend/editor/src/core/components/shared/RenameFileDialog.tsx b/frontend/editor/src/core/components/shared/RenameFileDialog.tsx new file mode 100644 index 0000000000..83e00f6ba3 --- /dev/null +++ b/frontend/editor/src/core/components/shared/RenameFileDialog.tsx @@ -0,0 +1,140 @@ +import { useEffect, useState } from "react"; +import { useTranslation } from "react-i18next"; +import { Alert, Group, Modal, Stack, TextInput } from "@mantine/core"; +import ErrorOutlineIcon from "@mui/icons-material/ErrorOutlined"; + +import { Button } from "@app/ui/Button"; +import { splitFileName } from "@app/utils/fileUtils"; + +/** Characters Windows/macOS reject in a filename, which is also what a download saves as. */ +const ILLEGAL_NAME_CHARS = /[\\/:*?"<>|]/; + +interface RenameFileDialogProps { + opened: boolean; + /** Current name, including its extension. */ + fileName: string; + onClose: () => void; + /** Gets the new full name. Throwing keeps the dialog open with the message. */ + onSubmit: (name: string) => void | Promise; +} + +/** + * Renames one library file. Only the base name is editable - the extension is + * shown but fixed, so a rename can't leave the file claiming the wrong type. + */ +export function RenameFileDialog({ + opened, + fileName, + onClose, + onSubmit, +}: RenameFileDialogProps) { + const { t } = useTranslation(); + const [base, extension] = splitFileName(fileName); + const [value, setValue] = useState(base); + const [submitting, setSubmitting] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + if (opened) { + setValue(base); + setSubmitting(false); + setError(null); + } + }, [opened, base]); + + const submit = async () => { + const nextBase = value.trim(); + if (!nextBase) return; + if (ILLEGAL_NAME_CHARS.test(nextBase)) { + setError( + t( + "fileSidebar.rename.illegalCharacters", + "A file name can't contain \\ / : * ? \" < > |", + ), + ); + return; + } + const nextName = `${nextBase}${extension}`; + if (nextName === fileName) { + onClose(); + return; + } + setSubmitting(true); + setError(null); + try { + await onSubmit(nextName); + onClose(); + } catch (err) { + // Stay open on failure: closing would look like the rename worked. + setError( + err instanceof Error + ? err.message + : t("fileSidebar.rename.error", "Could not rename the file."), + ); + } finally { + setSubmitting(false); + } + }; + + return ( + + + setValue(e.currentTarget.value)} + onFocus={(e) => e.currentTarget.select()} + onKeyDown={(e) => { + if (e.key === "Enter") { + e.preventDefault(); + void submit(); + } + }} + maxLength={200} + aria-label={t("fileSidebar.rename.label", "File name")} + rightSection={ + extension ? ( + + {extension} + + ) : undefined + } + rightSectionWidth={extension ? extension.length * 8 + 12 : undefined} + rightSectionPointerEvents="none" + /> + {error && ( + } + variant="light" + role="alert" + > + {error} + + )} + + + + + + + ); +} diff --git a/frontend/editor/src/core/components/shared/WorkbenchBar.tsx b/frontend/editor/src/core/components/shared/WorkbenchBar.tsx index 241d658bad..ad38fb1974 100644 --- a/frontend/editor/src/core/components/shared/WorkbenchBar.tsx +++ b/frontend/editor/src/core/components/shared/WorkbenchBar.tsx @@ -229,7 +229,8 @@ export default function WorkbenchBar({ try { const result = await downloadFile({ data: new Blob([buffer], { type: "application/pdf" }), - filename: fileToExport.name, + // Stub name, not File.name: a rename only writes the stub. + filename: stub?.name ?? fileToExport.name, localPath: forceNewFile ? undefined : stub?.localFilePath, fileId: stub?.id, }); @@ -281,7 +282,8 @@ export default function WorkbenchBar({ try { const result = await downloadRaw({ data: enforced[idx], - filename: file.name, + // Stub name, not File.name: a rename only writes the stub. + filename: stub?.name ?? file.name, localPath: forceNewFile ? undefined : stub?.localFilePath, fileId: stub?.id, }); diff --git a/frontend/editor/src/core/hooks/useFileHandler.ts b/frontend/editor/src/core/hooks/useFileHandler.ts index 2f26648fd1..6fcbcb2350 100644 --- a/frontend/editor/src/core/hooks/useFileHandler.ts +++ b/frontend/editor/src/core/hooks/useFileHandler.ts @@ -13,6 +13,10 @@ export const useFileHandler = () => { selectFiles?: boolean; /** Persist to IDB without dispatching to workspace state. */ skipWorkspaceDispatch?: boolean; + /** Defaults to true; false keeps an archive intact (e.g. duplicating one). */ + autoUnzip?: boolean; + /** Skip the upload metric - the file isn't new to the system (e.g. a copy). */ + skipUploadTracking?: boolean; } = {}, ): Promise => { // Merge default options with passed options - passed options take precedence diff --git a/frontend/editor/src/core/tests/stubbed/file-actions-menu.spec.ts b/frontend/editor/src/core/tests/stubbed/file-actions-menu.spec.ts new file mode 100644 index 0000000000..cafc188462 --- /dev/null +++ b/frontend/editor/src/core/tests/stubbed/file-actions-menu.spec.ts @@ -0,0 +1,167 @@ +import path from "path"; +import type { Page } from "@playwright/test"; +import { test, expect } from "@app/tests/helpers/stub-test-base"; +import { uploadFiles } from "@app/tests/helpers/ui-helpers"; + +// Per-file actions live behind a kebab on two surfaces - the file sidebar and +// the My Files grid. They must offer the same file actions on both. + +const SAMPLE = path.join(import.meta.dirname, "../test-fixtures/sample.pdf"); + +const rows = (page: Page) => page.locator(".file-sidebar-file-item"); + +/** Hover the first sidebar row (the kebab only shows on hover) and open it. */ +async function openKebab(page: Page): Promise { + const row = rows(page).first(); + await row.hover(); + await row.locator(".file-sidebar-kebab-btn").click(); + await expect(page.getByRole("menu")).toBeVisible(); +} + +test("the kebab lists the file's actions under its full name", async ({ + page, +}) => { + await uploadFiles(page, SAMPLE); + await openKebab(page); + + const menu = page.getByRole("menu"); + await expect(menu.locator(".file-sidebar-kebab-header-name")).toHaveText( + "sample.pdf", + ); + // Type · size · date - the row itself has no space for the size. + await expect(menu.locator(".file-sidebar-kebab-header-meta")).toContainText( + "PDF", + ); + // A lone upload lands in the viewer, so the toggle offers the way out. + await expect( + menu.getByRole("menuitem", { name: "Close viewer" }), + ).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Download" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Rename" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Duplicate" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Delete" })).toBeVisible(); +}); + +test("Download saves the file under its current name", async ({ page }) => { + await uploadFiles(page, SAMPLE); + await openKebab(page); + + const download = page.waitForEvent("download"); + await page.getByRole("menuitem", { name: "Download" }).click(); + expect((await download).suggestedFilename()).toBe("sample.pdf"); +}); + +test("Rename updates the row and survives a reload", async ({ page }) => { + await uploadFiles(page, SAMPLE); + await openKebab(page); + await page.getByRole("menuitem", { name: "Rename" }).click(); + + // Only the base name is editable; the extension is re-applied on submit. + const input = page.getByLabel("File name"); + await expect(input).toHaveValue("sample"); + await input.fill("quarterly report"); + await page.getByRole("button", { name: "Rename" }).click(); + + await expect(rows(page).first()).toContainText("quarterly report.pdf"); + + // The name is metadata in IndexedDB, so it must outlive the page. + await page.reload(); + await expect(rows(page).first()).toContainText("quarterly report.pdf"); +}); + +test("Duplicate adds a copy to the library", async ({ page }) => { + await uploadFiles(page, SAMPLE); + await openKebab(page); + await page.getByRole("menuitem", { name: "Duplicate" }).click(); + + await expect(rows(page)).toHaveCount(2); + await expect(rows(page).filter({ hasText: "sample (copy).pdf" })).toHaveCount( + 1, + ); +}); + +test("a duplicate inherits the original's classification", async ({ page }) => { + // The copy is byte-identical, so it must land in the same category group - + // it inherits the label rather than waiting on the idle backfill to re-parse. + await uploadFiles( + page, + path.join( + import.meta.dirname, + "../test-fixtures/classification/classified_invoice.pdf", + ), + ); + const financial = page + .locator(".file-sidebar-group") + .filter({ hasText: "Financial" }); + await expect(financial).toBeVisible({ timeout: 15_000 }); + + await openKebab(page); + await page.getByRole("menuitem", { name: "Duplicate" }).click(); + + // The copy carries the label straight away - its row shows the label chip... + const copy = rows(page) + .filter({ hasText: "classified_invoice (copy).pdf" }) + .first(); + await expect(copy).toContainText("Invoice", { timeout: 5_000 }); + // ...and it counts towards the same category group. + await expect(financial.locator(".file-sidebar-group-count")).toHaveText("2"); +}); + +// ─── My Files grid: the same actions, same behaviour ──────────────────────── + +const cards = (page: Page) => page.locator(".files-page-card:not(.is-folder)"); + +/** Upload a file, cross to My Files, and open the card's kebab. */ +async function openCardKebab(page: Page): Promise { + await uploadFiles(page, SAMPLE); + await page.getByTestId("my-files-button").click(); + const card = cards(page).filter({ hasText: "sample.pdf" }).first(); + await expect(card).toBeVisible(); + await card.getByRole("button", { name: /File actions/i }).click(); + await expect(page.getByRole("menu")).toBeVisible(); +} + +test("My Files offers the same file actions as the sidebar", async ({ + page, +}) => { + await openCardKebab(page); + + const menu = page.getByRole("menu"); + await expect( + menu.getByRole("menuitem", { name: "Add to workspace" }), + ).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Move to…" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Download" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Rename" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Duplicate" })).toBeVisible(); + await expect(menu.getByRole("menuitem", { name: "Delete" })).toBeVisible(); +}); + +test("My Files Download saves the file under its current name", async ({ + page, +}) => { + await openCardKebab(page); + + const download = page.waitForEvent("download"); + await page.getByRole("menuitem", { name: "Download" }).click(); + expect((await download).suggestedFilename()).toBe("sample.pdf"); +}); + +test("My Files Rename updates the card", async ({ page }) => { + await openCardKebab(page); + await page.getByRole("menuitem", { name: "Rename" }).click(); + + await page.getByLabel("File name").fill("statement"); + await page.getByRole("button", { name: "Rename" }).click(); + + await expect(cards(page).filter({ hasText: "statement.pdf" })).toHaveCount(1); +}); + +test("My Files Duplicate adds a copy", async ({ page }) => { + await openCardKebab(page); + await page.getByRole("menuitem", { name: "Duplicate" }).click(); + + await expect( + cards(page).filter({ hasText: "sample (copy).pdf" }), + ).toHaveCount(1); +}); diff --git a/frontend/editor/src/core/utils/duplicateFile.test.ts b/frontend/editor/src/core/utils/duplicateFile.test.ts new file mode 100644 index 0000000000..88c7cce686 --- /dev/null +++ b/frontend/editor/src/core/utils/duplicateFile.test.ts @@ -0,0 +1,130 @@ +import { describe, it, expect, vi, beforeEach } from "vitest"; +import type { StirlingFile, StirlingFileStub } from "@app/types/fileContext"; +import type { FileId } from "@app/types/file"; +import type { FolderId } from "@app/types/folder"; + +const getStirlingFile = vi.fn(); +const updateFileMetadata = vi.fn(); + +vi.mock("@app/services/fileStorage", () => ({ + fileStorage: { + getStirlingFile: (...args: unknown[]) => getStirlingFile(...args), + updateFileMetadata: (...args: unknown[]) => updateFileMetadata(...args), + }, +})); + +const { copyNameFor, duplicateStoredFile } = + await import("@app/utils/duplicateFile"); + +/** + * A duplicate is byte-identical to its source, so everything derived from those + * bytes (classification labels, thumbnail) and its place in the library (folder) + * is inherited rather than re-derived. These lock that contract. + */ + +const stub = (extra: Partial = {}): StirlingFileStub => + ({ + id: "src-id" as FileId, + name: "report.pdf", + size: 10, + type: "application/pdf", + lastModified: 1, + ...extra, + }) as StirlingFileStub; + +const asStirlingFile = (name: string): StirlingFile => + Object.assign(new File(["%PDF-1.7"], name, { type: "application/pdf" }), { + fileId: "new-id" as FileId, + quickKey: "k", + }) as StirlingFile; + +type AddFilesOptions = { + selectFiles?: boolean; + skipWorkspaceDispatch?: boolean; + autoUnzip?: boolean; + skipUploadTracking?: boolean; +}; +const addFiles = vi.fn< + (files: File[], options: AddFilesOptions) => Promise +>(async () => [asStirlingFile("report (copy).pdf")]); + +beforeEach(() => { + vi.clearAllMocks(); + getStirlingFile.mockResolvedValue(asStirlingFile("report.pdf")); + updateFileMetadata.mockResolvedValue(true); + addFiles.mockResolvedValue([asStirlingFile("report (copy).pdf")]); +}); + +describe("copyNameFor", () => { + it("keeps the extension and counts up until the name is free", () => { + expect(copyNameFor("report.pdf", [])).toBe("report (copy).pdf"); + expect(copyNameFor("report.pdf", ["report (copy).pdf"])).toBe( + "report (copy 2).pdf", + ); + expect( + copyNameFor("report.pdf", ["report (copy).pdf", "report (copy 2).pdf"]), + ).toBe("report (copy 3).pdf"); + }); + + it("handles a name with no extension", () => { + expect(copyNameFor("scan", [])).toBe("scan (copy)"); + }); +}); + +describe("duplicateStoredFile", () => { + it("inherits labels, folder and a persisted thumbnail", async () => { + const id = await duplicateStoredFile( + stub({ + classificationLabels: ["invoice"], + folderId: "folder-1" as FolderId, + thumbnailUrl: "data:image/png;base64,AAA", + }), + [], + addFiles, + ); + + expect(id).toBe("new-id"); + const [, updates] = updateFileMetadata.mock.calls[0]; + expect(updates.classificationLabels).toEqual(["invoice"]); + expect(updates.folderId).toBe("folder-1"); + expect(updates.thumbnail).toBe("data:image/png;base64,AAA"); + expect(updates.thumbnailStoredAt).toEqual(expect.any(Number)); + }); + + it("drops a blob: thumbnail - it would be dead on the next load", async () => { + await duplicateStoredFile( + stub({ classificationLabels: ["invoice"], thumbnailUrl: "blob:abc" }), + [], + addFiles, + ); + + const [, updates] = updateFileMetadata.mock.calls[0]; + expect(updates).not.toHaveProperty("thumbnail"); + }); + + it("writes nothing when the source has no derived metadata", async () => { + await duplicateStoredFile(stub(), [], addFiles); + + expect(updateFileMetadata).not.toHaveBeenCalled(); + }); + + it("copies the library entry without touching the workbench or metrics", async () => { + await duplicateStoredFile(stub(), ["report (copy).pdf"], addFiles); + + const [files, options] = addFiles.mock.calls[0]; + expect(files[0].name).toBe("report (copy 2).pdf"); + expect(options).toMatchObject({ + selectFiles: false, + skipWorkspaceDispatch: true, + autoUnzip: false, + skipUploadTracking: true, + }); + }); + + it("returns null when the source bytes are gone", async () => { + getStirlingFile.mockResolvedValue(null); + + expect(await duplicateStoredFile(stub(), [], addFiles)).toBeNull(); + expect(addFiles).not.toHaveBeenCalled(); + }); +}); diff --git a/frontend/editor/src/core/utils/duplicateFile.ts b/frontend/editor/src/core/utils/duplicateFile.ts new file mode 100644 index 0000000000..5bad75b513 --- /dev/null +++ b/frontend/editor/src/core/utils/duplicateFile.ts @@ -0,0 +1,85 @@ +import type { StirlingFile, StirlingFileStub } from "@app/types/fileContext"; +import type { FileId } from "@app/types/file"; +import { fileStorage } from "@app/services/fileStorage"; +import { splitFileName } from "@app/utils/fileUtils"; + +/** The subset of `useFileHandler().addFiles` a duplicate needs. */ +type AddFilesFn = ( + files: File[], + options: { + selectFiles?: boolean; + skipWorkspaceDispatch?: boolean; + autoUnzip?: boolean; + skipUploadTracking?: boolean; + }, +) => Promise; + +/** "report.pdf" → "report (copy).pdf", counting up until the name is free. */ +export function copyNameFor(name: string, taken: Iterable): string { + const [base, extension] = splitFileName(name); + const used = new Set(taken); + let candidate = `${base} (copy)${extension}`; + for (let n = 2; used.has(candidate); n++) { + candidate = `${base} (copy ${n})${extension}`; + } + return candidate; +} + +/** + * Copies a stored file into the library under a free "(copy)" name. + * + * The copy stays out of the workbench - it's an archive of the current bytes, + * not a file the user asked to work on. That skips the ingest side-effects that + * hang off the workspace dispatch, so the derived metadata is inherited from + * the source instead: the bytes are identical, so its classification labels and + * thumbnail are too, and re-deriving them would only re-parse the same PDF. The + * copy also lands in the source's folder rather than back at the root. + * + * @returns the new file's id, or null if the source has no readable bytes. + */ +export async function duplicateStoredFile( + stub: StirlingFileStub, + existingNames: Iterable, + addFiles: AddFilesFn, +): Promise { + const source = await fileStorage.getStirlingFile(stub.id); + if (!source) return null; + + const [copy] = await addFiles( + [ + new File([source], copyNameFor(stub.name, existingNames), { + type: source.type, + }), + ], + { + selectFiles: false, + skipWorkspaceDispatch: true, + // Duplicating an archive must yield one copy, not its contents scattered + // across the library. + autoUnzip: false, + // A local copy is not a new document entering the system. + skipUploadTracking: true, + }, + ); + if (!copy) return null; + + const inherited: Parameters[1] = {}; + if (stub.classificationLabels) { + inherited.classificationLabels = stub.classificationLabels; + } + // A copy belongs beside its original. Safe to set on a local-only file: the + // server is authoritative for folderId only on files it actually holds. + if (stub.folderId) { + inherited.folderId = stub.folderId; + } + // Blob URLs die with the session, so only a persisted thumbnail is worth + // carrying over; without one the row falls back to lazy regeneration. + if (stub.thumbnailUrl && !stub.thumbnailUrl.startsWith("blob:")) { + inherited.thumbnail = stub.thumbnailUrl; + inherited.thumbnailStoredAt = Date.now(); + } + if (Object.keys(inherited).length > 0) { + await fileStorage.updateFileMetadata(copy.fileId, inherited); + } + return copy.fileId; +} diff --git a/frontend/editor/src/core/utils/fileUtils.ts b/frontend/editor/src/core/utils/fileUtils.ts index 6a256a9073..13002db52e 100644 --- a/frontend/editor/src/core/utils/fileUtils.ts +++ b/frontend/editor/src/core/utils/fileUtils.ts @@ -73,6 +73,17 @@ export function getFilenameWithoutExtension( return preserveCase ? withoutExtension : withoutExtension.toLowerCase(); } +/** + * Splits a filename into its base and extension, keeping the dot on the + * extension so `base + extension` round-trips. A name with no extension (or a + * leading-dot name like ".env") gets an empty extension. + * @example splitFileName('report.pdf') // ['report', '.pdf'] + */ +export function splitFileName(name: string): [string, string] { + const dot = name.lastIndexOf("."); + return dot > 0 ? [name.slice(0, dot), name.slice(dot)] : [name, ""]; +} + /** * Checks if a file is a PDF based on extension and MIME type * @param file - File or file-like object with name and type properties From 96a00cebd18ba703f7c5719fa348d31885cd6543 Mon Sep 17 00:00:00 2001 From: Anthony Stirling <77850077+Frooodle@users.noreply.github.com> Date: Thu, 20 Aug 2026 12:00:03 +0000 Subject: [PATCH 35/59] Pdf ua converter testing (#7301) # Description of Changes Adds a PDF/UA converter, an accessibility report, and PDF/A conformance level A. **New: `POST /api/v1/convert/pdf/ua`** (Convert tool, "PDF/UA" target). Tags an untagged PDF, marks decorative content as artifacts, embeds missing fonts and applies the document-level PDF/UA requirements (title, language, tab order, form-field descriptions), then validates with veraPDF. The `pdfuaid` declaration is written only if validation passes, so a returned file never claims more than it delivers; response headers report whether it was declared, how many checks still fail and how many images still need a description. **New: `POST /api/v1/security/accessibility-report`.** Reports what fails, what the converter can fix on its own, what needs a person, and lists the figures needing a description with the keys the conversion accepts back. Read-only; does not modify the file. Capped at 100 MB / 2000 pages and weighted `LARGE_WEIGHT`, since it runs a full veraPDF pass plus the converter's layout analysis over every page. **PDF/A level A.** `pdfa-1a`, `pdfa-2a` and `pdfa-3a` output formats on the existing `/api/v1/convert/pdf/pdfa` endpoint. Level A is level B plus tagging, so the document is tagged after Ghostscript (which discards any structure tree it is given) and the level A claim is written only if veraPDF agrees. Optional `pdfUa=true` additionally declares PDF/UA alongside PDF/A, again only if it validates. Honesty rules the implementation holds to: - **Never claim a level that was not reached.** If tagging fails, the file is returned at level B and is named `_PDFA-2b.pdf`, not `_PDFA-2a.pdf`. With `strict=true` the request fails outright rather than returning a level B file against a level A request, and a level B pass no longer satisfies a strict level A request. - **Never relabel a document's language.** The requested language (default `en-GB`) is applied only when the document declares none; a French PDF stays French unless the caller sets `overrideLanguage`, and ignoring a requested language is reported as a warning. - **Never invent alternative text.** Descriptions come from the caller. The Convert panel can list the images needing one (via the report endpoint) and send them back per figure; any image left undescribed blocks the conformance claim rather than being papered over. - **Never certify hidden content.** Marking images decorative, or suppressing text that could not be tagged reliably, withdraws the claim instead of passing the checker by hiding content. PDF/UA-1 and PDF/UA-2 are both offered; UA-2 raises the file to PDF 2.0 and namespaces the structure tree, and its test asserts conformance rather than merely reporting it. Convert steps saved in Automations/Pipelines round-trip their PDF/UA settings (profile, language, override, title, font embedding, descriptions). --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details. --- .../SPDF/config/EndpointConfiguration.java | 19 +- .../service/PdfaLevelAServiceInterface.java | 22 + .../config/EndpointConfigurationGapTest.java | 32 +- .../api/converters/ConvertPDFToPDFA.java | 171 +++- .../api/converters/PdfToPdfARequest.java | 12 +- .../software/SPDF/service/VeraPDFService.java | 2 + .../converters/ConvertPDFToPDFAGapTest.java | 136 ++- .../converters/ConvertPDFToPDFAMoreTest.java | 5 +- .../VeraPDFServicePdfaFixtureTest.java | 17 +- app/proprietary/build.gradle | 9 + .../api/converters/ConvertPdfToPdfUa.java | 176 ++++ .../AccessibilityReportController.java | 67 ++ .../api/converters/PdfToPdfUaRequest.java | 73 ++ .../model/api/ua/AccessibilityIssue.java | 38 + .../model/api/ua/AccessibilityReport.java | 63 ++ .../api/ua/AccessibilityReportRequest.java | 19 + .../model/api/ua/FigureDescriptor.java | 18 + .../model/api/ua/PdfUaConversionOutcome.java | 26 + .../model/api/ua/UaValidationResult.java | 18 + .../proprietary/pdf/ua/ArtifactType.java | 23 + .../software/proprietary/pdf/ua/BBox.java | 48 + .../proprietary/pdf/ua/DocumentStructure.java | 84 ++ .../proprietary/pdf/ua/LayoutAnalyzer.java | 831 ++++++++++++++++++ .../proprietary/pdf/ua/MarkableOp.java | 44 + .../pdf/ua/MarkedContentInjector.java | 284 ++++++ .../proprietary/pdf/ua/PageContent.java | 33 + .../pdf/ua/PdfUaIdentificationSchema.java | 47 + .../pdf/ua/PdfUaMetadataWriter.java | 224 +++++ .../proprietary/pdf/ua/PdfUaProfile.java | 47 + .../proprietary/pdf/ua/PdfUaTagger.java | 303 +++++++ .../proprietary/pdf/ua/SourceFacts.java | 59 ++ .../proprietary/pdf/ua/StructBlock.java | 134 +++ .../proprietary/pdf/ua/StructTreeWriter.java | 295 +++++++ .../proprietary/pdf/ua/StructType.java | 62 ++ .../pdf/ua/TaggedContentExtractor.java | 630 +++++++++++++ .../proprietary/pdf/ua/TaggingOptions.java | 63 ++ .../proprietary/pdf/ua/TaggingResult.java | 42 + .../proprietary/pdf/ua/TextLineInfo.java | 42 + .../software/proprietary/pdf/ua/WordInfo.java | 18 + .../service/ua/AccessibilityAuditService.java | 187 ++++ .../service/ua/FontEmbeddingService.java | 254 ++++++ .../service/ua/PdfUaConversionService.java | 251 ++++++ .../service/ua/PdfUaValidationService.java | 245 ++++++ .../service/ua/PdfaAccessibilityService.java | 297 +++++++ .../pdf/ua/LayoutAnalyzerTest.java | 286 ++++++ .../pdf/ua/MarkedContentInjectorTest.java | 164 ++++ .../pdf/ua/MarkedContentSafetyTest.java | 193 ++++ .../pdf/ua/PdfUaFormAndDeclarationTest.java | 171 ++++ .../proprietary/pdf/ua/PdfUaLanguageTest.java | 79 ++ .../pdf/ua/PdfUaMetadataWriterTest.java | 137 +++ .../proprietary/pdf/ua/PdfUaModelTest.java | 160 ++++ .../pdf/ua/VectorAndHeadingTest.java | 155 ++++ .../service/ua/AltTextRoundTripTest.java | 115 +++ .../service/ua/PdfUa2ProfileTest.java | 92 ++ .../service/ua/PdfUaBenchmarkTest.java | 341 +++++++ .../ua/PdfUaConversionIntegrationTest.java | 231 +++++ .../service/ua/PdfUaHardeningTest.java | 322 +++++++ .../service/ua/PdfUaHttpEndpointTest.java | 183 ++++ .../service/ua/PdfUaRealCorpusTest.java | 249 ++++++ .../service/ua/PdfUaSampleDumpTest.java | 103 +++ .../service/ua/PdfUaServicesTest.java | 319 +++++++ .../service/ua/PdfUaTestDocuments.java | 390 ++++++++ .../service/ua/PdfaLevelATest.java | 202 +++++ .../TaggedContentExtractorRealFilesTest.java | 99 +++ engine/src/stirling/models/tool_io.py | 4 + engine/src/stirling/models/tool_models.py | 89 ++ .../public/locales/en-US/translation.toml | 23 + .../tools/convert/ConvertSettings.tsx | 15 + .../ConvertToPdfUaSettings.selection.test.tsx | 149 ++++ .../convert/ConvertToPdfUaSettings.test.ts | 34 + .../tools/convert/ConvertToPdfUaSettings.tsx | 280 ++++++ .../tools/convert/ConvertToPdfaSettings.tsx | 11 + .../src/core/constants/convertConstants.ts | 6 + .../tools/convert/convertPdfUaAltText.test.ts | 92 ++ .../tools/convert/useConvertOperation.ts | 55 +- .../tools/convert/useConvertParameters.ts | 17 + .../hooks/tools/shared/toolAutomation.test.ts | 46 + .../tests/convert/ConvertIntegration.test.tsx | 96 ++ .../editor/src/core/types/toolApiTypes.ts | 53 ++ frontend/editor/src/core/types/toolIO.ts | 10 + 80 files changed, 10373 insertions(+), 68 deletions(-) create mode 100644 app/common/src/main/java/stirling/software/common/service/PdfaLevelAServiceInterface.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/controller/api/converters/ConvertPdfToPdfUa.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/controller/api/security/AccessibilityReportController.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/converters/PdfToPdfUaRequest.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityIssue.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReport.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReportRequest.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/FigureDescriptor.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/PdfUaConversionOutcome.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/UaValidationResult.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/ArtifactType.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/BBox.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/DocumentStructure.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzer.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkableOp.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkedContentInjector.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PageContent.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaIdentificationSchema.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriter.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaProfile.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaTagger.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/SourceFacts.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructBlock.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructTreeWriter.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructType.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggedContentExtractor.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingOptions.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingResult.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TextLineInfo.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/WordInfo.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/service/ua/AccessibilityAuditService.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/service/ua/FontEmbeddingService.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaConversionService.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaValidationService.java create mode 100644 app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfaAccessibilityService.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzerTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentInjectorTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentSafetyTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaFormAndDeclarationTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaLanguageTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriterTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaModelTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/VectorAndHeadingTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/AltTextRoundTripTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUa2ProfileTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaBenchmarkTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaConversionIntegrationTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHardeningTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHttpEndpointTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaRealCorpusTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaSampleDumpTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaServicesTest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaTestDocuments.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfaLevelATest.java create mode 100644 app/proprietary/src/test/java/stirling/software/proprietary/service/ua/TaggedContentExtractorRealFilesTest.java create mode 100644 frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.selection.test.tsx create mode 100644 frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.test.ts create mode 100644 frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.tsx create mode 100644 frontend/editor/src/core/hooks/tools/convert/convertPdfUaAltText.test.ts diff --git a/app/common/src/main/java/stirling/software/SPDF/config/EndpointConfiguration.java b/app/common/src/main/java/stirling/software/SPDF/config/EndpointConfiguration.java index ff1a880010..5e8f7fe336 100644 --- a/app/common/src/main/java/stirling/software/SPDF/config/EndpointConfiguration.java +++ b/app/common/src/main/java/stirling/software/SPDF/config/EndpointConfiguration.java @@ -6,6 +6,7 @@ import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.stereotype.Service; @@ -13,6 +14,7 @@ import lombok.Getter; import lombok.extern.slf4j.Slf4j; import stirling.software.common.model.ApplicationProperties; +import stirling.software.common.service.PdfaLevelAServiceInterface; @Service @Slf4j @@ -51,12 +53,16 @@ public class EndpointConfiguration { private Map groupDisableReasons = new ConcurrentHashMap<>(); private Map> endpointAlternatives = new ConcurrentHashMap<>(); private final boolean runningProOrHigher; + private final boolean pdfUaAvailable; public EndpointConfiguration( ApplicationProperties applicationProperties, - @Qualifier("runningProOrHigher") boolean runningProOrHigher) { + @Qualifier("runningProOrHigher") boolean runningProOrHigher, + @Autowired(required = false) PdfaLevelAServiceInterface pdfaLevelAService) { this.applicationProperties = applicationProperties; this.runningProOrHigher = runningProOrHigher; + // The PDF/UA tagger ships in the proprietary module, and so do its endpoints. + this.pdfUaAvailable = pdfaLevelAService != null; init(); processEnvironmentConfigs(); } @@ -356,6 +362,7 @@ public class EndpointConfiguration { addEndpointToGroup("Convert", "pdf-to-img"); addEndpointToGroup("Convert", "img-to-pdf"); addEndpointToGroup("Convert", "pdf-to-pdfa"); + addEndpointToGroup("Convert", "pdf-to-ua"); addEndpointToGroup("Convert", "file-to-pdf"); addEndpointToGroup("Convert", "pdf-to-word"); addEndpointToGroup("Convert", "pdf-to-presentation"); @@ -395,6 +402,7 @@ public class EndpointConfiguration { // Backend-only endpoints (not in frontend tool registry endpoints) addEndpointToGroup("Security", "redact"); addEndpointToGroup("Security", "verify-pdf"); + addEndpointToGroup("Security", "accessibility-report"); addEndpointToGroup("Security", "sign"); // Adding endpoints to "Other" group @@ -529,6 +537,8 @@ public class EndpointConfiguration { addEndpointToGroup("Java", "json-to-pdf"); addEndpointToGroup("Java", "pdf-to-video"); addEndpointToGroup("Java", "verify-pdf"); + addEndpointToGroup("Java", "pdf-to-ua"); + addEndpointToGroup("Java", "accessibility-report"); addEndpointToGroup("Java", "flatten"); addEndpointToGroup("Java", "unlock-pdf-forms"); addEndpointToGroup("Java", "validate-signature"); @@ -600,6 +610,8 @@ public class EndpointConfiguration { // veraPDF dependent endpoints addEndpointToGroup("veraPDF", "verify-pdf"); + addEndpointToGroup("veraPDF", "pdf-to-ua"); + addEndpointToGroup("veraPDF", "accessibility-report"); // Pdftohtml dependent endpoints addEndpointToGroup("Pdftohtml", "pdf-to-html"); @@ -630,6 +642,11 @@ public class EndpointConfiguration { disableGroup("enterprise"); } + if (!pdfUaAvailable) { + disableEndpoint("pdf-to-ua"); + disableEndpoint("accessibility-report"); + } + if (!applicationProperties.getSystem().isEnableUrlToPDF()) { disableEndpoint("url-to-pdf"); } diff --git a/app/common/src/main/java/stirling/software/common/service/PdfaLevelAServiceInterface.java b/app/common/src/main/java/stirling/software/common/service/PdfaLevelAServiceInterface.java new file mode 100644 index 0000000000..2ee55719b2 --- /dev/null +++ b/app/common/src/main/java/stirling/software/common/service/PdfaLevelAServiceInterface.java @@ -0,0 +1,22 @@ +package stirling.software.common.service; + +import java.util.List; + +/** + * Raises a converted PDF/A file from conformance level B to level A, which needs the tagging the + * PDF/UA tagger does. Implemented only in the proprietary module; core builds convert at level B. + */ +public interface PdfaLevelAServiceInterface { + + /** + * @param levelA true only when the file was tagged and validated, so the claim is never a guess + */ + record Result(byte[] pdfBytes, boolean levelA, List warnings) {} + + /** + * @param part PDF/A part, 1 to 3; part 1 keeps its PDF 1.4 version + * @param alsoDeclareUa additionally claim PDF/UA, but only if it validates + */ + Result upgradeToLevelA( + byte[] pdfBytes, int part, String language, String title, boolean alsoDeclareUa); +} diff --git a/app/common/src/test/java/stirling/software/SPDF/config/EndpointConfigurationGapTest.java b/app/common/src/test/java/stirling/software/SPDF/config/EndpointConfigurationGapTest.java index 6275b49343..afc6fa7020 100644 --- a/app/common/src/test/java/stirling/software/SPDF/config/EndpointConfigurationGapTest.java +++ b/app/common/src/test/java/stirling/software/SPDF/config/EndpointConfigurationGapTest.java @@ -17,6 +17,7 @@ import org.junit.jupiter.api.Test; import stirling.software.SPDF.config.EndpointConfiguration.DisableReason; import stirling.software.SPDF.config.EndpointConfiguration.EndpointAvailability; import stirling.software.common.model.ApplicationProperties; +import stirling.software.common.service.PdfaLevelAServiceInterface; /** * Unit tests for {@link EndpointConfiguration}. The class wires up its endpoint/group registry in @@ -32,7 +33,14 @@ class EndpointConfigurationGapTest { * Construct an EndpointConfiguration with the given pro flag and current applicationProperties. */ private EndpointConfiguration build(boolean runningProOrHigher) { - return new EndpointConfiguration(applicationProperties, runningProOrHigher); + return build(runningProOrHigher, null); + } + + /** The PDF/UA service is only present in proprietary builds, so it is injected separately. */ + private EndpointConfiguration build( + boolean runningProOrHigher, PdfaLevelAServiceInterface pdfaLevelAService) { + return new EndpointConfiguration( + applicationProperties, runningProOrHigher, pdfaLevelAService); } /** Default config: not pro, no removals, url-to-pdf disabled (default System flag is false). */ @@ -177,6 +185,28 @@ class EndpointConfigurationGapTest { } } + @Nested + @DisplayName("PDF/UA availability") + class PdfUaTests { + + @Test + @DisplayName("the PDF/UA endpoints are off when the proprietary tagger is absent") + void disabledWithoutTagger() { + EndpointConfiguration config = build(false, null); + assertFalse(config.isEndpointEnabled("pdf-to-ua")); + assertFalse(config.isEndpointEnabled("accessibility-report")); + } + + @Test + @DisplayName("they are on once the tagger is on the classpath") + void enabledWithTagger() { + EndpointConfiguration config = + build(false, (pdfBytes, part, language, title, alsoDeclareUa) -> null); + assertTrue(config.isEndpointEnabled("pdf-to-ua")); + assertTrue(config.isEndpointEnabled("accessibility-report")); + } + } + @Nested @DisplayName("group enable / disable") class GroupTests { diff --git a/app/core/src/main/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFA.java b/app/core/src/main/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFA.java index 49bf4e4895..0354817315 100644 --- a/app/core/src/main/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFA.java +++ b/app/core/src/main/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFA.java @@ -11,6 +11,7 @@ import java.time.Instant; import java.time.ZoneId; import java.time.ZonedDateTime; import java.util.*; +import java.util.Locale; import java.util.regex.Pattern; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -71,6 +72,7 @@ import org.apache.xmpbox.schema.PDFAIdentificationSchema; import org.apache.xmpbox.schema.XMPBasicSchema; import org.apache.xmpbox.xml.DomXmpParser; import org.apache.xmpbox.xml.XmpSerializer; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.core.io.Resource; import org.springframework.http.HttpStatus; import org.springframework.http.MediaType; @@ -83,7 +85,6 @@ import io.github.pixee.security.Filenames; import io.swagger.v3.oas.annotations.Operation; import lombok.Getter; -import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import stirling.software.SPDF.model.api.converters.PdfToPdfARequest; @@ -93,6 +94,7 @@ import stirling.software.common.configuration.RuntimePathConfig; import stirling.software.common.enumeration.ResourceWeight; import stirling.software.common.model.tool.ToolFormat; import stirling.software.common.model.tool.ToolIO; +import stirling.software.common.service.PdfaLevelAServiceInterface; import stirling.software.common.util.ExceptionUtils; import stirling.software.common.util.ProcessExecutor; import stirling.software.common.util.ProcessExecutor.ProcessExecutorResult; @@ -102,14 +104,26 @@ import stirling.software.common.util.WebResponseUtils; @ConvertApi @Slf4j -@RequiredArgsConstructor public class ConvertPDFToPDFA { private static final Pattern NON_PRINTABLE_ASCII = Pattern.compile("[^\\x20-\\x7E]"); private final RuntimePathConfig runtimePathConfig; private final stirling.software.SPDF.service.VeraPDFService veraPDFService; + // Level A needs the proprietary tagger; core builds convert at level B instead. + private final PdfaLevelAServiceInterface pdfaLevelAService; private final TempFileManager tempFileManager; + public ConvertPDFToPDFA( + RuntimePathConfig runtimePathConfig, + stirling.software.SPDF.service.VeraPDFService veraPDFService, + @Autowired(required = false) PdfaLevelAServiceInterface pdfaLevelAService, + TempFileManager tempFileManager) { + this.runtimePathConfig = runtimePathConfig; + this.veraPDFService = veraPDFService; + this.pdfaLevelAService = pdfaLevelAService; + this.tempFileManager = tempFileManager; + } + private static final String ICC_RESOURCE_PATH = "/icc/sRGB2014.icc"; private static final int PDFA_COMPATIBILITY_POLICY = 1; @@ -604,7 +618,10 @@ public class ConvertPDFToPDFA { return handlePdfXConversion(inputFile, outputFormat); } else { return handlePdfAConversion( - inputFile, outputFormat, request.getStrict() != null && request.getStrict()); + inputFile, + outputFormat, + request.getStrict() != null && request.getStrict(), + request.getPdfUa() != null && request.getPdfUa()); } } @@ -1815,8 +1832,64 @@ public class ConvertPDFToPDFA { return Files.readAllBytes(outputPdf); } + /** Tags a converted PDF/A for level A; must run after Ghostscript, which discards tags. */ + private PdfaLevelAServiceInterface.Result applyLevelA( + byte[] converted, + Path original, + PdfaProfile profile, + String baseFileName, + boolean declarePdfUa) { + if (!profile.requiresTagging()) { + return new PdfaLevelAServiceInterface.Result(converted, true, List.of()); + } + if (pdfaLevelAService == null) { + return new PdfaLevelAServiceInterface.Result( + converted, + false, + List.of( + "Level A tagging is not available in this build, so the file was left" + + " at conformance level B.")); + } + // Prefer the document's own title/language; hardcoding "en" mislabelled German reports. + // Read the original, not the converted bytes: Ghostscript discards /Lang, so probing its + // output always yields null and every document would be relabelled with the default. + String language = null; + String title = null; + try (PDDocument probe = Loader.loadPDF(original.toFile())) { + language = probe.getDocumentCatalog().getLanguage(); + title = probe.getDocumentInformation().getTitle(); + } catch (IOException e) { + log.debug("Could not read original title/language: {}", e.getMessage()); + } + if (language == null || language.isBlank()) { + try (PDDocument probe = Loader.loadPDF(converted)) { + language = probe.getDocumentCatalog().getLanguage(); + if (title == null || title.isBlank()) { + title = probe.getDocumentInformation().getTitle(); + } + } catch (IOException e) { + log.debug("Could not read converted title/language: {}", e.getMessage()); + } + } + PdfaLevelAServiceInterface.Result result = + pdfaLevelAService.upgradeToLevelA( + converted, + profile.getPart(), + language, + title != null && !title.isBlank() ? title : baseFileName, + declarePdfUa); + result.warnings().forEach(warning -> log.info("PDF/A level A: {}", warning)); + if (!result.levelA()) { + log.warn( + "{} requested but the document could not be tagged; returning level B", + profile.getDisplayName()); + } + return result; + } + private ResponseEntity handlePdfAConversion( - MultipartFile inputFile, String outputFormat, boolean strict) throws Exception { + MultipartFile inputFile, String outputFormat, boolean strict, boolean declarePdfUa) + throws Exception { PdfaProfile profile = PdfaProfile.fromRequest(outputFormat); // Get the original filename without extension @@ -1841,12 +1914,15 @@ public class ConvertPDFToPDFA { log.info("Using Ghostscript for PDF/A conversion to {}", profile.getDisplayName()); try { converted = convertWithGhostscript(inputPath, workingDir, profile); - String outputFilename = baseFileName + profile.outputSuffix(); + var levelA = + applyLevelA(converted, inputPath, profile, baseFileName, declarePdfUa); + converted = levelA.pdfBytes(); + String outputFilename = baseFileName + profile.outputSuffix(levelA.levelA()); validateAndWarnPdfA(converted, profile, "Ghostscript"); if (strict) { - verifyStrictCompliance(converted); + verifyStrictCompliance(converted, profile, levelA.levelA()); } TempFile tempOut = tempFileManager.createManagedTempFile(".pdf"); @@ -1867,13 +1943,15 @@ public class ConvertPDFToPDFA { } converted = convertWithPdfBoxMethod(inputPath, profile); - String outputFilename = baseFileName + profile.outputSuffix(); + var levelA = applyLevelA(converted, inputPath, profile, baseFileName, declarePdfUa); + converted = levelA.pdfBytes(); + String outputFilename = baseFileName + profile.outputSuffix(levelA.levelA()); // Validate with PDFBox preflight and warn if issues found validateAndWarnPdfA(converted, profile, "PDFBox/LibreOffice"); if (strict) { - verifyStrictCompliance(converted); + verifyStrictCompliance(converted, profile, levelA.levelA()); } TempFile tempOut = tempFileManager.createManagedTempFile(".pdf"); @@ -1889,11 +1967,56 @@ public class ConvertPDFToPDFA { } } - private void verifyStrictCompliance(byte[] pdfBytes) throws IOException { + /** True for a PDF/UA or WCAG result, which says nothing about archival conformance. */ + private static boolean isAccessibilityProfile( + stirling.software.SPDF.model.api.security.PDFVerificationResult result) { + String profile = result.getValidationProfile(); + if (profile == null) { + return false; + } + String normalised = profile.toLowerCase(Locale.ROOT); + return normalised.contains("ua") || normalised.contains("wcag"); + } + + /** + * True when a result speaks for the requested profile. Only archival results count, and a level + * B pass must never satisfy a level A request. + */ + private static boolean answersRequest( + PdfaProfile profile, + stirling.software.SPDF.model.api.security.PDFVerificationResult result) { + if (isAccessibilityProfile(result)) { + return false; + } + String standard = result.getStandard(); + if (standard == null || standard.length() < 2) { + return false; + } + if (standard.charAt(0) != Character.forDigit(profile.getPart(), 10)) { + return false; + } + return !profile.requiresTagging() || Character.toLowerCase(standard.charAt(1)) == 'a'; + } + + private void verifyStrictCompliance(byte[] pdfBytes, PdfaProfile profile, boolean levelAReached) + throws IOException { + // Tagging is the only route to level A, so an untagged file cannot answer a strict request. + if (!levelAReached) { + throw new ResponseStatusException( + HttpStatus.BAD_REQUEST, + "Strict PDF/A mode enabled: the document could not be tagged, so " + + profile.getDisplayName() + + " was not reached. It is valid at level B."); + } try (InputStream is = new ByteArrayInputStream(pdfBytes)) { List results = veraPDFService.validatePDF(is); - boolean isCompliant = results.stream().anyMatch(result -> result.isCompliant()); + boolean isCompliant = + results.stream() + .filter(result -> answersRequest(profile, result)) + .anyMatch( + stirling.software.SPDF.model.api.security.PDFVerificationResult + ::isCompliant); if (!isCompliant) { String details = results.stream() @@ -1901,7 +2024,9 @@ public class ConvertPDFToPDFA { .collect(Collectors.joining("; ")); throw new ResponseStatusException( HttpStatus.BAD_REQUEST, - "Strict PDF/A mode enabled: Conversion is not perfectly compliant. Details: " + "Strict PDF/A mode enabled: the output is not perfectly compliant with " + + profile.getDisplayName() + + ". Details: " + details); } } catch (Exception e) { @@ -2466,11 +2591,16 @@ public class ConvertPDFToPDFA { @Getter private enum PdfaProfile { - PDF_A_1B(1, "PDF/A-1b", "_PDFA-1b.pdf", "1.4", Format.PDF_A1B, "pdfa-1"), - PDF_A_2B(2, "PDF/A-2b", "_PDFA-2b.pdf", "1.7", null, "pdfa", "pdfa-2", "pdfa-2b"), - PDF_A_3B(3, "PDF/A-3b", "_PDFA-3b.pdf", "1.7", null, "pdfa-3", "pdfa-3b"); + PDF_A_1B(1, "B", "PDF/A-1b", "_PDFA-1b.pdf", "1.4", Format.PDF_A1B, "pdfa-1"), + PDF_A_2B(2, "B", "PDF/A-2b", "_PDFA-2b.pdf", "1.7", null, "pdfa", "pdfa-2", "pdfa-2b"), + PDF_A_3B(3, "B", "PDF/A-3b", "_PDFA-3b.pdf", "1.7", null, "pdfa-3", "pdfa-3b"), + // Level A = level B plus tagging, declared language and Unicode text; tagged post-convert. + PDF_A_1A(1, "A", "PDF/A-1a", "_PDFA-1a.pdf", "1.4", Format.PDF_A1B, "pdfa-1a"), + PDF_A_2A(2, "A", "PDF/A-2a", "_PDFA-2a.pdf", "1.7", null, "pdfa-2a"), + PDF_A_3A(3, "A", "PDF/A-3a", "_PDFA-3a.pdf", "1.7", null, "pdfa-3a"); private final int part; + private final String conformanceLevel; private final String displayName; private final String suffix; private final String compatibilityLevel; @@ -2479,12 +2609,14 @@ public class ConvertPDFToPDFA { PdfaProfile( int part, + String conformanceLevel, String displayName, String suffix, String compatibilityLevel, Format preflightFormat, String... requestTokens) { this.part = part; + this.conformanceLevel = conformanceLevel; this.displayName = displayName; this.suffix = suffix; this.compatibilityLevel = compatibilityLevel; @@ -2495,6 +2627,10 @@ public class ConvertPDFToPDFA { .toList(); } + boolean requiresTagging() { + return "A".equals(conformanceLevel); + } + static PdfaProfile fromRequest(String requestToken) { if (requestToken == null) { return PDF_A_2B; @@ -2508,8 +2644,11 @@ public class ConvertPDFToPDFA { return match.orElse(PDF_A_2B); } - String outputSuffix() { - return suffix; + /** + * Names the file at the level actually reached; a level A name over level B content lies. + */ + String outputSuffix(boolean levelAReached) { + return levelAReached ? suffix : "_PDFA-" + part + "b.pdf"; } Optional preflightFormat() { diff --git a/app/core/src/main/java/stirling/software/SPDF/model/api/converters/PdfToPdfARequest.java b/app/core/src/main/java/stirling/software/SPDF/model/api/converters/PdfToPdfARequest.java index bb0520a4ba..921663912b 100644 --- a/app/core/src/main/java/stirling/software/SPDF/model/api/converters/PdfToPdfARequest.java +++ b/app/core/src/main/java/stirling/software/SPDF/model/api/converters/PdfToPdfARequest.java @@ -14,9 +14,19 @@ public class PdfToPdfARequest extends PDFFile { @Schema( description = "The output format type (PDF/A or PDF/X)", requiredMode = Schema.RequiredMode.REQUIRED, - allowableValues = {"pdfa", "pdfa-1", "pdfa-2", "pdfa-2b", "pdfa-3", "pdfa-3b", "pdfx"}) + allowableValues = { + "pdfa", "pdfa-1", "pdfa-2", "pdfa-2b", "pdfa-3", "pdfa-3b", "pdfa-1a", "pdfa-2a", + "pdfa-3a", "pdfx" + }) private String outputFormat; + @Schema( + description = + "Also declare PDF/UA accessibility alongside PDF/A. Only applies to the level A" + + " formats, and the claim is written only if it validates.", + defaultValue = "false") + private Boolean pdfUa; + @Schema( description = "If true, the conversion will fail if the output is not perfectly compliant") diff --git a/app/core/src/main/java/stirling/software/SPDF/service/VeraPDFService.java b/app/core/src/main/java/stirling/software/SPDF/service/VeraPDFService.java index bb3c84534c..6361157b21 100644 --- a/app/core/src/main/java/stirling/software/SPDF/service/VeraPDFService.java +++ b/app/core/src/main/java/stirling/software/SPDF/service/VeraPDFService.java @@ -285,6 +285,8 @@ public class VeraPDFService { } } + // Never force PDF/UA here - it flags every ordinary document as non-compliant and doubles + // verify cost; /accessibility-report checks PDF/UA on demand. if (!hasPdfaDeclaration) { results.add(createNoPdfaDeclarationResult()); } diff --git a/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAGapTest.java b/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAGapTest.java index b64776665b..ea693ddd27 100644 --- a/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAGapTest.java +++ b/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAGapTest.java @@ -46,6 +46,7 @@ import stirling.software.SPDF.model.api.converters.PdfToPdfARequest; import stirling.software.SPDF.model.api.security.PDFVerificationResult; import stirling.software.SPDF.service.VeraPDFService; import stirling.software.common.configuration.RuntimePathConfig; +import stirling.software.common.service.PdfaLevelAServiceInterface; import stirling.software.common.util.TempFileManager; /** @@ -62,10 +63,12 @@ class ConvertPDFToPDFAGapTest { @Mock private RuntimePathConfig runtimePathConfig; @Mock private VeraPDFService veraPDFService; + @Mock private PdfaLevelAServiceInterface pdfaLevelAService; @Mock private TempFileManager tempFileManager; private ConvertPDFToPDFA newController() { - return new ConvertPDFToPDFA(runtimePathConfig, veraPDFService, tempFileManager); + return new ConvertPDFToPDFA( + runtimePathConfig, veraPDFService, pdfaLevelAService, tempFileManager); } // ---- reflection helpers ---------------------------------------------------------------- @@ -161,9 +164,21 @@ class ConvertPDFToPDFAGapTest { } private String suffixOf(Object profile) throws Exception { - Method m = profile.getClass().getDeclaredMethod("outputSuffix"); + return suffixOf(profile, true); + } + + private String suffixOf(Object profile, boolean levelAReached) throws Exception { + Method m = profile.getClass().getDeclaredMethod("outputSuffix", boolean.class); m.setAccessible(true); - return (String) m.invoke(profile); + return (String) m.invoke(profile, levelAReached); + } + + @Test + @DisplayName("a level A profile falls back to the level B name when tagging failed") + void levelANotReachedIsNamedLevelB() throws Exception { + assertThat(suffixOf(resolveProfile("pdfa-1a"), false)).isEqualTo("_PDFA-1b.pdf"); + assertThat(suffixOf(resolveProfile("pdfa-2a"), false)).isEqualTo("_PDFA-2b.pdf"); + assertThat(suffixOf(resolveProfile("pdfa-3a"), true)).isEqualTo("_PDFA-3a.pdf"); } @Test @@ -717,6 +732,30 @@ class ConvertPDFToPDFAGapTest { @DisplayName("verifyStrictCompliance (VeraPDFService mocked)") class StrictCompliance { + private Object profile(String token) throws Exception { + Class enumClass = null; + for (Class inner : ConvertPDFToPDFA.class.getDeclaredClasses()) { + if (inner.getSimpleName().equals("PdfaProfile")) { + enumClass = inner; + } + } + Method m = enumClass.getDeclaredMethod("fromRequest", String.class); + m.setAccessible(true); + return m.invoke(null, token); + } + + private Throwable verify(String token, boolean levelAReached) throws Exception { + ConvertPDFToPDFA controller = newController(); + return catchThrowable( + () -> + invokeInstance( + controller, + "verifyStrictCompliance", + (Object) "dummy".getBytes(), + profile(token), + levelAReached)); + } + @Test @DisplayName("compliant result passes without throwing") void compliantPasses() throws Exception { @@ -726,14 +765,7 @@ class ConvertPDFToPDFAGapTest { ok.setComplianceSummary("PDF/A-1b compliant"); when(veraPDFService.validatePDF(any())).thenReturn(List.of(ok)); - ConvertPDFToPDFA controller = newController(); - assertThatCode( - () -> - invokeInstance( - controller, - "verifyStrictCompliance", - (Object) "dummy".getBytes())) - .doesNotThrowAnyException(); + assertThat(verify("pdfa-1", true)).isNull(); } @Test @@ -745,34 +777,70 @@ class ConvertPDFToPDFAGapTest { bad.setComplianceSummary("PDF/A-1b with errors"); when(veraPDFService.validatePDF(any())).thenReturn(List.of(bad)); - ConvertPDFToPDFA controller = newController(); - ResponseStatusException ex = - (ResponseStatusException) - catchThrowable( - () -> - invokeInstance( - controller, - "verifyStrictCompliance", - (Object) "dummy".getBytes())); + ResponseStatusException ex = (ResponseStatusException) verify("pdfa-1", true); assertThat(ex).isNotNull(); assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST); assertThat(ex.getReason()).contains("PDF/A-1b with errors"); } + @Test + @DisplayName("a level B pass does not satisfy a level A request") + void levelBDoesNotSatisfyLevelA() throws Exception { + PDFVerificationResult ok = new PDFVerificationResult(); + ok.setCompliant(true); + ok.setStandard("1b"); + ok.setComplianceSummary("PDF/A-1b compliant"); + when(veraPDFService.validatePDF(any())).thenReturn(List.of(ok)); + + ResponseStatusException ex = (ResponseStatusException) verify("pdfa-1a", true); + assertThat(ex).isNotNull(); + assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST); + assertThat(ex.getReason()).contains("PDF/A-1a"); + } + + @Test + @DisplayName("a level A result satisfies a level A request") + void levelASatisfiesLevelA() throws Exception { + PDFVerificationResult ok = new PDFVerificationResult(); + ok.setCompliant(true); + ok.setStandard("2a"); + ok.setComplianceSummary("PDF/A-2a compliant"); + when(veraPDFService.validatePDF(any())).thenReturn(List.of(ok)); + + assertThat(verify("pdfa-2a", true)).isNull(); + } + + @Test + @DisplayName("untagged output fails a level A request before validation runs") + void untaggedLevelARequestFails() throws Exception { + ResponseStatusException ex = (ResponseStatusException) verify("pdfa-2a", false); + assertThat(ex).isNotNull(); + assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST); + assertThat(ex.getReason()).contains("could not be tagged"); + verifyNoInteractions(veraPDFService); + } + + @Test + @DisplayName("a compliant PDF/UA result never satisfies a strict PDF/A request") + void accessibilityResultIsIgnored() throws Exception { + PDFVerificationResult ua = new PDFVerificationResult(); + ua.setCompliant(true); + ua.setStandard("ua1"); + ua.setValidationProfile("ua1"); + ua.setComplianceSummary("PDF/UA-1 compliant"); + when(veraPDFService.validatePDF(any())).thenReturn(List.of(ua)); + + ResponseStatusException ex = (ResponseStatusException) verify("pdfa-2b", true); + assertThat(ex).isNotNull(); + assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST); + } + @Test @DisplayName("empty result list is treated as non-compliant -> 400") void emptyResultsTreatedNonCompliant() throws Exception { when(veraPDFService.validatePDF(any())).thenReturn(Collections.emptyList()); - ConvertPDFToPDFA controller = newController(); - ResponseStatusException ex = - (ResponseStatusException) - catchThrowable( - () -> - invokeInstance( - controller, - "verifyStrictCompliance", - (Object) "dummy".getBytes())); + ResponseStatusException ex = (ResponseStatusException) verify("pdfa-1", true); assertThat(ex).isNotNull(); assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST); } @@ -782,15 +850,7 @@ class ConvertPDFToPDFAGapTest { void serviceErrorWrappedAs500() throws Exception { when(veraPDFService.validatePDF(any())).thenThrow(new IOException("boom")); - ConvertPDFToPDFA controller = newController(); - ResponseStatusException ex = - (ResponseStatusException) - catchThrowable( - () -> - invokeInstance( - controller, - "verifyStrictCompliance", - (Object) "dummy".getBytes())); + ResponseStatusException ex = (ResponseStatusException) verify("pdfa-1", true); assertThat(ex).isNotNull(); assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.INTERNAL_SERVER_ERROR); } diff --git a/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAMoreTest.java b/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAMoreTest.java index e9d9b9ce1d..d56a283464 100644 --- a/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAMoreTest.java +++ b/app/core/src/test/java/stirling/software/SPDF/controller/api/converters/ConvertPDFToPDFAMoreTest.java @@ -42,6 +42,7 @@ import org.springframework.mock.web.MockMultipartFile; import stirling.software.SPDF.model.api.converters.PdfToPdfARequest; import stirling.software.SPDF.service.VeraPDFService; import stirling.software.common.configuration.RuntimePathConfig; +import stirling.software.common.service.PdfaLevelAServiceInterface; import stirling.software.common.util.ProcessExecutor; import stirling.software.common.util.ProcessExecutor.ProcessExecutorResult; import stirling.software.common.util.TempFile; @@ -63,10 +64,12 @@ class ConvertPDFToPDFAMoreTest { @Mock private RuntimePathConfig runtimePathConfig; @Mock private VeraPDFService veraPDFService; + @Mock private PdfaLevelAServiceInterface pdfaLevelAService; @Mock private TempFileManager tempFileManager; private ConvertPDFToPDFA newController() { - return new ConvertPDFToPDFA(runtimePathConfig, veraPDFService, tempFileManager); + return new ConvertPDFToPDFA( + runtimePathConfig, veraPDFService, pdfaLevelAService, tempFileManager); } private static ResponseEntity streamingOk(byte[] bytes) { diff --git a/app/core/src/test/java/stirling/software/SPDF/service/VeraPDFServicePdfaFixtureTest.java b/app/core/src/test/java/stirling/software/SPDF/service/VeraPDFServicePdfaFixtureTest.java index 38043780d9..e071ece3fb 100644 --- a/app/core/src/test/java/stirling/software/SPDF/service/VeraPDFServicePdfaFixtureTest.java +++ b/app/core/src/test/java/stirling/software/SPDF/service/VeraPDFServicePdfaFixtureTest.java @@ -90,7 +90,9 @@ class VeraPDFServicePdfaFixtureTest { () -> service.validatePDF(new ByteArrayInputStream(pdfBytes)), "Empty veraPDF flavour list must not surface as IndexOutOfBoundsException"); - assertEquals(1, results.size()); + // One result: PDF/UA is checked by the dedicated accessibility-report endpoint, not here. + assertEquals(1, results.size(), () -> "Expected a single PDF/A result, got: " + results); + PDFVerificationResult result = results.get(0); assertEquals("not-pdfa", result.getStandard()); assertFalse(result.isDeclaredPdfa()); @@ -161,13 +163,22 @@ class VeraPDFServicePdfaFixtureTest { } } + /** The PDF/A result; every document is also checked against PDF/UA, so filter that one out. */ private PDFVerificationResult onlyResult(byte[] pdfBytes) throws Exception { List results = service.validatePDF(new ByteArrayInputStream(pdfBytes)); assertNotNull(results); - assertEquals(1, results.size(), () -> "Expected a single result, got: " + results); - return results.get(0); + List pdfaResults = + results.stream().filter(r -> !isUaResult(r)).toList(); + assertEquals( + 1, pdfaResults.size(), () -> "Expected a single PDF/A result, got: " + results); + return pdfaResults.get(0); + } + + private static boolean isUaResult(PDFVerificationResult result) { + String profile = result.getValidationProfile(); + return profile != null && profile.toLowerCase().contains("ua"); } private static String messages(PDFVerificationResult result) { diff --git a/app/proprietary/build.gradle b/app/proprietary/build.gradle index c20241dbb5..b884cb18be 100644 --- a/app/proprietary/build.gradle +++ b/app/proprietary/build.gradle @@ -37,6 +37,15 @@ dependencies { // https://mvnrepository.com/artifact/com.bucket4j/bucket4j_jdk17 implementation "org.bouncycastle:bcprov-jdk18on:$bouncycastleVersion" + // PDF/UA tagging and its validation oracle. + implementation 'org.verapdf:validation-model:1.30.2' + // CVE-2025-66453: Explicit rhino 1.7.15 to override verapdf's 1.7.13 + implementation "org.mozilla:rhino:${rhinoVersion}" + // veraPDF still uses javax.xml.bind, not the new jakarta namespace + implementation 'javax.xml.bind:jaxb-api:2.3.1' + runtimeOnly 'com.sun.xml.bind:jaxb-impl:2.3.9' + runtimeOnly 'com.sun.xml.bind:jaxb-core:4.0.9' + implementation "com.google.code.gson:gson:${gsonVersion}" // jinjava/jjwt transitively request older Jackson 2 versions; declare the current diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/controller/api/converters/ConvertPdfToPdfUa.java b/app/proprietary/src/main/java/stirling/software/proprietary/controller/api/converters/ConvertPdfToPdfUa.java new file mode 100644 index 0000000000..7f5241388f --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/controller/api/converters/ConvertPdfToPdfUa.java @@ -0,0 +1,176 @@ +package stirling.software.proprietary.controller.api.converters; + +import java.io.IOException; +import java.nio.file.Files; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.regex.Pattern; + +import org.springframework.core.io.Resource; +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.web.bind.annotation.ModelAttribute; +import org.springframework.web.multipart.MultipartFile; + +import io.github.pixee.security.Filenames; +import io.swagger.v3.oas.annotations.Operation; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import stirling.software.common.annotations.AutoJobPostMapping; +import stirling.software.common.annotations.api.ConvertApi; +import stirling.software.common.enumeration.ResourceWeight; +import stirling.software.common.model.tool.ToolFormat; +import stirling.software.common.model.tool.ToolIO; +import stirling.software.common.util.ExceptionUtils; +import stirling.software.common.util.TempFile; +import stirling.software.common.util.TempFileManager; +import stirling.software.common.util.WebResponseUtils; +import stirling.software.proprietary.model.api.converters.PdfToPdfUaRequest; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; +import stirling.software.proprietary.service.ua.PdfUaConversionService; + +/** Converts a PDF to PDF/UA; response headers say whether the result actually conforms. */ +@ConvertApi +@Slf4j +@RequiredArgsConstructor +public class ConvertPdfToPdfUa { + + private static final String HEADER_DECLARED = "X-Stirling-UA-Declared"; + private static final String HEADER_FAILURES = "X-Stirling-UA-Failures"; + private static final String HEADER_ALT_NEEDED = "X-Stirling-UA-Figures-Needing-Alt"; + private static final String HEADER_WARNINGS = "X-Stirling-UA-Warnings"; + + /** Any line ending, so descriptions pasted from any platform parse the same. */ + private static final Pattern NEWLINE = Pattern.compile("\\R"); + + private final PdfUaConversionService conversionService; + private final TempFileManager tempFileManager; + + @AutoJobPostMapping( + consumes = MediaType.MULTIPART_FORM_DATA_VALUE, + value = "/pdf/ua", + resourceWeight = ResourceWeight.LARGE_WEIGHT) + @ToolIO(produces = ToolFormat.PDF) + @Operation( + summary = "Convert a PDF to PDF/UA-1 or PDF/UA-2", + description = + "Tags the document, marks decorative content as artifacts, embeds fonts and" + + " applies the document-level requirements of PDF/UA, then validates" + + " the result. A conformance declaration is written only if validation" + + " passes, so the returned file never claims more than it delivers.") + public ResponseEntity pdfToPdfUa(@ModelAttribute PdfToPdfUaRequest request) + throws IOException { + + MultipartFile input = request.getFileInput(); + if (input == null || input.isEmpty()) { + throw ExceptionUtils.createPdfFileRequiredException(); + } + + String originalName = Filenames.toSimpleFileName(input.getOriginalFilename()); + String stem = stripExtension(originalName == null ? "document" : originalName); + PdfUaProfile profile = PdfUaProfile.fromRequest(request.getProfile()); + + TaggingOptions options = + TaggingOptions.builder() + .profile(profile) + .title(request.getTitle()) + .fallbackTitle(stem) + // Only used when the document declares no language of its own. + .language( + request.getLanguage() == null || request.getLanguage().isBlank() + ? "en-GB" + : request.getLanguage()) + .overrideLanguage( + request.getOverrideLanguage() != null + && request.getOverrideLanguage()) + .existingTags(existingTags(request.getExistingTags())) + .figurePolicy(figurePolicy(request.getFigurePolicy())) + .embedFonts(request.getEmbedFonts() == null || request.getEmbedFonts()) + .altTextByFigure(parseAltText(request.getAltText())) + .build(); + + PdfUaConversionOutcome outcome = conversionService.convert(input.getBytes(), options); + + log.info( + "Converted '{}' to {}: declared={}, {} remaining failure(s)", + originalName, + profile.displayName(), + outcome.declared(), + outcome.validation().totalFailures()); + + outcome.warnings().forEach(warning -> log.info("PDF/UA warning: {}", warning)); + + // Streamed from a temp file so a large conversion does not hold a second heap copy. + String suffix = outcome.declared() ? "_pdfua" + profile.part() : "_tagged"; + TempFile tempOut = tempFileManager.createManagedTempFile(".pdf"); + try { + Files.write(tempOut.getPath(), outcome.pdfBytes()); + } catch (IOException e) { + tempOut.close(); + throw e; + } + ResponseEntity response = + WebResponseUtils.pdfFileToWebResponse(tempOut, stem + suffix + ".pdf"); + + return ResponseEntity.status(response.getStatusCode()) + .headers(response.getHeaders()) + .header(HEADER_DECLARED, String.valueOf(outcome.declared())) + .header(HEADER_FAILURES, String.valueOf(outcome.validation().totalFailures())) + .header( + HEADER_ALT_NEEDED, + String.valueOf(outcome.tagging().figuresNeedingAltText())) + // Count only: warning text is multi-line prose, which HTTP headers mangle. + .header(HEADER_WARNINGS, String.valueOf(outcome.warnings().size())) + .body(response.getBody()); + } + + /** + * Parses newline-separated {@code key=description} pairs, keyed as the report hands them out. + * Only the first "=" splits, since a description may contain one. + */ + public static Map parseAltText(String raw) { + if (raw == null || raw.isBlank()) { + return Map.of(); + } + Map parsed = new LinkedHashMap<>(); + for (String line : NEWLINE.split(raw)) { + int split = line.indexOf('='); + if (split <= 0) { + continue; + } + String key = line.substring(0, split).strip(); + String description = line.substring(split + 1).strip(); + if (!key.isEmpty() && !description.isEmpty()) { + parsed.put(key, description); + } + } + return parsed; + } + + private static TaggingOptions.ExistingTags existingTags(String value) { + if (value == null) { + return TaggingOptions.ExistingTags.AUTO; + } + return switch (value.trim().toLowerCase()) { + case "keep" -> TaggingOptions.ExistingTags.KEEP; + case "rebuild" -> TaggingOptions.ExistingTags.REBUILD; + default -> TaggingOptions.ExistingTags.AUTO; + }; + } + + private static TaggingOptions.FigurePolicy figurePolicy(String value) { + if (value != null && value.trim().equalsIgnoreCase("mark-decorative")) { + return TaggingOptions.FigurePolicy.MARK_DECORATIVE; + } + return TaggingOptions.FigurePolicy.REQUIRE_ALT; + } + + private static String stripExtension(String filename) { + int dot = filename.lastIndexOf('.'); + return dot > 0 ? filename.substring(0, dot) : filename; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/controller/api/security/AccessibilityReportController.java b/app/proprietary/src/main/java/stirling/software/proprietary/controller/api/security/AccessibilityReportController.java new file mode 100644 index 0000000000..043734dad8 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/controller/api/security/AccessibilityReportController.java @@ -0,0 +1,67 @@ +package stirling.software.proprietary.controller.api.security; + +import java.io.IOException; + +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.web.bind.annotation.ModelAttribute; +import org.springframework.web.multipart.MultipartFile; + +import io.swagger.v3.oas.annotations.Operation; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import stirling.software.common.annotations.AutoJobPostMapping; +import stirling.software.common.annotations.api.SecurityApi; +import stirling.software.common.enumeration.ResourceWeight; +import stirling.software.common.model.tool.ToolFormat; +import stirling.software.common.model.tool.ToolIO; +import stirling.software.common.util.ExceptionUtils; +import stirling.software.proprietary.model.api.ua.AccessibilityReport; +import stirling.software.proprietary.model.api.ua.AccessibilityReportRequest; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.service.ua.AccessibilityAuditService; + +/** Reports how accessible a document is, without modifying it. */ +@SecurityApi +@RequiredArgsConstructor +@Slf4j +public class AccessibilityReportController { + + private final AccessibilityAuditService auditService; + + @ToolIO(produces = ToolFormat.JSON) + @Operation( + summary = "Report a document's accessibility standing", + description = + "Validates the document against PDF/UA and reports what fails, which failures" + + " can be fixed automatically, and which checks still need a person." + + " Does not modify the file.") + // Costs a full veraPDF pass plus the converter's own layout analysis over every page. + @AutoJobPostMapping( + value = "/accessibility-report", + consumes = MediaType.MULTIPART_FORM_DATA_VALUE, + resourceWeight = ResourceWeight.LARGE_WEIGHT) + public ResponseEntity report( + @ModelAttribute AccessibilityReportRequest request) { + + MultipartFile file = request.getFileInput(); + if (file == null || file.isEmpty()) { + throw ExceptionUtils.createPdfFileRequiredException(); + } + PdfUaProfile profile = PdfUaProfile.fromRequest(request.getProfile()); + try { + AccessibilityReport report = auditService.audit(file.getBytes(), profile); + log.info( + "Accessibility report for '{}': tagged={}, {} issue(s)", + file.getOriginalFilename(), + report.isTagged(), + report.getIssues().size()); + return ResponseEntity.ok(report); + } catch (IOException e) { + throw ExceptionUtils.createRuntimeException( + "error.ioException", "Could not read the PDF: {0}", e, e.getMessage()); + } + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/converters/PdfToPdfUaRequest.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/converters/PdfToPdfUaRequest.java new file mode 100644 index 0000000000..8981398001 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/converters/PdfToPdfUaRequest.java @@ -0,0 +1,73 @@ +package stirling.software.proprietary.model.api.converters; + +import io.swagger.v3.oas.annotations.media.Schema; + +import lombok.Data; +import lombok.EqualsAndHashCode; + +import stirling.software.common.model.api.PDFFile; + +@Data +@EqualsAndHashCode(callSuper = true) +public class PdfToPdfUaRequest extends PDFFile { + + @Schema( + description = "PDF/UA conformance level to target", + defaultValue = "ua1", + allowableValues = {"ua1", "ua2"}) + private String profile; + + @Schema( + description = + "Document title, required by PDF/UA. Falls back to the first heading, then the" + + " filename.") + private String title; + + @Schema( + description = + "Document language as a BCP-47 tag, for example en-GB. Applied only when the" + + " document does not already declare one, unless overrideLanguage is" + + " set.", + defaultValue = "en-GB") + private String language; + + @Schema( + description = + "Replace the language the document already declares. Off by default, so a" + + " document is never relabelled into a language it is not written in.", + defaultValue = "false") + private Boolean overrideLanguage; + + @Schema( + description = + "What to do with an existing structure tree: keep it, rebuild it, or decide" + + " automatically", + defaultValue = "auto", + allowableValues = {"auto", "keep", "rebuild"}) + private String existingTags; + + @Schema( + description = + "How to treat images with no description. require-alt leaves them undescribed so" + + " the report asks for input; mark-decorative treats every image as" + + " decoration.", + defaultValue = "require-alt", + allowableValues = {"require-alt", "mark-decorative"}) + private String figurePolicy; + + @Schema( + description = + "Embed fonts the document references but does not carry. Required for" + + " conformance and needs Ghostscript.", + defaultValue = "true") + private Boolean embedFonts; + + @Schema( + description = + "Alternative descriptions for figures, as key=text pairs separated by newlines." + + " Keys come from the accessibility-report endpoint's" + + " figuresNeedingDescription list, for example \"0:12=Bar chart of" + + " quarterly revenue\". Descriptions are never invented, so without" + + " these an illustrated document cannot claim conformance.") + private String altText; +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityIssue.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityIssue.java new file mode 100644 index 0000000000..0c2a86ad99 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityIssue.java @@ -0,0 +1,38 @@ +package stirling.software.proprietary.model.api.ua; + +import io.swagger.v3.oas.annotations.media.Schema; + +import lombok.Data; + +/** One accessibility problem, grouped across all of its occurrences. */ +@Data +@Schema(description = "A single accessibility issue found in a document") +public class AccessibilityIssue { + + @Schema(description = "ISO 14289 clause, e.g. 7.3") + private String clause; + + @Schema(description = "Test number within the clause") + private String testNumber; + + @Schema(description = "Plain-English description of the problem") + private String message; + + @Schema(description = "The validator's own wording, for support and debugging") + private String technicalMessage; + + @Schema(description = "error or warning") + private String severity = "error"; + + @Schema(description = "Standard the check came from, e.g. PDF/UA-1") + private String specification; + + @Schema(description = "Where the problem was found, when the validator reports it") + private String location; + + @Schema(description = "How many times this issue occurs") + private int occurrences; + + @Schema(description = "True when the converter can fix this without human input") + private boolean autoFixable; +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReport.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReport.java new file mode 100644 index 0000000000..bc810635d1 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReport.java @@ -0,0 +1,63 @@ +package stirling.software.proprietary.model.api.ua; + +import java.util.List; + +import io.swagger.v3.oas.annotations.media.Schema; + +import lombok.Data; + +/** + * A document's accessibility standing. The machine/human split is load-bearing: veraPDF covers only + * about half of the Matterhorn Protocol, so a clean automated pass is not "accessible". + */ +@Data +@Schema(description = "Accessibility standing of a document") +public class AccessibilityReport { + + @Schema(description = "Profile the document was checked against, e.g. PDF/UA-1") + private String profile; + + @Schema(description = "Whether the document has a structure tree at all") + private boolean tagged; + + @Schema(description = "Whether the document declares PDF/UA conformance in its metadata") + private boolean declaresConformance; + + @Schema(description = "Whether every automated check passed") + private boolean passesAutomatedChecks; + + @Schema(description = "Automated checks that failed, grouped by rule") + private List issues = List.of(); + + @Schema(description = "Things a person still has to verify; automation cannot decide these") + private List humanChecks = List.of(); + + @Schema(description = "How many of the failing checks the converter can fix on its own") + private int automaticallyFixable; + + @Schema(description = "How many need information from the user, such as alternative text") + private int needsInput; + + @Schema( + description = + "Figures that need an alternative description. Each carries the key to pass" + + " back in the conversion request's altTextByFigure map, so a caller" + + " can enumerate what is missing and then supply it.") + private List figuresNeedingDescription = List.of(); + + @Schema(description = "Document-level facts that drive most failures") + private Summary summary = new Summary(); + + @Data + @Schema(description = "Quick document-level facts") + public static class Summary { + private int pages; + private boolean hasTitle; + private boolean displaysDocTitle; + private boolean hasLanguage; + private boolean allFontsEmbedded; + private int unembeddedFonts; + private int figures; + private boolean encrypted; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReportRequest.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReportRequest.java new file mode 100644 index 0000000000..178d2637b2 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/AccessibilityReportRequest.java @@ -0,0 +1,19 @@ +package stirling.software.proprietary.model.api.ua; + +import io.swagger.v3.oas.annotations.media.Schema; + +import lombok.Data; +import lombok.EqualsAndHashCode; + +import stirling.software.common.model.api.PDFFile; + +@Data +@EqualsAndHashCode(callSuper = true) +public class AccessibilityReportRequest extends PDFFile { + + @Schema( + description = "Profile to check against", + defaultValue = "ua1", + allowableValues = {"ua1", "ua2"}) + private String profile; +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/FigureDescriptor.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/FigureDescriptor.java new file mode 100644 index 0000000000..1c960d87e7 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/FigureDescriptor.java @@ -0,0 +1,18 @@ +package stirling.software.proprietary.model.api.ua; + +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * One figure needing an alternative description, which is never invented. key is the + * altTextByFigure key "pageIndex:ordinal"; page is 1-based; kind is "figure" or "formula". + */ +@Schema(description = "A figure that needs an alternative description") +public record FigureDescriptor( + String key, + int page, + String kind, + float x, + float y, + float width, + float height, + String existingAlt) {} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/PdfUaConversionOutcome.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/PdfUaConversionOutcome.java new file mode 100644 index 0000000000..dcef97c916 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/PdfUaConversionOutcome.java @@ -0,0 +1,26 @@ +package stirling.software.proprietary.model.api.ua; + +import java.util.List; + +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * Result of a PDF/UA conversion. + * + * @param declared whether a {@code pdfuaid} conformance claim was written into {@code pdfBytes} + */ +@Schema(description = "Result of converting a document to PDF/UA") +public record PdfUaConversionOutcome( + byte[] pdfBytes, + boolean declared, + UaValidationResult validation, + TaggingSummary tagging, + List warnings) { + + @Schema(description = "What the tagging pass produced") + public record TaggingSummary( + boolean rebuiltStructure, + int taggedElements, + int artifacts, + int figuresNeedingAltText) {} +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/UaValidationResult.java b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/UaValidationResult.java new file mode 100644 index 0000000000..5e494c35be --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/model/api/ua/UaValidationResult.java @@ -0,0 +1,18 @@ +package stirling.software.proprietary.model.api.ua; + +import java.util.List; + +import io.swagger.v3.oas.annotations.media.Schema; + +/** + * Outcome of validating against one PDF/UA profile. compliant means every automated check passed, + * which is not the same as usable by assistive technology; totalFailures is ungrouped. + */ +@Schema(description = "Result of validating a document against a PDF/UA profile") +public record UaValidationResult( + String profile, boolean compliant, List issues, int totalFailures) { + + public boolean hasIssues() { + return !issues.isEmpty(); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/ArtifactType.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/ArtifactType.java new file mode 100644 index 0000000000..78df1fbc27 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/ArtifactType.java @@ -0,0 +1,23 @@ +package stirling.software.proprietary.pdf.ua; + +/** Artifact subtypes (ISO 32000-1 14.8.2.2). Artifacts are excluded from the structure tree. */ +public enum ArtifactType { + /** Running heads, folios, page numbers. Required by PDF/UA-1 clause 7.8. */ + PAGINATION("Pagination"), + /** Rules, boxes, and other layout ornamentation. */ + LAYOUT("Layout"), + /** Cut marks and colour bars. */ + PAGE("Page"), + /** Background graphics with no informational content. */ + BACKGROUND("Background"); + + private final String subtype; + + ArtifactType(String subtype) { + this.subtype = subtype; + } + + public String subtype() { + return subtype; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/BBox.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/BBox.java new file mode 100644 index 0000000000..f2fbf97438 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/BBox.java @@ -0,0 +1,48 @@ +package stirling.software.proprietary.pdf.ua; + +/** An axis-aligned rectangle in PDF user space, with y increasing upwards. */ +public record BBox(float x0, float y0, float x1, float y1) { + + public static final BBox EMPTY = new BBox(0, 0, 0, 0); + + public static BBox of(float x, float y, float width, float height) { + return new BBox(x, y, x + width, y + height); + } + + public float width() { + return x1 - x0; + } + + public float height() { + return y1 - y0; + } + + public float centreX() { + return (x0 + x1) / 2f; + } + + public BBox union(BBox other) { + if (other == null || other.isEmpty()) { + return this; + } + if (isEmpty()) { + return other; + } + return new BBox( + Math.min(x0, other.x0), + Math.min(y0, other.y0), + Math.max(x1, other.x1), + Math.max(y1, other.y1)); + } + + public boolean isEmpty() { + return x1 <= x0 || y1 <= y0; + } + + /** Horizontal overlap with another box as a fraction of the narrower box's width. */ + public float horizontalOverlap(BBox other) { + float overlap = Math.min(x1, other.x1) - Math.max(x0, other.x0); + float narrower = Math.min(width(), other.width()); + return narrower <= 0 ? 0 : Math.max(0, overlap) / narrower; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/DocumentStructure.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/DocumentStructure.java new file mode 100644 index 0000000000..5922895453 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/DocumentStructure.java @@ -0,0 +1,84 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.ArrayList; +import java.util.List; +import java.util.function.Consumer; + +import lombok.Getter; +import lombok.Setter; + +/** The derived logical structure of a document, ready for serialisation into a structure tree. */ +@Getter +@Setter +public class DocumentStructure { + + /** Top-level blocks in document reading order. */ + private final List blocks = new ArrayList<>(); + + /** Warnings raised during analysis, surfaced in the conversion report. */ + private final List warnings = new ArrayList<>(); + + private String title; + private String language; + + /** True when real text was wrapped as artifacts, which blocks any conformance claim. */ + private boolean textSuppressed; + + /** Body text size used as the baseline for heading detection, in points. */ + private float bodyFontSize; + + public void add(StructBlock block) { + blocks.add(block); + } + + public void warn(String message) { + if (!warnings.contains(message)) { + warnings.add(message); + } + } + + public void visit(Consumer visitor) { + blocks.forEach(block -> block.visit(visitor)); + } + + public int count(StructType type) { + int[] total = {0}; + visit( + block -> { + if (block.getType() == type) { + total[0]++; + } + }); + return total[0]; + } + + public int artifactCount() { + int[] total = {0}; + visit( + block -> { + if (block.isArtifact()) { + total[0]++; + } + }); + return total[0]; + } + + /** Figures with no alternative description, the most common PDF/UA failure. */ + public List figuresWithoutAlt() { + List missing = new ArrayList<>(); + visit( + block -> { + if ((block.getType() == StructType.FIGURE + || block.getType() == StructType.FORMULA) + && (block.getAlt() == null || block.getAlt().isBlank()) + && (block.getActualText() == null || block.getActualText().isBlank())) { + missing.add(block); + } + }); + return missing; + } + + public boolean isEmpty() { + return blocks.isEmpty(); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzer.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzer.java new file mode 100644 index 0000000000..11c510d6ea --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzer.java @@ -0,0 +1,831 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.IdentityHashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +import lombok.extern.slf4j.Slf4j; + +/** + * Derives a logical structure from extracted lines and graphics, reusing {@code HeadingDetector}'s + * heuristics. Degrades to paragraphs rather than guessing, since a wrong tag misleads readers. + */ +@Slf4j +public class LayoutAnalyzer { + + private static final Pattern BULLET = Pattern.compile("^[•‣◦⁃∙·▪●■o\\-\\*\\+]\\s+.*"); + private static final Pattern ORDERED = + Pattern.compile("^(\\d{1,3}|[a-zA-Z]|[ivxlcIVXLC]{1,5})[\\.\\)]\\s+.*"); + private static final Pattern PAGE_NUMBER = + Pattern.compile( + "^(page\\s+)?\\d{1,4}(\\s*(of|/)\\s*\\d{1,4})?$", Pattern.CASE_INSENSITIVE); + private static final Pattern DIGITS = Pattern.compile("\\d+"); + + /** Fraction of page height treated as the running head / foot band. */ + private static final float MARGIN_BAND = 0.10f; + + /** A line must exceed the body size by this ratio before it can be a heading. */ + private static final float HEADING_RATIO = 1.10f; + + /** Sizes within this many points are treated as the same heading tier. */ + private static final float TIER_TOLERANCE = 0.4f; + + private static final int MAX_HEADING_WORDS = 12; + + /** Word gap beyond this multiple of the font size separates table cells. */ + private static final float CELL_GAP_RATIO = 1.2f; + + /** Images smaller than this in either dimension are decoration, not content. */ + private static final float MIN_FIGURE_SIZE = 12f; + + /** A size used by more than this share of lines is body text, however large the median says. */ + private static final float MAX_HEADING_LINE_SHARE = 0.2f; + + /** Consecutive lines sharing a size are a text block; headings appear alone. */ + private static final int MAX_HEADING_RUN = 3; + + /** A vector thinner than this in either dimension is a rule or border, not a drawing. */ + private static final float MIN_VECTOR_THICKNESS = 3f; + + /** Vector clusters smaller than this are ornament; larger ones are probably a chart. */ + private static final float MIN_VECTOR_FIGURE_SIZE = 40f; + + /** A drawing is built from several strokes; one big rectangle is a panel, not a chart. */ + private static final int MIN_VECTOR_FIGURE_OPS = 4; + + /** More text than this inside the region means shading behind content, not a drawing. */ + private static final int MAX_LINES_INSIDE_FIGURE = 2; + + public DocumentStructure analyse(List pages) { + DocumentStructure structure = new DocumentStructure(); + float bodySize = bodyFontSize(pages); + structure.setBodyFontSize(bodySize); + Map tiers = headingTiers(pages, bodySize); + Map> artifactLines = repeatedMarginLines(pages, bodySize); + + for (PageContent page : pages) { + analysePage( + page, + structure, + bodySize, + tiers, + artifactLines.getOrDefault(page.pageIndex(), List.of())); + } + + List suppressedPages = + pages.stream() + .filter(PageContent::linesDropped) + .map(PageContent::pageIndex) + .toList(); + if (!suppressedPages.isEmpty()) { + structure.setTextSuppressed(true); + structure.warn( + "Text on page(s) " + + suppressedPages.stream() + .map(i -> String.valueOf(i + 1)) + .collect(Collectors.joining(", ")) + + " could not be tagged reliably and was marked as artifacts. The" + + " converter will not claim conformance while real text is hidden" + + " from assistive technology."); + } + + normaliseHeadingLevels(structure); + structure.setTitle(deriveTitle(structure)); + return structure; + } + + // --- Document-wide statistics ----------------------------------------- + + /** Character-weighted median line size, which is far more stable than a plain median. */ + static float bodyFontSize(List pages) { + Map weights = new HashMap<>(); + for (PageContent page : pages) { + for (TextLineInfo line : page.lines()) { + if (line.dominantFontSize() > 0 && !line.isBlank()) { + weights.merge(line.dominantFontSize(), line.charCount(), Integer::sum); + } + } + } + if (weights.isEmpty()) { + return 0f; + } + int total = weights.values().stream().mapToInt(Integer::intValue).sum(); + List> sorted = + weights.entrySet().stream().sorted(Map.Entry.comparingByKey()).toList(); + int seen = 0; + for (Map.Entry entry : sorted) { + seen += entry.getValue(); + if (seen >= total / 2) { + return entry.getKey(); + } + } + return sorted.get(sorted.size() - 1).getKey(); + } + + /** Maps each distinct heading size to a 1-based level, largest size first. */ + static Map headingTiers(List pages, float bodySize) { + if (bodySize <= 0) { + return Map.of(); + } + // A size used by a large share of the lines is body text, whatever the median says. + Map lineCounts = new HashMap<>(); + int totalLines = 0; + for (PageContent page : pages) { + for (TextLineInfo line : page.lines()) { + if (!line.isBlank()) { + lineCounts.merge(line.dominantFontSize(), 1, Integer::sum); + totalLines++; + } + } + } + int headingLineCeiling = Math.max(1, (int) (totalLines * MAX_HEADING_LINE_SHARE)); + + // Headings do not cluster; a run of same-size lines is a text block, not headings. + Map longestRun = new HashMap<>(); + for (PageContent page : pages) { + Float runSize = null; + int runLength = 0; + for (TextLineInfo line : page.lines()) { + if (line.isBlank()) { + continue; + } + float size = line.dominantFontSize(); + if (runSize != null && Float.compare(size, runSize) == 0) { + runLength++; + } else { + runSize = size; + runLength = 1; + } + int seen = longestRun.getOrDefault(size, 0); + if (runLength > seen) { + longestRun.put(size, runLength); + } + } + } + + List sizes = new ArrayList<>(); + for (PageContent page : pages) { + for (TextLineInfo line : page.lines()) { + if (isHeadingCandidate(line) + && line.dominantFontSize() > bodySize * HEADING_RATIO + && lineCounts.getOrDefault(line.dominantFontSize(), 0) <= headingLineCeiling + && longestRun.getOrDefault(line.dominantFontSize(), 0) < MAX_HEADING_RUN) { + sizes.add(line.dominantFontSize()); + } + } + } + List distinct = sizes.stream().distinct().sorted(Comparator.reverseOrder()).toList(); + + Map tiers = new LinkedHashMap<>(); + int level = 0; + Float previous = null; + for (Float size : distinct) { + if (previous == null || previous - size > TIER_TOLERANCE) { + level = Math.min(level + 1, 6); + previous = size; + } + tiers.put(size, level); + } + return tiers; + } + + /** + * Claims a line's operators word run by word run; claiming the whole ordinal interval would + * swallow anything drawn between them, an image included. + */ + private static void claimLine(StructBlock block, TextLineInfo line) { + // Sort by ordinal, not position: merging out-of-order runs silently drops them to + // /Artifact, hiding them from assistive technology while the file still validates. + List words = + line.words().stream() + .filter(w -> !w.isBlank()) + .sorted(Comparator.comparingInt(WordInfo::startOrdinal)) + .toList(); + if (words.isEmpty()) { + block.addRange(line.startOrdinal(), line.endOrdinal()); + return; + } + int start = words.get(0).startOrdinal(); + int end = words.get(0).endOrdinal(); + for (int i = 1; i < words.size(); i++) { + WordInfo word = words.get(i); + if (word.startOrdinal() <= end + 1) { + end = Math.max(end, word.endOrdinal()); + } else { + block.addRange(start, end); + start = word.startOrdinal(); + end = word.endOrdinal(); + } + } + block.addRange(start, end); + } + + static boolean isHeadingCandidate(TextLineInfo line) { + String text = line.text().strip(); + if (text.isEmpty() || line.wordCount() > MAX_HEADING_WORDS) { + return false; + } + char last = text.charAt(text.length() - 1); + return last != '.' && last != '!' && last != '?'; + } + + /** + * Finds lines in the head/foot bands whose text repeats across pages. Digits are masked first + * so that "Page 4" and "Page 5" count as the same running foot. + */ + static Map> repeatedMarginLines(List pages) { + return repeatedMarginLines(pages, bodyFontSize(pages)); + } + + static Map> repeatedMarginLines( + List pages, float bodySize) { + Map> result = new HashMap<>(); + if (pages.isEmpty()) { + return result; + } + Map counts = new HashMap<>(); + Map> candidates = new HashMap<>(); + + for (PageContent page : pages) { + float height = page.mediaBox().height(); + if (height <= 0) { + continue; + } + float topEdge = page.mediaBox().y1() - height * MARGIN_BAND; + float bottomEdge = page.mediaBox().y0() + height * MARGIN_BAND; + List inBand = new ArrayList<>(); + for (TextLineInfo line : page.lines()) { + if (line.bbox().y0() >= topEdge || line.bbox().y1() <= bottomEdge) { + inBand.add(line); + counts.merge(mask(line.text()), 1, Integer::sum); + } + } + candidates.put(page.pageIndex(), inBand); + } + + int threshold = Math.max(2, pages.size() / 2); + for (Map.Entry> entry : candidates.entrySet()) { + List artifacts = new ArrayList<>(); + for (TextLineInfo line : entry.getValue()) { + boolean repeats = + pages.size() >= 3 && counts.getOrDefault(mask(line.text()), 0) >= threshold; + boolean pageNumber = PAGE_NUMBER.matcher(line.text().strip()).matches(); + // Masked digits merge "Section 1" and "Section 2"; size is the tie-break that stops + // a real heading being demoted, as running heads are never larger than body text. + boolean looksLikeChrome = + bodySize <= 0 || line.dominantFontSize() <= bodySize * 1.05f; + if (pageNumber || (repeats && looksLikeChrome)) { + artifacts.add(line); + } + } + result.put(entry.getKey(), artifacts); + } + return result; + } + + private static String mask(String text) { + return DIGITS.matcher(text.strip().toLowerCase()).replaceAll("#").replaceAll("\\s+", " "); + } + + // --- Per-page analysis ------------------------------------------------- + + private void analysePage( + PageContent page, + DocumentStructure structure, + float bodySize, + Map tiers, + List marginArtifacts) { + + for (TextLineInfo line : marginArtifacts) { + StructBlock artifact = StructBlock.artifact(ArtifactType.PAGINATION, page.pageIndex()); + claimLine(artifact, line); + artifact.setBbox(line.bbox()); + artifact.setText(line.text()); + structure.add(artifact); + } + + // Identity set, not List.contains: TextLineInfo is a record whose equals walks its word + // list, so a linear scan per line is quadratic with a deep comparison inside it. + java.util.Set marginSet = Collections.newSetFromMap(new IdentityHashMap<>()); + marginSet.addAll(marginArtifacts); + List body = + page.lines().stream() + .filter(line -> !line.isBlank() && !marginSet.contains(line)) + .sorted(readingOrder(page)) + .toList(); + + List blocks = new ArrayList<>(); + int index = 0; + while (index < body.size()) { + TextLineInfo line = body.get(index); + + int tableEnd = tableRunEnd(body, index); + if (tableEnd > index) { + StructBlock table = buildTable(body.subList(index, tableEnd + 1), page.pageIndex()); + if (table != null) { + blocks.add(table); + index = tableEnd + 1; + continue; + } + } + + int listEnd = listRunEnd(body, index); + if (listEnd > index) { + blocks.add(buildList(body.subList(index, listEnd + 1), page.pageIndex())); + index = listEnd + 1; + continue; + } + + Integer level = headingLevel(line, tiers); + if (level != null) { + StructBlock heading = new StructBlock(StructType.heading(level), page.pageIndex()); + claimLine(heading, line); + heading.setBbox(line.bbox()); + heading.setText(line.text()); + blocks.add(heading); + index++; + continue; + } + + int paragraphEnd = paragraphRunEnd(body, index, tiers, bodySize); + blocks.add(buildParagraph(body.subList(index, paragraphEnd + 1), page.pageIndex())); + index = paragraphEnd + 1; + } + + // Form XObject text is attributed to its Do, so a Figure too would double-claim it. + Set claimed = new HashSet<>(); + for (StructBlock block : blocks) { + block.visit( + node -> + node.getRanges() + .forEach( + range -> { + for (int i = range.start(); i <= range.end(); i++) { + claimed.add(i); + } + })); + } + blocks.addAll(buildGraphics(page, structure, claimed)); + blocks.forEach(structure::add); + } + + /** + * Orders lines top-to-bottom, splitting into columns first when the page is clearly + * multi-column. Without this, a two-column page reads as interleaved half-sentences. + */ + private Comparator readingOrder(PageContent page) { + Float gutter = detectGutter(page); + if (gutter == null) { + return Comparator.comparingDouble((TextLineInfo l) -> -l.bbox().y1()) + .thenComparingDouble(l -> l.bbox().x0()); + } + return Comparator.comparingInt((TextLineInfo l) -> l.bbox().centreX() < gutter ? 0 : 1) + .thenComparingDouble(l -> -l.bbox().y1()) + .thenComparingDouble(l -> l.bbox().x0()); + } + + /** + * Returns the x of a vertical gutter when the page is two-column, else null. A gutter must sit + * near the middle, be crossed by almost no line, and have substantial text on both sides. + */ + static Float detectGutter(PageContent page) { + List lines = page.lines().stream().filter(line -> !line.isBlank()).toList(); + if (lines.size() < 8) { + return null; + } + float pageWidth = page.mediaBox().width(); + if (pageWidth <= 0) { + return null; + } + float centre = page.mediaBox().x0() + pageWidth / 2f; + long crossing = + lines.stream() + .filter( + line -> + line.bbox().x0() < centre - 5 + && line.bbox().x1() > centre + 5) + .count(); + if (crossing > lines.size() * 0.1) { + return null; + } + long left = lines.stream().filter(line -> line.bbox().centreX() < centre).count(); + long right = lines.size() - left; + boolean balanced = left > lines.size() * 0.25 && right > lines.size() * 0.25; + return balanced ? centre : null; + } + + private static Integer headingLevel(TextLineInfo line, Map tiers) { + if (!isHeadingCandidate(line)) { + return null; + } + return tiers.get(line.dominantFontSize()); + } + + // --- Paragraphs -------------------------------------------------------- + + private static int paragraphRunEnd( + List lines, int start, Map tiers, float bodySize) { + int end = start; + for (int i = start + 1; i < lines.size(); i++) { + TextLineInfo previous = lines.get(i - 1); + TextLineInfo current = lines.get(i); + if (headingLevel(current, tiers) != null || startsListItem(current)) { + break; + } + float gap = previous.bbox().y0() - current.bbox().y1(); + float leading = Math.max(bodySize, current.bbox().height()); + boolean sameBlock = gap < leading * 0.8f && gap > -leading; + boolean sentenceEnded = endsSentence(previous.text()); + if (!sameBlock || (sentenceEnded && gap > leading * 0.4f)) { + break; + } + end = i; + } + return end; + } + + private static boolean endsSentence(String text) { + String stripped = text.strip(); + if (stripped.isEmpty()) { + return false; + } + char last = stripped.charAt(stripped.length() - 1); + return last == '.' || last == '!' || last == '?'; + } + + private static StructBlock buildParagraph(List lines, int pageIndex) { + StructBlock paragraph = new StructBlock(StructType.P, pageIndex); + BBox box = BBox.EMPTY; + StringBuilder text = new StringBuilder(); + for (TextLineInfo line : lines) { + claimLine(paragraph, line); + box = box.union(line.bbox()); + if (text.length() > 0) { + text.append(' '); + } + text.append(line.text().strip()); + } + paragraph.setBbox(box); + paragraph.setText(text.toString()); + return paragraph; + } + + // --- Lists ------------------------------------------------------------- + + static boolean startsListItem(TextLineInfo line) { + String text = line.text().strip(); + return BULLET.matcher(text).matches() || ORDERED.matcher(text).matches(); + } + + private static int listRunEnd(List lines, int start) { + if (!startsListItem(lines.get(start))) { + return start; + } + float indent = lines.get(start).bbox().x0(); + int end = start; + for (int i = start + 1; i < lines.size(); i++) { + TextLineInfo line = lines.get(i); + boolean isItem = startsListItem(line) && Math.abs(line.bbox().x0() - indent) < 6f; + boolean isContinuation = !startsListItem(line) && line.bbox().x0() > indent + 2f; + if (!isItem && !isContinuation) { + break; + } + end = i; + } + // A single marker is a stray character, not a list. + long items = + lines.subList(start, end + 1).stream() + .filter(LayoutAnalyzer::startsListItem) + .count(); + return items >= 2 ? end : start; + } + + private static StructBlock buildList(List lines, int pageIndex) { + StructBlock list = new StructBlock(StructType.L, pageIndex); + list.setListNumbering(listNumbering(lines.get(0))); + BBox box = BBox.EMPTY; + StructBlock currentBody = null; + + for (TextLineInfo line : lines) { + box = box.union(line.bbox()); + if (startsListItem(line) || currentBody == null) { + StructBlock item = new StructBlock(StructType.LI, pageIndex); + StructBlock body = new StructBlock(StructType.LBODY, pageIndex); + claimLine(body, line); + body.setBbox(line.bbox()); + body.setText(line.text()); + item.addChild(body); + item.setBbox(line.bbox()); + list.addChild(item); + currentBody = body; + } else { + claimLine(currentBody, line); + currentBody.setBbox(currentBody.getBbox().union(line.bbox())); + currentBody.setText(currentBody.getText() + " " + line.text().strip()); + } + } + list.setBbox(box); + return list; + } + + private static String listNumbering(TextLineInfo first) { + String text = first.text().strip(); + if (BULLET.matcher(text).matches()) { + return "Disc"; + } + char c = text.charAt(0); + if (Character.isDigit(c)) { + return "Decimal"; + } + if ("ivxlc".indexOf(Character.toLowerCase(c)) >= 0 && text.length() > 1) { + return Character.isUpperCase(c) ? "UpperRoman" : "LowerRoman"; + } + return Character.isUpperCase(c) ? "UpperAlpha" : "LowerAlpha"; + } + + // --- Tables ------------------------------------------------------------ + + /** Splits a line into cells wherever the gap between words exceeds the cell threshold. */ + static List> splitCells(TextLineInfo line) { + List words = line.words().stream().filter(w -> !w.isBlank()).toList(); + List> cells = new ArrayList<>(); + if (words.isEmpty()) { + return cells; + } + float threshold = Math.max(line.dominantFontSize(), 1f) * CELL_GAP_RATIO; + List current = new ArrayList<>(); + current.add(words.get(0)); + for (int i = 1; i < words.size(); i++) { + float gap = words.get(i).bbox().x0() - words.get(i - 1).bbox().x1(); + if (gap > threshold) { + cells.add(List.copyOf(current)); + current = new ArrayList<>(); + } + current.add(words.get(i)); + } + cells.add(List.copyOf(current)); + return cells; + } + + /** + * Index of the last line of a table run starting at {@code start}, or {@code start} if none. + */ + private static int tableRunEnd(List lines, int start) { + int end = start; + for (int i = start; i < lines.size(); i++) { + if (splitCells(lines.get(i)).size() < 2) { + break; + } + end = i; + } + return end > start ? end : start; + } + + /** + * Builds a Table when the run really looks tabular and each cell owns its own operators. + * Returns null when it does not, so the caller falls back to paragraphs. + */ + private static StructBlock buildTable(List rows, int pageIndex) { + if (rows.size() < 2) { + return null; + } + List>> grid = new ArrayList<>(); + for (TextLineInfo row : rows) { + if (!row.wordsAreSeparable()) { + log.debug("Table row shares operators between cells; falling back to paragraphs"); + return null; + } + grid.add(splitCells(row)); + } + int columns = grid.get(0).size(); + long consistent = grid.stream().filter(row -> row.size() == columns).count(); + if (columns < 2 || consistent < Math.max(2, grid.size() * 0.6)) { + return null; + } + + boolean headerRow = looksLikeHeader(rows, grid); + StructBlock table = new StructBlock(StructType.TABLE, pageIndex); + BBox box = BBox.EMPTY; + + for (int r = 0; r < grid.size(); r++) { + List> cells = grid.get(r); + if (cells.size() != columns) { + continue; + } + StructBlock tr = new StructBlock(StructType.TR, pageIndex); + boolean isHeader = headerRow && r == 0; + for (List cell : cells) { + StructBlock td = + new StructBlock(isHeader ? StructType.TH : StructType.TD, pageIndex); + if (isHeader) { + td.setScope("Column"); + } + BBox cellBox = BBox.EMPTY; + StringBuilder text = new StringBuilder(); + int from = cell.get(0).startOrdinal(); + int to = cell.get(cell.size() - 1).endOrdinal(); + for (WordInfo word : cell) { + cellBox = cellBox.union(word.bbox()); + if (text.length() > 0) { + text.append(' '); + } + text.append(word.text()); + } + td.addRange(from, to); + td.setBbox(cellBox); + td.setText(text.toString()); + tr.addChild(td); + box = box.union(cellBox); + } + tr.setBbox(box); + table.addChild(tr); + } + table.setBbox(box); + if (table.getChildren().size() < 2) { + return null; + } + // Clause 7.5 needs equal cell counts per row; a ragged table fails validation outright. + long distinctWidths = + table.getChildren().stream() + .map(row -> row.getChildren().size()) + .distinct() + .count(); + if (distinctWidths != 1) { + log.debug("Discarding a table whose rows have different cell counts"); + return null; + } + return table; + } + + /** The first row is a header when it is bold, or when only later rows carry numbers. */ + private static boolean looksLikeHeader( + List rows, List>> grid) { + if (rows.get(0).bold()) { + return true; + } + boolean firstHasDigits = DIGITS.matcher(rows.get(0).text()).find(); + boolean laterHasDigits = + rows.subList(1, rows.size()).stream() + .anyMatch(row -> DIGITS.matcher(row.text()).find()); + return !firstHasDigits && laterHasDigits; + } + + // --- Graphics ---------------------------------------------------------- + + private List buildGraphics( + PageContent page, DocumentStructure structure, java.util.Set claimed) { + List blocks = new ArrayList<>(); + boolean warnedForms = false; + + // Vectors cluster: a chart is many strokes in one region, a rule is a single thin one. + java.util.Set vectorFigureOrdinals = vectorFigureOrdinals(page, claimed); + + for (MarkableOp op : page.ops()) { + if (op.kind() == MarkableOp.Kind.TEXT || claimed.contains(op.ordinal())) { + continue; + } + BBox box = op.bbox(); + + if (op.kind() == MarkableOp.Kind.VECTOR) { + StructBlock block; + if (vectorFigureOrdinals.contains(op.ordinal())) { + block = new StructBlock(StructType.FIGURE, page.pageIndex()); + } else { + block = StructBlock.artifact(ArtifactType.LAYOUT, page.pageIndex()); + } + block.addRange(op.ordinal(), op.ordinal()); + block.setBbox(box); + blocks.add(block); + continue; + } + + boolean decorative = box.width() < MIN_FIGURE_SIZE || box.height() < MIN_FIGURE_SIZE; + if (decorative) { + StructBlock artifact = StructBlock.artifact(ArtifactType.LAYOUT, page.pageIndex()); + artifact.addRange(op.ordinal(), op.ordinal()); + artifact.setBbox(box); + blocks.add(artifact); + continue; + } + + if (op.kind() == MarkableOp.Kind.FORM && !warnedForms) { + structure.warn( + "Content inside form XObjects was tagged as a single region because its" + + " text is not separately addressable; review those areas."); + warnedForms = true; + } + + StructBlock figure = new StructBlock(StructType.FIGURE, page.pageIndex()); + figure.addRange(op.ordinal(), op.ordinal()); + figure.setBbox(box); + blocks.add(figure); + } + return blocks; + } + + /** + * Finds vector operators belonging to a substantial drawing rather than page furniture; thin + * paths are rules and table borders, and a short run is ornament. + */ + private static Set vectorFigureOrdinals( + PageContent page, java.util.Set claimed) { + // A chart's plot area is mostly empty, while shading sits behind the text it decorates. + Set result = new HashSet<>(); + List run = new ArrayList<>(); + BBox extent = BBox.EMPTY; + + for (MarkableOp op : page.ops()) { + boolean substantial = + op.kind() == MarkableOp.Kind.VECTOR + && !claimed.contains(op.ordinal()) + && !op.bbox().isEmpty() + && op.bbox().width() >= MIN_VECTOR_THICKNESS + && op.bbox().height() >= MIN_VECTOR_THICKNESS; + if (substantial) { + run.add(op); + extent = extent.isEmpty() ? op.bbox() : extent.union(op.bbox()); + continue; + } + flushVectorRun(run, extent, page.lines(), result); + run = new ArrayList<>(); + extent = BBox.EMPTY; + } + flushVectorRun(run, extent, page.lines(), result); + return result; + } + + private static void flushVectorRun( + List run, + BBox extent, + List lines, + java.util.Set result) { + if (run.size() < MIN_VECTOR_FIGURE_OPS + || extent.width() < MIN_VECTOR_FIGURE_SIZE + || extent.height() < MIN_VECTOR_FIGURE_SIZE) { + return; + } + if (overlappingLines(extent, lines) > MAX_LINES_INSIDE_FIGURE) { + return; + } + run.forEach(op -> result.add(op.ordinal())); + } + + /** How many text lines sit within the region a vector cluster covers. */ + private static int overlappingLines(BBox extent, List lines) { + int count = 0; + for (TextLineInfo line : lines) { + BBox box = line.bbox(); + boolean inside = + box.x0() >= extent.x0() - 2 + && box.x1() <= extent.x1() + 2 + && box.y0() >= extent.y0() - 2 + && box.y1() <= extent.y1() + 2; + if (inside) { + count++; + } + } + return count; + } + + // --- Post-processing --------------------------------------------------- + + /** + * Rewrites heading levels so no level is skipped, which PDF/UA-1 clause 7.4 requires. A + * document that jumps H1 to H3 is remapped to H1, H2 while preserving relative depth. + */ + static void normaliseHeadingLevels(DocumentStructure structure) { + List headings = new ArrayList<>(); + structure.visit( + block -> { + if (block.getType().isHeading()) { + headings.add(block); + } + }); + int previous = 0; + for (StructBlock heading : headings) { + int level = heading.getType().headingLevel(); + int adjusted = level > previous + 1 ? previous + 1 : level; + heading.setType(StructType.heading(adjusted)); + previous = adjusted; + } + } + + /** Uses the first top-level heading as the title when the document has no metadata title. */ + private static String deriveTitle(DocumentStructure structure) { + for (StructBlock block : structure.getBlocks()) { + if (block.getType().isHeading() && !block.getText().isBlank()) { + return block.getText().strip(); + } + } + return null; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkableOp.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkableOp.java new file mode 100644 index 0000000000..c48319260f --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkableOp.java @@ -0,0 +1,44 @@ +package stirling.software.proprietary.pdf.ua; + +/** + * One operator in a page content stream that may be wrapped in a marked-content sequence. The + * ordinal counts only markable operators, joining text extraction to token rewriting. + */ +public record MarkableOp(int ordinal, Kind kind, BBox bbox, String resourceName) { + + public enum Kind { + /** Tj, TJ, ' or " */ + TEXT, + /** Do referencing an image XObject */ + IMAGE, + /** Do referencing a form XObject */ + FORM, + /** BI ... ID ... EI */ + INLINE_IMAGE, + /** A path-painting or shading operator: rules, borders, fills, logos */ + VECTOR; + + public boolean isGraphic() { + return this == IMAGE || this == INLINE_IMAGE; + } + } + + /** + * Operator names counted as markable; both passes must agree on this set. Path painting is + * included because clause 7.1 needs visible rules and borders tagged or artifacted. + */ + public static boolean isMarkableOperator(String name) { + return switch (name) { + case "Tj", "TJ", "'", "\"", "Do", "BI" -> true; + default -> isPathPainting(name); + }; + } + + /** Painting operators only: {@code n} ends a path without marking the page. */ + public static boolean isPathPainting(String name) { + return switch (name) { + case "S", "s", "f", "F", "f*", "B", "B*", "b", "b*", "sh" -> true; + default -> false; + }; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkedContentInjector.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkedContentInjector.java new file mode 100644 index 0000000000..c9d6330a60 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/MarkedContentInjector.java @@ -0,0 +1,284 @@ +package stirling.software.proprietary.pdf.ua; + +import java.io.IOException; +import java.io.OutputStream; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.pdfbox.contentstream.operator.Operator; +import org.apache.pdfbox.cos.COSBase; +import org.apache.pdfbox.cos.COSDictionary; +import org.apache.pdfbox.cos.COSInteger; +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdfparser.PDFStreamParser; +import org.apache.pdfbox.pdfwriter.ContentStreamWriter; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.common.PDStream; + +import lombok.extern.slf4j.Slf4j; + +/** + * Rewrites a page stream so every markable operator sits inside a marked-content sequence: claimed + * content gets an MCID, everything else /Artifact, satisfying PDF/UA-1 clause 7.1 by construction. + */ +@Slf4j +public class MarkedContentInjector { + + private static final COSName ARTIFACT = COSName.getPDFName("Artifact"); + private static final COSName MCID = COSName.getPDFName("MCID"); + private static final COSName ACTUAL_TEXT = COSName.getPDFName("ActualText"); + private static final COSName ALT = COSName.getPDFName("Alt"); + + /** Operators that force an open sequence to close so nesting stays legal. */ + private static boolean isBoundary(String name) { + return "BT".equals(name) || "ET".equals(name) || "q".equals(name) || "Q".equals(name); + } + + private static boolean isMarkedContentOperator(String name) { + return "BDC".equals(name) || "BMC".equals(name) || "EMC".equals(name); + } + + /** + * Path-construction operators; ISO 32000-1 forbids marked content inside a path object, so a + * sequence wrapping a fill or stroke must open before the path starts. + */ + private static boolean isPathConstruction(String name) { + return switch (name) { + case "m", "l", "c", "v", "y", "h", "re" -> true; + default -> false; + }; + } + + private static boolean opensMarkedContent(String name) { + return "BDC".equals(name) || "BMC".equals(name); + } + + /** + * True for an optional-content sequence; stripping an {@code /OC} wrapper would make hidden + * layers such as watermarks or redaction overlays visible. + */ + private static boolean isOptionalContent(String name, List operands) { + return opensMarkedContent(name) + && !operands.isEmpty() + && operands.get(0) instanceof COSName tag + && "OC".equals(tag.getName()); + } + + /** + * True when a sequence supplies replacement text for its glyphs; dropping it leaves a screen + * reader with the font's own mapping, which for a ligature says nothing useful. + */ + private static boolean carriesReplacementText(String name, List operands) { + if (!opensMarkedContent(name)) { + return false; + } + for (COSBase operand : operands) { + if (operand instanceof COSDictionary properties + && (properties.containsKey(ACTUAL_TEXT) + || properties.containsKey(ALT) + || properties.containsKey(COSName.E))) { + return true; + } + } + return false; + } + + /** The source's own ids mean nothing once the tree is rebuilt, so they are dropped. */ + private static void stripStaleMcid(List operands) { + for (COSBase operand : operands) { + if (operand instanceof COSDictionary properties) { + properties.removeItem(MCID); + } + } + } + + /** Wraps every markable operator on the page; returns the next unused marked content id. */ + public int inject( + PDDocument document, + PDPage page, + List blocks, + int nextMcid, + boolean stripExisting) + throws IOException { + + Map owners = ownersByOrdinal(blocks); + List tokens = parse(page); + List output = new ArrayList<>(tokens.size() + owners.size() * 4); + + List operands = new ArrayList<>(); + // Tracks, for each surviving source sequence, whether its closer should be kept. + Deque keptSequences = new ArrayDeque<>(); + StructBlock openBlock = null; + boolean open = false; + int ordinal = -1; + int mcid = nextMcid; + int pathStart = -1; + + for (Object token : tokens) { + if (!(token instanceof Operator operator)) { + operands.add((COSBase) token); + continue; + } + String name = operator.getName(); + + if (stripExisting && isMarkedContentOperator(name)) { + boolean keep; + if (opensMarkedContent(name)) { + keep = + isOptionalContent(name, operands) + || carriesReplacementText(name, operands); + if (keep) { + stripStaleMcid(operands); + } + keptSequences.push(keep); + } else { + // A closer is kept exactly when its matching opener was. + keep = !keptSequences.isEmpty() && keptSequences.pop(); + } + if (!keep) { + operands.clear(); + continue; + } + // Close our own sequence first so the two never interleave illegally. + if (open) { + output.add(Operator.getOperator("EMC")); + open = false; + openBlock = null; + } + output.addAll(operands); + output.add(operator); + operands.clear(); + continue; + } + + if (isBoundary(name) && open) { + output.add(Operator.getOperator("EMC")); + open = false; + openBlock = null; + } + + // Remember where the current path object began so a sequence wrapping its painting + // operator can be opened before it rather than inside it. + if (isPathConstruction(name)) { + if (pathStart < 0) { + pathStart = output.size(); + } + } else if (!MarkableOp.isPathPainting(name) && !"n".equals(name)) { + pathStart = -1; + } + + if (MarkableOp.isMarkableOperator(name)) { + ordinal++; + StructBlock owner = owners.get(ordinal); + if (!open || owner != openBlock) { + boolean insidePath = MarkableOp.isPathPainting(name) && pathStart >= 0; + if (open) { + // Close before the path began, so the EMC also stays outside the path. + output.add( + insidePath ? pathStart : output.size(), + Operator.getOperator("EMC")); + if (insidePath) { + pathStart++; + } + } + int at = insidePath ? pathStart : output.size(); + mcid = openSequenceAt(output, at, owner, mcid); + open = true; + openBlock = owner; + } + } + + output.addAll(operands); + output.add(operator); + operands.clear(); + + if (MarkableOp.isPathPainting(name) || "n".equals(name)) { + pathStart = -1; + } + } + + if (open) { + output.add(Operator.getOperator("EMC")); + } + + write(document, page, output); + return mcid; + } + + /** Emits the opening BDC/BMC at a given position and records the id on the owning block. */ + private int openSequenceAt(List output, int at, StructBlock owner, int mcid) { + List opening = new ArrayList<>(3); + if (owner == null) { + opening.add(ARTIFACT); + opening.add(Operator.getOperator("BMC")); + } else if (owner.isArtifact()) { + COSDictionary properties = new COSDictionary(); + if (owner.getArtifactType() != null) { + properties.setName(COSName.TYPE, owner.getArtifactType().subtype()); + } + opening.add(ARTIFACT); + opening.add(properties); + opening.add(Operator.getOperator("BDC")); + } else { + COSDictionary properties = new COSDictionary(); + properties.setItem(MCID, COSInteger.get(mcid)); + opening.add(COSName.getPDFName(owner.getType().tag())); + opening.add(properties); + opening.add(Operator.getOperator("BDC")); + owner.getMcids().add(mcid); + mcid++; + } + output.addAll(at, opening); + return mcid; + } + + /** + * Maps each claimed ordinal to its block. Overlapping claims are dropped rather than merged: + * two structure elements sharing content would make the reading order ambiguous. + */ + static Map ownersByOrdinal(List blocks) { + Map owners = new HashMap<>(); + for (StructBlock block : blocks) { + block.visit( + node -> { + for (StructBlock.OrdinalRange range : node.getRanges()) { + for (int i = range.start(); i <= range.end(); i++) { + StructBlock existing = owners.putIfAbsent(i, node); + if (existing != null && existing != node) { + log.debug( + "Ordinal {} claimed by both {} and {}; keeping the first", + i, + existing, + node); + } + } + } + }); + } + return owners; + } + + private static List parse(PDPage page) throws IOException { + PDFStreamParser parser = new PDFStreamParser(page); + List tokens = new ArrayList<>(); + Object token; + while ((token = parser.parseNextToken()) != null) { + tokens.add(token); + } + return tokens; + } + + private static void write(PDDocument document, PDPage page, List tokens) + throws IOException { + PDStream stream = new PDStream(document); + try (OutputStream out = stream.createOutputStream(COSName.FLATE_DECODE)) { + new ContentStreamWriter(out).writeTokens(tokens); + } + page.setContents(stream); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PageContent.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PageContent.java new file mode 100644 index 0000000000..6c74212621 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PageContent.java @@ -0,0 +1,33 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.List; + +/** + * Everything the layout analyser needs about one page. carriesTextSemantics: existing marked + * content has ActualText/Alt/expansion a rebuild would discard. linesDropped: text became + * artifacts. + */ +public record PageContent( + int pageIndex, + List lines, + List ops, + int markableCount, + boolean preExistingMarkedContent, + boolean carriesTextSemantics, + boolean linesDropped, + BBox mediaBox) { + + public boolean hasText() { + return lines.stream().anyMatch(line -> !line.isBlank()); + } + + /** Markable operators that draw graphics rather than text. */ + public List graphics() { + return ops.stream().filter(op -> op.kind().isGraphic()).toList(); + } + + /** Form XObject invocations, which are tagged as a unit because their text is opaque here. */ + public List forms() { + return ops.stream().filter(op -> op.kind() == MarkableOp.Kind.FORM).toList(); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaIdentificationSchema.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaIdentificationSchema.java new file mode 100644 index 0000000000..d37c2569c9 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaIdentificationSchema.java @@ -0,0 +1,47 @@ +package stirling.software.proprietary.pdf.ua; + +import org.apache.xmpbox.XMPMetadata; +import org.apache.xmpbox.schema.XMPSchema; +import org.apache.xmpbox.type.IntegerType; +import org.apache.xmpbox.type.StructuredType; + +/** + * The {@code pdfuaid} XMP conformance schema, which XMPBox does not ship. Only write it once + * validation has passed - it is a compliance claim. + */ +@StructuredType( + preferedPrefix = PdfUaIdentificationSchema.PREFERRED_PREFIX, + namespace = PdfUaIdentificationSchema.NAMESPACE) +public class PdfUaIdentificationSchema extends XMPSchema { + + public static final String PREFERRED_PREFIX = "pdfuaid"; + public static final String NAMESPACE = "http://www.aiim.org/pdfua/ns/id/"; + + public static final String PART = "part"; + public static final String REV = "rev"; + + public PdfUaIdentificationSchema(XMPMetadata metadata) { + super(metadata); + } + + public PdfUaIdentificationSchema(XMPMetadata metadata, String prefix) { + super(metadata, prefix); + } + + /** Sets {@code pdfuaid:part}, the conformance level (1 or 2). */ + public void setPart(int part) { + addProperty(new IntegerType(getMetadata(), getNamespace(), getPrefix(), PART, part)); + } + + /** Sets {@code pdfuaid:rev}, the four-digit revision year used by PDF/UA-2. */ + public void setRevision(int year) { + addProperty(new IntegerType(getMetadata(), getNamespace(), getPrefix(), REV, year)); + } + + public Integer getPart() { + if (getProperty(PART) instanceof IntegerType part) { + return part.getValue(); + } + return null; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriter.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriter.java new file mode 100644 index 0000000000..7f58654c87 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriter.java @@ -0,0 +1,224 @@ +package stirling.software.proprietary.pdf.ua; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDDocumentCatalog; +import org.apache.pdfbox.pdmodel.PDDocumentInformation; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.common.PDMetadata; +import org.apache.pdfbox.pdmodel.interactive.form.PDAcroForm; +import org.apache.pdfbox.pdmodel.interactive.form.PDField; +import org.apache.pdfbox.pdmodel.interactive.viewerpreferences.PDViewerPreferences; +import org.apache.xmpbox.XMPMetadata; +import org.apache.xmpbox.schema.DublinCoreSchema; +import org.apache.xmpbox.schema.XMPSchema; +import org.apache.xmpbox.xml.DomXmpParser; +import org.apache.xmpbox.xml.XmpSerializer; + +import lombok.extern.slf4j.Slf4j; + +/** Applies the document-level PDF/UA requirements: title, language, tab order, declaration. */ +@Slf4j +public class PdfUaMetadataWriter { + + private static final COSName TABS = COSName.getPDFName("Tabs"); + private static final COSName SUSPECTS = COSName.getPDFName("Suspects"); + + /** + * Applies everything except the conformance declaration. Clause 7.1 requires a title, so a + * blank one falls back to the existing metadata title. + */ + public List applyDocumentRequirements( + PDDocument document, String title, String language, PdfUaProfile profile) + throws IOException { + return applyDocumentRequirements(document, title, language, profile, false); + } + + public List applyDocumentRequirements( + PDDocument document, + String title, + String language, + PdfUaProfile profile, + boolean preserveVersion) + throws IOException { + + List warnings = new ArrayList<>(); + PDDocumentCatalog catalog = document.getDocumentCatalog(); + + if (language != null && !language.isBlank()) { + catalog.setLanguage(language); + } + + String effectiveTitle = resolveTitle(document, title); + if (effectiveTitle != null) { + PDDocumentInformation info = document.getDocumentInformation(); + info.setTitle(effectiveTitle); + document.setDocumentInformation(info); + } + + // Without this a viewer shows the filename instead of the title, which defeats the point. + PDViewerPreferences preferences = catalog.getViewerPreferences(); + if (preferences == null) { + preferences = new PDViewerPreferences(catalog.getCOSObject()); + } + preferences.setDisplayDocTitle(true); + catalog.setViewerPreferences(preferences); + + // Clause 7.18.1: every page needs an explicit tab order. + for (PDPage page : document.getPages()) { + page.getCOSObject().setName(TABS, "S"); + } + + // A structure tree flagged as suspect is not conforming. + if (catalog.getMarkInfo() != null) { + catalog.getMarkInfo().getCOSObject().removeItem(SUSPECTS); + } + + if (!preserveVersion && document.getVersion() < profile.pdfVersion()) { + document.setVersion(profile.pdfVersion()); + } + + warnings.addAll(describeFormFields(document)); + writeXmp(document, effectiveTitle, language, null); + return warnings; + } + + /** + * Gives every form field the {@code /TU} description clause 7.18.1 requires, reusing its + * authored partial name. Unnamed fields are reported, never given a useless placeholder. + */ + private static List describeFormFields(PDDocument document) { + List warnings = new ArrayList<>(); + PDAcroForm form = document.getDocumentCatalog().getAcroForm(); + if (form == null) { + return warnings; + } + int unnamed = 0; + for (PDField field : form.getFieldTree()) { + String existing = field.getAlternateFieldName(); + if (existing != null && !existing.isBlank()) { + continue; + } + String partialName = field.getPartialName(); + if (partialName == null || partialName.isBlank()) { + unnamed++; + continue; + } + field.setAlternateFieldName(partialName); + } + if (unnamed > 0) { + warnings.add( + unnamed + + " form field(s) have neither a description nor a name, so no tooltip" + + " could be derived. Add one for each before claiming conformance."); + } + return warnings; + } + + /** + * Strips the {@code pdfuaid} declaration when validation fails after it was written, so the + * returned file does not assert conformance it lacks. + */ + public void removeConformanceDeclaration(PDDocument document) throws IOException { + PDDocumentCatalog catalog = document.getDocumentCatalog(); + XMPMetadata metadata = loadOrCreate(catalog); + XMPSchema identification = metadata.getSchema(PdfUaIdentificationSchema.NAMESPACE); + if (identification == null) { + return; + } + metadata.removeSchema(identification); + serialiseInto(document, metadata); + } + + /** Writes the {@code pdfuaid:part} declaration. Only call this after validation has passed. */ + public void declareConformance(PDDocument document, PdfUaProfile profile) throws IOException { + writeXmp(document, resolveTitle(document, null), documentLanguage(document), profile); + } + + private String resolveTitle(PDDocument document, String preferred) { + if (preferred != null && !preferred.isBlank()) { + return preferred.strip(); + } + String existing = document.getDocumentInformation().getTitle(); + return existing != null && !existing.isBlank() ? existing.strip() : null; + } + + private static String documentLanguage(PDDocument document) { + return document.getDocumentCatalog().getLanguage(); + } + + /** + * Rewrites the XMP packet, preserving what was there. A malformed packet is replaced, since an + * unparseable one fails validation on its own. + */ + private void writeXmp(PDDocument document, String title, String language, PdfUaProfile profile) + throws IOException { + + PDDocumentCatalog catalog = document.getDocumentCatalog(); + XMPMetadata metadata = loadOrCreate(catalog); + + if (title != null) { + DublinCoreSchema dublinCore = metadata.getDublinCoreSchema(); + if (dublinCore == null) { + dublinCore = metadata.createAndAddDublinCoreSchema(); + } + dublinCore.setTitle(title); + if (language != null + && !language.isBlank() + && (dublinCore.getLanguages() == null + || !dublinCore.getLanguages().contains(language))) { + dublinCore.addLanguage(language); + } + } + + if (profile != null) { + // Re-converting an already-declared file must not leave two pdfuaid schemas. + XMPSchema stale = metadata.getSchema(PdfUaIdentificationSchema.NAMESPACE); + if (stale != null) { + metadata.removeSchema(stale); + } + PdfUaIdentificationSchema identification = new PdfUaIdentificationSchema(metadata); + identification.setPart(profile.part()); + if (profile.revision() > 0) { + identification.setRevision(profile.revision()); + } + metadata.addSchema(identification); + } + + serialiseInto(document, metadata); + } + + private static void serialiseInto(PDDocument document, XMPMetadata metadata) + throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + try { + new XmpSerializer().serialize(metadata, out, true); + } catch (javax.xml.transform.TransformerException e) { + throw new IOException("Could not serialise XMP metadata", e); + } + PDMetadata pdMetadata = new PDMetadata(document); + pdMetadata.importXMPMetadata(out.toByteArray()); + document.getDocumentCatalog().setMetadata(pdMetadata); + } + + private XMPMetadata loadOrCreate(PDDocumentCatalog catalog) { + PDMetadata existing = catalog.getMetadata(); + if (existing != null) { + try { + DomXmpParser parser = new DomXmpParser(); + // Strict parsing rejects pdfuaid, silently discarding a packet we just wrote. + parser.setStrictParsing(false); + return parser.parse(new ByteArrayInputStream(existing.toByteArray())); + } catch (Exception e) { + log.debug("Replacing unparseable XMP packet: {}", e.getMessage()); + } + } + return XMPMetadata.createXMPMetadata(); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaProfile.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaProfile.java new file mode 100644 index 0000000000..9523f5c790 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaProfile.java @@ -0,0 +1,47 @@ +package stirling.software.proprietary.pdf.ua; + +/** The PDF/UA conformance level a conversion targets. */ +public enum PdfUaProfile { + /** ISO 14289-1, layered on PDF 1.7. */ + UA1(1, 1.7f, 0), + /** ISO 14289-2: needs PDF 2.0, namespaced structure types and a revision year. */ + UA2(2, 2.0f, 2024); + + private final int part; + private final float pdfVersion; + private final int revision; + + PdfUaProfile(int part, float pdfVersion, int revision) { + this.part = part; + this.pdfVersion = pdfVersion; + this.revision = revision; + } + + public int part() { + return part; + } + + public float pdfVersion() { + return pdfVersion; + } + + /** The {@code pdfuaid:rev} year, or 0 when the profile does not use one. */ + public int revision() { + return revision; + } + + public String displayName() { + return "PDF/UA-" + part; + } + + public static PdfUaProfile fromRequest(String value) { + if (value == null || value.isBlank()) { + return UA1; + } + String normalised = value.trim().toLowerCase().replace("/", "").replace("-", ""); + return switch (normalised) { + case "ua2", "pdfua2", "2" -> UA2; + default -> UA1; + }; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaTagger.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaTagger.java new file mode 100644 index 0000000000..7f5dcc3739 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/PdfUaTagger.java @@ -0,0 +1,303 @@ +package stirling.software.proprietary.pdf.ua; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDDocumentCatalog; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureElement; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureTreeRoot; + +import lombok.extern.slf4j.Slf4j; + +/** + * Tags an untagged PDF and applies the document-level PDF/UA requirements. Content must be marked + * before the tree can reference it, and conformance is declared elsewhere, only after validation. + */ +@Slf4j +public class PdfUaTagger { + + private final TaggedContentExtractor extractor = new TaggedContentExtractor(); + private final LayoutAnalyzer analyzer = new LayoutAnalyzer(); + private final MarkedContentInjector injector = new MarkedContentInjector(); + private final PdfUaMetadataWriter metadataWriter = new PdfUaMetadataWriter(); + + public TaggingResult tag(PDDocument document, TaggingOptions options) throws IOException { + boolean alreadyTagged = hasUsableStructureTree(document); + boolean rebuild = + switch (options.getExistingTags()) { + case KEEP -> false; + case REBUILD -> true; + case AUTO -> !alreadyTagged; + }; + + List languageWarnings = new ArrayList<>(); + String language = resolveLanguage(document, options, languageWarnings); + + if (!rebuild) { + log.info("Keeping existing structure tree; applying document requirements only"); + DocumentStructure kept = new DocumentStructure(); + languageWarnings.forEach(kept::warn); + metadataWriter + .applyDocumentRequirements( + document, + options.getTitle(), + language, + options.getProfile(), + options.isPreservePdfVersion()) + .forEach(kept::warn); + return new TaggingResult(kept, false); + } + + // Types the old tree carried, so a rebuild that cannot reproduce them can say so. Font + // embedding may already have deleted the tree, so fall back to what the source had. + Set discardedTypes = + alreadyTagged + ? structureTypes(document) + : options.getSourceFacts().structureTypes(); + + if (alreadyTagged) { + stripStructure(document); + } + + List pages = extractor.extract(document); + DocumentStructure structure = analyzer.analyse(pages); + structure.setLanguage(language); + languageWarnings.forEach(structure::warn); + applyFigurePolicy(structure, options); + + if (structure.isEmpty()) { + structure.warn( + "No taggable content was found; the document may be a scan with no text layer."); + } + + injectMarkedContent(document, structure, pages); + new StructTreeWriter().write(document, structure, options.getProfile()); + // Losing the tree to the embedder is a different problem from a requested rebuild, and + // the advice that helps differs too, so tell them apart. + boolean lostToEmbedder = !alreadyTagged && options.getSourceFacts().hasUsableTree(); + warnAboutFlattenedStructure( + discardedTypes, structureTypes(document), structure, lostToEmbedder); + + String title = resolveTitle(options, structure); + if (title == null) { + structure.warn( + "No document title could be derived. PDF/UA requires one, so supply a title."); + } + metadataWriter + .applyDocumentRequirements( + document, + title, + language, + options.getProfile(), + options.isPreservePdfVersion()) + .forEach(structure::warn); + + return new TaggingResult(structure, true); + } + + /** + * Keeps the language the document already declares. Overwriting it relabels, say, a French file + * as English, and no validator can catch that. + */ + private static String resolveLanguage( + PDDocument document, TaggingOptions options, List warnings) { + String existing = document.getDocumentCatalog().getLanguage(); + if (existing == null || existing.isBlank()) { + // Font embedding discards /Lang, so without this a rewritten French document would + // silently take the caller's default language. + existing = options.getSourceFacts().language(); + } + String requested = options.getLanguage(); + if (existing == null || existing.isBlank() || options.isOverrideLanguage()) { + return requested; + } + if (requested != null && !requested.isBlank() && !requested.equalsIgnoreCase(existing)) { + warnings.add( + "The document already declares its language as '" + + existing + + "', so the requested '" + + requested + + "' was ignored. Ask to override the language to change it."); + } + return existing; + } + + /** Explicit title first, then the first heading, then the caller's fallback. */ + private static String resolveTitle(TaggingOptions options, DocumentStructure structure) { + for (String candidate : + new String[] { + options.getTitle(), structure.getTitle(), options.getFallbackTitle() + }) { + if (candidate != null && !candidate.isBlank()) { + return candidate.strip(); + } + } + return null; + } + + /** Writes the conformance declaration. Separate from tagging so validation can gate it. */ + public void declareConformance(PDDocument document, PdfUaProfile profile) throws IOException { + metadataWriter.declareConformance(document, profile); + } + + /** Withdraws the conformance claim, for a document that turned out not to validate. */ + public void withdrawConformance(PDDocument document) throws IOException { + metadataWriter.removeConformanceDeclaration(document); + } + + /** Wraps content page by page; marked content ids restart on each page. */ + private void injectMarkedContent( + PDDocument document, DocumentStructure structure, List pages) + throws IOException { + Map markableCounts = new LinkedHashMap<>(); + pages.forEach(page -> markableCounts.put(page.pageIndex(), page.markableCount())); + Map> byPage = new LinkedHashMap<>(); + for (StructBlock block : structure.getBlocks()) { + byPage.computeIfAbsent(block.getPageIndex(), k -> new ArrayList<>()).add(block); + } + for (int pageIndex = 0; pageIndex < document.getNumberOfPages(); pageIndex++) { + List blocks = byPage.getOrDefault(pageIndex, List.of()); + // Nothing to wrap, and rewriting costs a parse and recompress for an identical stream. + if (blocks.isEmpty() && markableCounts.getOrDefault(pageIndex, 0) == 0) { + continue; + } + injector.inject(document, document.getPage(pageIndex), blocks, 0, true); + } + } + + /** Applies alt text supplied by the caller, or demotes images to artifacts on request. */ + private static void applyFigurePolicy(DocumentStructure structure, TaggingOptions options) { + int[] suppressed = {0}; + structure.visit( + block -> { + if (block.getType() != StructType.FIGURE) { + return; + } + if (options.getFigurePolicy() == TaggingOptions.FigurePolicy.MARK_DECORATIVE) { + block.setType(StructType.ARTIFACT); + block.setArtifactType(ArtifactType.LAYOUT); + suppressed[0]++; + return; + } + int ordinal = + block.getRanges().isEmpty() ? -1 : block.getRanges().get(0).start(); + String alt = options.altTextFor(block.getPageIndex(), ordinal); + if (alt != null && !alt.isBlank()) { + block.setAlt(alt); + } + }); + // Marking images decorative validates by hiding content, so never report it as clean. + if (suppressed[0] > 0) { + structure.warn( + suppressed[0] + + " image(s) were marked as decoration and are now hidden from" + + " assistive technology. Confirm none of them carried meaning."); + } + int missing = structure.figuresWithoutAlt().size(); + if (missing > 0) { + structure.warn( + missing + + " figure(s) have no alternative description. PDF/UA requires one for" + + " every image that carries meaning."); + } + } + + /** + * A tree is only worth keeping when wired up: kids, a parent tree, and a marked catalog. + * Keeping one that fails any of those leaves the document permanently unfixable. + */ + public static boolean hasUsableStructureTree(PDDocument document) { + PDDocumentCatalog catalog = document.getDocumentCatalog(); + PDStructureTreeRoot root = catalog.getStructureTreeRoot(); + if (root == null) { + return false; + } + try { + boolean hasKids = root.getKids() != null && !root.getKids().isEmpty(); + boolean hasParentTree = root.getParentTree() != null; + boolean marked = catalog.getMarkInfo() != null && catalog.getMarkInfo().isMarked(); + return hasKids && hasParentTree && marked; + } catch (RuntimeException e) { + log.debug("Unreadable structure tree, treating as absent: {}", e.getMessage()); + return false; + } + } + + /** + * A rebuild derives structure from layout, so semantics the old tree carried can vanish - a + * table becomes loose paragraphs. Validators cannot see that loss, so it has to be reported. + */ + private static void warnAboutFlattenedStructure( + Set before, + Set after, + DocumentStructure structure, + boolean lostToEmbedder) { + List lost = + MEANINGFUL_TYPES.stream() + .filter(type -> before.contains(type) && !after.contains(type)) + .toList(); + if (lost.isEmpty()) { + return; + } + // Keeping the tags cannot help once the embedder has deleted them, so do not suggest it. + String remedy = + lostToEmbedder + ? " Embedding the missing fonts rewrote the document and deleted its" + + " original tags. Turn off font embedding to keep them." + : " Keep the existing tags instead to preserve it."; + structure.warn( + "Rebuilding the tags could not reproduce " + + String.join(", ", lost) + + " structure, so that content is now plain paragraphs." + + remedy); + } + + /** Structure whose loss changes what a screen reader conveys, not just how it is nested. */ + private static final List MEANINGFUL_TYPES = + List.of("Table", "TH", "Formula", "L", "LI", "TOC", "Note"); + + private static Set structureTypes(PDDocument document) { + Set types = new HashSet<>(); + try { + PDStructureTreeRoot root = document.getDocumentCatalog().getStructureTreeRoot(); + if (root != null) { + collectTypes(root.getKids(), types, 0); + } + } catch (RuntimeException e) { + log.debug("Could not read structure types: {}", e.getMessage()); + } + return types; + } + + private static void collectTypes(Object node, Set types, int depth) { + // Structure trees can be deep or, in damaged files, cyclic; cap rather than overflow. + if (node == null || depth > 64) { + return; + } + if (node instanceof List list) { + list.forEach(child -> collectTypes(child, types, depth + 1)); + } else if (node instanceof PDStructureElement element) { + types.add(element.getStructureType()); + collectTypes(element.getKids(), types, depth + 1); + } + } + + private static void stripStructure(PDDocument document) { + PDDocumentCatalog catalog = document.getDocumentCatalog(); + catalog.getCOSObject().removeItem(COSName.getPDFName("StructTreeRoot")); + catalog.getCOSObject().removeItem(COSName.getPDFName("MarkInfo")); + document.getPages() + .forEach( + page -> + page.getCOSObject() + .removeItem(COSName.getPDFName("StructParents"))); + log.info("Removed existing structure tree before rebuilding"); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/SourceFacts.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/SourceFacts.java new file mode 100644 index 0000000000..4f058d6c48 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/SourceFacts.java @@ -0,0 +1,59 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.HashSet; +import java.util.Set; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureElement; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureTreeRoot; + +import lombok.extern.slf4j.Slf4j; + +/** + * What the document said about itself before anything rewrote it. Font embedding shells out to + * Ghostscript, which returns a file with no structure tree, no {@code /Lang} and no XMP, so a + * tagger reading the rewritten document sees an untagged, language-less file and cannot tell that + * anything was lost. These facts are captured from the original and carried past that stage. + * + * @param language the catalog {@code /Lang} the author declared, or null + * @param structureTypes every structure element type the original tree contained + * @param hasUsableTree whether the original had a structure tree worth preserving + */ +@Slf4j +public record SourceFacts(String language, Set structureTypes, boolean hasUsableTree) { + + private static final int MAX_DEPTH = 64; + + /** Facts for a document nothing has rewritten, used when font embedding did not run. */ + public static final SourceFacts NONE = new SourceFacts(null, Set.of(), false); + + public static SourceFacts of(PDDocument document) { + String language = null; + Set types = new HashSet<>(); + boolean usable = false; + try { + language = document.getDocumentCatalog().getLanguage(); + usable = PdfUaTagger.hasUsableStructureTree(document); + PDStructureTreeRoot root = document.getDocumentCatalog().getStructureTreeRoot(); + if (root != null) { + collect(root.getKids(), types, 0); + } + } catch (RuntimeException e) { + log.debug("Could not read source facts: {}", e.getMessage()); + } + return new SourceFacts(language, Set.copyOf(types), usable); + } + + private static void collect(Object node, Set types, int depth) { + // Damaged files can present a cyclic tree; cap rather than overflow the stack. + if (node == null || depth > MAX_DEPTH) { + return; + } + if (node instanceof java.util.List list) { + list.forEach(child -> collect(child, types, depth + 1)); + } else if (node instanceof PDStructureElement element) { + types.add(element.getStructureType()); + collect(element.getKids(), types, depth + 1); + } + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructBlock.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructBlock.java new file mode 100644 index 0000000000..d2f3e452b1 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructBlock.java @@ -0,0 +1,134 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.ArrayList; +import java.util.List; +import java.util.function.Consumer; + +import lombok.Getter; +import lombok.Setter; + +/** + * One node of the derived logical structure: either page content (ranges of markable operator + * ordinals) or child blocks. Containers with no content are pruned before serialisation. + */ +@Getter +@Setter +public class StructBlock { + + /** A contiguous, inclusive run of markable operator ordinals within one page stream. */ + public record OrdinalRange(int start, int end) { + public boolean contains(int ordinal) { + return ordinal >= start && ordinal <= end; + } + + public int size() { + return end - start + 1; + } + } + + private StructType type; + private ArtifactType artifactType; + private int pageIndex; + private BBox bbox = BBox.EMPTY; + private String text = ""; + + private final List ranges = new ArrayList<>(); + private final List children = new ArrayList<>(); + + /** {@code /Alt} - required on Figure and Formula for PDF/UA. */ + private String alt; + + /** {@code /ActualText} - replacement text for content whose glyphs do not spell the word. */ + private String actualText; + + /** {@code /Lang} - set only where it differs from the document default. */ + private String lang; + + /** {@code /Scope} on a TH: Row, Column or Both. */ + private String scope; + + /** {@code /ListNumbering} on an L. */ + private String listNumbering; + + /** Unique {@code /ID}, required on Note and FENote elements. */ + private String id; + + /** + * Marked content ids assigned during injection; one block yields several when split, since a + * sequence must nest inside BT/ET and q/Q rather than straddle them. + */ + private final List mcids = new ArrayList<>(); + + /** True when the source content was already inside a marked-content sequence. */ + private boolean preMarked; + + public StructBlock(StructType type, int pageIndex) { + this.type = type; + this.pageIndex = pageIndex; + } + + public static StructBlock artifact(ArtifactType artifactType, int pageIndex) { + StructBlock block = new StructBlock(StructType.ARTIFACT, pageIndex); + block.artifactType = artifactType; + return block; + } + + public StructBlock addChild(StructBlock child) { + children.add(child); + return this; + } + + public StructBlock addRange(int start, int end) { + ranges.add(new OrdinalRange(start, end)); + return this; + } + + public boolean isArtifact() { + return type == StructType.ARTIFACT; + } + + /** Depth-first walk over this block and all descendants. */ + public void visit(Consumer visitor) { + visitor.accept(this); + for (StructBlock child : children) { + child.visit(visitor); + } + } + + /** Total number of ordinals owned by this block and its descendants. */ + public int contentCount() { + int total = ranges.stream().mapToInt(OrdinalRange::size).sum(); + for (StructBlock child : children) { + total += child.contentCount(); + } + return total; + } + + /** Concatenated text of this block and its descendants, in tree order. */ + public String collectText() { + StringBuilder sb = new StringBuilder(); + visit( + block -> { + if (!block.text.isBlank()) { + if (sb.length() > 0) { + sb.append(' '); + } + sb.append(block.text.strip()); + } + }); + return sb.toString(); + } + + @Override + public String toString() { + return type.tag() + + (artifactType != null ? "[" + artifactType.subtype() + "]" : "") + + "(p" + + pageIndex + + ", " + + ranges.size() + + " ranges, " + + children.size() + + " kids)"; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructTreeWriter.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructTreeWriter.java new file mode 100644 index 0000000000..f6b5d89b4c --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructTreeWriter.java @@ -0,0 +1,295 @@ +package stirling.software.proprietary.pdf.ua; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import org.apache.pdfbox.cos.COSArray; +import org.apache.pdfbox.cos.COSBase; +import org.apache.pdfbox.cos.COSDictionary; +import org.apache.pdfbox.cos.COSInteger; +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.common.PDNumberTreeNode; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDMarkInfo; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDObjectReference; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureElement; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureTreeRoot; +import org.apache.pdfbox.pdmodel.documentinterchange.taggedpdf.PDListAttributeObject; +import org.apache.pdfbox.pdmodel.documentinterchange.taggedpdf.PDTableAttributeObject; +import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotation; +import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotationLink; +import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotationWidget; + +import lombok.extern.slf4j.Slf4j; + +/** + * Serialises a {@link DocumentStructure} into a PDF structure tree. Must run after {@link + * MarkedContentInjector}, which assigns the marked content ids this writer references. + */ +@Slf4j +public class StructTreeWriter { + + private static final COSName STRUCT_PARENT = COSName.getPDFName("StructParent"); + private static final COSName NUMS = COSName.getPDFName("Nums"); + private static final String PDF2_STANDARD_NAMESPACE = "http://iso.org/pdf2/ssn"; + + /** Per-page marked content id to owning element, built while walking the tree. */ + private final Map> mcidOwners = new LinkedHashMap<>(); + + private COSDictionary standardNamespace; + private int nextParentKey; + + public void write(PDDocument document, DocumentStructure structure, PdfUaProfile profile) + throws IOException { + PDStructureTreeRoot root = new PDStructureTreeRoot(); + PDStructureElement documentElement = + new PDStructureElement(StructType.DOCUMENT.tag(), root); + if (structure.getLanguage() != null) { + documentElement.setLanguage(structure.getLanguage()); + } + if (profile == PdfUaProfile.UA2) { + applyNamespace(documentElement, document); + } + + for (StructBlock block : structure.getBlocks()) { + if (block.isArtifact()) { + continue; + } + PDStructureElement child = buildElement(document, block, documentElement, profile); + if (child != null) { + documentElement.appendKid(child); + } + } + + root.appendKid(documentElement); + buildParentTree(document, root); + registerNamespaces(root); + + PDMarkInfo markInfo = new PDMarkInfo(); + markInfo.setMarked(true); + document.getDocumentCatalog().setMarkInfo(markInfo); + document.getDocumentCatalog().setStructureTreeRoot(root); + } + + /** Recursively builds an element, returning null when the block carries no content at all. */ + private PDStructureElement buildElement( + PDDocument document, + StructBlock block, + PDStructureElement parent, + PdfUaProfile profile) { + + // Prune on assigned MCIDs, not claimed ranges: form-XObject lines all resolve to one Do, + // and emitting the losers would announce empty paragraphs to a screen reader. + if (!carriesContent(block)) { + return null; + } + StructType type = effectiveType(block, profile); + PDStructureElement element = new PDStructureElement(type.tag(), parent); + PDPage page = document.getPage(block.getPageIndex()); + element.setPage(page); + + if (profile == PdfUaProfile.UA2) { + applyNamespace(element, document); + } + applyAttributes(block, element); + + for (int mcid : block.getMcids()) { + element.appendKid(mcid); + mcidOwners + .computeIfAbsent(block.getPageIndex(), k -> new LinkedHashMap<>()) + .put(mcid, element); + } + + for (StructBlock child : block.getChildren()) { + PDStructureElement childElement = buildElement(document, child, element, profile); + if (childElement != null) { + element.appendKid(childElement); + } + } + return element; + } + + /** True when this block, or something beneath it, was actually given marked content. */ + private static boolean carriesContent(StructBlock block) { + if (!block.getMcids().isEmpty()) { + return true; + } + return block.getChildren().stream().anyMatch(StructTreeWriter::carriesContent); + } + + /** PDF/UA-2 replaces Note with FENote for footnotes. */ + private static StructType effectiveType(StructBlock block, PdfUaProfile profile) { + if (profile == PdfUaProfile.UA2 && block.getType() == StructType.NOTE) { + return StructType.FENOTE; + } + return block.getType(); + } + + private static void applyAttributes(StructBlock block, PDStructureElement element) { + if (block.getAlt() != null && !block.getAlt().isBlank()) { + element.setAlternateDescription(block.getAlt()); + } + if (block.getActualText() != null && !block.getActualText().isBlank()) { + element.setActualText(block.getActualText()); + } + if (block.getLang() != null && !block.getLang().isBlank()) { + element.setLanguage(block.getLang()); + } + if (block.getId() != null && !block.getId().isBlank()) { + element.setElementIdentifier(block.getId()); + } + if (block.getScope() != null) { + PDTableAttributeObject table = new PDTableAttributeObject(); + table.setScope(block.getScope()); + element.addAttribute(table); + } + if (block.getListNumbering() != null) { + PDListAttributeObject list = new PDListAttributeObject(); + list.setListNumbering(block.getListNumbering()); + element.addAttribute(list); + } + } + + /** PDF/UA-2 requires every element to declare the standard structure namespace. */ + private void applyNamespace(PDStructureElement element, PDDocument document) { + element.getCOSObject().setItem(COSName.getPDFName("NS"), standardNamespace()); + } + + /** The PDF 2.0 standard structure namespace, created once per document. */ + private COSDictionary standardNamespace() { + if (standardNamespace == null) { + standardNamespace = new COSDictionary(); + standardNamespace.setName(COSName.TYPE, "Namespace"); + standardNamespace.setString(COSName.getPDFName("NS"), PDF2_STANDARD_NAMESPACE); + } + return standardNamespace; + } + + private void registerNamespaces(PDStructureTreeRoot root) { + if (standardNamespace == null) { + return; + } + COSArray namespaces = new COSArray(); + namespaces.add(standardNamespace); + root.getCOSObject().setItem(COSName.getPDFName("Namespaces"), namespaces); + } + + /** + * Builds {@code /ParentTree}: per page, an array indexed by marked content id keyed on {@code + * /StructParents}, plus one entry per annotation keyed on {@code /StructParent}. + */ + private void buildParentTree(PDDocument document, PDStructureTreeRoot root) { + COSArray nums = new COSArray(); + nextParentKey = 0; + + for (int pageIndex = 0; pageIndex < document.getNumberOfPages(); pageIndex++) { + Map owners = mcidOwners.get(pageIndex); + if (owners == null || owners.isEmpty()) { + continue; + } + PDPage page = document.getPage(pageIndex); + int key = nextParentKey++; + page.setStructParents(key); + + int maxMcid = owners.keySet().stream().mapToInt(Integer::intValue).max().orElse(-1); + COSArray entries = new COSArray(); + for (int mcid = 0; mcid <= maxMcid; mcid++) { + PDStructureElement owner = owners.get(mcid); + entries.add( + owner != null ? owner.getCOSObject() : org.apache.pdfbox.cos.COSNull.NULL); + } + nums.add(COSInteger.get(key)); + nums.add(entries); + } + + List annotationEntries = tagAnnotations(document, root); + for (int i = 0; i + 1 < annotationEntries.size(); i += 2) { + nums.add(annotationEntries.get(i)); + nums.add(annotationEntries.get(i + 1)); + } + + COSDictionary parentTreeDict = new COSDictionary(); + parentTreeDict.setItem(NUMS, nums); + root.setParentTree(new PDNumberTreeNode(parentTreeDict, PDStructureElement.class)); + root.setParentTreeNextKey(nextParentKey); + } + + /** + * Clause 7.18: every visible annotation needs a structure element so it is reachable from the + * tree. Links become Link elements, anything else an Annot. + */ + private List tagAnnotations(PDDocument document, PDStructureTreeRoot root) { + List entries = new ArrayList<>(); + PDStructureElement documentElement = firstDocumentElement(root); + if (documentElement == null) { + return entries; + } + for (int pageIndex = 0; pageIndex < document.getNumberOfPages(); pageIndex++) { + PDPage page = document.getPage(pageIndex); + List annotations; + try { + annotations = page.getAnnotations(); + } catch (IOException e) { + log.debug("Could not read annotations on page {}: {}", pageIndex, e.getMessage()); + continue; + } + for (PDAnnotation annotation : annotations) { + if (annotation == null + || annotation.isHidden() + || annotation.isNoView() + || "Popup".equals(annotation.getSubtype())) { + continue; + } + PDStructureElement element = + new PDStructureElement(annotationType(annotation), documentElement); + element.setPage(page); + + PDObjectReference reference = new PDObjectReference(); + reference.setReferencedObject(annotation); + element.appendKid(reference); + documentElement.appendKid(element); + + int key = nextParentKey++; + annotation.getCOSObject().setInt(STRUCT_PARENT, key); + entries.add(COSInteger.get(key)); + entries.add(element.getCOSObject()); + + if (annotation.getContents() == null || annotation.getContents().isBlank()) { + annotation.setContents(defaultContents(annotation)); + } + } + } + return entries; + } + + /** Clause 7.18.4: widgets need a Form element, links a Link element, everything else Annot. */ + private static String annotationType(PDAnnotation annotation) { + if (annotation instanceof PDAnnotationWidget) { + return StructType.FORM.tag(); + } + if (annotation instanceof PDAnnotationLink) { + return StructType.LINK.tag(); + } + return "Annot"; + } + + private static String defaultContents(PDAnnotation annotation) { + if (annotation instanceof PDAnnotationLink link && link.getAction() != null) { + return "Link"; + } + return annotation.getSubtype() == null ? "Annotation" : annotation.getSubtype(); + } + + private static PDStructureElement firstDocumentElement(PDStructureTreeRoot root) { + for (Object kid : root.getKids()) { + if (kid instanceof PDStructureElement element) { + return element; + } + } + return null; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructType.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructType.java new file mode 100644 index 0000000000..b4634a58c6 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/StructType.java @@ -0,0 +1,62 @@ +package stirling.software.proprietary.pdf.ua; + +/** + * PDF standard structure types emitted by the tagger (ISO 32000-1 14.8.4), limited to the PDF/UA + * subset. {@link #ARTIFACT} is not one: it marks content in the stream and stays out of the tree. + */ +public enum StructType { + DOCUMENT("Document"), + PART("Part"), + SECT("Sect"), + H1("H1"), + H2("H2"), + H3("H3"), + H4("H4"), + H5("H5"), + H6("H6"), + P("P"), + L("L"), + LI("LI"), + LBL("Lbl"), + LBODY("LBody"), + TABLE("Table"), + TR("TR"), + TH("TH"), + TD("TD"), + FIGURE("Figure"), + CAPTION("Caption"), + FORMULA("Formula"), + NOTE("Note"), + FENOTE("FENote"), + LINK("Link"), + /** Wraps a widget annotation; PDF/UA-1 clause 7.18.4 requires widgets to sit inside one. */ + FORM("Form"), + SPAN("Span"), + ARTIFACT("Artifact"); + + private final String tag; + + StructType(String tag) { + this.tag = tag; + } + + /** The name written into the PDF {@code /S} entry. */ + public String tag() { + return tag; + } + + public boolean isHeading() { + return this == H1 || this == H2 || this == H3 || this == H4 || this == H5 || this == H6; + } + + /** Heading level 1-6, or 0 when this is not a heading. */ + public int headingLevel() { + return isHeading() ? ordinal() - H1.ordinal() + 1 : 0; + } + + /** The heading type for a 1-based level, clamped to the H1-H6 range. */ + public static StructType heading(int level) { + int clamped = Math.max(1, Math.min(6, level)); + return values()[H1.ordinal() + clamped - 1]; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggedContentExtractor.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggedContentExtractor.java new file mode 100644 index 0000000000..01c86215d4 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggedContentExtractor.java @@ -0,0 +1,630 @@ +package stirling.software.proprietary.pdf.ua; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.Map; + +import org.apache.pdfbox.contentstream.operator.Operator; +import org.apache.pdfbox.cos.COSBase; +import org.apache.pdfbox.cos.COSDictionary; +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdfparser.PDFStreamParser; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDResources; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.font.PDFontDescriptor; +import org.apache.pdfbox.pdmodel.font.PDType3Font; +import org.apache.pdfbox.pdmodel.graphics.PDXObject; +import org.apache.pdfbox.pdmodel.graphics.form.PDFormXObject; +import org.apache.pdfbox.pdmodel.graphics.form.PDTransparencyGroup; +import org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject; +import org.apache.pdfbox.text.PDFTextStripper; +import org.apache.pdfbox.text.TextPosition; +import org.apache.pdfbox.util.Matrix; +import org.apache.pdfbox.util.Vector; + +import lombok.extern.slf4j.Slf4j; + +/** + * Extracts text lines and graphic ops from page streams, tagging each with its operator ordinal. + * Both passes count the same operators in the same order, so ordinals cross-reference. + */ +@Slf4j +public class TaggedContentExtractor { + + /** Glyph size below which a run is treated as noise rather than a line. */ + private static final float MIN_FONT_SIZE = 0.5f; + + public List extract(PDDocument document) throws IOException { + LineCollector collector = new LineCollector(); + collector.setSortByPosition(true); + collector.setStartPage(1); + collector.setEndPage(document.getNumberOfPages()); + collector.writeText(document, Writer.nullWriter()); + + List pages = new ArrayList<>(document.getNumberOfPages()); + for (int i = 0; i < document.getNumberOfPages(); i++) { + PDPage page = document.getPage(i); + List ops = collector.opsFor(i); + List lines = collector.linesFor(i); + boolean dropped = false; + if (ops.size() < maxOrdinal(lines) + 1) { + // Untrusted ordinals: drop the lines so the page is untaggable rather than + // mis-tagged, and flag it so the caller refuses to declare conformance. + log.warn( + "Ordinal mismatch on page {} (ops={}, text={}); skipping page", + i, + ops.size(), + maxOrdinal(lines) + 1); + dropped = !lines.isEmpty(); + lines = List.of(); + } + pages.add( + new PageContent( + i, + lines, + ops, + ops.size(), + collector.preMarkedOn(i), + collector.textSemanticsOn(i), + dropped, + normalisedBox(page))); + } + return pages; + } + + /** + * The page box in the space of extracted line coordinates: origin-zero, width and height + * swapped for 90/270 rotations, because the text engine reports in the rotated frame. + */ + static BBox normalisedBox(PDPage page) { + PDRectangle mediaBox = page.getMediaBox(); + boolean sideways = page.getRotation() % 180 != 0; + float width = sideways ? mediaBox.getHeight() : mediaBox.getWidth(); + float height = sideways ? mediaBox.getWidth() : mediaBox.getHeight(); + return new BBox(0, 0, width, height); + } + + /** Counts images with the token scan alone, skipping the expensive text pass. */ + public int countGraphics(PDDocument document) { + int total = 0; + for (int i = 0; i < document.getNumberOfPages(); i++) { + try { + PDResources resources = document.getPage(i).getResources(); + PDFStreamParser parser = new PDFStreamParser(document.getPage(i)); + List operands = new ArrayList<>(); + Object token; + while ((token = parser.parseNextToken()) != null) { + if (!(token instanceof Operator operator)) { + operands.add((COSBase) token); + continue; + } + if (isGraphicOperator(operator.getName(), operands, resources)) { + total++; + } + operands.clear(); + } + } catch (IOException e) { + log.debug("Could not scan page {} for graphics: {}", i, e.getMessage()); + } + } + return total; + } + + /** True for an inline image, or a Do that resolves to an image XObject. */ + private static boolean isGraphicOperator( + String name, List operands, PDResources resources) { + if ("BI".equals(name)) { + return true; + } + if (!"Do".equals(name) || resources == null || operands.size() != 1) { + return false; + } + if (!(operands.get(0) instanceof COSName resourceName)) { + return false; + } + try { + return resources.getXObject(resourceName) instanceof PDImageXObject; + } catch (IOException e) { + return false; + } + } + + private static int maxOrdinal(List lines) { + return lines.stream().mapToInt(TextLineInfo::endOrdinal).max().orElse(-1); + } + + static BBox toBBox(PDRectangle rect) { + return new BBox( + rect.getLowerLeftX(), + rect.getLowerLeftY(), + rect.getUpperRightX(), + rect.getUpperRightY()); + } + + // --- Operator classification ------------------------------------------- + + private static boolean isPathConstruction(String name) { + return switch (name) { + case "m", "l", "c", "v", "y", "re" -> true; + default -> false; + }; + } + + /** True when a sequence carries replacement or alternative text, which a rebuild would drop. */ + private static boolean carriesTextSemantics(List operands) { + for (COSBase operand : operands) { + if (operand instanceof COSDictionary dictionary + && (dictionary.containsKey(COSName.getPDFName("ActualText")) + || dictionary.containsKey(COSName.getPDFName("Alt")) + || dictionary.containsKey(COSName.E))) { + return true; + } + } + return false; + } + + /** + * Describes one markable operator, placed with the engine's own matrix rather than a + * hand-rolled q/Q/cm stack that would get nesting and form matrices wrong. + */ + private static MarkableOp classify( + String name, + List operands, + PDResources resources, + Matrix ctm, + BBox pathBox, + int ordinal) { + + if ("BI".equals(name)) { + return new MarkableOp(ordinal, MarkableOp.Kind.INLINE_IMAGE, unitSquare(ctm), null); + } + if (MarkableOp.isPathPainting(name)) { + return new MarkableOp(ordinal, MarkableOp.Kind.VECTOR, pathBox, null); + } + if (!"Do".equals(name)) { + return new MarkableOp(ordinal, MarkableOp.Kind.TEXT, BBox.EMPTY, null); + } + COSName resourceName = + operands.size() == 1 && operands.get(0) instanceof COSName n ? n : null; + if (resourceName == null || resources == null) { + return new MarkableOp(ordinal, MarkableOp.Kind.FORM, unitSquare(ctm), null); + } + try { + PDXObject xobject = resources.getXObject(resourceName); + if (xobject instanceof PDImageXObject) { + return new MarkableOp( + ordinal, MarkableOp.Kind.IMAGE, unitSquare(ctm), resourceName.getName()); + } + if (xobject instanceof PDFormXObject form) { + return new MarkableOp( + ordinal, MarkableOp.Kind.FORM, formBox(form, ctm), resourceName.getName()); + } + } catch (IOException e) { + log.debug("Could not resolve XObject {}: {}", resourceName.getName(), e.getMessage()); + } + return new MarkableOp( + ordinal, MarkableOp.Kind.FORM, unitSquare(ctm), resourceName.getName()); + } + + /** + * Extends the running path box with one path-construction operator's points; without it every + * vector had an empty box and charts and vector logos vanished from the structure tree. + */ + private static BBox extendPath(BBox current, String name, List operands, Matrix ctm) { + int pairs = + switch (name) { + case "m", "l" -> 1; + case "re" -> 2; + case "v", "y" -> 2; + case "c" -> 3; + default -> 0; + }; + if (pairs == 0 || operands.size() < pairs * 2) { + return current; + } + + // Deliberately allocation-free; the obvious version cost a third of the extraction budget. + float minX = current.isEmpty() ? Float.MAX_VALUE : current.x0(); + float minY = current.isEmpty() ? Float.MAX_VALUE : current.y0(); + float maxX = current.isEmpty() ? -Float.MAX_VALUE : current.x1(); + float maxY = current.isEmpty() ? -Float.MAX_VALUE : current.y1(); + + for (int pair = 0; pair < pairs; pair++) { + Float x = numberAt(operands, pair * 2); + Float y = numberAt(operands, pair * 2 + 1); + if (x == null || y == null) { + continue; + } + float px = x; + float py = y; + // "re" gives origin plus size, so the second pair is a corner offset from the first. + if ("re".equals(name) && pair == 1) { + Float ox = numberAt(operands, 0); + Float oy = numberAt(operands, 1); + if (ox == null || oy == null) { + continue; + } + px = ox + x; + py = oy + y; + } + float tx = ctm.getScaleX() * px + ctm.getShearX() * py + ctm.getTranslateX(); + float ty = ctm.getShearY() * px + ctm.getScaleY() * py + ctm.getTranslateY(); + minX = Math.min(minX, tx); + minY = Math.min(minY, ty); + maxX = Math.max(maxX, tx); + maxY = Math.max(maxY, ty); + } + return maxX < minX ? current : new BBox(minX, minY, maxX, maxY); + } + + private static Float numberAt(List operands, int index) { + return index < operands.size() + && operands.get(index) instanceof org.apache.pdfbox.cos.COSNumber number + ? number.floatValue() + : null; + } + + /** The unit square mapped through the CTM, which is how images are placed. */ + private static BBox unitSquare(Matrix ctm) { + return transformBox(new BBox(0, 0, 1, 1), ctm); + } + + private static BBox formBox(PDFormXObject form, Matrix ctm) { + PDRectangle box = form.getBBox(); + if (box == null) { + return unitSquare(ctm); + } + Matrix combined = form.getMatrix() != null ? form.getMatrix().multiply(ctm) : ctm; + return transformBox(toBBox(box), combined); + } + + private static BBox transformBox(BBox box, Matrix m) { + float[] xs = new float[4]; + float[] ys = new float[4]; + float[][] corners = { + {box.x0(), box.y0()}, {box.x1(), box.y0()}, + {box.x0(), box.y1()}, {box.x1(), box.y1()} + }; + for (int i = 0; i < 4; i++) { + Vector v = m.transform(new Vector(corners[i][0], corners[i][1])); + xs[i] = v.getX(); + ys[i] = v.getY(); + } + float minX = Math.min(Math.min(xs[0], xs[1]), Math.min(xs[2], xs[3])); + float maxX = Math.max(Math.max(xs[0], xs[1]), Math.max(xs[2], xs[3])); + float minY = Math.min(Math.min(ys[0], ys[1]), Math.min(ys[2], ys[3])); + float maxY = Math.max(Math.max(ys[0], ys[1]), Math.max(ys[2], ys[3])); + return new BBox(minX, minY, maxX, maxY); + } + + // --- Text pass --------------------------------------------------------- + + /** Marker recorded for each glyph so a finished line knows where it came from. */ + private record GlyphOrigin(int ordinal, boolean marked) {} + + private static final class LineCollector extends PDFTextStripper { + + private final Map> byPage = new HashMap<>(); + private final Map> opsByPage = new HashMap<>(); + private final Map preMarkedByPage = new HashMap<>(); + private final Map textSemanticsByPage = new HashMap<>(); + private final Map origins = new IdentityHashMap<>(); + private final List lineBuffer = new ArrayList<>(); + private final List lineWords = new ArrayList<>(); + private final StringBuilder lineText = new StringBuilder(); + + private int ordinal = -1; + private int markedDepth; + private int nestedDepth; + private BBox pathBox = BBox.EMPTY; + private int syntheticDepth; + private float pageHeight; + private int pageIndex; + + LineCollector() throws IOException { + super(); + } + + List linesFor(int index) { + return byPage.getOrDefault(index, List.of()); + } + + List opsFor(int index) { + return opsByPage.getOrDefault(index, List.of()); + } + + boolean preMarkedOn(int index) { + return preMarkedByPage.getOrDefault(index, false); + } + + boolean textSemanticsOn(int index) { + return textSemanticsByPage.getOrDefault(index, false); + } + + @Override + protected void startPage(PDPage page) throws IOException { + ordinal = -1; + markedDepth = 0; + nestedDepth = 0; + syntheticDepth = 0; + pathBox = BBox.EMPTY; + origins.clear(); + lineBuffer.clear(); + lineWords.clear(); + lineText.setLength(0); + // Dir-adjusted glyph coordinates live in the rotated frame, so the flip must too. + pageHeight = normalisedBox(page).height(); + pageIndex = getCurrentPageNo() - 1; + super.startPage(page); + } + + @Override + protected void endPage(PDPage page) throws IOException { + flushLine(); + super.endPage(page); + } + + /** + * Counts only operators physically present in the page's own stream: PDFBox re-enters here + * with synthetic calls for {@code '} and {@code "}, and descends into form XObjects. + */ + @Override + protected void processOperator(Operator operator, List operands) + throws IOException { + String name = operator.getName(); + if (nestedDepth == 0 && syntheticDepth == 0) { + if (isPathConstruction(name)) { + pathBox = + extendPath( + pathBox, + name, + operands, + getGraphicsState().getCurrentTransformationMatrix()); + } + if (MarkableOp.isMarkableOperator(name)) { + ordinal++; + // Classified here rather than in a second parse of the same stream: the engine + // already has the operands and the live transformation matrix. + opsByPage + .computeIfAbsent(pageIndex, k -> new ArrayList<>()) + .add( + classify( + name, + operands, + getResources(), + getGraphicsState().getCurrentTransformationMatrix(), + pathBox, + ordinal)); + if (MarkableOp.isPathPainting(name)) { + pathBox = BBox.EMPTY; + } + } else if ("n".equals(name)) { + pathBox = BBox.EMPTY; + } else if ("BDC".equals(name) || "BMC".equals(name)) { + markedDepth++; + preMarkedByPage.put(pageIndex, true); + if (carriesTextSemantics(operands)) { + textSemanticsByPage.put(pageIndex, true); + } + } else if ("EMC".equals(name) && markedDepth > 0) { + markedDepth--; + } + } + boolean synthesises = "'".equals(name) || "\"".equals(name); + if (synthesises) { + syntheticDepth++; + } + try { + super.processOperator(operator, operands); + } finally { + if (synthesises) { + syntheticDepth--; + } + } + } + + @Override + public void showForm(PDFormXObject form) throws IOException { + nestedDepth++; + try { + super.showForm(form); + } finally { + nestedDepth--; + } + } + + @Override + public void showTransparencyGroup(PDTransparencyGroup group) throws IOException { + nestedDepth++; + try { + super.showTransparencyGroup(group); + } finally { + nestedDepth--; + } + } + + @Override + protected void showType3Glyph( + Matrix textRenderingMatrix, + PDType3Font font, + int code, + org.apache.pdfbox.util.Vector displacement) + throws IOException { + nestedDepth++; + try { + super.showType3Glyph(textRenderingMatrix, font, code, displacement); + } finally { + nestedDepth--; + } + } + + @Override + protected void processChildStream( + org.apache.pdfbox.contentstream.PDContentStream contentStream, PDPage page) + throws IOException { + nestedDepth++; + try { + super.processChildStream(contentStream, page); + } finally { + nestedDepth--; + } + } + + @Override + protected void processTextPosition(TextPosition text) { + origins.put(text, new GlyphOrigin(ordinal, markedDepth > 0)); + super.processTextPosition(text); + } + + @Override + protected void writeString(String text, List positions) { + lineText.append(text); + lineBuffer.addAll(positions); + WordInfo word = buildWord(text, positions); + if (word != null) { + lineWords.add(word); + } + } + + private WordInfo buildWord(String text, List positions) { + if (text == null || text.isBlank() || positions.isEmpty()) { + return null; + } + Bounds bounds = new Bounds(); + for (TextPosition tp : positions) { + bounds.accept(tp, pageHeight, origins.get(tp)); + } + if (bounds.end < 0) { + return null; + } + return new WordInfo( + text, + bounds.box(), + bounds.start, + bounds.end, + bounds.dominantSize(), + bounds.bold); + } + + @Override + protected void writeWordSeparator() { + lineText.append(' '); + } + + @Override + protected void writeLineSeparator() { + flushLine(); + } + + @Override + protected void writeParagraphSeparator() { + flushLine(); + } + + private void flushLine() { + if (lineBuffer.isEmpty()) { + lineText.setLength(0); + lineWords.clear(); + return; + } + TextLineInfo line = buildLine(); + lineBuffer.clear(); + lineWords.clear(); + lineText.setLength(0); + if (line != null) { + byPage.computeIfAbsent(pageIndex, k -> new ArrayList<>()).add(line); + } + } + + private TextLineInfo buildLine() { + String text = lineText.toString(); + if (text.isBlank()) { + return null; + } + Bounds bounds = new Bounds(); + for (TextPosition tp : lineBuffer) { + bounds.accept(tp, pageHeight, origins.get(tp)); + } + if (bounds.end < 0) { + return null; + } + return new TextLineInfo( + pageIndex, + text, + bounds.box(), + bounds.dominantSize(), + bounds.bold, + bounds.start, + bounds.end, + bounds.marked, + List.copyOf(lineWords)); + } + } + + /** Accumulates glyph geometry, ordinals and font signals for a word or a line. */ + private static final class Bounds { + private float minX = Float.MAX_VALUE; + private float maxX = -Float.MAX_VALUE; + private float minY = Float.MAX_VALUE; + private float maxY = -Float.MAX_VALUE; + private int start = Integer.MAX_VALUE; + private int end = -1; + private boolean marked; + private boolean bold; + private final Map sizeCounts = new HashMap<>(); + + void accept(TextPosition tp, float pageHeight, GlyphOrigin origin) { + float top = pageHeight - tp.getYDirAdj(); + float bottom = top - Math.max(tp.getHeightDir(), 0); + minX = Math.min(minX, tp.getXDirAdj()); + maxX = Math.max(maxX, tp.getXDirAdj() + tp.getWidthDirAdj()); + minY = Math.min(minY, bottom); + maxY = Math.max(maxY, top); + + if (origin != null) { + start = Math.min(start, origin.ordinal()); + end = Math.max(end, origin.ordinal()); + marked |= origin.marked(); + } + float size = tp.getFontSizeInPt(); + if (size > MIN_FONT_SIZE) { + sizeCounts.merge(round(size), 1, Integer::sum); + } + bold |= isBold(tp); + } + + BBox box() { + return new BBox(minX, minY, maxX, maxY); + } + + float dominantSize() { + return sizeCounts.entrySet().stream() + .max(Map.Entry.comparingByValue()) + .map(Map.Entry::getKey) + .orElse(0f); + } + + private static float round(float value) { + return Math.round(value * 10f) / 10f; + } + + private static boolean isBold(TextPosition tp) { + if (tp.getFont() == null) { + return false; + } + String name = tp.getFont().getName(); + if (name != null && name.toLowerCase().contains("bold")) { + return true; + } + PDFontDescriptor descriptor = tp.getFont().getFontDescriptor(); + return descriptor != null + && (descriptor.getFontWeight() >= 600 || descriptor.isForceBold()); + } + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingOptions.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingOptions.java new file mode 100644 index 0000000000..bba15ed09f --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingOptions.java @@ -0,0 +1,63 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.Map; + +import lombok.Builder; +import lombok.Getter; + +/** Inputs that change how a document is tagged. */ +@Getter +@Builder(toBuilder = true) +public class TaggingOptions { + + /** What to do when the source already has a structure tree. */ + public enum ExistingTags { + /** Leave the tree alone and fix only document-level requirements. */ + KEEP, + /** Discard the tree and derive a new one. */ + REBUILD, + /** Keep a usable tree, rebuild an empty or trivially broken one. */ + AUTO + } + + /** How images with no alternative description are handled. */ + public enum FigurePolicy { + /** Leave undescribed so validation fails honestly; a faked {@code /Alt} helps nobody. */ + REQUIRE_ALT, + /** Treat every image as decoration and mark it as an artifact. */ + MARK_DECORATIVE + } + + @Builder.Default private PdfUaProfile profile = PdfUaProfile.UA1; + + /** BCP-47 language tag for the document, for example {@code en-GB}. */ + private String language; + + /** Replace a language the document already declares. Off, so a French file stays French. */ + @Builder.Default private boolean overrideLanguage = false; + + private String title; + + /** Last resort when no title is given and none can be derived; pass the uploaded filename. */ + private String fallbackTitle; + + /** Embed any font the document references but does not carry, which clause 7.21 requires. */ + @Builder.Default private boolean embedFonts = true; + + /** Leave the PDF version alone; raising it would break PDF/A-1, defined on PDF 1.4. */ + @Builder.Default private boolean preservePdfVersion = false; + + @Builder.Default private ExistingTags existingTags = ExistingTags.AUTO; + + @Builder.Default private FigurePolicy figurePolicy = FigurePolicy.REQUIRE_ALT; + + /** Alternative descriptions supplied by the caller, keyed by "pageIndex:ordinal". */ + @Builder.Default private Map altTextByFigure = Map.of(); + + /** What the document said before font embedding rewrote it; see {@link SourceFacts}. */ + @Builder.Default private SourceFacts sourceFacts = SourceFacts.NONE; + + public String altTextFor(int pageIndex, int ordinal) { + return altTextByFigure.get(pageIndex + ":" + ordinal); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingResult.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingResult.java new file mode 100644 index 0000000000..d54fad5de8 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TaggingResult.java @@ -0,0 +1,42 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.ArrayList; +import java.util.List; + +import lombok.Getter; + +/** What a tagging run produced, for the conversion report. */ +@Getter +public class TaggingResult { + + private final List warnings = new ArrayList<>(); + private final DocumentStructure structure; + private final boolean rebuilt; + private final int taggedElements; + private final int artifacts; + private final int figuresNeedingAlt; + + /** True when text was hidden as artifacts; the caller must not declare conformance. */ + private final boolean contentSuppressed; + + public TaggingResult(DocumentStructure structure, boolean rebuilt) { + this.structure = structure; + this.rebuilt = rebuilt; + this.warnings.addAll(structure.getWarnings()); + int[] elements = {0}; + structure.visit( + block -> { + if (!block.isArtifact()) { + elements[0]++; + } + }); + this.taggedElements = elements[0]; + this.artifacts = structure.artifactCount(); + this.figuresNeedingAlt = structure.figuresWithoutAlt().size(); + this.contentSuppressed = structure.isTextSuppressed(); + } + + public boolean needsHumanReview() { + return figuresNeedingAlt > 0 || !warnings.isEmpty(); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TextLineInfo.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TextLineInfo.java new file mode 100644 index 0000000000..67f37d1c8d --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/TextLineInfo.java @@ -0,0 +1,42 @@ +package stirling.software.proprietary.pdf.ua; + +import java.util.List; + +/** + * A run of text on one baseline, with the operator ordinals that produced it. {@code preMarked} + * means the source stream already wrapped this text in BDC/EMC. + */ +public record TextLineInfo( + int pageIndex, + String text, + BBox bbox, + float dominantFontSize, + boolean bold, + int startOrdinal, + int endOrdinal, + boolean preMarked, + List words) { + + public boolean isBlank() { + return text == null || text.isBlank(); + } + + public int charCount() { + return text == null ? 0 : text.strip().length(); + } + + public int wordCount() { + return (int) words.stream().filter(w -> !w.isBlank()).count(); + } + + /** True when every word occupies its own operator run, so cells can be tagged separately. */ + public boolean wordsAreSeparable() { + List real = words.stream().filter(w -> !w.isBlank()).toList(); + for (int i = 1; i < real.size(); i++) { + if (!real.get(i - 1).isSeparableFrom(real.get(i))) { + return false; + } + } + return true; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/WordInfo.java b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/WordInfo.java new file mode 100644 index 0000000000..515775b6f8 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/pdf/ua/WordInfo.java @@ -0,0 +1,18 @@ +package stirling.software.proprietary.pdf.ua; + +/** + * A whitespace-delimited run of glyphs, with the operator ordinals that produced it. Cell detection + * needs both: geometry to find cells, ordinals to tell whether they can be tagged separately. + */ +public record WordInfo( + String text, BBox bbox, int startOrdinal, int endOrdinal, float fontSize, boolean bold) { + + public boolean isBlank() { + return text == null || text.isBlank(); + } + + /** True when this word shares no operator with the other, so both can carry their own MCID. */ + public boolean isSeparableFrom(WordInfo other) { + return endOrdinal < other.startOrdinal || other.endOrdinal < startOrdinal; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/AccessibilityAuditService.java b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/AccessibilityAuditService.java new file mode 100644 index 0000000000..869d489a04 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/AccessibilityAuditService.java @@ -0,0 +1,187 @@ +package stirling.software.proprietary.service.ua; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDDocumentCatalog; +import org.apache.pdfbox.pdmodel.interactive.viewerpreferences.PDViewerPreferences; +import org.springframework.stereotype.Service; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import stirling.software.common.util.ExceptionUtils; +import stirling.software.proprietary.model.api.ua.AccessibilityIssue; +import stirling.software.proprietary.model.api.ua.AccessibilityReport; +import stirling.software.proprietary.model.api.ua.FigureDescriptor; +import stirling.software.proprietary.model.api.ua.UaValidationResult; +import stirling.software.proprietary.pdf.ua.BBox; +import stirling.software.proprietary.pdf.ua.DocumentStructure; +import stirling.software.proprietary.pdf.ua.LayoutAnalyzer; +import stirling.software.proprietary.pdf.ua.PageContent; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.StructBlock; +import stirling.software.proprietary.pdf.ua.StructType; +import stirling.software.proprietary.pdf.ua.TaggedContentExtractor; + +/** Produces an accessibility report without changing the document. */ +@Service +@Slf4j +@RequiredArgsConstructor +public class AccessibilityAuditService { + + /** Checks no validator can make; omitting them implies the work does not exist. */ + private static final List HUMAN_CHECKS = + List.of( + "Is the reading order correct for someone who cannot see the layout?", + "Does each alternative description convey what the image is for, not just what" + + " it looks like?", + "Are headings used for structure rather than for visual emphasis?", + "Is any information conveyed by colour alone also available another way?", + "Do tables have headers that identify the right rows and columns?", + "Is the document language correct, including for quoted passages?", + "Do links describe their destination rather than saying 'click here'?"); + + /** The report walks every page and validates, so it carries the conversion's own caps. */ + private static final long MAX_INPUT_BYTES = 100L * 1024 * 1024; + + private static final int MAX_PAGES = 2000; + + private final PdfUaValidationService validationService; + + public AccessibilityReport audit(byte[] pdfBytes, PdfUaProfile profile) throws IOException { + enforceLimits(pdfBytes); + AccessibilityReport report = new AccessibilityReport(); + report.setProfile(profile.displayName()); + + UaValidationResult validation = validationService.validate(pdfBytes, profile); + report.setIssues(validation.issues()); + report.setPassesAutomatedChecks(validation.compliant()); + report.setHumanChecks(HUMAN_CHECKS); + + int fixable = 0; + int needsInput = 0; + for (AccessibilityIssue issue : validation.issues()) { + if (issue.isAutoFixable()) { + fixable++; + } else { + needsInput++; + } + } + report.setAutomaticallyFixable(fixable); + report.setNeedsInput(needsInput); + + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + populateSummary(document, report); + report.setFiguresNeedingDescription(figuresNeedingDescription(document)); + } catch (IOException e) { + log.debug("Could not inspect document for the summary: {}", e.getMessage()); + } + return report; + } + + /** + * Rejects before the expensive pass. An unreadable file is left to the report itself to say. + */ + private static void enforceLimits(byte[] pdfBytes) { + if (pdfBytes.length > MAX_INPUT_BYTES) { + throw ExceptionUtils.createIllegalArgumentException( + "error.fileTooLarge", + "This PDF is {0} MB. The accessibility report is limited to {1} MB.", + pdfBytes.length / (1024 * 1024), + MAX_INPUT_BYTES / (1024 * 1024)); + } + int pages; + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + pages = document.getNumberOfPages(); + } catch (IOException e) { + return; + } + if (pages > MAX_PAGES) { + throw ExceptionUtils.createIllegalArgumentException( + "error.tooManyPages", + "This PDF has {0} pages. The accessibility report is limited to {1} pages;" + + " split it first.", + pages, + MAX_PAGES); + } + } + + private void populateSummary(PDDocument document, AccessibilityReport report) + throws IOException { + PDDocumentCatalog catalog = document.getDocumentCatalog(); + AccessibilityReport.Summary summary = report.getSummary(); + + report.setTagged(catalog.getStructureTreeRoot() != null); + report.setDeclaresConformance(declaresUa(document)); + + summary.setPages(document.getNumberOfPages()); + summary.setEncrypted(document.isEncrypted()); + summary.setHasLanguage(catalog.getLanguage() != null && !catalog.getLanguage().isBlank()); + + String title = document.getDocumentInformation().getTitle(); + summary.setHasTitle(title != null && !title.isBlank()); + + PDViewerPreferences preferences = catalog.getViewerPreferences(); + summary.setDisplaysDocTitle(preferences != null && preferences.displayDocTitle()); + + Set unembedded = FontEmbeddingService.findUnembeddedFonts(document); + summary.setUnembeddedFonts(unembedded.size()); + summary.setAllFontsEmbedded(unembedded.isEmpty()); + + try { + summary.setFigures(new TaggedContentExtractor().countGraphics(document)); + } catch (Exception e) { + log.debug("Could not count figures: {}", e.getMessage()); + } + } + + /** + * Lists the figures a conversion would leave undescribed, running the converter's own analysis + * because counting raster images would miss vector charts and existing descriptions. + */ + private List figuresNeedingDescription(PDDocument document) { + try { + List pages = new TaggedContentExtractor().extract(document); + DocumentStructure structure = new LayoutAnalyzer().analyse(pages); + List figures = new ArrayList<>(); + for (StructBlock block : structure.figuresWithoutAlt()) { + int ordinal = block.getRanges().isEmpty() ? -1 : block.getRanges().get(0).start(); + BBox box = block.getBbox(); + figures.add( + new FigureDescriptor( + block.getPageIndex() + ":" + ordinal, + block.getPageIndex() + 1, + block.getType() == StructType.FORMULA ? "formula" : "figure", + box.x0(), + box.y0(), + box.width(), + box.height(), + block.getAlt())); + } + return figures; + } catch (Exception e) { + log.debug("Could not enumerate figures: {}", e.getMessage()); + return List.of(); + } + } + + /** True when the XMP packet carries a pdfuaid identifier. */ + private static boolean declaresUa(PDDocument document) { + try { + var metadata = document.getDocumentCatalog().getMetadata(); + if (metadata == null) { + return false; + } + String xmp = + new String(metadata.toByteArray(), java.nio.charset.StandardCharsets.UTF_8); + return xmp.contains("pdfuaid"); + } catch (IOException e) { + return false; + } + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/FontEmbeddingService.java b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/FontEmbeddingService.java new file mode 100644 index 0000000000..9c38b54f6b --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/FontEmbeddingService.java @@ -0,0 +1,254 @@ +package stirling.software.proprietary.service.ua; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.stream.Stream; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDResources; +import org.apache.pdfbox.pdmodel.font.PDFont; +import org.springframework.stereotype.Service; + +import lombok.extern.slf4j.Slf4j; + +import stirling.software.common.util.ProcessExecutor; +import stirling.software.common.util.ProcessExecutor.ProcessExecutorResult; + +/** + * Embeds any font the document references but does not carry, as PDF/UA-1 clause 7.21 requires. + * Ghostscript does the embedding and discards the structure tree, so this must run before tagging. + */ +@Service +@Slf4j +public class FontEmbeddingService { + + public boolean hasUnembeddedFonts(byte[] pdfBytes) { + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + return !findUnembeddedFonts(document).isEmpty(); + } catch (IOException e) { + log.debug("Could not inspect fonts: {}", e.getMessage()); + return false; + } + } + + public static Set findUnembeddedFonts(PDDocument document) { + Set missing = new HashSet<>(); + for (PDPage page : document.getPages()) { + PDResources resources = page.getResources(); + if (resources == null) { + continue; + } + for (COSName name : resources.getFontNames()) { + try { + PDFont font = resources.getFont(name); + if (font != null && !font.isEmbedded()) { + missing.add(font.getName()); + } + } catch (IOException e) { + log.debug("Could not read font {}: {}", name.getName(), e.getMessage()); + } + } + } + return missing; + } + + /** + * Returns the document with all fonts embedded, or the input unchanged. Never throws: failing + * to embed is a reportable shortfall, not a reason to abandon the conversion. + */ + public Result embedFonts(byte[] pdfBytes) { + Set missing; + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + missing = findUnembeddedFonts(document); + } catch (IOException e) { + return new Result( + pdfBytes, false, Set.of(), "Could not inspect fonts: " + e.getMessage()); + } + if (missing.isEmpty()) { + return new Result(pdfBytes, false, Set.of(), null); + } + if (!isGhostscriptAvailable()) { + return new Result( + pdfBytes, + false, + missing, + "Ghostscript is not installed, so " + + missing.size() + + " unembedded font(s) could not be embedded. PDF/UA requires every font" + + " to be embedded."); + } + + Path workingDir = null; + try { + workingDir = Files.createTempDirectory("pdfua_fonts_"); + Path input = workingDir.resolve("input.pdf"); + Path output = workingDir.resolve("output.pdf"); + Files.write(input, pdfBytes); + + ProcessExecutorResult result = + ProcessExecutor.getInstance(ProcessExecutor.Processes.GHOSTSCRIPT) + .runCommandWithOutputHandling(command(input, output, workingDir)); + + if (result.getRc() != 0 || !Files.exists(output)) { + return new Result( + pdfBytes, + false, + missing, + "Font embedding failed with code " + result.getRc()); + } + byte[] embedded = Files.readAllBytes(output); + + // Ghostscript can exit 0 having written a blank page, so keep the original rather than + // return an empty document. + if (!survived(pdfBytes, embedded)) { + log.warn("Ghostscript produced a degenerate document; keeping the original"); + return new Result( + pdfBytes, + false, + missing, + "Font embedding was skipped because the embedder returned a document that" + + " had lost content. " + + missing.size() + + " font(s) remain unembedded."); + } + + // It can also exit 0 while simply leaving fonts unembedded. + Set remaining; + try (PDDocument check = Loader.loadPDF(embedded)) { + remaining = findUnembeddedFonts(check); + } + if (!remaining.isEmpty()) { + return new Result( + embedded, + true, + remaining, + remaining.size() + + " font(s) could not be embedded (" + + String.join(", ", remaining) + + "). PDF/UA requires every font to be embedded."); + } + log.info("Embedded {} previously unembedded font(s)", missing.size()); + return new Result(embedded, true, missing, null); + + } catch (Exception e) { + log.warn("Font embedding failed: {}", e.getMessage()); + return new Result(pdfBytes, false, missing, "Font embedding failed: " + e.getMessage()); + } finally { + deleteQuietly(workingDir); + } + } + + /** + * True when the rewritten document still holds the original's content. A collapse in page count + * or content-stream size is the only signature of a failed rewrite the exit code hides. + */ + private static boolean survived(byte[] original, byte[] rewritten) { + try (PDDocument before = Loader.loadPDF(original); + PDDocument after = Loader.loadPDF(rewritten)) { + if (after.getNumberOfPages() != before.getNumberOfPages()) { + return false; + } + long beforeBytes = contentBytes(before); + long afterBytes = contentBytes(after); + if (beforeBytes == 0) { + return true; + } + return afterBytes * 20L >= beforeBytes; + } catch (IOException e) { + log.debug("Could not compare documents after embedding: {}", e.getMessage()); + return false; + } + } + + private static long contentBytes(PDDocument document) { + long total = 0; + for (PDPage page : document.getPages()) { + try (InputStream in = page.getContents()) { + if (in != null) { + byte[] buffer = new byte[8192]; + int read; + while ((read = in.read(buffer)) > 0) { + total += read; + } + } + } catch (IOException e) { + log.debug("Could not measure page content: {}", e.getMessage()); + } + } + return total; + } + + private static List command(Path input, Path output, Path workingDir) { + List command = new ArrayList<>(); + command.add("gs"); + command.add("--permit-file-read=" + workingDir.toAbsolutePath()); + command.add("--permit-file-write=" + workingDir.toAbsolutePath()); + command.add("-sDEVICE=pdfwrite"); + command.add("-dEmbedAllFonts=true"); + command.add("-dSubsetFonts=true"); + command.add("-dCompressFonts=true"); + command.add("-dNOSUBSTFONTS=false"); + command.add("-dPDFSETTINGS=/prepress"); + command.add("-dNOPAUSE"); + command.add("-dBATCH"); + command.add("-sOutputFile=" + output.toAbsolutePath()); + command.add(input.toAbsolutePath().toString()); + return command; + } + + /** Cached after the first probe: availability does not change mid-process. */ + private volatile Boolean ghostscriptAvailable; + + private boolean isGhostscriptAvailable() { + Boolean cached = ghostscriptAvailable; + if (cached != null) { + return cached; + } + boolean available; + try { + ProcessExecutorResult result = + ProcessExecutor.getInstance(ProcessExecutor.Processes.GHOSTSCRIPT) + .runCommandWithOutputHandling(List.of("gs", "--version")); + available = result.getRc() == 0; + } catch (Exception e) { + log.debug("Ghostscript availability check failed: {}", e.getMessage()); + available = false; + } + ghostscriptAvailable = available; + return available; + } + + private static void deleteQuietly(Path directory) { + if (directory == null) { + return; + } + try (Stream stream = Files.walk(directory)) { + stream.sorted(Comparator.reverseOrder()) + .forEach( + path -> { + try { + Files.deleteIfExists(path); + } catch (IOException e) { + log.debug("Could not delete {}", path); + } + }); + } catch (IOException e) { + log.debug("Could not clean {}", directory); + } + } + + /** + * @param warning non-null when fonts remain unembedded, for the conversion report + */ + public record Result(byte[] pdfBytes, boolean changed, Set fonts, String warning) {} +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaConversionService.java b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaConversionService.java new file mode 100644 index 0000000000..1d0920c109 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaConversionService.java @@ -0,0 +1,251 @@ +package stirling.software.proprietary.service.ua; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.encryption.InvalidPasswordException; +import org.apache.pdfbox.pdmodel.interactive.form.PDAcroForm; +import org.springframework.stereotype.Service; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.model.api.ua.UaValidationResult; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.PdfUaTagger; +import stirling.software.proprietary.pdf.ua.SourceFacts; +import stirling.software.proprietary.pdf.ua.TaggingOptions; +import stirling.software.proprietary.pdf.ua.TaggingResult; + +/** + * Converts a PDF to PDF/UA. The declaration is written first and withdrawn unless validation + * passes, so a returned file either conforms or does not claim to. + */ +@Service +@Slf4j +@RequiredArgsConstructor +public class PdfUaConversionService { + + private final PdfUaValidationService validationService; + private final FontEmbeddingService fontEmbeddingService; + private final stirling.software.common.service.CustomPDFDocumentFactory pdfDocumentFactory; + + /** Matches the cap GetInfoOnPDF already applies to comparable whole-document work. */ + private static final long MAX_INPUT_BYTES = 100L * 1024 * 1024; + + /** Beyond this the structure model alone runs to hundreds of megabytes. */ + private static final int MAX_PAGES = 2000; + + public PdfUaConversionOutcome convert(byte[] input, TaggingOptions options) throws IOException { + if (input.length > MAX_INPUT_BYTES) { + throw new IOException( + "This PDF is " + + (input.length / (1024 * 1024)) + + " MB. PDF/UA conversion is limited to " + + (MAX_INPUT_BYTES / (1024 * 1024)) + + " MB."); + } + PdfUaProfile profile = options.getProfile(); + List warnings = new ArrayList<>(); + + // Read the document's own facts before anything rewrites it. Font embedding runs + // Ghostscript over the whole file, which discards the structure tree, /Lang and XFA, so + // every guard and every "what did the source say" question must be answered from here. + SourceFacts facts; + try (PDDocument original = load(input)) { + rejectUnsupportedSource(original); + warnSignatures(original, warnings); + facts = SourceFacts.of(original); + } + + byte[] source = input; + if (options.isEmbedFonts()) { + // Must precede tagging: the embedder rewrites the file and drops any structure tree. + FontEmbeddingService.Result fonts = fontEmbeddingService.embedFonts(input); + source = fonts.pdfBytes(); + if (fonts.warning() != null) { + warnings.add(fonts.warning()); + } + source = keepTagsOverFonts(input, source, facts, options, warnings); + } + + TaggingOptions effective = options.toBuilder().sourceFacts(facts).build(); + + // Tag and declare in one pass; the claim is withdrawn below if validation disagrees. + byte[] declared; + TaggingResult taggingResult; + PdfUaTagger tagger = new PdfUaTagger(); + try (PDDocument document = load(source)) { + rejectEncrypted(document); + taggingResult = tagger.tag(document, effective); + warnings.addAll(taggingResult.getWarnings()); + tagger.declareConformance(document, profile); + declared = save(document); + } + + UaValidationResult validation = validationService.validate(declared, profile); + + // A validator cannot see text hidden behind artifact markers, so a clean verdict over + // suppressed content would be a false claim. + boolean honest = !taggingResult.isContentSuppressed(); + + if (validation.compliant() && honest) { + log.info("{} conversion passed validation", profile.displayName()); + return new PdfUaConversionOutcome( + declared, true, validation, summary(taggingResult), warnings); + } + + byte[] undeclared; + try (PDDocument document = load(declared)) { + tagger.withdrawConformance(document); + undeclared = save(document); + } + + if (!validation.compliant()) { + warnings.add( + "The document could not be made " + + profile.displayName() + + " conformant, so no conformance claim was written. " + + validation.totalFailures() + + " automated check(s) still fail."); + } + log.info( + "{} conversion left undeclared: {} failures, suppressedText={}", + profile.displayName(), + validation.totalFailures(), + !honest); + return new PdfUaConversionOutcome( + undeclared, false, validation, summary(taggingResult), warnings); + } + + private static PdfUaConversionOutcome.TaggingSummary summary(TaggingResult result) { + return new PdfUaConversionOutcome.TaggingSummary( + result.isRebuilt(), + result.getTaggedElements(), + result.getArtifacts(), + result.getFiguresNeedingAlt()); + } + + /** + * Tagging rewrites the content streams a signature covers, so the conversion still runs but the + * caller has to know the signature will no longer verify. + */ + private static void warnSignatures(PDDocument document, List warnings) { + int signatures = document.getSignatureDictionaries().size(); + if (signatures > 0) { + warnings.add( + signatures + + " digital signature(s) will stop verifying: tagging rewrites the" + + " content streams they cover. Convert first, then re-sign."); + } + } + + /** Replaces PDFBox's "incorrect password" wording, baffling when the caller supplied none. */ + private PDDocument load(byte[] bytes) throws IOException { + try { + // The factory spills large documents to a temp-file cache instead of the heap. + return pdfDocumentFactory.load(bytes); + } catch (IOException | RuntimeException e) { + // The factory wraps the parse failure, so check the cause chain rather than the type. + if (mentionsPassword(e)) { + throw new IOException( + "This PDF is encrypted. Remove the password before converting it to" + + " PDF/UA.", + e); + } + throw e; + } + } + + private static boolean mentionsPassword(Throwable error) { + for (Throwable cause = error; cause != null; cause = cause.getCause()) { + if (cause instanceof InvalidPasswordException) { + return true; + } + String message = cause.getMessage(); + if (message != null) { + String lower = message.toLowerCase(Locale.ROOT); + if (lower.contains("password") || lower.contains("decrypt")) { + return true; + } + } + } + return false; + } + + /** XFA is forbidden by PDF/UA-1 clause 7.15; encrypted or huge files cannot be restructured. */ + /** + * Under KEEP nothing rebuilds a tree, so if the embedder deleted one we would hand back an + * untagged document. Fonts are not worth the whole structure; give the tags back instead. + */ + private byte[] keepTagsOverFonts( + byte[] input, + byte[] embedded, + SourceFacts facts, + TaggingOptions options, + List warnings) + throws IOException { + if (options.getExistingTags() != TaggingOptions.ExistingTags.KEEP + || !facts.hasUsableTree() + || embedded == input) { + return embedded; + } + boolean survived; + try (PDDocument rewritten = load(embedded)) { + survived = PdfUaTagger.hasUsableStructureTree(rewritten); + } + if (survived) { + return embedded; + } + warnings.add( + "Embedding the missing fonts would have deleted the document's existing tags, so" + + " the tags were kept and the fonts left unembedded. Turn off font" + + " embedding to silence this, or rebuild the tags to embed them."); + return input; + } + + /** + * Checks that must see the document as the author wrote it. Font embedding strips XFA, so + * running this afterwards would let a dynamic form through unnoticed, and it would push a + * document we are about to reject through the whole embedder first. + */ + private static void rejectUnsupportedSource(PDDocument document) throws IOException { + if (document.getNumberOfPages() > MAX_PAGES) { + throw new IOException( + "This PDF has " + + document.getNumberOfPages() + + " pages. PDF/UA conversion is limited to " + + MAX_PAGES + + " pages; split it first."); + } + PDAcroForm form = document.getDocumentCatalog().getAcroForm(); + if (form != null && form.xfaIsDynamic()) { + throw new IOException( + "Dynamic XFA forms are not permitted by PDF/UA. Flatten the form first."); + } + } + + /** + * Deliberately checked on the working document rather than the source. Permissions-only + * encryption with an empty user password is common in published documents, the embedder + * resolves it, and those files convert usefully; rejecting them up front would fail a document + * for a password its author never set. + */ + private static void rejectEncrypted(PDDocument document) throws IOException { + if (document.isEncrypted()) { + throw new IOException( + "Encrypted PDFs cannot be converted to PDF/UA. Remove the password first."); + } + } + + private static byte[] save(PDDocument document) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaValidationService.java b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaValidationService.java new file mode 100644 index 0000000000..18d9fcaee7 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfUaValidationService.java @@ -0,0 +1,245 @@ +package stirling.software.proprietary.service.ua; + +import java.io.ByteArrayInputStream; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import org.springframework.stereotype.Service; +import org.verapdf.gf.foundry.VeraGreenfieldFoundryProvider; +import org.verapdf.pdfa.Foundries; +import org.verapdf.pdfa.PDFAParser; +import org.verapdf.pdfa.PDFAValidator; +import org.verapdf.pdfa.flavours.PDFAFlavour; +import org.verapdf.pdfa.results.TestAssertion; +import org.verapdf.pdfa.results.ValidationResult; + +import jakarta.annotation.PostConstruct; + +import lombok.extern.slf4j.Slf4j; + +import stirling.software.proprietary.model.api.ua.AccessibilityIssue; +import stirling.software.proprietary.model.api.ua.UaValidationResult; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; + +/** + * Validates a document against a PDF/UA profile using veraPDF, the oracle a conversion is declared + * against. It checks only the machine-verifiable subset, so a clean result is not "accessible". + */ +@Service +@Slf4j +public class PdfUaValidationService { + + /** Plain-English text and remediability for the clauses users actually hit. */ + private static final Map CLAUSES = buildClauseTable(); + + record ClauseInfo(String message, boolean autoFixable) {} + + @PostConstruct + public void initialise() { + try { + VeraGreenfieldFoundryProvider.initialise(); + } catch (Exception e) { + log.error("Failed to initialise veraPDF for PDF/UA validation", e); + } + } + + public UaValidationResult validate(byte[] pdfBytes, PdfUaProfile profile) { + PDFAFlavour flavour = flavourFor(profile); + try (PDFAParser parser = + Foundries.defaultInstance() + .createParser(new ByteArrayInputStream(pdfBytes), flavour)) { + + PDFAValidator validator = Foundries.defaultInstance().createValidator(flavour, false); + ValidationResult result = validator.validate(parser); + return toResult(profile, result); + + } catch (Exception e) { + log.warn("PDF/UA validation failed for {}: {}", profile.displayName(), e.getMessage()); + AccessibilityIssue issue = new AccessibilityIssue(); + issue.setMessage("Validation could not run: " + e.getMessage()); + issue.setSeverity("error"); + issue.setClause("n/a"); + return new UaValidationResult(profile.displayName(), false, List.of(issue), 0); + } + } + + public static PDFAFlavour flavourFor(PdfUaProfile profile) { + return profile == PdfUaProfile.UA2 ? PDFAFlavour.PDFUA_2 : PDFAFlavour.PDFUA_1; + } + + /** + * Whether the bytes really validate as PDF/A level A for the given part. Tagging is necessary + * for level A but not sufficient, so the claim is only written once veraPDF agrees. + */ + public boolean validatesAsPdfaLevelA(byte[] pdfBytes, int part) { + PDFAFlavour flavour = + switch (part) { + case 1 -> PDFAFlavour.PDFA_1_A; + case 2 -> PDFAFlavour.PDFA_2_A; + case 3 -> PDFAFlavour.PDFA_3_A; + default -> null; + }; + if (flavour == null) { + log.warn("No PDF/A level A flavour for part {}", part); + return false; + } + try (PDFAParser parser = + Foundries.defaultInstance() + .createParser(new ByteArrayInputStream(pdfBytes), flavour)) { + PDFAValidator validator = Foundries.defaultInstance().createValidator(flavour, false); + return validator.validate(parser).isCompliant(); + } catch (Exception e) { + log.warn("Level A validation could not run: {}", e.getMessage()); + return false; + } + } + + /** + * Groups repeated failures of the same rule so a report lists issues, not thousands of lines. + */ + private static UaValidationResult toResult(PdfUaProfile profile, ValidationResult result) { + Map grouped = new LinkedHashMap<>(); + int total = 0; + + for (TestAssertion assertion : result.getTestAssertions()) { + if (assertion.getStatus() != TestAssertion.Status.FAILED) { + continue; + } + total++; + String clause = + assertion.getRuleId() != null ? assertion.getRuleId().getClause() : "unknown"; + int test = assertion.getRuleId() != null ? assertion.getRuleId().getTestNumber() : 0; + String key = clause + "-" + test; + + AccessibilityIssue issue = + grouped.computeIfAbsent( + key, + k -> { + AccessibilityIssue created = new AccessibilityIssue(); + created.setClause(clause); + created.setTestNumber(String.valueOf(test)); + created.setSeverity("error"); + ClauseInfo info = lookupClause(clause); + created.setMessage( + info != null ? info.message() : assertion.getMessage()); + created.setTechnicalMessage(assertion.getMessage()); + created.setAutoFixable(info != null && info.autoFixable()); + created.setSpecification(profile.displayName()); + return created; + }); + issue.setOccurrences(issue.getOccurrences() + 1); + if (issue.getLocation() == null && assertion.getLocation() != null) { + issue.setLocation(assertion.getLocation().toString()); + } + } + + List issues = new ArrayList<>(grouped.values()); + return new UaValidationResult( + profile.displayName(), result.isCompliant() && total == 0, issues, total); + } + + /** + * Finds the most specific entry covering a clause by walking up the dotted hierarchy. String + * prefixes would be wrong: {@code 7.1} prefixes {@code 7.18.1} without being its ancestor. + */ + static ClauseInfo lookupClause(String clause) { + if (clause == null) { + return null; + } + String current = clause; + while (!current.isEmpty()) { + ClauseInfo info = CLAUSES.get(current); + if (info != null) { + return info; + } + int dot = current.lastIndexOf('.'); + if (dot < 0) { + return null; + } + current = current.substring(0, dot); + } + return null; + } + + private static Map buildClauseTable() { + Map table = new LinkedHashMap<>(); + table.put( + "7.1", + new ClauseInfo( + "Document is not tagged, or some content is neither tagged nor marked as an artifact.", + true)); + table.put( + "7.2", + new ClauseInfo( + "Text cannot be mapped to Unicode, or the document language is not declared.", + true)); + table.put( + "7.3", + new ClauseInfo("An image or graphic has no alternative description.", false)); + table.put( + "7.4", + new ClauseInfo( + "Heading levels skip a level, or headings are nested incorrectly.", true)); + table.put( + "7.5", + new ClauseInfo("A table is missing header cells or header associations.", false)); + table.put( + "7.6", new ClauseInfo("A list is not structured as list items with bodies.", true)); + table.put( + "7.7", + new ClauseInfo("A mathematical expression has no alternative description.", false)); + table.put( + "7.8", + new ClauseInfo("Running heads or page numbers are not marked as artifacts.", true)); + table.put("7.9", new ClauseInfo("A note is missing a unique identifier.", true)); + // Tagging does not touch optional content groups, so this needs the authoring tool. + table.put("7.10", new ClauseInfo("An optional content group has no name.", false)); + // The attachment's own /AFRelationship and /Desc are not something tagging can supply. + table.put( + "7.11", + new ClauseInfo( + "An embedded file is missing its relationship or description.", false)); + table.put( + "7.15", + new ClauseInfo( + "The document uses a dynamic XFA form, which PDF/UA does not allow.", + false)); + table.put( + "7.16", + new ClauseInfo( + "Security settings prevent assistive technology from reading the content.", + true)); + table.put("7.17", new ClauseInfo("Navigation aids such as page labels are missing.", true)); + table.put( + "7.18", + new ClauseInfo( + "An annotation is missing a description, tab order, or structure entry.", + true)); + table.put( + "7.20", + new ClauseInfo( + "A form or group XObject is not marked as content or as an artifact.", + false)); + // Most font defects (CIDFont, CMap, metrics, encoding) need the font itself repaired. + table.put( + "7.21", + new ClauseInfo("A font in the document does not meet PDF/UA rules.", false)); + // The one font defect embedding does fix. + table.put("7.21.4.1", new ClauseInfo("A font used in the document is not embedded.", true)); + // ToUnicode gaps need the font itself repaired, which embedding does not do. + table.put( + "7.21.7", + new ClauseInfo( + "A font does not map every character it uses to Unicode, so extracted text" + + " may be wrong.", + false)); + table.put( + "5", + new ClauseInfo( + "The document does not declare PDF/UA conformance in its XMP metadata.", + true)); + return table; + } +} diff --git a/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfaAccessibilityService.java b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfaAccessibilityService.java new file mode 100644 index 0000000000..5a13b4ab65 --- /dev/null +++ b/app/proprietary/src/main/java/stirling/software/proprietary/service/ua/PdfaAccessibilityService.java @@ -0,0 +1,297 @@ +package stirling.software.proprietary.service.ua; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.List; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdfwriter.compress.CompressParameters; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.common.PDMetadata; +import org.apache.xmpbox.XMPMetadata; +import org.apache.xmpbox.schema.PDFAExtensionSchema; +import org.apache.xmpbox.schema.PDFAIdentificationSchema; +import org.apache.xmpbox.type.AbstractStructuredType; +import org.apache.xmpbox.type.ArrayProperty; +import org.apache.xmpbox.type.Cardinality; +import org.apache.xmpbox.type.PDFAPropertyType; +import org.apache.xmpbox.type.PDFASchemaType; +import org.apache.xmpbox.xml.DomXmpParser; +import org.apache.xmpbox.xml.XmpSerializer; +import org.springframework.stereotype.Service; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import stirling.software.common.service.PdfaLevelAServiceInterface; +import stirling.software.proprietary.pdf.ua.PdfUaIdentificationSchema; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.PdfUaTagger; +import stirling.software.proprietary.pdf.ua.TaggingOptions; +import stirling.software.proprietary.pdf.ua.TaggingResult; + +/** + * Raises a PDF/A file from conformance level B to level A, which adds the tagging the PDF/UA tagger + * already does. Must run after Ghostscript, which discards any structure tree it is given. + */ +@Service +@Slf4j +@RequiredArgsConstructor +public class PdfaAccessibilityService implements PdfaLevelAServiceInterface { + + /** + * Matches the PDF/UA converter's own cap; beyond this the structure model exhausts the heap. + */ + private static final int MAX_TAGGABLE_PAGES = 2000; + + private final PdfUaValidationService validationService; + + /** + * Tags a converted PDF/A and marks it conformance A, or returns it unchanged rather than + * claiming level A over untagged content. part is 1 to 3; part 1 keeps its PDF 1.4 version. + */ + public Result upgradeToLevelA(byte[] pdfBytes, int part, String language, String title) { + return upgradeToLevelA(pdfBytes, part, language, title, false); + } + + /** + * @param alsoDeclareUa additionally claim PDF/UA, but only if it validates + */ + @Override + public Result upgradeToLevelA( + byte[] pdfBytes, int part, String language, String title, boolean alsoDeclareUa) { + List warnings = new ArrayList<>(); + try { + byte[] tagged; + TaggingResult taggingResult; + + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + // Tagging holds a model of the whole document; without a cap a large file exhausts + // the heap, and OutOfMemoryError is an Error, so the catch below never sees it. + if (document.getNumberOfPages() > MAX_TAGGABLE_PAGES) { + warnings.add( + "This document has " + + document.getNumberOfPages() + + " pages, more than the " + + MAX_TAGGABLE_PAGES + + " that can be tagged, so it was left at conformance level B."); + return new Result(pdfBytes, false, warnings); + } + TaggingOptions options = + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language(language) + .title(title) + .fallbackTitle(title) + // Fonts were embedded on the PDF/A pass; a rewrite would undo it. + .embedFonts(false) + // PDF/A-1 is defined on PDF 1.4; raising it breaks conformance. + .preservePdfVersion(part == 1) + .existingTags(TaggingOptions.ExistingTags.AUTO) + .build(); + + taggingResult = new PdfUaTagger().tag(document, options); + warnings.addAll(taggingResult.getWarnings()); + tagged = save(document, part); + } + + if (taggingResult.getTaggedElements() == 0 && taggingResult.isRebuilt()) { + warnings.add( + "No taggable content was found, so the file cannot claim PDF/A level A." + + " It remains valid at level B."); + return new Result(pdfBytes, false, warnings); + } + if (taggingResult.isContentSuppressed()) { + warnings.add( + "Some text could not be tagged reliably and was marked as an artifact, so" + + " no level A claim was written. The file remains valid at level B."); + return new Result(tagged, false, warnings); + } + + byte[] declared = setConformance(tagged, part, "A"); + + // Tagging is necessary for level A but not sufficient: Unicode mappings are too. + if (!validationService.validatesAsPdfaLevelA(declared, part)) { + warnings.add( + "The document was tagged but does not validate as PDF/A-" + + part + + "a, so it was left at conformance level B."); + return new Result(setConformance(tagged, part, "B"), false, warnings); + } + + if (alsoDeclareUa) { + byte[] withUa = declarePdfUaAlongsidePdfa(declared, part); + var uaResult = validationService.validate(withUa, PdfUaProfile.UA1); + if (uaResult.compliant()) { + log.info("Upgraded PDF/A-{} to level A and declared PDF/UA", part); + return new Result(withUa, true, warnings); + } + // The archival upgrade stands on its own; only the accessibility claim is dropped. + warnings.add( + "PDF/UA was requested alongside PDF/A but " + + uaResult.totalFailures() + + " accessibility check(s) still fail, so no PDF/UA claim was" + + " written. The file is valid PDF/A-" + + part + + "a."); + } + + log.info("Upgraded PDF/A-{} to conformance level A", part); + return new Result(declared, true, warnings); + + } catch (Exception e) { + log.warn("Could not upgrade to PDF/A level A: {}", e.getMessage()); + warnings.add( + "Level A upgrade failed (" + + e.getMessage() + + "), so the file was left at conformance level B."); + return new Result(pdfBytes, false, warnings); + } + } + + /** + * Declares PDF/UA alongside PDF/A in one file. The extension schema is required: PDF/A forbids + * XMP properties no schema describes, and XMPBox has none for {@code pdfuaid}. + */ + static byte[] declarePdfUaAlongsidePdfa(byte[] pdfBytes, int part) throws Exception { + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + XMPMetadata xmp = parseOrCreate(document); + + PdfUaIdentificationSchema identification = new PdfUaIdentificationSchema(xmp); + identification.setPart(1); + xmp.addSchema(identification); + + addPdfUaExtensionSchema(xmp); + writeMetadata(document, xmp); + return save(document, part); + } + } + + /** + * Describes the pdfuaid namespace so a PDF/A validator accepts it. Fields are set individually, + * not by subclassing: XMPBox reads the namespace from an annotation, which is not inherited. + */ + private static void addPdfUaExtensionSchema(XMPMetadata xmp) { + PDFAExtensionSchema extension = + (PDFAExtensionSchema) xmp.getSchema(PDFAExtensionSchema.class); + if (extension == null) { + extension = xmp.createAndAddPDFAExtensionSchemaWithDefaultNS(); + } + + PDFAPropertyType partProperty = new PDFAPropertyType(xmp); + addField(xmp, partProperty, PDFAPropertyType.NAME, "part"); + addField(xmp, partProperty, PDFAPropertyType.VALUETYPE, "Integer"); + addField(xmp, partProperty, PDFAPropertyType.CATEGORY, "internal"); + addField( + xmp, + partProperty, + PDFAPropertyType.DESCRIPTION, + "Indicates which part of ISO 14289 the document conforms to"); + + PDFASchemaType schema = new PDFASchemaType(xmp); + addField(xmp, schema, PDFASchemaType.SCHEMA, "PDF/UA Universal Accessibility Schema"); + addField(xmp, schema, PDFASchemaType.NAMESPACE_URI, PdfUaIdentificationSchema.NAMESPACE); + addField(xmp, schema, PDFASchemaType.PREFIX, PdfUaIdentificationSchema.PREFERRED_PREFIX); + + ArrayProperty properties = + xmp.getTypeMapping() + .createArrayProperty( + schema.getNamespace(), + schema.getPrefix(), + PDFASchemaType.PROPERTY, + Cardinality.Seq); + properties.getContainer().addProperty(partProperty); + schema.getContainer().addProperty(properties); + + // A freshly created extension schema has no schemas bag yet, so make one. + ArrayProperty schemas = extension.getSchemasProperty(); + if (schemas == null) { + schemas = + xmp.getTypeMapping() + .createArrayProperty( + extension.getNamespace(), + extension.getPrefix(), + PDFAExtensionSchema.SCHEMAS, + Cardinality.Bag); + extension.addProperty(schemas); + } + schemas.getContainer().addProperty(schema); + } + + /** Adds one text field to a structured type, in that type's own namespace. */ + private static void addField( + XMPMetadata xmp, AbstractStructuredType target, String name, String value) { + target.getContainer() + .addProperty( + xmp.getTypeMapping() + .createText( + target.getNamespace(), target.getPrefix(), name, value)); + } + + private static XMPMetadata parseOrCreate(PDDocument document) throws Exception { + PDMetadata existing = document.getDocumentCatalog().getMetadata(); + if (existing == null) { + return XMPMetadata.createXMPMetadata(); + } + try (InputStream in = new ByteArrayInputStream(existing.toByteArray())) { + DomXmpParser parser = new DomXmpParser(); + parser.setStrictParsing(false); + return parser.parse(in); + } + } + + private static void writeMetadata(PDDocument document, XMPMetadata xmp) throws Exception { + ByteArrayOutputStream serialised = new ByteArrayOutputStream(); + new XmpSerializer().serialize(xmp, serialised, true); + PDMetadata metadata = new PDMetadata(document); + metadata.importXMPMetadata(serialised.toByteArray()); + document.getDocumentCatalog().setMetadata(metadata); + } + + /** Rewrites {@code pdfaid:conformance} without disturbing the rest of the packet. */ + static byte[] setConformance(byte[] pdfBytes, int part, String conformance) throws Exception { + try (PDDocument document = Loader.loadPDF(pdfBytes)) { + PDMetadata existing = document.getDocumentCatalog().getMetadata(); + XMPMetadata xmp; + if (existing != null) { + try (InputStream in = new ByteArrayInputStream(existing.toByteArray())) { + DomXmpParser parser = new DomXmpParser(); + parser.setStrictParsing(false); + xmp = parser.parse(in); + } + } else { + xmp = XMPMetadata.createXMPMetadata(); + } + + PDFAIdentificationSchema identification = + (PDFAIdentificationSchema) xmp.getSchema(PDFAIdentificationSchema.class); + if (identification == null) { + identification = xmp.createAndAddPDFAIdentificationSchema(); + } + identification.setPart(part); + identification.setConformance(conformance); + + ByteArrayOutputStream serialised = new ByteArrayOutputStream(); + new XmpSerializer().serialize(xmp, serialised, true); + PDMetadata metadata = new PDMetadata(document); + metadata.importXMPMetadata(serialised.toByteArray()); + document.getDocumentCatalog().setMetadata(metadata); + + return save(document, part); + } + } + + /** + * Part 1 is saved uncompressed: PDFBox's default object streams need PDF 1.5, which would push + * a PDF/A-1 file off its required 1.4 version. + */ + private static byte[] save(PDDocument document, int part) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save( + out, part == 1 ? CompressParameters.NO_COMPRESSION : new CompressParameters()); + return out.toByteArray(); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzerTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzerTest.java new file mode 100644 index 0000000000..e9aaab0a1a --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/LayoutAnalyzerTest.java @@ -0,0 +1,286 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +/** Unit tests for the heuristics that decide what a run of text means. */ +class LayoutAnalyzerTest { + + private static final BBox A4 = new BBox(0, 0, 595, 842); + + private static TextLineInfo line(String text, float size, float x, float y) { + return line(text, size, x, y, false, 0, 0); + } + + private static TextLineInfo line( + String text, float size, float x, float y, boolean bold, int start, int end) { + List words = new ArrayList<>(); + float cursor = x; + for (String token : text.strip().split("\\s+")) { + float width = token.length() * size * 0.5f; + words.add( + new WordInfo( + token, + new BBox(cursor, y, cursor + width, y + size), + start, + end, + size, + bold)); + cursor += width + size * 0.3f; + } + return new TextLineInfo( + 0, text, new BBox(x, y, cursor, y + size), size, bold, start, end, false, words); + } + + private static PageContent page(List lines) { + return new PageContent(0, lines, List.of(), lines.size(), false, false, false, A4); + } + + @Nested + @DisplayName("body font size") + class BodyFontSize { + + @Test + @DisplayName("weights by characters so one huge title does not skew the baseline") + void weightsByCharacterCount() { + List lines = + List.of( + line("A Very Large Title", 32, 50, 700), + line("Body text line one which is long", 11, 50, 650), + line("Body text line two which is long", 11, 50, 630), + line("Body text line three also long", 11, 50, 610)); + assertEquals(11f, LayoutAnalyzer.bodyFontSize(List.of(page(lines)))); + } + + @Test + @DisplayName("returns zero when there is no text") + void handlesEmptyDocument() { + assertEquals(0f, LayoutAnalyzer.bodyFontSize(List.of(page(List.of())))); + } + } + + @Nested + @DisplayName("heading detection") + class Headings { + + @Test + @DisplayName("assigns distinct sizes to descending levels") + void assignsTiers() { + List lines = + List.of( + line("Title", 24, 50, 800), + line("Chapter", 18, 50, 750), + line("Section", 14, 50, 700), + line("Body text that is long enough to set a baseline", 11, 50, 650)); + Map tiers = LayoutAnalyzer.headingTiers(List.of(page(lines)), 11f); + assertEquals(1, tiers.get(24f)); + assertEquals(2, tiers.get(18f)); + assertEquals(3, tiers.get(14f)); + assertNull(tiers.get(11f), "body size must not be a heading tier"); + } + + @Test + @DisplayName("rejects long lines and full sentences whatever their size") + void rejectsProse() { + assertFalse( + LayoutAnalyzer.isHeadingCandidate( + line("This line ends like a sentence does.", 20, 50, 700)), + "a line ending in a full stop reads as prose"); + assertFalse( + LayoutAnalyzer.isHeadingCandidate( + line( + "one two three four five six seven eight nine ten eleven twelve" + + " thirteen", + 20, + 50, + 700)), + "a long line is body text however large"); + assertTrue(LayoutAnalyzer.isHeadingCandidate(line("Financial Results", 20, 50, 700))); + } + + @Test + @DisplayName("boldness alone never promotes a line to a heading") + void boldIsNotAHeadingSignal() { + List lines = + List.of( + line("Bold Label", 11, 50, 700, true, 0, 0), + line("Body text long enough to set the baseline here", 11, 50, 650)); + assertTrue( + LayoutAnalyzer.headingTiers(List.of(page(lines)), 11f).isEmpty(), + "a bold line at body size is emphasis, not a heading"); + } + + @Test + @DisplayName("rewrites skipped levels so H1 is never followed by H3") + void normalisesSkippedLevels() { + DocumentStructure structure = new DocumentStructure(); + structure.add(new StructBlock(StructType.H1, 0)); + structure.add(new StructBlock(StructType.H3, 0)); + structure.add(new StructBlock(StructType.H4, 0)); + LayoutAnalyzer.normaliseHeadingLevels(structure); + + assertEquals(StructType.H1, structure.getBlocks().get(0).getType()); + assertEquals(StructType.H2, structure.getBlocks().get(1).getType()); + assertEquals(StructType.H3, structure.getBlocks().get(2).getType()); + } + } + + @Nested + @DisplayName("lists") + class Lists { + + @Test + @DisplayName("recognises bullet and ordered markers") + void recognisesMarkers() { + assertTrue(LayoutAnalyzer.startsListItem(line("• First item", 11, 50, 700))); + assertTrue(LayoutAnalyzer.startsListItem(line("- First item", 11, 50, 700))); + assertTrue(LayoutAnalyzer.startsListItem(line("1. First item", 11, 50, 700))); + assertTrue(LayoutAnalyzer.startsListItem(line("a) First item", 11, 50, 700))); + assertFalse(LayoutAnalyzer.startsListItem(line("Ordinary prose here", 11, 50, 700))); + } + } + + @Nested + @DisplayName("table cells") + class Tables { + + @Test + @DisplayName("splits a row at wide gaps but not at ordinary word spacing") + void splitsOnWideGaps() { + List words = + List.of( + new WordInfo("Region", new BBox(50, 700, 90, 711), 0, 0, 11, false), + new WordInfo("name", new BBox(93, 700, 125, 711), 0, 0, 11, false), + new WordInfo("Units", new BBox(250, 700, 285, 711), 1, 1, 11, false)); + TextLineInfo row = + new TextLineInfo( + 0, + "Region name Units", + new BBox(50, 700, 285, 711), + 11, + false, + 0, + 1, + false, + words); + List> cells = LayoutAnalyzer.splitCells(row); + assertEquals(2, cells.size(), "the small gap is a word space, the large one is a cell"); + assertEquals(2, cells.get(0).size()); + assertEquals("Units", cells.get(1).get(0).text()); + } + + @Test + @DisplayName("words sharing an operator cannot become separate cells") + void detectsInseparableWords() { + List shared = + List.of( + new WordInfo("A", new BBox(50, 700, 60, 711), 3, 3, 11, false), + new WordInfo("B", new BBox(250, 700, 260, 711), 3, 3, 11, false)); + TextLineInfo row = + new TextLineInfo( + 0, "A B", new BBox(50, 700, 260, 711), 11, false, 3, 3, false, shared); + assertFalse( + row.wordsAreSeparable(), + "cells drawn by one operator cannot carry separate marked content ids"); + } + } + + @Nested + @DisplayName("running heads") + class RunningHeads { + + @Test + @DisplayName("treats text repeating in the margin band across pages as an artifact") + void findsRepeatedMarginText() { + List pages = new ArrayList<>(); + for (int i = 0; i < 4; i++) { + List lines = + List.of( + line("Confidential Report", 9, 50, 800), + line("Body content for the page", 11, 50, 400), + line("Page " + (i + 1), 9, 300, 20)); + pages.add(new PageContent(i, lines, List.of(), 3, false, false, false, A4)); + } + Map> artifacts = LayoutAnalyzer.repeatedMarginLines(pages); + assertEquals( + 2, artifacts.get(0).size(), "the running head and the folio are artifacts"); + assertTrue( + artifacts.get(0).stream().noneMatch(l -> l.text().contains("Body content")), + "body text must never be demoted to an artifact"); + } + + @Test + @DisplayName("does not treat a one-off margin line as a running head") + void ignoresUniqueMarginText() { + List titles = List.of("Alpha", "Beta", "Gamma", "Delta"); + List pages = new ArrayList<>(); + for (int i = 0; i < 4; i++) { + List lines = + List.of( + line(titles.get(i) + " overview", 9, 50, 800), + line("Body content", 11, 50, 400)); + pages.add(new PageContent(i, lines, List.of(), 2, false, false, false, A4)); + } + assertTrue(LayoutAnalyzer.repeatedMarginLines(pages).get(0).isEmpty()); + } + + @Test + @DisplayName("a large heading high on the page stays a heading, not chrome") + void doesNotDemoteHeadingsNearTheTop() { + List pages = new ArrayList<>(); + for (int i = 0; i < 4; i++) { + List lines = + List.of( + // Masking digits makes these look identical across pages. + line("Section " + (i + 1), 20, 50, 800), + line("Body text long enough to set the baseline", 11, 50, 400)); + pages.add(new PageContent(i, lines, List.of(), 2, false, false, false, A4)); + } + assertTrue( + LayoutAnalyzer.repeatedMarginLines(pages, 11f).get(0).isEmpty(), + "a heading larger than body text is content, wherever it sits"); + } + } + + @Nested + @DisplayName("columns") + class Columns { + + @Test + @DisplayName("detects a gutter when text sits in two balanced blocks") + void detectsTwoColumns() { + List lines = new ArrayList<>(); + for (int i = 0; i < 6; i++) { + lines.add(line("Left column text", 10, 50, 700 - i * 14)); + lines.add(line("Right column text", 10, 320, 700 - i * 14)); + } + assertNotNull(LayoutAnalyzer.detectGutter(page(lines))); + } + + @Test + @DisplayName("does not split a page whose lines span the full width") + void ignoresSingleColumn() { + List lines = new ArrayList<>(); + for (int i = 0; i < 10; i++) { + lines.add( + line( + "A full width line of prose that crosses the centre of the page", + 10, + 50, + 700 - i * 14)); + } + assertNull(LayoutAnalyzer.detectGutter(page(lines))); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentInjectorTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentInjectorTest.java new file mode 100644 index 0000000000..73d4c64817 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentInjectorTest.java @@ -0,0 +1,164 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDPageContentStream; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.font.PDFont; +import org.apache.pdfbox.pdmodel.font.PDType1Font; +import org.apache.pdfbox.pdmodel.font.Standard14Fonts; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** Tests the content-stream rewriting that makes tagging possible. */ +class MarkedContentInjectorTest { + + private static byte[] threeLinePdf() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = new PDType1Font(Standard14Fonts.FontName.HELVETICA); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + for (int i = 0; i < 3; i++) { + cs.beginText(); + cs.setFont(font, 12); + cs.newLineAtOffset(50, 700 - i * 20); + cs.showText("Line " + i); + cs.endText(); + } + } + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } + } + + private static String contentOf(PDDocument document) throws IOException { + try (InputStream in = document.getPage(0).getContents()) { + return new String(in.readAllBytes(), StandardCharsets.ISO_8859_1); + } + } + + @Test + @DisplayName("wraps claimed content in BDC/EMC with a marked content id") + void wrapsClaimedContent() throws Exception { + try (PDDocument document = Loader.loadPDF(threeLinePdf())) { + StructBlock paragraph = new StructBlock(StructType.P, 0); + paragraph.addRange(0, 1); + + int next = + new MarkedContentInjector() + .inject(document, document.getPage(0), List.of(paragraph), 0, true); + + String content = contentOf(document); + assertTrue(content.contains("/P"), "the structure type was not written"); + assertTrue(content.contains("/MCID"), "no marked content id was written"); + assertTrue(content.contains("BDC"), "no marked content sequence was opened"); + assertTrue(content.contains("EMC"), "no marked content sequence was closed"); + assertFalse(paragraph.getMcids().isEmpty(), "the block was given no marked content id"); + assertTrue(next > 0, "the id counter did not advance"); + } + } + + @Test + @DisplayName("marks unclaimed content as an artifact so nothing is left untagged") + void unclaimedContentBecomesArtifact() throws Exception { + try (PDDocument document = Loader.loadPDF(threeLinePdf())) { + StructBlock paragraph = new StructBlock(StructType.P, 0); + paragraph.addRange(0, 0); + + new MarkedContentInjector() + .inject(document, document.getPage(0), List.of(paragraph), 0, true); + + String content = contentOf(document); + assertTrue( + content.contains("/Artifact"), + "content nobody claimed must be marked as an artifact, or PDF/UA clause 7.1" + + " fails"); + } + } + + @Test + @DisplayName("opens and closes sequences in balanced pairs") + void sequencesAreBalanced() throws Exception { + try (PDDocument document = Loader.loadPDF(threeLinePdf())) { + StructBlock first = new StructBlock(StructType.P, 0); + first.addRange(0, 0); + StructBlock second = new StructBlock(StructType.H1, 0); + second.addRange(2, 2); + + new MarkedContentInjector() + .inject(document, document.getPage(0), List.of(first, second), 0, true); + + String content = contentOf(document); + int opens = count(content, "BDC") + count(content, "BMC"); + int closes = count(content, "EMC"); + assertEquals(opens, closes, "every opened sequence must be closed"); + } + } + + @Test + @DisplayName("rewriting does not change what a reader extracts") + void textIsUnchanged() throws Exception { + byte[] original = threeLinePdf(); + String before = extract(original); + + byte[] rewritten; + try (PDDocument document = Loader.loadPDF(original)) { + StructBlock paragraph = new StructBlock(StructType.P, 0); + paragraph.addRange(0, 2); + new MarkedContentInjector() + .inject(document, document.getPage(0), List.of(paragraph), 0, true); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + rewritten = out.toByteArray(); + } + assertEquals(before, extract(rewritten), "marked content operators must not render"); + } + + @Test + @DisplayName("two blocks claiming the same content keep the first, not both") + void overlappingClaimsAreResolved() { + StructBlock first = new StructBlock(StructType.P, 0); + first.addRange(0, 2); + StructBlock second = new StructBlock(StructType.H1, 0); + second.addRange(1, 1); + + Map owners = + MarkedContentInjector.ownersByOrdinal(List.of(first, second)); + assertSame(first, owners.get(1), "the first claim wins so reading order stays unambiguous"); + assertEquals(3, owners.size()); + } + + private static int count(String haystack, String needle) { + int total = 0; + int index = 0; + while ((index = haystack.indexOf(needle, index)) >= 0) { + total++; + index += needle.length(); + } + return total; + } + + private static String extract(byte[] pdf) throws IOException { + try (PDDocument document = Loader.loadPDF(pdf)) { + return new org.apache.pdfbox.text.PDFTextStripper() + .getText(document) + .replaceAll("\\s+", " ") + .strip(); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentSafetyTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentSafetyTest.java new file mode 100644 index 0000000000..29cc18c525 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/MarkedContentSafetyTest.java @@ -0,0 +1,193 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDPageContentStream; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.common.PDStream; +import org.apache.pdfbox.pdmodel.font.PDType1Font; +import org.apache.pdfbox.pdmodel.font.Standard14Fonts; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** Regression tests for rewriter damage a validator cannot see, so it still passes validation. */ +class MarkedContentSafetyTest { + + private static String contentOf(PDDocument document) throws IOException { + try (InputStream in = document.getPage(0).getContents()) { + return new String(in.readAllBytes(), StandardCharsets.ISO_8859_1); + } + } + + private static void setContent(PDDocument document, String content) throws IOException { + PDStream stream = new PDStream(document); + try (var out = stream.createOutputStream()) { + out.write(content.getBytes(StandardCharsets.ISO_8859_1)); + } + document.getPage(0).setContents(stream); + } + + private static PDDocument onePage() throws IOException { + PDDocument document = new PDDocument(); + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + cs.beginText(); + cs.setFont(new PDType1Font(Standard14Fonts.FontName.HELVETICA), 12); + cs.newLineAtOffset(50, 700); + cs.showText("visible"); + cs.endText(); + } + return document; + } + + @Test + @DisplayName("an optional-content layer survives the rebuild, so hidden content stays hidden") + void optionalContentIsPreserved() throws Exception { + try (PDDocument document = onePage()) { + String original = contentOf(document); + setContent(document, "/OC /MC0 BDC\n" + original + "\nEMC\n"); + + StructBlock paragraph = new StructBlock(StructType.P, 0); + paragraph.addRange(0, 0); + new MarkedContentInjector() + .inject(document, document.getPage(0), List.of(paragraph), 0, true); + + String rewritten = contentOf(document); + assertTrue( + rewritten.contains("/OC"), + "the optional-content wrapper was stripped, which would make a hidden" + + " DRAFT/CONFIDENTIAL or redaction layer permanently visible:\n" + + rewritten); + assertEquals( + countOf(rewritten, "BDC") + countOf(rewritten, "BMC"), + countOf(rewritten, "EMC"), + "marked content is unbalanced after preserving the layer"); + } + } + + @Test + @DisplayName("replacement text survives the rebuild so ligatures still read correctly") + void actualTextIsPreserved() throws Exception { + try (PDDocument document = onePage()) { + String original = contentOf(document); + // A generator marks an ffi ligature with what it really spells. + setContent( + document, "/Span <> BDC\n" + original + "\nEMC\n"); + + StructBlock paragraph = new StructBlock(StructType.P, 0); + paragraph.addRange(0, 0); + new MarkedContentInjector() + .inject(document, document.getPage(0), List.of(paragraph), 0, true); + + String rewritten = contentOf(document); + assertTrue( + rewritten.contains("ActualText"), + "dropping ActualText leaves a screen reader announcing the raw glyph:\n" + + rewritten); + assertFalse( + rewritten.contains("/MCID 7"), + "the source's own marked content id is meaningless after a rebuild"); + assertEquals( + countOf(rewritten, "BDC") + countOf(rewritten, "BMC"), + countOf(rewritten, "EMC"), + "marked content is unbalanced after preserving replacement text"); + } + } + + @Test + @DisplayName("a sequence wrapping a fill opens before the path, not inside it") + void markedContentNeverOpensInsideAPathObject() throws Exception { + try (PDDocument document = onePage()) { + setContent(document, "0 0 0 rg\n10 10 50 5 re\nf\n"); + + new MarkedContentInjector().inject(document, document.getPage(0), List.of(), 0, true); + + String rewritten = contentOf(document); + int reAt = rewritten.indexOf(" re"); + int openAt = Math.max(rewritten.indexOf("BMC"), rewritten.indexOf("BDC")); + assertTrue(openAt >= 0, "no sequence was opened at all: " + rewritten); + assertTrue( + openAt < reAt, + "ISO 32000-1 does not permit a marked-content operator inside a path object;" + + " the sequence must open before the path construction:\n" + + rewritten); + } + } + + @Test + @DisplayName("words drawn out of stream order are still claimed, not silently artifacted") + void outOfOrderWordsAreClaimed() { + // A line whose second word on the page was painted first: ordinals 1 then 0. + WordInfo right = new WordInfo("label", new BBox(50, 700, 90, 712), 1, 1, 11, false); + WordInfo left = new WordInfo("value", new BBox(200, 700, 240, 712), 0, 0, 11, false); + TextLineInfo line = + new TextLineInfo( + 0, + "label value", + new BBox(50, 700, 240, 712), + 11, + false, + 0, + 1, + false, + List.of(right, left)); + + PageContent page = + new PageContent( + 0, + List.of(line), + List.of(), + 2, + false, + false, + false, + new BBox(0, 0, 595, 842)); + DocumentStructure structure = new LayoutAnalyzer().analyse(List.of(page)); + + boolean[] claimed = new boolean[2]; + structure.visit( + block -> { + if (block.isArtifact()) { + return; + } + block.getRanges() + .forEach( + r -> { + for (int i = r.start(); i <= r.end() && i < 2; i++) { + claimed[i] = true; + } + }); + }); + assertTrue( + claimed[0] && claimed[1], + "an out-of-order word was left unclaimed and would be hidden from assistive" + + " technology while the file still validated"); + } + + private static int countOf(String haystack, String needle) { + int total = 0; + int index = 0; + while ((index = haystack.indexOf(needle, index)) >= 0) { + total++; + index += needle.length(); + } + return total; + } + + private static byte[] bytes(PDDocument document) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaFormAndDeclarationTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaFormAndDeclarationTest.java new file mode 100644 index 0000000000..bae4ebed9a --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaFormAndDeclarationTest.java @@ -0,0 +1,171 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureElement; +import org.apache.pdfbox.pdmodel.documentinterchange.logicalstructure.PDStructureNode; +import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotationWidget; +import org.apache.pdfbox.pdmodel.interactive.form.PDAcroForm; +import org.apache.pdfbox.pdmodel.interactive.form.PDTextField; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** Covers form-field descriptions, widget nesting and withdrawing a conformance claim. */ +class PdfUaFormAndDeclarationTest { + + /** A document with one named text field and one unnamed one. */ + private static PDDocument formDocument(boolean nameTheSecondField) throws IOException { + PDDocument document = new PDDocument(); + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + + PDAcroForm form = new PDAcroForm(document); + document.getDocumentCatalog().setAcroForm(form); + + PDTextField named = new PDTextField(form); + named.setPartialName("EmailAddress"); + addWidget(named, page, 700); + form.getFields().add(named); + + PDTextField second = new PDTextField(form); + if (nameTheSecondField) { + second.setPartialName("PostCode"); + } + addWidget(second, page, 650); + form.getFields().add(second); + + return document; + } + + private static void addWidget(PDTextField field, PDPage page, float y) throws IOException { + PDAnnotationWidget widget = field.getWidgets().get(0); + PDRectangle rectangle = new PDRectangle(); + rectangle.setLowerLeftX(50); + rectangle.setLowerLeftY(y); + rectangle.setUpperRightX(250); + rectangle.setUpperRightY(y + 18); + widget.setRectangle(rectangle); + widget.setPage(page); + page.getAnnotations().add(widget); + } + + private static String xmpOf(PDDocument document) throws IOException { + var metadata = document.getDocumentCatalog().getMetadata(); + assertNotNull(metadata, "no XMP packet"); + return new String(metadata.toByteArray(), StandardCharsets.UTF_8); + } + + @Test + @DisplayName("a form field gets its tooltip from its own name, not an invented one") + void derivesTooltipFromFieldName() throws Exception { + try (PDDocument document = formDocument(true)) { + List warnings = + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, "Form", "en", PdfUaProfile.UA1); + + PDAcroForm form = document.getDocumentCatalog().getAcroForm(); + assertEquals("EmailAddress", form.getField("EmailAddress").getAlternateFieldName()); + assertEquals("PostCode", form.getField("PostCode").getAlternateFieldName()); + assertTrue(warnings.isEmpty(), "nothing needed reporting: " + warnings); + } + } + + @Test + @DisplayName("a field with no name is reported rather than given a placeholder tooltip") + void reportsUnnameableField() throws Exception { + try (PDDocument document = formDocument(false)) { + List warnings = + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, "Form", "en", PdfUaProfile.UA1); + assertEquals(1, warnings.size()); + assertTrue(warnings.get(0).contains("form field"), warnings.get(0)); + } + } + + @Test + @DisplayName("an existing description is never overwritten") + void keepsExistingDescription() throws Exception { + try (PDDocument document = formDocument(true)) { + PDAcroForm form = document.getDocumentCatalog().getAcroForm(); + form.getField("EmailAddress").setAlternateFieldName("Your email address"); + + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, "Form", "en", PdfUaProfile.UA1); + assertEquals( + "Your email address", form.getField("EmailAddress").getAlternateFieldName()); + } + } + + @Test + @DisplayName("widget annotations are nested inside a Form structure element") + void widgetsAreNestedInFormElements() throws Exception { + try (PDDocument document = formDocument(true)) { + DocumentStructure structure = new DocumentStructure(); + StructBlock paragraph = new StructBlock(StructType.P, 0); + paragraph.getMcids().add(0); + structure.add(paragraph); + + new StructTreeWriter().write(document, structure, PdfUaProfile.UA1); + + var root = document.getDocumentCatalog().getStructureTreeRoot(); + assertTrue( + typesUnder(root).contains("Form"), + "clause 7.18.4 requires a widget to sit inside a Form element, found: " + + typesUnder(root)); + } + } + + private static List typesUnder(PDStructureNode node) { + List types = new java.util.ArrayList<>(); + for (Object kid : node.getKids()) { + if (kid instanceof PDStructureElement element) { + types.add(element.getStructureType()); + types.addAll(typesUnder(element)); + } + } + return types; + } + + @Test + @DisplayName("withdrawing conformance removes the claim but keeps the other metadata") + void withdrawingConformanceRemovesOnlyTheClaim() throws Exception { + try (PDDocument document = new PDDocument()) { + document.addPage(new PDPage()); + PdfUaTagger tagger = new PdfUaTagger(); + PdfUaMetadataWriter writer = new PdfUaMetadataWriter(); + + writer.applyDocumentRequirements(document, "Kept Title", "en-GB", PdfUaProfile.UA1); + writer.declareConformance(document, PdfUaProfile.UA1); + assertTrue(xmpOf(document).contains("pdfuaid")); + + tagger.withdrawConformance(document); + + String xmp = xmpOf(document); + assertFalse(xmp.contains("pdfuaid"), "the conformance claim should be gone"); + assertTrue(xmp.contains("Kept Title"), "the title should survive"); + assertEquals("en-GB", document.getDocumentCatalog().getLanguage()); + } + } + + @Test + @DisplayName("withdrawing conformance on a document that never claimed it is harmless") + void withdrawingIsIdempotent() throws Exception { + try (PDDocument document = new PDDocument()) { + document.addPage(new PDPage()); + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, "Title", "en", PdfUaProfile.UA1); + new PdfUaTagger().withdrawConformance(document); + assertFalse(xmpOf(document).contains("pdfuaid")); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaLanguageTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaLanguageTest.java new file mode 100644 index 0000000000..ed05e8de0b --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaLanguageTest.java @@ -0,0 +1,79 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** + * A relabelled language is invisible to every validator, so the tagger must not guess over one the + * document already declares. + */ +class PdfUaLanguageTest { + + private static PDDocument documentWithLanguage(String language) { + PDDocument document = new PDDocument(); + document.addPage(new PDPage()); + if (language != null) { + document.getDocumentCatalog().setLanguage(language); + } + return document; + } + + private static TaggingOptions.TaggingOptionsBuilder options() { + return TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Rapport") + .embedFonts(false); + } + + @Test + @DisplayName("keeps the language the document already declares") + void keepsExistingLanguage() throws Exception { + try (PDDocument document = documentWithLanguage("fr-FR")) { + TaggingResult result = new PdfUaTagger().tag(document, options().build()); + + assertEquals("fr-FR", document.getDocumentCatalog().getLanguage()); + assertTrue( + result.getWarnings().stream().anyMatch(w -> w.contains("fr-FR")), + "ignoring the requested language must be reported: " + result.getWarnings()); + } + } + + @Test + @DisplayName("applies the requested language when the document declares none") + void fillsInMissingLanguage() throws Exception { + try (PDDocument document = documentWithLanguage(null)) { + new PdfUaTagger().tag(document, options().build()); + + assertEquals("en-GB", document.getDocumentCatalog().getLanguage()); + } + } + + @Test + @DisplayName("replaces the declared language only when the caller asks") + void overridesOnRequest() throws Exception { + try (PDDocument document = documentWithLanguage("fr-FR")) { + new PdfUaTagger().tag(document, options().overrideLanguage(true).build()); + + assertEquals("en-GB", document.getDocumentCatalog().getLanguage()); + } + } + + @Test + @DisplayName("keeps the existing language when an existing structure tree is left alone") + void keepsExistingLanguageWithoutRebuilding() throws Exception { + try (PDDocument document = documentWithLanguage("de-DE")) { + new PdfUaTagger() + .tag( + document, + options().existingTags(TaggingOptions.ExistingTags.KEEP).build()); + + assertEquals("de-DE", document.getDocumentCatalog().getLanguage()); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriterTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriterTest.java new file mode 100644 index 0000000000..f953488b03 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaMetadataWriterTest.java @@ -0,0 +1,137 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** Tests the document-level requirements that have nothing to do with tagging. */ +class PdfUaMetadataWriterTest { + + private static PDDocument twoPageDocument() { + PDDocument document = new PDDocument(); + document.addPage(new PDPage()); + document.addPage(new PDPage()); + return document; + } + + private static String xmpOf(PDDocument document) throws IOException { + var metadata = document.getDocumentCatalog().getMetadata(); + assertNotNull(metadata, "no XMP packet was written"); + return new String(metadata.toByteArray(), StandardCharsets.UTF_8); + } + + @Test + @DisplayName("sets title, language, tab order and the display-title flag") + void appliesDocumentRequirements() throws Exception { + try (PDDocument document = twoPageDocument()) { + new PdfUaMetadataWriter() + .applyDocumentRequirements( + document, "Annual Report", "en-GB", PdfUaProfile.UA1); + + assertEquals("en-GB", document.getDocumentCatalog().getLanguage()); + assertEquals("Annual Report", document.getDocumentInformation().getTitle()); + assertTrue( + document.getDocumentCatalog().getViewerPreferences().displayDocTitle(), + "without DisplayDocTitle a viewer shows the filename instead of the title"); + + for (PDPage page : document.getPages()) { + assertEquals( + "S", + page.getCOSObject().getNameAsString(COSName.getPDFName("Tabs")), + "clause 7.18.1 requires an explicit tab order on every page"); + } + } + } + + @Test + @DisplayName("writes dc:title into the XMP packet, not just the info dictionary") + void writesDublinCoreTitle() throws Exception { + try (PDDocument document = twoPageDocument()) { + new PdfUaMetadataWriter() + .applyDocumentRequirements( + document, "Annual Report", "en-GB", PdfUaProfile.UA1); + assertTrue(xmpOf(document).contains("Annual Report")); + } + } + + @Test + @DisplayName("does not declare conformance as part of applying requirements") + void doesNotDeclareEarly() throws Exception { + try (PDDocument document = twoPageDocument()) { + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, "Report", "en", PdfUaProfile.UA1); + assertFalse( + xmpOf(document).contains("pdfuaid"), + "the conformance claim must wait until validation has passed"); + } + } + + @Test + @DisplayName("declaring conformance writes pdfuaid with the right part") + void declaresConformance() throws Exception { + try (PDDocument document = twoPageDocument()) { + PdfUaMetadataWriter writer = new PdfUaMetadataWriter(); + writer.applyDocumentRequirements(document, "Report", "en", PdfUaProfile.UA1); + writer.declareConformance(document, PdfUaProfile.UA1); + + String xmp = xmpOf(document); + assertTrue(xmp.contains("pdfuaid"), "no PDF/UA identifier was written"); + assertTrue(xmp.contains("part"), "no conformance part was written"); + } + } + + @Test + @DisplayName("UA-2 raises the PDF version to 2.0") + void ua2RaisesVersion() throws Exception { + try (PDDocument document = twoPageDocument()) { + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, "Report", "en", PdfUaProfile.UA2); + assertEquals(2.0f, document.getVersion()); + } + } + + @Test + @DisplayName("keeps an existing title when none is supplied") + void keepsExistingTitle() throws Exception { + try (PDDocument document = twoPageDocument()) { + var info = document.getDocumentInformation(); + info.setTitle("Original Title"); + document.setDocumentInformation(info); + + new PdfUaMetadataWriter() + .applyDocumentRequirements(document, null, "en", PdfUaProfile.UA1); + assertEquals("Original Title", document.getDocumentInformation().getTitle()); + } + } + + @Test + @DisplayName("survives a round trip through save and reload") + void survivesRoundTrip() throws Exception { + byte[] saved; + try (PDDocument document = twoPageDocument()) { + PdfUaMetadataWriter writer = new PdfUaMetadataWriter(); + writer.applyDocumentRequirements(document, "Round Trip", "fr-FR", PdfUaProfile.UA1); + writer.declareConformance(document, PdfUaProfile.UA1); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + saved = out.toByteArray(); + } + try (PDDocument reloaded = Loader.loadPDF(saved)) { + assertEquals("fr-FR", reloaded.getDocumentCatalog().getLanguage()); + assertEquals("Round Trip", reloaded.getDocumentInformation().getTitle()); + assertTrue(xmpOf(reloaded).contains("pdfuaid")); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaModelTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaModelTest.java new file mode 100644 index 0000000000..de189f2f74 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/PdfUaModelTest.java @@ -0,0 +1,160 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +/** Tests for the small types the tagger is built from. */ +class PdfUaModelTest { + + @Nested + @DisplayName("structure types") + class Types { + + @Test + @DisplayName("maps levels to heading tags and back") + void headingLevelsRoundTrip() { + for (int level = 1; level <= 6; level++) { + assertEquals(level, StructType.heading(level).headingLevel()); + assertEquals("H" + level, StructType.heading(level).tag()); + } + } + + @Test + @DisplayName("clamps out-of-range levels rather than throwing") + void clampsLevels() { + assertEquals(StructType.H1, StructType.heading(0)); + assertEquals(StructType.H1, StructType.heading(-3)); + assertEquals(StructType.H6, StructType.heading(9)); + } + + @Test + @DisplayName("reports zero for types that are not headings") + void nonHeadingsHaveNoLevel() { + assertEquals(0, StructType.P.headingLevel()); + assertFalse(StructType.TABLE.isHeading()); + } + } + + @Nested + @DisplayName("markable operators") + class Markable { + + @ParameterizedTest + @ValueSource(strings = {"Tj", "TJ", "'", "\"", "Do", "BI", "S", "f", "f*", "B", "sh"}) + @DisplayName("counts text, XObjects and path painting") + void counted(String operator) { + assertTrue(MarkableOp.isMarkableOperator(operator), operator + " should be markable"); + } + + @ParameterizedTest + @ValueSource(strings = {"q", "Q", "cm", "BT", "ET", "Tf", "Td", "n", "W", "gs", "re"}) + @DisplayName("ignores operators that paint nothing") + void notCounted(String operator) { + assertFalse( + MarkableOp.isMarkableOperator(operator), operator + " should not be markable"); + } + + @Test + @DisplayName("n ends a path without painting, so it is not content") + void pathEndIsNotPainting() { + assertFalse(MarkableOp.isPathPainting("n")); + assertTrue(MarkableOp.isPathPainting("f")); + } + } + + @Nested + @DisplayName("profiles") + class Profiles { + + @Test + @DisplayName("parses the shapes a caller might send") + void parsesRequestValues() { + assertEquals(PdfUaProfile.UA1, PdfUaProfile.fromRequest("ua1")); + assertEquals(PdfUaProfile.UA1, PdfUaProfile.fromRequest(null)); + assertEquals(PdfUaProfile.UA1, PdfUaProfile.fromRequest("")); + assertEquals(PdfUaProfile.UA1, PdfUaProfile.fromRequest("nonsense")); + assertEquals(PdfUaProfile.UA2, PdfUaProfile.fromRequest("ua2")); + assertEquals(PdfUaProfile.UA2, PdfUaProfile.fromRequest("PDF/UA-2")); + } + + @Test + @DisplayName("UA-2 requires PDF 2.0") + void ua2NeedsPdf2() { + assertEquals(2.0f, PdfUaProfile.UA2.pdfVersion()); + assertEquals(1.7f, PdfUaProfile.UA1.pdfVersion()); + } + } + + @Nested + @DisplayName("bounding boxes") + class Boxes { + + @Test + @DisplayName("union of an empty box is the other box") + void unionWithEmpty() { + BBox box = new BBox(10, 10, 20, 20); + assertEquals(box, box.union(BBox.EMPTY)); + assertEquals(box, BBox.EMPTY.union(box)); + } + + @Test + @DisplayName("union covers both boxes") + void unionCoversBoth() { + BBox union = new BBox(0, 0, 10, 10).union(new BBox(20, 5, 30, 25)); + assertEquals(new BBox(0, 0, 30, 25), union); + } + + @Test + @DisplayName("reports horizontal overlap as a fraction of the narrower box") + void overlapIsRelative() { + BBox wide = new BBox(0, 0, 100, 10); + BBox narrow = new BBox(40, 0, 60, 10); + assertEquals(1.0f, wide.horizontalOverlap(narrow)); + assertEquals(0f, wide.horizontalOverlap(new BBox(200, 0, 220, 10))); + } + } + + @Nested + @DisplayName("structure blocks") + class Blocks { + + @Test + @DisplayName("counts content across the whole subtree") + void countsDescendantContent() { + StructBlock table = new StructBlock(StructType.TABLE, 0); + StructBlock row = new StructBlock(StructType.TR, 0); + StructBlock cell = new StructBlock(StructType.TD, 0); + cell.addRange(3, 5); + row.addChild(cell); + table.addChild(row); + assertEquals(3, table.contentCount()); + } + + @Test + @DisplayName("collects text in tree order") + void collectsText() { + StructBlock list = new StructBlock(StructType.L, 0); + StructBlock first = new StructBlock(StructType.LI, 0); + first.setText("one"); + StructBlock second = new StructBlock(StructType.LI, 0); + second.setText("two"); + list.addChild(first).addChild(second); + assertEquals("one two", list.collectText()); + } + + @Test + @DisplayName("an artifact is not a structure element") + void artifactsAreDistinct() { + StructBlock artifact = StructBlock.artifact(ArtifactType.PAGINATION, 0); + assertTrue(artifact.isArtifact()); + assertEquals("Pagination", artifact.getArtifactType().subtype()); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/VectorAndHeadingTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/VectorAndHeadingTest.java new file mode 100644 index 0000000000..aa1b13e3b9 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/pdf/ua/VectorAndHeadingTest.java @@ -0,0 +1,155 @@ +package stirling.software.proprietary.pdf.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +/** Tests the heuristics that decide what counts as a drawing and what counts as a heading. */ +class VectorAndHeadingTest { + + private static final BBox A4 = new BBox(0, 0, 595, 842); + + private static TextLineInfo line(String text, float size, float x, float y) { + List words = new ArrayList<>(); + float cursor = x; + for (String token : text.strip().split("\\s+")) { + float width = token.length() * size * 0.5f; + words.add( + new WordInfo( + token, + new BBox(cursor, y, cursor + width, y + size), + 0, + 0, + size, + false)); + cursor += width + size * 0.3f; + } + return new TextLineInfo( + 0, text, new BBox(x, y, cursor, y + size), size, false, 0, 0, false, words); + } + + private static MarkableOp vector(int ordinal, BBox box) { + return new MarkableOp(ordinal, MarkableOp.Kind.VECTOR, box, null); + } + + private static DocumentStructure analyse(List lines, List ops) { + PageContent page = new PageContent(0, lines, ops, ops.size(), false, false, false, A4); + return new LayoutAnalyzer().analyse(List.of(page)); + } + + private static long countOf(DocumentStructure structure, StructType type) { + long[] total = {0}; + structure.visit( + block -> { + if (block.getType() == type) { + total[0]++; + } + }); + return total[0]; + } + + @Test + @DisplayName("a cluster of substantial strokes becomes a figure, not silent decoration") + void chartBecomesAFigure() { + List bars = new ArrayList<>(); + for (int i = 0; i < 6; i++) { + bars.add(vector(i, new BBox(100 + i * 20, 400, 115 + i * 20, 400 + 30 + i * 10))); + } + DocumentStructure structure = analyse(List.of(), bars); + + assertTrue( + countOf(structure, StructType.FIGURE) > 0, + "a bar chart drawn with path operators must not vanish as decoration"); + assertTrue( + structure.figuresWithoutAlt().size() > 0, + "the report must say the chart needs a description"); + } + + @Test + @DisplayName("thin rules and table borders stay artifacts") + void tableRulesStayDecoration() { + List rules = new ArrayList<>(); + for (int i = 0; i < 8; i++) { + rules.add(vector(i, new BBox(60, 700 - i * 20, 540, 701 - i * 20))); + } + DocumentStructure structure = analyse(List.of(), rules); + + assertEquals( + 0, + countOf(structure, StructType.FIGURE), + "horizontal rules are page furniture and must not demand alt text"); + assertEquals(0, structure.figuresWithoutAlt().size()); + } + + @Test + @DisplayName("a lone box is ornament, not a chart") + void singleBoxIsNotAFigure() { + DocumentStructure structure = + analyse(List.of(), List.of(vector(0, new BBox(60, 400, 500, 700)))); + assertEquals(0, countOf(structure, StructType.FIGURE)); + } + + @Test + @DisplayName("shaded table rows behind text are not mistaken for a chart") + void shadedTableRowsAreNotFigures() { + List shading = new ArrayList<>(); + List rows = new ArrayList<>(); + for (int i = 0; i < 6; i++) { + float y = 600 - i * 20; + // A filled row background, tall enough to pass the thinness test. + shading.add(vector(i, new BBox(60, y, 540, y + 16))); + rows.add(line("Expense line item " + i + " amount", 10, 64, y + 3)); + } + DocumentStructure structure = analyse(rows, shading); + + assertEquals( + 0, + countOf(structure, StructType.FIGURE), + "row shading sits behind the text it decorates and is not a drawing"); + } + + @Test + @DisplayName("small print dominating an invoice does not promote addresses to headings") + void smallPrintDoesNotCreateHeadings() { + List lines = new ArrayList<>(); + // Address block at ordinary 11pt. + lines.add(line("Acme Industries Limited", 11, 60, 780)); + lines.add(line("14 Example Street", 11, 60, 765)); + lines.add(line("Manchester M1 2AB", 11, 60, 750)); + // 40 lines of 9pt line-item small print, which dominates the character count. + for (int i = 0; i < 40; i++) { + lines.add(line("Item " + i + " widget assembly part number " + i, 9, 60, 700 - i * 12)); + } + + Map tiers = + LayoutAnalyzer.headingTiers( + List.of(new PageContent(0, lines, List.of(), 0, false, false, false, A4)), + 9f); + assertNull( + tiers.get(11f), + "11pt address lines are body text on an invoice, not headings: " + tiers); + } + + @Test + @DisplayName("a genuinely rare large size is still a heading") + void realHeadingsSurvive() { + List lines = new ArrayList<>(); + lines.add(line("Annual Report", 24, 60, 780)); + for (int i = 0; i < 40; i++) { + lines.add( + line("Body prose line number " + i + " continues here", 11, 60, 700 - i * 12)); + } + Map tiers = + LayoutAnalyzer.headingTiers( + List.of(new PageContent(0, lines, List.of(), 0, false, false, false, A4)), + 11f); + assertEquals(1, tiers.get(24f), "a rare large size is exactly what a heading looks like"); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/AltTextRoundTripTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/AltTextRoundTripTest.java new file mode 100644 index 0000000000..0f8e52d9ec --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/AltTextRoundTripTest.java @@ -0,0 +1,115 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.Map; + +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.controller.api.converters.ConvertPdfToPdfUa; +import stirling.software.proprietary.model.api.ua.AccessibilityReport; +import stirling.software.proprietary.model.api.ua.FigureDescriptor; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** + * The alt-text loop end to end: the report hands out keys the conversion accepts. The converter + * never invents descriptions, so a caller must be able to supply them. + */ +class AltTextRoundTripTest { + + private static PdfUaConversionService conversion; + private static AccessibilityAuditService audit; + + @BeforeAll + static void setUp() { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + conversion = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + audit = new AccessibilityAuditService(validation); + } + + private static TaggingOptions.TaggingOptionsBuilder options() { + return TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Illustrated") + .embedFonts(false); + } + + @Test + @DisplayName("the report names the figures that need describing, with usable keys") + void reportEnumeratesFigures() throws Exception { + byte[] input = PdfUaTestDocuments.imageDocument(); + AccessibilityReport report = audit.audit(input, PdfUaProfile.UA1); + + assertFalse( + report.getFiguresNeedingDescription().isEmpty(), + "a document with an undescribed image must say which figure needs text"); + + FigureDescriptor figure = report.getFiguresNeedingDescription().get(0); + assertTrue(figure.key().matches("\\d+:\\d+"), "key should be pageIndex:ordinal: " + figure); + assertEquals(1, figure.page(), "pages are reported 1-based for humans"); + assertTrue(figure.width() > 0 && figure.height() > 0, "figure should carry its box"); + } + + @Test + @DisplayName("feeding the report's key back makes the document conform") + void suppliedDescriptionClosesTheLoop() throws Exception { + byte[] input = PdfUaTestDocuments.imageDocument(); + + PdfUaConversionOutcome before = conversion.convert(input, options().build()); + assertFalse(before.declared(), "an undescribed image must block the claim"); + + String key = + audit.audit(input, PdfUaProfile.UA1).getFiguresNeedingDescription().get(0).key(); + PdfUaConversionOutcome after = + conversion.convert( + input, options().altTextByFigure(Map.of(key, "A blue rectangle")).build()); + + assertEquals( + 0, + after.tagging().figuresNeedingAltText(), + "the description supplied against the report's own key was not applied"); + assertTrue(after.declared(), "with every figure described the document should conform"); + } + + @Test + @DisplayName("the request's key=text form parses the way the report emits keys") + void parsesTheWireFormat() { + Map parsed = + ConvertPdfToPdfUa.parseAltText( + "0:12=Bar chart of quarterly revenue\r\n" + + "1:3=Company logo\n" + + " \n" + + "malformed-line\n" + + "2:7=Diagram showing the approval flow = end to end"); + + assertEquals(3, parsed.size(), "blank and malformed lines are skipped: " + parsed); + assertEquals("Bar chart of quarterly revenue", parsed.get("0:12")); + assertEquals("Company logo", parsed.get("1:3")); + assertEquals( + "Diagram showing the approval flow = end to end", + parsed.get("2:7"), + "only the first equals splits, so descriptions may contain one"); + } + + @Test + @DisplayName("no descriptions supplied means none invented") + void emptyInputInventsNothing() { + assertTrue(ConvertPdfToPdfUa.parseAltText(null).isEmpty()); + assertTrue(ConvertPdfToPdfUa.parseAltText(" ").isEmpty()); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUa2ProfileTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUa2ProfileTest.java new file mode 100644 index 0000000000..9cf342824f --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUa2ProfileTest.java @@ -0,0 +1,92 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.cos.COSName; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** PDF/UA-2 is not just a metadata number: it needs PDF 2.0 and namespaced structure types. */ +class PdfUa2ProfileTest { + + private static PdfUaConversionService conversion; + + @BeforeAll + static void setUp() { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + conversion = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + } + + private static PdfUaConversionOutcome convertUa2(byte[] input) throws Exception { + return conversion.convert( + input, + TaggingOptions.builder() + .profile(PdfUaProfile.UA2) + .language("en-GB") + .title("UA-2 Document") + .embedFonts(false) + .build()); + } + + @Test + @DisplayName("raises the file to PDF 2.0 and namespaces the structure tree") + void producesPdf2WithNamespaces() throws Exception { + PdfUaConversionOutcome outcome = convertUa2(PdfUaTestDocuments.headingHierarchy()); + + try (PDDocument document = Loader.loadPDF(outcome.pdfBytes())) { + assertEquals(2.0f, document.getVersion(), "UA-2 is defined on PDF 2.0"); + + var root = document.getDocumentCatalog().getStructureTreeRoot(); + assertNotNull(root, "no structure tree was written"); + assertNotNull( + root.getCOSObject().getDictionaryObject(COSName.getPDFName("Namespaces")), + "UA-2 requires the standard structure namespace to be declared"); + } + } + + @Test + @DisplayName("validates against the PDF/UA-2 profile, not the UA-1 one") + void validatesAgainstUa2() throws Exception { + PdfUaConversionOutcome outcome = convertUa2(PdfUaTestDocuments.simpleDocument()); + assertEquals("PDF/UA-2", outcome.validation().profile()); + } + + @Test + @DisplayName("reaches UA-2 conformance and declares it") + void reachesUa2Conformance() throws Exception { + PdfUaConversionOutcome outcome = convertUa2(PdfUaTestDocuments.simpleDocument()); + + String failures = + outcome.validation().issues().stream() + .map(issue -> issue.getClause() + ": " + issue.getTechnicalMessage()) + .collect(java.util.stream.Collectors.joining("; ")); + assertEquals(0, outcome.validation().totalFailures(), "UA-2 checks failed: " + failures); + assertTrue(outcome.declared(), "a conforming UA-2 file must carry the declaration"); + assertTrue(outcome.pdfBytes().length > 0); + } + + @Test + @DisplayName("an illustrated document still cannot claim UA-2 without descriptions") + void undescribedImageBlocksTheUa2Claim() throws Exception { + PdfUaConversionOutcome outcome = convertUa2(PdfUaTestDocuments.imageDocument()); + assertFalse(outcome.declared(), "an undescribed image must block the claim"); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaBenchmarkTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaBenchmarkTest.java new file mode 100644 index 0000000000..2c4ca6db2a --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaBenchmarkTest.java @@ -0,0 +1,341 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDPageContentStream; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.font.PDFont; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.pdf.ua.DocumentStructure; +import stirling.software.proprietary.pdf.ua.LayoutAnalyzer; +import stirling.software.proprietary.pdf.ua.PageContent; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.PdfUaTagger; +import stirling.software.proprietary.pdf.ua.TaggedContentExtractor; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** + * Measures where conversion time and memory go; meant to be read, not to gate CI. Assertions catch + * only order-of-magnitude regressions - wall-clock numbers are no contract. + */ +class PdfUaBenchmarkTest { + + private static PdfUaConversionService service; + private static PdfUaValidationService validation; + + @BeforeAll + static void setUp() { + validation = new PdfUaValidationService(); + validation.initialise(); + service = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + } + + /** A realistic page: heading, prose, a small table, a bullet list. */ + private static byte[] document(int pages) throws IOException { + try (PDDocument document = new PDDocument()) { + PDFont font = null; + for (int p = 0; p < pages; p++) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + if (font == null) { + font = PdfUaTestDocuments.font(document); + } + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 790; + write(cs, font, 9, 60, 810, "Benchmark Corpus Running Head"); + write(cs, font, 18, 60, y, "Section " + (p + 1)); + y -= 30; + for (int line = 0; line < 22; line++) { + write( + cs, + font, + 11, + 60, + y, + "Body line " + line + " of section " + (p + 1) + " with prose."); + y -= 15; + } + for (int row = 0; row < 4; row++) { + cs.beginText(); + cs.setFont(font, 11); + cs.newLineAtOffset(60, y); + cs.showText("Row " + row); + cs.newLineAtOffset(160, 0); + cs.showText(String.valueOf(row * 120)); + cs.newLineAtOffset(140, 0); + cs.showText(String.valueOf(row * 480)); + cs.endText(); + y -= 16; + } + write(cs, font, 11, 60, y - 10, "• First bullet point"); + write(cs, font, 11, 60, y - 25, "• Second bullet point"); + write(cs, font, 9, 300, 30, "Page " + (p + 1)); + } + } + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } + } + + private static void write( + PDPageContentStream cs, PDFont font, float size, float x, float y, String text) + throws IOException { + cs.beginText(); + cs.setFont(font, size); + cs.newLineAtOffset(x, y); + cs.showText(text); + cs.endText(); + } + + private static TaggingOptions options() { + return TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Benchmark") + .embedFonts(false) + .existingTags(TaggingOptions.ExistingTags.REBUILD) + .build(); + } + + private static long usedHeap() { + Runtime runtime = Runtime.getRuntime(); + System.gc(); + return runtime.totalMemory() - runtime.freeMemory(); + } + + @Test + @DisplayName("reports throughput and memory across document sizes") + void throughputAcrossSizes() throws Exception { + int[] sizes = {1, 10, 50, 150}; + StringBuilder report = + new StringBuilder("\nPDF/UA conversion throughput\n") + .append( + String.format( + " %-7s %-10s %-12s %-12s %-10s %s%n", + "pages", + "input", + "convert ms", + "ms/page", + "pages/s", + "heap MB")); + + // Warm up so the first timed run is not measuring class loading and JIT. + service.convert(document(5), options()); + + for (int pages : sizes) { + byte[] input = document(pages); + long heapBefore = usedHeap(); + long start = System.nanoTime(); + var outcome = service.convert(input, options()); + long elapsedMs = (System.nanoTime() - start) / 1_000_000; + long heapDelta = (usedHeap() - heapBefore) / (1024 * 1024); + + assertTrue(outcome.pdfBytes().length > 0); + report.append( + String.format( + Locale.ROOT, + " %-7d %-10s %-12d %-12.2f %-10.1f %d%n", + pages, + humanBytes(input.length), + elapsedMs, + elapsedMs / (double) pages, + pages * 1000.0 / Math.max(elapsedMs, 1), + Math.max(heapDelta, 0))); + } + System.out.println(report); + } + + @Test + @DisplayName("breaks conversion down by phase so optimisation has a target") + void phaseBreakdown() throws Exception { + byte[] input = document(60); + + // Warm up. + try (PDDocument warm = Loader.loadPDF(input)) { + new TaggedContentExtractor().extract(warm); + } + + long parseMs; + long extractMs; + long analyseMs; + long tagMs; + List pages; + DocumentStructure structure; + + long t0 = System.nanoTime(); + try (PDDocument document = Loader.loadPDF(input)) { + parseMs = ms(t0); + + long t1 = System.nanoTime(); + pages = new TaggedContentExtractor().extract(document); + extractMs = ms(t1); + + long t2 = System.nanoTime(); + structure = new LayoutAnalyzer().analyse(pages); + analyseMs = ms(t2); + } + + long t3 = System.nanoTime(); + try (PDDocument document = Loader.loadPDF(input)) { + new PdfUaTagger().tag(document, options()); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + } + tagMs = ms(t3); + + long t4 = System.nanoTime(); + var outcome = service.convert(input, options()); + long totalMs = ms(t4); + + long t5 = System.nanoTime(); + validation.validate(outcome.pdfBytes(), PdfUaProfile.UA1); + long validateMs = ms(t5); + + System.out.printf( + Locale.ROOT, + "%nPhase breakdown over %d pages (%d blocks)%n" + + " parse %5d ms%n" + + " extract %5d ms (text pass + token scan)%n" + + " analyse %5d ms%n" + + " tag end-to-end %5d ms (includes parse, extract, analyse, inject, write)%n" + + " validate %5d ms (veraPDF)%n" + + " full convert %5d ms (tag + declare + validate)%n", + 60, + structure.getBlocks().size(), + parseMs, + extractMs, + analyseMs, + tagMs, + validateMs, + totalMs); + + assertTrue(pages.size() == 60, "extractor lost pages"); + } + + @Test + @DisplayName("splits the tagging pass into its own sub-phases") + void taggingSubPhases() throws Exception { + byte[] input = document(60); + try (PDDocument warm = Loader.loadPDF(input)) { + new TaggedContentExtractor().extract(warm); + } + + long extractMs; + long analyseMs; + long injectMs; + long treeMs; + long saveMs; + + try (PDDocument document = Loader.loadPDF(input)) { + long t = System.nanoTime(); + List pages = new TaggedContentExtractor().extract(document); + extractMs = ms(t); + + t = System.nanoTime(); + DocumentStructure structure = new LayoutAnalyzer().analyse(pages); + analyseMs = ms(t); + + t = System.nanoTime(); + var injector = new stirling.software.proprietary.pdf.ua.MarkedContentInjector(); + var byPage = + new java.util.LinkedHashMap< + Integer, List>(); + structure + .getBlocks() + .forEach( + b -> + byPage.computeIfAbsent(b.getPageIndex(), k -> new ArrayList<>()) + .add(b)); + for (int p = 0; p < document.getNumberOfPages(); p++) { + injector.inject( + document, document.getPage(p), byPage.getOrDefault(p, List.of()), 0, true); + } + injectMs = ms(t); + + t = System.nanoTime(); + new stirling.software.proprietary.pdf.ua.StructTreeWriter() + .write(document, structure, PdfUaProfile.UA1); + treeMs = ms(t); + + t = System.nanoTime(); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + saveMs = ms(t); + } + + System.out.printf( + Locale.ROOT, + "%nTagging sub-phases over 60 pages%n" + + " extract %5d ms%n" + + " analyse %5d ms%n" + + " inject %5d ms%n" + + " struct tree %5d ms%n" + + " save %5d ms%n", + extractMs, + analyseMs, + injectMs, + treeMs, + saveMs); + } + + @Test + @DisplayName("memory stays proportional to document size, not quadratic") + void memoryScales() throws Exception { + List rows = new ArrayList<>(); + long previousPerPage = 0; + boolean blewUp = false; + + for (int pages : new int[] {20, 80, 200}) { + byte[] input = document(pages); + long before = usedHeap(); + var outcome = service.convert(input, options()); + long after = usedHeap(); + long perPageKb = Math.max(after - before, 0) / 1024 / pages; + rows.add( + String.format( + Locale.ROOT, + " %-6d pages in %-9s out %-9s ~%d KB/page retained", + pages, + humanBytes(input.length), + humanBytes(outcome.pdfBytes().length), + perPageKb)); + // Per-page cost should stay roughly flat; a big jump means something accumulates. + if (previousPerPage > 0 && perPageKb > previousPerPage * 4 && perPageKb > 200) { + blewUp = true; + } + previousPerPage = Math.max(perPageKb, 1); + } + System.out.println("\nMemory scaling\n" + String.join("\n", rows)); + assertTrue(!blewUp, "per-page memory grew superlinearly: " + rows); + } + + private static long ms(long startNanos) { + return (System.nanoTime() - startNanos) / 1_000_000; + } + + private static String humanBytes(int bytes) { + return bytes < 1024 * 1024 + ? (bytes / 1024) + " KB" + : String.format(Locale.ROOT, "%.1f MB", bytes / 1024.0 / 1024.0); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaConversionIntegrationTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaConversionIntegrationTest.java new file mode 100644 index 0000000000..e785957b7b --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaConversionIntegrationTest.java @@ -0,0 +1,231 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.concurrent.Callable; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.text.PDFTextStripper; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.model.api.ua.AccessibilityIssue; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.model.api.ua.UaValidationResult; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** + * End-to-end conversion over the fixture corpus, validated with veraPDF. The corpus is deliberately + * varied: what breaks a tagger is rarely the simple case. + */ +class PdfUaConversionIntegrationTest { + + private static PdfUaConversionService service; + + @BeforeAll + static void setUp() { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + service = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + } + + /** Fixtures already embed fonts, so the Ghostscript pass is off to keep tests hermetic. */ + private static TaggingOptions options() { + return TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Test Document") + .embedFonts(false) + .build(); + } + + private static PdfUaConversionOutcome convert(byte[] input) throws IOException { + return service.convert(input, options()); + } + + private static String extractText(byte[] pdf) throws IOException { + try (PDDocument document = Loader.loadPDF(pdf)) { + PDFTextStripper stripper = new PDFTextStripper(); + stripper.setSortByPosition(true); + return PdfUaRealCorpusTest.normalise(stripper.getText(document)); + } + } + + @Test + @DisplayName("every fixture converts without error and gains a structure tree") + void corpusConverts() throws Exception { + Map> corpus = corpus(); + StringBuilder report = new StringBuilder("\nPDF/UA conversion over the fixture corpus\n"); + + for (Map.Entry> entry : corpus.entrySet()) { + byte[] input = + entry.getKey().equals("empty") + ? entry.getValue().call() + : entry.getValue().call(); + PdfUaConversionOutcome outcome = convert(input); + + assertNotNull(outcome.pdfBytes(), entry.getKey() + " produced no output"); + report.append( + String.format( + " %-18s declared=%-5s failures=%-3d elements=%-3d artifacts=%-3d altNeeded=%d%n", + entry.getKey(), + outcome.declared(), + outcome.validation().totalFailures(), + outcome.tagging().taggedElements(), + outcome.tagging().artifacts(), + outcome.tagging().figuresNeedingAltText())); + for (AccessibilityIssue issue : outcome.validation().issues()) { + report.append( + String.format( + " clause %-6s x%-4d %s%n", + issue.getClause(), + issue.getOccurrences(), + issue.getTechnicalMessage())); + } + } + System.out.println(report); + } + + @Test + @DisplayName("tagging never changes the text content of a page") + void textIsPreserved() throws Exception { + for (Map.Entry> entry : corpus().entrySet()) { + byte[] input = entry.getValue().call(); + String before = extractText(input); + String after = extractText(convert(input).pdfBytes()); + assertEquals(before, after, "text changed for fixture " + entry.getKey()); + } + } + + @Test + @DisplayName("a simple document gains a structure tree with headings and paragraphs") + void simpleDocumentIsTagged() throws Exception { + PdfUaConversionOutcome outcome = convert(PdfUaTestDocuments.simpleDocument()); + try (PDDocument document = Loader.loadPDF(outcome.pdfBytes())) { + assertNotNull( + document.getDocumentCatalog().getStructureTreeRoot(), + "no structure tree was written"); + assertTrue( + document.getDocumentCatalog().getMarkInfo() != null + && document.getDocumentCatalog().getMarkInfo().isMarked(), + "MarkInfo/Marked was not set"); + assertEquals("en-GB", document.getDocumentCatalog().getLanguage()); + assertEquals("Test Document", document.getDocumentInformation().getTitle()); + } + assertTrue(outcome.tagging().taggedElements() > 0, "nothing was tagged"); + } + + @Test + @DisplayName("running heads and page numbers become artifacts, not content") + void runningHeadersBecomeArtifacts() throws Exception { + PdfUaConversionOutcome outcome = convert(PdfUaTestDocuments.runningHeadersDocument()); + assertTrue( + outcome.tagging().artifacts() >= 4, + "expected the repeated header on each page to become an artifact, got " + + outcome.tagging().artifacts()); + } + + @Test + @DisplayName("an image is tagged as a figure and reported as needing alt text") + void imagesNeedAltText() throws Exception { + PdfUaConversionOutcome outcome = convert(PdfUaTestDocuments.imageDocument()); + assertEquals(1, outcome.tagging().figuresNeedingAltText()); + assertFalse( + outcome.declared(), + "a document with an undescribed image must not claim conformance"); + } + + @Test + @DisplayName("supplying alt text lets an illustrated document conform") + void suppliedAltTextIsApplied() throws Exception { + byte[] input = PdfUaTestDocuments.imageDocument(); + PdfUaConversionOutcome probe = convert(input); + assertEquals(1, probe.tagging().figuresNeedingAltText()); + + TaggingOptions withAlt = + options().toBuilder() + .altTextByFigure(Map.of(figureKey(input), "A blue rectangle")) + .build(); + PdfUaConversionOutcome outcome = service.convert(input, withAlt); + assertEquals( + 0, + outcome.tagging().figuresNeedingAltText(), + "alt text supplied by the caller was not applied"); + } + + @Test + @DisplayName("an empty document does not crash the converter") + void emptyDocumentIsHandled() throws Exception { + PdfUaConversionOutcome outcome = convert(PdfUaTestDocuments.emptyDocument()); + assertNotNull(outcome.pdfBytes()); + assertFalse(outcome.warnings().isEmpty(), "an empty document should warn"); + } + + @Test + @DisplayName("an un-OCRed scan is reported rather than silently declared conformant") + void scannedDocumentIsNotDeclared() throws Exception { + PdfUaConversionOutcome outcome = convert(PdfUaTestDocuments.scannedDocument()); + assertFalse(outcome.declared(), "a scan with no text layer must not claim conformance"); + } + + @Test + @DisplayName("validation of an untagged document reports the missing structure") + void untaggedDocumentFailsValidation() throws Exception { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + UaValidationResult result = + validation.validate(PdfUaTestDocuments.simpleDocument(), PdfUaProfile.UA1); + assertFalse(result.compliant(), "an untagged document cannot be PDF/UA compliant"); + assertTrue(result.hasIssues()); + } + + /** The key the tagger uses for figure alt text is "pageIndex:firstOrdinal". */ + private static String figureKey(byte[] input) throws IOException { + try (PDDocument document = Loader.loadPDF(input)) { + var pages = + new stirling.software.proprietary.pdf.ua.TaggedContentExtractor() + .extract(document); + for (var page : pages) { + for (var op : page.graphics()) { + return page.pageIndex() + ":" + op.ordinal(); + } + } + } + return "0:0"; + } + + private static Map> corpus() { + Map> corpus = new LinkedHashMap<>(); + corpus.put("simple", PdfUaTestDocuments::simpleDocument); + corpus.put("headings", PdfUaTestDocuments::headingHierarchy); + corpus.put("lists", PdfUaTestDocuments::listDocument); + corpus.put("table", PdfUaTestDocuments::tableDocument); + corpus.put("image", PdfUaTestDocuments::imageDocument); + corpus.put("runningHeads", PdfUaTestDocuments::runningHeadersDocument); + corpus.put("twoColumn", PdfUaTestDocuments::twoColumnDocument); + corpus.put("link", PdfUaTestDocuments::linkDocument); + corpus.put("empty", PdfUaTestDocuments::emptyDocument); + corpus.put("scanned", PdfUaTestDocuments::scannedDocument); + corpus.put("formXObject", PdfUaTestDocuments::formXObjectDocument); + corpus.put("multiStream", PdfUaTestDocuments::multiStreamDocument); + corpus.put("rotated", PdfUaTestDocuments::rotatedDocument); + corpus.put("offsetMediaBox", PdfUaTestDocuments::offsetMediaBoxDocument); + return corpus; + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHardeningTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHardeningTest.java new file mode 100644 index 0000000000..cf72013a13 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHardeningTest.java @@ -0,0 +1,322 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.contentstream.operator.Operator; +import org.apache.pdfbox.pdfparser.PDFStreamParser; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.text.PDFTextStripper; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.pdf.ua.PageContent; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.StructBlock; +import stirling.software.proprietary.pdf.ua.StructType; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** Covers the document shapes and failure modes the first round of tests missed. */ +class PdfUaHardeningTest { + + private static PdfUaConversionService service; + + @BeforeAll + static void setUp() { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + service = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + } + + private static PdfUaConversionOutcome convert(byte[] input) throws IOException { + return service.convert( + input, + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Hardening") + .embedFonts(false) + .build()); + } + + private static String extract(byte[] pdf) throws IOException { + try (PDDocument document = Loader.loadPDF(pdf)) { + PDFTextStripper stripper = new PDFTextStripper(); + stripper.setSortByPosition(true); + return PdfUaRealCorpusTest.normalise(stripper.getText(document)); + } + } + + @Nested + @DisplayName("document shapes") + class Shapes { + + @Test + @DisplayName("text inside a form XObject is attributed to the Do and tagged as prose") + void formXObjectTextIsTagged() throws Exception { + byte[] input = PdfUaTestDocuments.formXObjectDocument(); + assertTrue( + extract(input).contains("inside the form XObject"), + "fixture must actually draw text inside a form"); + + PdfUaConversionOutcome outcome = convert(input); + assertTrue(outcome.declared(), warnings(outcome)); + assertEquals(extract(input), extract(outcome.pdfBytes())); + assertEquals( + 0, + outcome.tagging().figuresNeedingAltText(), + "a form whose text is reachable must not degrade to an undescribed figure"); + } + + @Test + @DisplayName("a page built from multiple content streams converts as one sequence") + void multiStreamPageConverts() throws Exception { + byte[] input = PdfUaTestDocuments.multiStreamDocument(); + PdfUaConversionOutcome outcome = convert(input); + assertTrue(outcome.declared(), warnings(outcome)); + assertEquals(extract(input), extract(outcome.pdfBytes())); + } + + @Test + @DisplayName("a rotated page keeps its text and converts") + void rotatedPageConverts() throws Exception { + byte[] input = PdfUaTestDocuments.rotatedDocument(); + PdfUaConversionOutcome outcome = convert(input); + assertEquals(extract(input), extract(outcome.pdfBytes())); + assertTrue(outcome.tagging().taggedElements() > 0, "rotated text was not tagged"); + } + + @Test + @DisplayName("a MediaBox that does not start at the origin does not break analysis") + void offsetMediaBoxConverts() throws Exception { + byte[] input = PdfUaTestDocuments.offsetMediaBoxDocument(); + PdfUaConversionOutcome outcome = convert(input); + assertTrue(outcome.declared(), warnings(outcome)); + assertEquals(extract(input), extract(outcome.pdfBytes())); + } + + private static String warnings(PdfUaConversionOutcome outcome) { + return "warnings: " + + String.join(" | ", outcome.warnings()) + + " issues: " + + outcome.validation().issues(); + } + } + + @Nested + @DisplayName("pre-marked content") + class PreMarked { + + @Test + @DisplayName("existing BDC/EMC operators are stripped before new ones are written") + void stripsExistingMarkedContent() throws Exception { + byte[] premarked = premarkedDocument(); + PdfUaConversionOutcome outcome = convert(premarked); + + try (PDDocument document = Loader.loadPDF(outcome.pdfBytes())) { + Counts counts = countMarkedContent(document.getPage(0)); + assertEquals( + counts.opens(), + counts.closes(), + "unbalanced marked content after stripping and re-injection"); + assertFalse( + contentOf(document).contains("/OldTag"), + "the source's own marked content survived the rebuild"); + } + assertEquals(extract(premarked), extract(outcome.pdfBytes())); + } + + /** A document whose stream already contains a BDC sequence under a custom tag. */ + private static byte[] premarkedDocument() throws Exception { + byte[] plain = PdfUaTestDocuments.simpleDocument(); + try (PDDocument document = Loader.loadPDF(plain)) { + PDPage page = document.getPage(0); + String content; + try (InputStream in = page.getContents()) { + content = new String(in.readAllBytes(), StandardCharsets.ISO_8859_1); + } + String wrapped = "/OldTag <> BDC\n" + content + "\nEMC\n"; + var stream = new org.apache.pdfbox.pdmodel.common.PDStream(document); + try (var out = stream.createOutputStream()) { + out.write(wrapped.getBytes(StandardCharsets.ISO_8859_1)); + } + page.setContents(stream); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } + } + + private record Counts(int opens, int closes) {} + + private static Counts countMarkedContent(PDPage page) throws IOException { + int opens = 0; + int closes = 0; + PDFStreamParser parser = new PDFStreamParser(page); + Object token; + while ((token = parser.parseNextToken()) != null) { + if (token instanceof Operator operator) { + switch (operator.getName()) { + case "BDC", "BMC" -> opens++; + case "EMC" -> closes++; + default -> {} + } + } + } + return new Counts(opens, closes); + } + + private static String contentOf(PDDocument document) throws IOException { + try (InputStream in = document.getPage(0).getContents()) { + return new String(in.readAllBytes(), StandardCharsets.ISO_8859_1); + } + } + } + + @Nested + @DisplayName("honesty rules") + class Honesty { + + @Test + @DisplayName("suppressed text blocks the conformance claim even when validation passes") + void suppressedTextBlocksDeclaration() { + var structure = new stirling.software.proprietary.pdf.ua.DocumentStructure(); + structure.setTextSuppressed(true); + var result = new stirling.software.proprietary.pdf.ua.TaggingResult(structure, true); + assertTrue( + result.isContentSuppressed(), + "the suppression flag must survive into the tagging result"); + } + + @Test + @DisplayName("dropped lines are reported per page by the analyser") + void analyserWarnsOnDroppedLines() { + PageContent dropped = + new PageContent( + 0, + List.of(), + List.of(), + 5, + false, + false, + true, + new stirling.software.proprietary.pdf.ua.BBox(0, 0, 595, 842)); + var structure = + new stirling.software.proprietary.pdf.ua.LayoutAnalyzer() + .analyse(List.of(dropped)); + assertTrue(structure.isTextSuppressed()); + assertTrue( + structure.getWarnings().stream().anyMatch(w -> w.contains("page(s) 1")), + "warning should name the affected page: " + structure.getWarnings()); + } + } + + @Nested + @DisplayName("clause table") + class Clauses { + + @Test + @DisplayName("subclauses resolve to their parent entry, not to a string prefix") + void subclauseLookupWalksSegments() { + assertNotNull(PdfUaValidationService.lookupClause("7.21.4.1"), "7.21.4.1 -> 7.21"); + assertNotNull(PdfUaValidationService.lookupClause("7.18.1"), "7.18.1 -> 7.18"); + var toUnicode = PdfUaValidationService.lookupClause("7.21.7"); + assertNotNull(toUnicode); + assertFalse( + toUnicode.autoFixable(), + "a missing ToUnicode map is not fixable by embedding fonts"); + assertNull(PdfUaValidationService.lookupClause("9.9.9")); + assertNull(PdfUaValidationService.lookupClause(null)); + } + } + + @Nested + @DisplayName("range claiming") + class Claiming { + + @Test + @DisplayName("a figure drawn between two text runs on one line stays a figure") + void interleavedFigureIsNotSwallowed() throws Exception { + // Words at ordinals 0 and 2 with an image at ordinal 1 between them. + var line = + new stirling.software.proprietary.pdf.ua.TextLineInfo( + 0, + "left right", + new stirling.software.proprietary.pdf.ua.BBox(50, 700, 400, 712), + 11, + false, + 0, + 2, + false, + List.of( + new stirling.software.proprietary.pdf.ua.WordInfo( + "left", + new stirling.software.proprietary.pdf.ua.BBox( + 50, 700, 90, 712), + 0, + 0, + 11, + false), + new stirling.software.proprietary.pdf.ua.WordInfo( + "right", + new stirling.software.proprietary.pdf.ua.BBox( + 360, 700, 400, 712), + 2, + 2, + 11, + false))); + var image = + new stirling.software.proprietary.pdf.ua.MarkableOp( + 1, + stirling.software.proprietary.pdf.ua.MarkableOp.Kind.IMAGE, + new stirling.software.proprietary.pdf.ua.BBox(150, 650, 350, 760), + "Im0"); + PageContent page = + new PageContent( + 0, + List.of(line), + List.of(image), + 3, + false, + false, + false, + new stirling.software.proprietary.pdf.ua.BBox(0, 0, 595, 842)); + + var structure = + new stirling.software.proprietary.pdf.ua.LayoutAnalyzer() + .analyse(List.of(page)); + List figures = new java.util.ArrayList<>(); + structure.visit( + block -> { + if (block.getType() == StructType.FIGURE) { + figures.add(block); + } + }); + assertEquals( + 1, + figures.size(), + "the image between the words must survive as its own figure"); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHttpEndpointTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHttpEndpointTest.java new file mode 100644 index 0000000000..90c40af82b --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaHttpEndpointTest.java @@ -0,0 +1,183 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.multipart; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.header; +import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; + +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.springframework.mock.web.MockMultipartFile; +import org.springframework.test.web.servlet.MockMvc; +import org.springframework.test.web.servlet.MvcResult; +import org.springframework.test.web.servlet.setup.MockMvcBuilders; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.common.util.TempFileManager; +import stirling.software.proprietary.controller.api.converters.ConvertPdfToPdfUa; +import stirling.software.proprietary.controller.api.security.AccessibilityReportController; + +/** + * Exercises the endpoints over HTTP, not through the service layer. Covers route mapping, multipart + * binding, and the headers and JSON a client depends on. + */ +class PdfUaHttpEndpointTest { + + private static MockMvc convertMvc; + private static MockMvc reportMvc; + private static final ObjectMapper JSON = new ObjectMapper(); + + @BeforeAll + static void setUp() throws Exception { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + PdfUaConversionService conversion = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + + // A real TempFileManager, so the streamed response path is exercised rather than mocked. + TempFileManager tempFiles = + new TempFileManager( + new stirling.software.common.util.TempFileRegistry(), + new stirling.software.common.model.ApplicationProperties()); + + // Stand in for the app's advice, which lives in core; without one every rejection is a 500. + var advice = new BadRequestAdvice(); + convertMvc = + MockMvcBuilders.standaloneSetup(new ConvertPdfToPdfUa(conversion, tempFiles)) + .setControllerAdvice(advice) + .build(); + reportMvc = + MockMvcBuilders.standaloneSetup( + new AccessibilityReportController( + new AccessibilityAuditService(validation))) + .setControllerAdvice(advice) + .build(); + } + + private static MockMultipartFile upload(byte[] pdf, String name) { + return new MockMultipartFile("fileInput", name, "application/pdf", pdf); + } + + /** Mirrors the one rule these endpoints rely on: a rejected input is a 400, not a 500. */ + @org.springframework.web.bind.annotation.RestControllerAdvice + static class BadRequestAdvice { + @org.springframework.web.bind.annotation.ExceptionHandler(IllegalArgumentException.class) + org.springframework.http.ResponseEntity badRequest(IllegalArgumentException ex) { + return org.springframework.http.ResponseEntity.badRequest().body(ex.getMessage()); + } + } + + @Test + @DisplayName("POST /api/v1/convert/pdf/ua returns a PDF and reports what it did in headers") + void conversionEndpointResponds() throws Exception { + MvcResult result = + convertMvc + .perform( + multipart("/api/v1/convert/pdf/ua") + .file(upload(PdfUaTestDocuments.simpleDocument(), "in.pdf")) + .param("language", "en-GB") + .param("title", "Over The Wire") + .param("embedFonts", "false")) + .andExpect(status().isOk()) + .andExpect(header().exists("X-Stirling-UA-Declared")) + .andExpect(header().exists("X-Stirling-UA-Failures")) + .andReturn(); + + byte[] body = result.getResponse().getContentAsByteArray(); + assertTrue(body.length > 0, "no document came back"); + assertEquals( + "%PDF", + new String(body, 0, 4, java.nio.charset.StandardCharsets.ISO_8859_1), + "the response body is not a PDF"); + assertEquals( + "true", + result.getResponse().getHeader("X-Stirling-UA-Declared"), + "a simple embedded-font document should convert and be declared conformant"); + } + + @Test + @DisplayName("POST /api/v1/security/accessibility-report returns the figure inventory as JSON") + void reportEndpointResponds() throws Exception { + MvcResult result = + reportMvc + .perform( + multipart("/api/v1/security/accessibility-report") + .file(upload(PdfUaTestDocuments.imageDocument(), "in.pdf")) + .param("profile", "ua1")) + .andExpect(status().isOk()) + .andReturn(); + + JsonNode json = JSON.readTree(result.getResponse().getContentAsString()); + assertEquals("PDF/UA-1", json.get("profile").asText()); + assertNotNull(json.get("summary"), "the report should carry a summary"); + + JsonNode figures = json.get("figuresNeedingDescription"); + assertNotNull(figures, "the field a caller needs to supply alt text is missing"); + assertTrue(figures.isArray() && figures.size() > 0, "the image should be listed: " + json); + assertTrue( + figures.get(0).get("key").asText().matches("\\d+:\\d+"), + "the key must be usable in a follow-up conversion request"); + } + + @Test + @DisplayName("alt text supplied as form data reaches the converter over HTTP") + void altTextBindsFromFormData() throws Exception { + byte[] input = PdfUaTestDocuments.imageDocument(); + + // Discover the key the way a client would, through the report endpoint. + MvcResult reported = + reportMvc + .perform( + multipart("/api/v1/security/accessibility-report") + .file(upload(input, "in.pdf"))) + .andExpect(status().isOk()) + .andReturn(); + String key = + JSON.readTree(reported.getResponse().getContentAsString()) + .get("figuresNeedingDescription") + .get(0) + .get("key") + .asText(); + + MvcResult converted = + convertMvc + .perform( + multipart("/api/v1/convert/pdf/ua") + .file(upload(input, "in.pdf")) + .param("embedFonts", "false") + .param("altText", key + "=A blue rectangle")) + .andExpect(status().isOk()) + .andReturn(); + + assertEquals( + "0", + converted.getResponse().getHeader("X-Stirling-UA-Figures-Needing-Alt"), + "the description posted as form data was not applied"); + } + + @Test + @DisplayName("a request with no file is rejected rather than processed") + void missingFileIsRejected() throws Exception { + convertMvc + .perform( + multipart("/api/v1/convert/pdf/ua") + .file( + new MockMultipartFile( + "fileInput", + "e.pdf", + "application/pdf", + new byte[0]))) + .andExpect(status().is4xxClientError()); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaRealCorpusTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaRealCorpusTest.java new file mode 100644 index 0000000000..9ebda08736 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaRealCorpusTest.java @@ -0,0 +1,249 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.stream.Stream; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.text.PDFTextStripper; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** + * Runs the converter over every PDF in the repository, where real-tool output breaks assumptions. A + * clean refusal counts as a pass; nothing may crash or make a false conformance claim. + */ +class PdfUaRealCorpusTest { + + private static PdfUaConversionService service; + private static Path repoRoot; + + /** Files the converter is expected to refuse rather than process. */ + private static final List EXPECTED_REJECTS = List.of("encrypted.pdf", "corrupted.pdf"); + + @BeforeAll + static void setUp() { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + service = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + repoRoot = findRepoRoot(); + } + + private static Path findRepoRoot() { + Path current = Path.of("").toAbsolutePath(); + while (current != null && !Files.exists(current.resolve("settings.gradle"))) { + current = current.getParent(); + } + return current; + } + + private record Outcome(String name, String status, int failures, int elements, int artifacts) {} + + private static TaggingOptions.TaggingOptionsBuilder options(String fallbackTitle) { + return TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .fallbackTitle(fallbackTitle) + .existingTags(TaggingOptions.ExistingTags.REBUILD); + } + + @Test + @DisplayName("converts every PDF in the repository without crashing or lying about conformance") + void realCorpusConverts() throws Exception { + assertNotNull(repoRoot, "could not locate the repository root"); + List pdfs = findPdfs(); + assertTrue(pdfs.size() >= 20, "expected a substantial corpus, found " + pdfs.size()); + + List outcomes = new ArrayList<>(); + List crashes = new ArrayList<>(); + java.util.Map clauseFiles = new java.util.TreeMap<>(); + java.util.Map clauseText = new java.util.HashMap<>(); + java.util.Map clauseExamples = new java.util.HashMap<>(); + + for (Path pdf : pdfs) { + String name = repoRoot.relativize(pdf).toString().replace('\\', '/'); + byte[] input; + try { + input = Files.readAllBytes(pdf); + } catch (IOException e) { + continue; + } + try { + String stem = pdf.getFileName().toString().replaceFirst("\\.pdf$", ""); + + // Fidelity is a tagging property, so measure it with font embedding off. + PdfUaConversionOutcome taggedOnly = + service.convert(input, options(stem).embedFonts(false).build()); + assertTextPreserved(name, input, taggedOnly.pdfBytes()); + + PdfUaConversionOutcome outcome = service.convert(input, options(stem).build()); + // Full pipeline too: Ghostscript can exit 0 having blanked the document. + assertTextPreserved(name + " (with font embedding)", input, outcome.pdfBytes()); + outcomes.add( + new Outcome( + name, + outcome.declared() ? "CONFORMS" : "improved", + outcome.validation().totalFailures(), + outcome.tagging().taggedElements(), + outcome.tagging().artifacts())); + outcome.validation() + .issues() + .forEach( + issue -> { + clauseFiles.merge(issue.getClause(), 1, Integer::sum); + clauseText.putIfAbsent( + issue.getClause(), issue.getTechnicalMessage()); + clauseExamples.putIfAbsent(issue.getClause(), name); + }); + + } catch (IOException e) { + // A refusal with an explanation is an acceptable outcome. + outcomes.add(new Outcome(name, "refused: " + e.getMessage(), 0, 0, 0)); + } catch (RuntimeException e) { + crashes.add(name + " -> " + e); + outcomes.add(new Outcome(name, "CRASH: " + e, 0, 0, 0)); + } + } + + System.out.println(render(outcomes)); + + StringBuilder clauses = + new StringBuilder("\nBlocking clauses, by number of files affected\n"); + clauseFiles.entrySet().stream() + .sorted(java.util.Map.Entry.comparingByValue().reversed()) + .forEach( + e -> + clauses.append( + String.format( + " clause %-9s %-3d files e.g. %s%n %s%n", + e.getKey(), + e.getValue(), + clauseExamples.get(e.getKey()), + abbreviate(clauseText.get(e.getKey()))))); + System.out.println(clauses); + + List unexpectedCrashes = + crashes.stream() + .filter(c -> EXPECTED_REJECTS.stream().noneMatch(c::contains)) + .toList(); + assertTrue( + unexpectedCrashes.isEmpty(), + "converter crashed on: " + String.join("; ", unexpectedCrashes)); + } + + /** Tagging must not change extracted text; a diff means the rewrite corrupted the page. */ + private static void assertTextPreserved(String name, byte[] before, byte[] after) { + String textBefore = safeExtract(before); + if (textBefore == null) { + // The source itself is unreadable, so there is nothing to compare against. + return; + } + String textAfter = safeExtract(after); + assertTrue( + textAfter != null, "the converted file could not be read back at all for " + name); + assertTrue( + textBefore.equals(textAfter), + "tagging changed extracted text for " + + name + + "\n before: " + + preview(textBefore) + + "\n after: " + + preview(textAfter)); + } + + private static String abbreviate(String text) { + if (text == null) { + return ""; + } + return text.length() <= 110 ? text : text.substring(0, 110) + "..."; + } + + private static String preview(String text) { + return text.length() <= 160 ? text : text.substring(0, 160) + "..."; + } + + private static String safeExtract(byte[] pdf) { + try (PDDocument document = Loader.loadPDF(pdf)) { + PDFTextStripper stripper = new PDFTextStripper(); + stripper.setSortByPosition(true); + return normalise(stripper.getText(document)); + } catch (Exception e) { + return null; + } + } + + /** Drops invisible formatting characters: a rebuild legitimately loses soft hyphens. */ + static String normalise(String text) { + StringBuilder sb = new StringBuilder(text.length()); + text.codePoints() + .forEach( + cp -> { + if (Character.getType(cp) != Character.FORMAT && cp != 0x00AD) { + sb.appendCodePoint(cp); + } + }); + return sb.toString().replaceAll("\\s+", " ").strip(); + } + + private List findPdfs() throws IOException { + try (Stream stream = Files.walk(repoRoot)) { + return stream.filter(Files::isRegularFile) + .filter(p -> p.toString().toLowerCase().endsWith(".pdf")) + .filter(p -> !p.toString().contains("node_modules")) + .filter(p -> !p.toString().contains(File_BUILD)) + .filter(p -> !p.toString().contains(".git")) + .sorted(Comparator.comparing(Path::toString)) + .toList(); + } + } + + private static final String File_BUILD = "build" + java.io.File.separator; + + private static String render(List outcomes) { + StringBuilder sb = new StringBuilder("\nPDF/UA conversion over the repository corpus\n"); + long conforming = outcomes.stream().filter(o -> "CONFORMS".equals(o.status())).count(); + long refused = outcomes.stream().filter(o -> o.status().startsWith("refused")).count(); + sb.append( + String.format( + " %d files: %d conform, %d improved but not conformant, %d refused%n%n", + outcomes.size(), + conforming, + outcomes.size() - conforming - refused, + refused)); + for (Outcome outcome : outcomes) { + sb.append( + String.format( + " %-62s %-10s fail=%-4d el=%-5d art=%d%n", + outcome.name().length() > 60 + ? "..." + outcome.name().substring(outcome.name().length() - 57) + : outcome.name(), + outcome.status().length() > 10 + ? outcome.status().substring(0, 10) + : outcome.status(), + outcome.failures(), + outcome.elements(), + outcome.artifacts())); + } + return sb.toString(); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaSampleDumpTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaSampleDumpTest.java new file mode 100644 index 0000000000..46cdb37c76 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaSampleDumpTest.java @@ -0,0 +1,103 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Stream; + +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** + * Dumps converted output for independent checkers; validating PDFBox with PDFBox is circular. Off + * by default as it writes outside the build directory: run with {@code DUMP_UA_SAMPLES=}. + */ +@EnabledIfEnvironmentVariable(named = "DUMP_UA_SAMPLES", matches = ".+") +class PdfUaSampleDumpTest { + + private static PdfUaConversionService service; + private static Path repoRoot; + + @BeforeAll + static void setUp() { + PdfUaValidationService validation = new PdfUaValidationService(); + validation.initialise(); + service = + new PdfUaConversionService( + validation, + new FontEmbeddingService(), + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + repoRoot = Path.of("").toAbsolutePath(); + while (repoRoot != null && !Files.exists(repoRoot.resolve("settings.gradle"))) { + repoRoot = repoRoot.getParent(); + } + } + + @Test + @DisplayName("writes original and converted pairs for external validation") + void dumpSamples() throws Exception { + Path out = Path.of(System.getenv("DUMP_UA_SAMPLES")); + Files.createDirectories(out); + + List pdfs; + try (Stream stream = Files.walk(repoRoot)) { + pdfs = + stream.filter(Files::isRegularFile) + .filter(p -> p.toString().toLowerCase().endsWith(".pdf")) + .filter(p -> !p.toString().contains("node_modules")) + .filter(p -> !p.toString().contains(java.io.File.separator + "build")) + .filter(p -> !p.toString().contains(".git")) + .sorted() + .toList(); + } + + List manifest = new ArrayList<>(); + int written = 0; + for (Path pdf : pdfs) { + String stem = pdf.getFileName().toString().replaceFirst("\\.pdf$", ""); + byte[] input; + try { + input = Files.readAllBytes(pdf); + } catch (Exception e) { + continue; + } + try { + PdfUaConversionOutcome outcome = + service.convert( + input, + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .fallbackTitle(stem) + .existingTags(TaggingOptions.ExistingTags.REBUILD) + .build()); + Files.write(out.resolve(stem + "__before.pdf"), input); + Files.write(out.resolve(stem + "__after.pdf"), outcome.pdfBytes()); + manifest.add( + stem + + "\tdeclared=" + + outcome.declared() + + "\tfailures=" + + outcome.validation().totalFailures()); + written++; + } catch (Exception e) { + manifest.add(stem + "\tREFUSED\t" + e.getMessage()); + } + } + Files.write(out.resolve("manifest.tsv"), manifest); + System.out.println("Wrote " + written + " before/after pairs to " + out); + assertTrue(written > 10, "expected a usable sample set, wrote " + written); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaServicesTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaServicesTest.java new file mode 100644 index 0000000000..9d2e14ee0d --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaServicesTest.java @@ -0,0 +1,319 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDPageContentStream; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.encryption.AccessPermission; +import org.apache.pdfbox.pdmodel.encryption.StandardProtectionPolicy; +import org.apache.pdfbox.pdmodel.font.PDType1Font; +import org.apache.pdfbox.pdmodel.font.Standard14Fonts; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import stirling.software.common.service.CustomPDFDocumentFactory; +import stirling.software.common.service.PdfMetadataService; +import stirling.software.proprietary.model.api.ua.AccessibilityReport; +import stirling.software.proprietary.model.api.ua.PdfUaConversionOutcome; +import stirling.software.proprietary.model.api.ua.UaValidationResult; +import stirling.software.proprietary.pdf.ua.PdfUaProfile; +import stirling.software.proprietary.pdf.ua.TaggingOptions; + +/** Tests for the services that validate, audit and convert. */ +class PdfUaServicesTest { + + private static PdfUaValidationService validation; + private static PdfUaConversionService conversion; + private static AccessibilityAuditService audit; + private static FontEmbeddingService fonts; + + @BeforeAll + static void setUp() { + validation = new PdfUaValidationService(); + validation.initialise(); + fonts = new FontEmbeddingService(); + conversion = + new PdfUaConversionService( + validation, + fonts, + new CustomPDFDocumentFactory( + org.mockito.Mockito.mock(PdfMetadataService.class))); + audit = new AccessibilityAuditService(validation); + } + + /** Uses a standard 14 font deliberately: never embedded, which clause 7.21 forbids. */ + private static byte[] unembeddedFontPdf() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + cs.beginText(); + cs.setFont(new PDType1Font(Standard14Fonts.FontName.HELVETICA), 12); + cs.newLineAtOffset(50, 700); + cs.showText("Hello accessibility"); + cs.endText(); + } + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } + } + + private static byte[] manyPages(int pages) throws IOException { + try (PDDocument document = new PDDocument()) { + for (int i = 0; i < pages; i++) { + document.addPage(new PDPage()); + } + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } + } + + private static byte[] encryptedPdf() throws IOException { + try (PDDocument document = new PDDocument()) { + document.addPage(new PDPage()); + AccessPermission permissions = new AccessPermission(); + document.protect(new StandardProtectionPolicy("owner", "user", permissions)); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } + } + + @Nested + @DisplayName("validation") + class Validation { + + @Test + @DisplayName("an untagged document fails and the failures are grouped by rule") + void untaggedFails() throws Exception { + UaValidationResult result = validation.validate(unembeddedFontPdf(), PdfUaProfile.UA1); + assertFalse(result.compliant()); + assertTrue(result.hasIssues()); + assertTrue( + result.totalFailures() >= result.issues().size(), + "grouping must not invent failures"); + assertTrue( + result.issues().stream().allMatch(i -> i.getOccurrences() > 0), + "every grouped issue should count its occurrences"); + } + + @Test + @DisplayName("malformed input reports a failure instead of throwing") + void malformedInputIsReported() { + UaValidationResult result = + validation.validate("not a pdf".getBytes(), PdfUaProfile.UA1); + assertFalse(result.compliant()); + assertFalse(result.issues().isEmpty()); + } + + @Test + @DisplayName("issues carry plain-English text as well as the validator's own wording") + void issuesAreReadable() throws Exception { + UaValidationResult result = validation.validate(unembeddedFontPdf(), PdfUaProfile.UA1); + assertTrue( + result.issues().stream() + .allMatch(i -> i.getMessage() != null && !i.getMessage().isBlank())); + } + } + + @Nested + @DisplayName("auditing") + class Auditing { + + @Test + @DisplayName("reports the document facts that drive most failures") + void reportsSummary() throws Exception { + AccessibilityReport report = audit.audit(unembeddedFontPdf(), PdfUaProfile.UA1); + + assertFalse(report.isTagged(), "the fixture has no structure tree"); + assertFalse(report.isDeclaresConformance()); + assertFalse(report.isPassesAutomatedChecks()); + assertEquals(1, report.getSummary().getPages()); + assertFalse(report.getSummary().isAllFontsEmbedded()); + assertTrue(report.getSummary().getUnembeddedFonts() > 0); + assertFalse(report.getSummary().isHasLanguage()); + assertFalse(report.getSummary().isHasTitle()); + } + + @Test + @DisplayName("always lists the checks a person still has to make") + void listsHumanChecks() throws Exception { + AccessibilityReport report = audit.audit(unembeddedFontPdf(), PdfUaProfile.UA1); + assertFalse( + report.getHumanChecks().isEmpty(), + "a report showing only automated results implies the rest does not exist"); + } + + @Test + @DisplayName("splits failures into automatically fixable and needs-input") + void splitsRemediability() throws Exception { + AccessibilityReport report = audit.audit(unembeddedFontPdf(), PdfUaProfile.UA1); + assertEquals( + report.getIssues().size(), + report.getAutomaticallyFixable() + report.getNeedsInput()); + } + + @Test + @DisplayName("refuses a document past the page cap the conversion also applies") + void refusesTooManyPages() throws Exception { + byte[] oversized = manyPages(2001); + assertThrows( + IllegalArgumentException.class, + () -> audit.audit(oversized, PdfUaProfile.UA1), + "an uncapped report walks every page of any document a caller uploads"); + } + + @Test + @DisplayName("a converted document reports as tagged and conformant") + void reportsAfterConversion() throws Exception { + PdfUaConversionOutcome outcome = + conversion.convert( + unembeddedFontPdf(), + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Converted") + .build()); + AccessibilityReport report = audit.audit(outcome.pdfBytes(), PdfUaProfile.UA1); + assertTrue(report.isTagged()); + assertTrue(report.getSummary().isHasTitle()); + assertTrue(report.getSummary().isHasLanguage()); + assertTrue(report.getSummary().isDisplaysDocTitle()); + } + } + + @Nested + @DisplayName("font embedding") + class Fonts { + + @Test + @DisplayName("detects a standard 14 font as unembedded") + void detectsUnembedded() throws Exception { + assertTrue(fonts.hasUnembeddedFonts(unembeddedFontPdf())); + } + + @Test + @DisplayName("embeds fonts, or explains why it could not") + void embedsOrExplains() throws Exception { + FontEmbeddingService.Result result = fonts.embedFonts(unembeddedFontPdf()); + assertNotNull(result.pdfBytes()); + if (result.changed()) { + assertFalse( + fonts.hasUnembeddedFonts(result.pdfBytes()), + "embedding reported success but fonts are still missing"); + } else { + assertNotNull( + result.warning(), "failing to embed must be explained, not passed over"); + } + } + + @Test + @DisplayName("leaves a document alone when every font is already embedded") + void skipsWhenNothingToDo() throws Exception { + byte[] embedded = PdfUaTestDocuments.simpleDocument(); + FontEmbeddingService.Result result = fonts.embedFonts(embedded); + assertFalse(result.changed()); + assertEquals(embedded.length, result.pdfBytes().length); + } + } + + @Nested + @DisplayName("conversion") + class Conversion { + + @Test + @DisplayName("refuses an encrypted document with an explanation") + void refusesEncrypted() throws Exception { + byte[] encrypted = encryptedPdf(); + IOException error = + assertThrows( + IOException.class, + () -> + conversion.convert( + encrypted, + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .build())); + assertTrue(error.getMessage().toLowerCase().contains("encrypted")); + } + + @Test + @DisplayName("keeping existing tags does not rebuild the tree") + void keepRespectsExistingTags() throws Exception { + byte[] tagged = + conversion + .convert( + PdfUaTestDocuments.simpleDocument(), + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("First pass") + .embedFonts(false) + .build()) + .pdfBytes(); + + PdfUaConversionOutcome second = + conversion.convert( + tagged, + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Second pass") + .embedFonts(false) + .existingTags(TaggingOptions.ExistingTags.KEEP) + .build()); + + assertFalse(second.tagging().rebuiltStructure(), "KEEP must not rebuild"); + try (PDDocument document = Loader.loadPDF(second.pdfBytes())) { + assertEquals("Second pass", document.getDocumentInformation().getTitle()); + } + } + + @Test + @DisplayName("marking images decorative removes the alt-text blocker") + void decorativePolicyClearsFigures() throws Exception { + PdfUaConversionOutcome outcome = + conversion.convert( + PdfUaTestDocuments.imageDocument(), + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Decorative") + .embedFonts(false) + .figurePolicy(TaggingOptions.FigurePolicy.MARK_DECORATIVE) + .build()); + assertEquals(0, outcome.tagging().figuresNeedingAltText()); + assertTrue(outcome.declared(), "with no undescribed figures the file should conform"); + } + + @Test + @DisplayName("converting twice produces the same conformance verdict") + void conversionIsStable() throws Exception { + byte[] input = PdfUaTestDocuments.headingHierarchy(); + TaggingOptions options = + TaggingOptions.builder() + .profile(PdfUaProfile.UA1) + .language("en-GB") + .title("Stable") + .embedFonts(false) + .build(); + PdfUaConversionOutcome first = conversion.convert(input, options); + PdfUaConversionOutcome second = conversion.convert(first.pdfBytes(), options); + assertEquals(first.declared(), second.declared()); + } + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaTestDocuments.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaTestDocuments.java new file mode 100644 index 0000000000..589edbc1a2 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfUaTestDocuments.java @@ -0,0 +1,390 @@ +package stirling.software.proprietary.service.ua; + +import java.awt.Color; +import java.awt.image.BufferedImage; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDFormContentStream; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDPageContentStream; +import org.apache.pdfbox.pdmodel.PDResources; +import org.apache.pdfbox.pdmodel.common.PDRectangle; +import org.apache.pdfbox.pdmodel.font.PDFont; +import org.apache.pdfbox.pdmodel.font.PDType0Font; +import org.apache.pdfbox.pdmodel.graphics.form.PDFormXObject; +import org.apache.pdfbox.pdmodel.graphics.image.LosslessFactory; +import org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject; +import org.apache.pdfbox.pdmodel.interactive.action.PDActionURI; +import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotationLink; +import org.apache.pdfbox.util.Matrix; + +/** + * Builds the fixture corpus used by the PDF/UA tests. Fonts are embedded deliberately: the standard + * 14 fail clause 7.21 and would mask every result. + */ +final class PdfUaTestDocuments { + + private static final String FONT_RESOURCE = "/static/fonts/DejaVuSans.ttf"; + // The font ships with core's resources, which are not on this module's classpath. + private static final String FONT_REPO_PATH = + "app/core/src/main/resources/static/fonts/DejaVuSans.ttf"; + private static final float MARGIN = 60f; + + private PdfUaTestDocuments() {} + + static PDFont font(PDDocument document) throws IOException { + try (InputStream in = PdfUaTestDocuments.class.getResourceAsStream(FONT_RESOURCE)) { + if (in != null) { + return PDType0Font.load(document, in, true); + } + } + Path repoRoot = Path.of("").toAbsolutePath(); + while (repoRoot != null && !Files.exists(repoRoot.resolve("settings.gradle"))) { + repoRoot = repoRoot.getParent(); + } + Path font = repoRoot == null ? null : repoRoot.resolve(FONT_REPO_PATH); + if (font == null || !Files.exists(font)) { + throw new IOException( + "Test font not found: " + FONT_RESOURCE + " or " + FONT_REPO_PATH); + } + try (InputStream in = Files.newInputStream(font)) { + return PDType0Font.load(document, in, true); + } + } + + /** A heading followed by two paragraphs: the simplest thing that should convert cleanly. */ + static byte[] simpleDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 760; + y = text(cs, font, 20, MARGIN, y, "Quarterly Report"); + y -= 14; + y = text(cs, font, 11, MARGIN, y, "This document summarises the results for the"); + y = text(cs, font, 11, MARGIN, y, "period and outlines the outlook for next year."); + y -= 14; + text(cs, font, 11, MARGIN, y, "A second paragraph follows the first one here."); + } + return bytes(document); + } + } + + /** Three heading tiers, to exercise level assignment and the no-skipped-levels rule. */ + static byte[] headingHierarchy() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 760; + y = text(cs, font, 24, MARGIN, y, "Annual Review"); + y -= 12; + y = text(cs, font, 11, MARGIN, y, "Introductory prose sits under the title here."); + y -= 16; + y = text(cs, font, 17, MARGIN, y, "Financial Results"); + y -= 10; + y = text(cs, font, 11, MARGIN, y, "Revenue grew steadily across every region."); + y -= 16; + y = text(cs, font, 13, MARGIN, y, "Europe"); + y -= 10; + text(cs, font, 11, MARGIN, y, "European revenue rose by eleven per cent."); + } + return bytes(document); + } + } + + /** A bulleted and a numbered list. */ + static byte[] listDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 760; + y = text(cs, font, 18, MARGIN, y, "Checklist"); + y -= 14; + y = text(cs, font, 11, MARGIN, y, "• Review the source document"); + y = text(cs, font, 11, MARGIN, y, "• Check every heading level"); + y = text(cs, font, 11, MARGIN, y, "• Describe each image"); + y -= 16; + y = text(cs, font, 11, MARGIN, y, "1. Open the file"); + y = text(cs, font, 11, MARGIN, y, "2. Run the converter"); + text(cs, font, 11, MARGIN, y, "3. Validate the result"); + } + return bytes(document); + } + } + + /** A three-column table whose cells each occupy their own text-showing operator. */ + static byte[] tableDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 760; + y = text(cs, font, 18, MARGIN, y, "Regional Totals"); + y -= 20; + String[][] rows = { + {"Region", "Units", "Revenue"}, + {"North", "1200", "48000"}, + {"South", "980", "39200"}, + {"East", "1430", "57200"} + }; + float[] columns = {MARGIN, MARGIN + 160, MARGIN + 300}; + for (String[] row : rows) { + tableRow(cs, font, 11, columns, y, row); + y -= 20; + } + } + return bytes(document); + } + } + + /** A page with a real image, which must end up as a Figure needing alternative text. */ + static byte[] imageDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + BufferedImage bitmap = new BufferedImage(120, 90, BufferedImage.TYPE_INT_RGB); + java.awt.Graphics2D graphics = bitmap.createGraphics(); + graphics.setColor(Color.BLUE); + graphics.fillRect(0, 0, 120, 90); + graphics.dispose(); + PDImageXObject image = LosslessFactory.createFromImage(document, bitmap); + + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 760; + y = text(cs, font, 18, MARGIN, y, "Illustrated Page"); + y -= 20; + y = text(cs, font, 11, MARGIN, y, "The chart below shows the trend."); + cs.drawImage(image, MARGIN, y - 120, 180, 100); + } + return bytes(document); + } + } + + /** Four pages sharing a running head and a page number, which must become artifacts. */ + static byte[] runningHeadersDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDFont font = null; + for (int i = 1; i <= 4; i++) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + if (font == null) { + font = font(document); + } + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + text(cs, font, 9, MARGIN, 810, "Confidential Internal Report"); + float y = 750; + y = text(cs, font, 16, MARGIN, y, "Section " + i); + y -= 12; + text(cs, font, 11, MARGIN, y, "Body text for section number " + i + " here."); + text(cs, font, 9, 300, 30, "Page " + i); + } + } + return bytes(document); + } + } + + /** Two columns of prose, to exercise reading order. */ + static byte[] twoColumnDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float left = MARGIN; + float right = 320; + float y = 740; + for (int i = 1; i <= 8; i++) { + text(cs, font, 10, left, y - i * 16, "Left column line number " + i); + text(cs, font, 10, right, y - i * 16, "Right column line number " + i); + } + } + return bytes(document); + } + } + + /** A page carrying a link annotation, which must be reachable from the structure tree. */ + static byte[] linkDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + float y = 760; + y = text(cs, font, 18, MARGIN, y, "Useful Links"); + text(cs, font, 11, MARGIN, y - 20, "Visit the project home page for details."); + } + PDAnnotationLink link = new PDAnnotationLink(); + PDRectangle rectangle = new PDRectangle(); + rectangle.setLowerLeftX(MARGIN); + rectangle.setLowerLeftY(725); + rectangle.setUpperRightX(MARGIN + 200); + rectangle.setUpperRightY(740); + link.setRectangle(rectangle); + PDActionURI action = new PDActionURI(); + action.setURI("https://example.org"); + link.setAction(action); + page.getAnnotations().add(link); + return bytes(document); + } + } + + /** A page with no content at all. */ + static byte[] emptyDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + document.addPage(new PDPage(PDRectangle.A4)); + return bytes(document); + } + } + + /** A page whose only content is a full-page image, standing in for an un-OCRed scan. */ + static byte[] scannedDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + BufferedImage bitmap = new BufferedImage(600, 850, BufferedImage.TYPE_INT_RGB); + java.awt.Graphics2D graphics = bitmap.createGraphics(); + graphics.setColor(Color.WHITE); + graphics.fillRect(0, 0, 600, 850); + graphics.setColor(Color.BLACK); + graphics.drawString("scanned page", 40, 60); + graphics.dispose(); + PDImageXObject image = LosslessFactory.createFromImage(document, bitmap); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + cs.drawImage(image, 0, 0, PDRectangle.A4.getWidth(), PDRectangle.A4.getHeight()); + } + return bytes(document); + } + } + + /** Text drawn inside a form XObject, attributed to the Do operator that invoked it. */ + static byte[] formXObjectDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + + PDFormXObject form = new PDFormXObject(document); + form.setBBox(new PDRectangle(220, 40)); + form.setResources(new PDResources()); + try (PDFormContentStream fcs = new PDFormContentStream(form)) { + fcs.beginText(); + fcs.setFont(font, 11); + fcs.newLineAtOffset(4, 14); + fcs.showText("Text living inside the form XObject"); + fcs.endText(); + } + + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + text(cs, font, 18, MARGIN, 760, "Page With Embedded Form"); + text(cs, font, 11, MARGIN, 730, "Ordinary page text sits above the form."); + cs.saveGraphicsState(); + cs.transform(Matrix.getTranslateInstance(MARGIN, 650)); + cs.drawForm(form); + cs.restoreGraphicsState(); + } + return bytes(document); + } + } + + /** Content split across two streams (a PDF array) - the parser must see one sequence. */ + static byte[] multiStreamDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + text(cs, font, 18, MARGIN, 760, "First Stream Heading"); + } + try (PDPageContentStream cs = + new PDPageContentStream( + document, page, PDPageContentStream.AppendMode.APPEND, true)) { + text(cs, font, 11, MARGIN, 720, "Second stream paragraph appended later."); + } + return bytes(document); + } + } + + /** A landscape page via /Rotate 90, which flips the frame the text engine reports in. */ + static byte[] rotatedDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(PDRectangle.A4); + page.setRotation(90); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + // Drawn rotated so the text reads upright on the rotated page. + cs.transform(Matrix.getRotateInstance(Math.toRadians(90), 595, 0)); + text(cs, font, 18, MARGIN, 500, "Rotated Page Title"); + text(cs, font, 11, MARGIN, 470, "Body text on a landscape page."); + } + return bytes(document); + } + } + + /** A MediaBox whose origin is not (0,0), which some scanners produce. */ + static byte[] offsetMediaBoxDocument() throws IOException { + try (PDDocument document = new PDDocument()) { + PDPage page = new PDPage(new PDRectangle(100, 200, 595, 842)); + document.addPage(page); + PDFont font = font(document); + try (PDPageContentStream cs = new PDPageContentStream(document, page)) { + text(cs, font, 18, 160, 960, "Offset Origin Title"); + text(cs, font, 11, 160, 930, "Text on a page whose MediaBox starts at 100,200."); + } + return bytes(document); + } + } + + // --- helpers ----------------------------------------------------------- + + private static float text( + PDPageContentStream cs, PDFont font, float size, float x, float y, String value) + throws IOException { + cs.beginText(); + cs.setFont(font, size); + cs.newLineAtOffset(x, y); + cs.showText(value); + cs.endText(); + return y - size * 1.35f; + } + + /** + * Emits one row with a separate show-text operator per cell, so each cell gets its own MCID. + */ + private static void tableRow( + PDPageContentStream cs, + PDFont font, + float size, + float[] columns, + float y, + String[] values) + throws IOException { + cs.beginText(); + cs.setFont(font, size); + cs.newLineAtOffset(columns[0], y); + cs.showText(values[0]); + for (int i = 1; i < values.length; i++) { + cs.newLineAtOffset(columns[i] - columns[i - 1], 0); + cs.showText(values[i]); + } + cs.endText(); + } + + private static byte[] bytes(PDDocument document) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + document.save(out); + return out.toByteArray(); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfaLevelATest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfaLevelATest.java new file mode 100644 index 0000000000..367b9d1171 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/PdfaLevelATest.java @@ -0,0 +1,202 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.verapdf.pdfa.Foundries; +import org.verapdf.pdfa.PDFAParser; +import org.verapdf.pdfa.flavours.PDFAFlavour; +import org.verapdf.pdfa.results.TestAssertion; +import org.verapdf.pdfa.results.ValidationResult; + +/** + * Proves tagging raises a PDF/A file from level B to the accessible level A. veraPDF is the + * arbiter: the claim only counts if the validator agrees. + */ +class PdfaLevelATest { + + private static PdfaAccessibilityService service; + private static Path repoRoot; + + @BeforeAll + static void setUp() { + PdfUaValidationService uaValidation = new PdfUaValidationService(); + uaValidation.initialise(); + service = new PdfaAccessibilityService(uaValidation); + repoRoot = Path.of("").toAbsolutePath(); + while (repoRoot != null && !Files.exists(repoRoot.resolve("settings.gradle"))) { + repoRoot = repoRoot.getParent(); + } + } + + private static byte[] fixture(String name) throws Exception { + return Files.readAllBytes( + repoRoot.resolve("app/core/src/test/resources/pdfa").resolve(name)); + } + + private static String xmpOf(byte[] pdf) throws Exception { + try (PDDocument document = Loader.loadPDF(pdf)) { + var metadata = document.getDocumentCatalog().getMetadata(); + assertNotNull(metadata, "no XMP packet"); + return new String(metadata.toByteArray(), StandardCharsets.UTF_8); + } + } + + /** The flavour the file declares in its XMP, which is what a validator picks up by itself. */ + private static String declaredStandard(byte[] pdf) throws Exception { + try (PDFAParser parser = + Foundries.defaultInstance().createParser(new ByteArrayInputStream(pdf))) { + List flavours = parser.getFlavours(); + return flavours == null || flavours.isEmpty() ? null : flavours.get(0).getId(); + } + } + + private static ValidationResult validate(byte[] pdf, PDFAFlavour flavour) throws Exception { + try (PDFAParser parser = + Foundries.defaultInstance().createParser(new ByteArrayInputStream(pdf), flavour)) { + return Foundries.defaultInstance().createValidator(flavour, false).validate(parser); + } + } + + private static List failures(ValidationResult result) { + return result.getTestAssertions().stream() + .filter(assertion -> assertion.getStatus() == TestAssertion.Status.FAILED) + .map(TestAssertion::getMessage) + .toList(); + } + + @Test + @DisplayName("a level B file gains a structure tree and a conformance A claim") + void upgradesLevelBToLevelA() throws Exception { + byte[] levelB = fixture("valid-pdfa-2b.pdf"); + + try (PDDocument before = Loader.loadPDF(levelB)) { + assertEquals( + null, + before.getDocumentCatalog().getStructureTreeRoot(), + "the fixture should start untagged, or the test proves nothing"); + } + + PdfaAccessibilityService.Result result = + service.upgradeToLevelA(levelB, 2, "en-GB", "Archived Report"); + assertTrue(result.levelA(), "upgrade failed: " + result.warnings()); + + try (PDDocument after = Loader.loadPDF(result.pdfBytes())) { + assertNotNull( + after.getDocumentCatalog().getStructureTreeRoot(), "no structure tree written"); + assertTrue(after.getDocumentCatalog().getMarkInfo().isMarked()); + assertEquals("en-GB", after.getDocumentCatalog().getLanguage()); + } + + String xmp = xmpOf(result.pdfBytes()); + assertTrue(xmp.contains("part"), "pdfaid:part missing"); + assertTrue( + xmp.contains(">A<") || xmp.contains("conformance=\"A\""), + "conformance was not raised to A: " + xmp); + } + + @Test + @DisplayName("the upgraded file still validates as PDF/A, now at level A") + void upgradedFileStillValidates() throws Exception { + byte[] levelB = fixture("valid-pdfa-2b.pdf"); + PdfaAccessibilityService.Result result = + service.upgradeToLevelA(levelB, 2, "en-GB", "Archived Report"); + assertTrue(result.levelA(), "upgrade failed: " + result.warnings()); + + assertEquals( + "2a", declaredStandard(result.pdfBytes()), "the file should now declare PDF/A-2a"); + + ValidationResult pdfa = validate(result.pdfBytes(), PDFAFlavour.PDFA_2_A); + assertTrue(pdfa.isCompliant(), () -> "PDF/A-2a validation failed: " + failures(pdfa)); + } + + @Test + @DisplayName("PDF/A-1 keeps its 1.4 version, since level A must not change the part") + void partOneKeepsItsVersion() throws Exception { + byte[] levelB = fixture("valid-pdfa-1b.pdf"); + float versionBefore; + try (PDDocument document = Loader.loadPDF(levelB)) { + versionBefore = document.getVersion(); + } + + PdfaAccessibilityService.Result result = + service.upgradeToLevelA(levelB, 1, "en", "Archived"); + try (PDDocument document = Loader.loadPDF(result.pdfBytes())) { + assertEquals( + versionBefore, + document.getVersion(), + "raising the PDF version would break PDF/A-1 conformance"); + } + } + + @Test + @DisplayName("a document with nothing to tag is left at level B rather than mislabelled") + void refusesToClaimLevelAWithoutTags() throws Exception { + byte[] blank; + try (PDDocument document = new PDDocument()) { + document.addPage(new org.apache.pdfbox.pdmodel.PDPage()); + var out = new java.io.ByteArrayOutputStream(); + document.save(out); + blank = out.toByteArray(); + } + + PdfaAccessibilityService.Result result = service.upgradeToLevelA(blank, 2, "en", "Empty"); + assertFalse(result.levelA(), "an untaggable document must not claim level A"); + assertFalse(result.warnings().isEmpty(), "the refusal should be explained"); + } + + @Test + @DisplayName("setting conformance leaves the rest of the XMP packet intact") + void conformanceRewritePreservesPacket() throws Exception { + byte[] levelB = fixture("valid-pdfa-2b.pdf"); + byte[] rewritten = PdfaAccessibilityService.setConformance(levelB, 2, "A"); + + assertEquals("2a", declaredStandard(rewritten), "the rewritten packet should declare 2a"); + } + + @Test + @DisplayName("a file can declare PDF/A and PDF/UA at once without breaking either") + void combinedPdfaAndPdfUa() throws Exception { + byte[] levelB = fixture("valid-pdfa-2b.pdf"); + PdfaAccessibilityService.Result upgraded = + service.upgradeToLevelA(levelB, 2, "en-GB", "Archived and Accessible"); + assertTrue(upgraded.levelA(), "upgrade failed: " + upgraded.warnings()); + + byte[] both = PdfaAccessibilityService.declarePdfUaAlongsidePdfa(upgraded.pdfBytes(), 2); + + String xmp = xmpOf(both); + assertTrue(xmp.contains("pdfuaid"), "no PDF/UA identifier"); + assertTrue( + xmp.contains("pdfaSchema") || xmp.contains("schemas"), + "PDF/A requires an extension schema describing pdfuaid, none found: " + xmp); + + assertEquals( + "2a", declaredStandard(both), "the combined file should still declare PDF/A-2a"); + + ValidationResult pdfa = validate(both, PDFAFlavour.PDFA_2_A); + assertTrue( + pdfa.isCompliant(), + () -> "adding the PDF/UA identifier broke PDF/A: " + failures(pdfa)); + } + + @Test + @DisplayName("PDFAFlavour exposes the level A profiles the converter now targets") + void flavoursExistForLevelA() { + assertNotNull(PDFAFlavour.PDFA_1_A); + assertNotNull(PDFAFlavour.PDFA_2_A); + assertNotNull(PDFAFlavour.PDFA_3_A); + } +} diff --git a/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/TaggedContentExtractorRealFilesTest.java b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/TaggedContentExtractorRealFilesTest.java new file mode 100644 index 0000000000..b42a429007 --- /dev/null +++ b/app/proprietary/src/test/java/stirling/software/proprietary/service/ua/TaggedContentExtractorRealFilesTest.java @@ -0,0 +1,99 @@ +package stirling.software.proprietary.service.ua; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Stream; + +import org.apache.pdfbox.Loader; +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.text.PDFTextStripper; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import stirling.software.proprietary.pdf.ua.PageContent; +import stirling.software.proprietary.pdf.ua.TaggedContentExtractor; + +/** + * Guards the invariant the tagger rests on: the token pass and text pass must agree on ordinals. + * They can silently disagree, and the extractor then drops the page rather than mis-tag it. + */ +class TaggedContentExtractorRealFilesTest { + + private static Path repoRoot; + + @BeforeAll + static void setUp() { + repoRoot = Path.of("").toAbsolutePath(); + while (repoRoot != null && !Files.exists(repoRoot.resolve("settings.gradle"))) { + repoRoot = repoRoot.getParent(); + } + } + + @Test + @DisplayName("pages with extractable text always yield lines across the repository corpus") + void ordinalsAgreeOnRealFiles() throws Exception { + assertNotNull(repoRoot, "could not locate the repository root"); + List dropped = new ArrayList<>(); + int inspected = 0; + + for (Path pdf : findPdfs()) { + byte[] bytes; + try { + bytes = Files.readAllBytes(pdf); + } catch (Exception e) { + continue; + } + try (PDDocument document = Loader.loadPDF(bytes)) { + if (document.getNumberOfPages() > 30) { + continue; + } + inspected++; + List pages = new TaggedContentExtractor().extract(document); + + for (PageContent page : pages) { + if (page.markableCount() == 0 || !hasText(document, page.pageIndex())) { + continue; + } + if (page.lines().isEmpty() && page.forms().isEmpty()) { + dropped.add(repoRoot.relativize(pdf) + " page " + page.pageIndex()); + } + } + } catch (Exception e) { + // Unreadable files are covered by the conversion tests. + } + } + + assertTrue(inspected > 15, "expected to inspect a real corpus, saw " + inspected); + assertTrue( + dropped.isEmpty(), + "the two extraction passes disagreed, so these pages were skipped: " + dropped); + } + + private static boolean hasText(PDDocument document, int pageIndex) { + try { + PDFTextStripper stripper = new PDFTextStripper(); + stripper.setStartPage(pageIndex + 1); + stripper.setEndPage(pageIndex + 1); + return !stripper.getText(document).isBlank(); + } catch (Exception e) { + return false; + } + } + + private List findPdfs() throws Exception { + try (Stream stream = Files.walk(repoRoot)) { + return stream.filter(Files::isRegularFile) + .filter(p -> p.toString().toLowerCase().endsWith(".pdf")) + .filter(p -> !p.toString().contains("node_modules")) + .filter(p -> !p.toString().contains(java.io.File.separator + "build")) + .filter(p -> !p.toString().contains(".git")) + .toList(); + } + } +} diff --git a/engine/src/stirling/models/tool_io.py b/engine/src/stirling/models/tool_io.py index da0cd5b9ae..c9da9cce90 100644 --- a/engine/src/stirling/models/tool_io.py +++ b/engine/src/stirling/models/tool_io.py @@ -126,6 +126,7 @@ TOOL_IO: dict[ToolEndpoint, ToolIOSpec] = { ) ], ), + ToolEndpoint.PDF_TO_UA: ToolIOSpec(accepts=[ToolFormat.PDF], produces=ToolFormat.PDF, arity=ToolArity.SISO), ToolEndpoint.PDF_TO_VECTOR: ToolIOSpec( accepts=[ToolFormat.PDF], produces=ToolFormat.IMAGE, @@ -246,6 +247,9 @@ TOOL_IO: dict[ToolEndpoint, ToolIOSpec] = { ToolEndpoint.SCANNER_EFFECT: ToolIOSpec(accepts=[ToolFormat.PDF], produces=ToolFormat.PDF, arity=ToolArity.SISO), ToolEndpoint.UNLOCK_PDF_FORMS: ToolIOSpec(accepts=[ToolFormat.PDF], produces=ToolFormat.PDF, arity=ToolArity.SISO), ToolEndpoint.UPDATE_METADATA: ToolIOSpec(accepts=[ToolFormat.PDF], produces=ToolFormat.PDF, arity=ToolArity.SISO), + ToolEndpoint.ACCESSIBILITY_REPORT: ToolIOSpec( + accepts=[ToolFormat.PDF], produces=ToolFormat.JSON, arity=ToolArity.SISO + ), ToolEndpoint.ADD_PASSWORD: ToolIOSpec( accepts=[ToolFormat.PDF], produces=ToolFormat.PDF_ENCRYPTED, diff --git a/engine/src/stirling/models/tool_models.py b/engine/src/stirling/models/tool_models.py index 712e3575af..6650942b6f 100644 --- a/engine/src/stirling/models/tool_models.py +++ b/engine/src/stirling/models/tool_models.py @@ -11,6 +11,19 @@ from pydantic import Field, RootModel, SecretStr from stirling.models.base import ApiModel +class Profile(StrEnum): + """ + Profile to check against + """ + + ua1 = "ua1" + ua2 = "ua2" + + +class AccessibilityReportParams(ApiModel): + profile: Profile = Field(Profile.ua1, description="Profile to check against") + + class AddCommentsParams(ApiModel): comments: str = Field( ..., @@ -843,11 +856,18 @@ class OutputFormat1(StrEnum): pdfa_2b = "pdfa-2b" pdfa_3 = "pdfa-3" pdfa_3b = "pdfa-3b" + pdfa_1a = "pdfa-1a" + pdfa_2a = "pdfa-2a" + pdfa_3a = "pdfa-3a" pdfx = "pdfx" class PdfToPdfaParams(ApiModel): output_format: OutputFormat1 = Field(..., description="The output format type (PDF/A or PDF/X)") + pdf_ua: bool = Field( + False, + description="Also declare PDF/UA accessibility alongside PDF/A. Only applies to the level A formats, and the claim is written only if it validates.", + ) strict: bool | None = Field( None, description="If true, the conversion will fail if the output is not perfectly compliant" ) @@ -886,6 +906,65 @@ class PdfToTextParams(ApiModel): output_format: OutputFormat3 = Field(..., description="The output Text or RTF format") +class ExistingTags(StrEnum): + """ + What to do with an existing structure tree: keep it, rebuild it, or decide automatically + """ + + auto = "auto" + keep = "keep" + rebuild = "rebuild" + + +class FigurePolicy(StrEnum): + """ + How to treat images with no description. require-alt leaves them undescribed so the report asks for input; mark-decorative treats every image as decoration. + """ + + require_alt = "require-alt" + mark_decorative = "mark-decorative" + + +class Profile1(StrEnum): + """ + PDF/UA conformance level to target + """ + + ua1 = "ua1" + ua2 = "ua2" + + +class PdfToUaParams(ApiModel): + alt_text: str | None = Field( + None, + description='Alternative descriptions for figures, as key=text pairs separated by newlines. Keys come from the accessibility-report endpoint\'s figuresNeedingDescription list, for example "0:12=Bar chart of quarterly revenue". Descriptions are never invented, so without these an illustrated document cannot claim conformance.', + ) + embed_fonts: bool = Field( + True, + description="Embed fonts the document references but does not carry. Required for conformance and needs Ghostscript.", + ) + existing_tags: ExistingTags = Field( + ExistingTags.auto, + description="What to do with an existing structure tree: keep it, rebuild it, or decide automatically", + ) + figure_policy: FigurePolicy = Field( + FigurePolicy.require_alt, + description="How to treat images with no description. require-alt leaves them undescribed so the report asks for input; mark-decorative treats every image as decoration.", + ) + language: str = Field( + "en-GB", + description="Document language as a BCP-47 tag, for example en-GB. Applied only when the document does not already declare one, unless overrideLanguage is set.", + ) + override_language: bool = Field( + False, + description="Replace the language the document already declares. Off by default, so a document is never relabelled into a language it is not written in.", + ) + profile: Profile1 = Field(Profile1.ua1, description="PDF/UA conformance level to target") + title: str | None = Field( + None, description="Document title, required by PDF/UA. Falls back to the first heading, then the filename." + ) + + class OutputFormat4(StrEnum): """ Target vector format extension @@ -1451,6 +1530,7 @@ class Model( | PdfToPdfaParams | PdfToPresentationParams | PdfToTextParams + | PdfToUaParams | PdfToVectorParams | PdfToWordParams | PdfToXlsxParams @@ -1495,6 +1575,7 @@ class Model( | ScannerEffectParams | UnlockPdfFormsParams | UpdateMetadataParams + | AccessibilityReportParams | AddPasswordParams | AddWatermarkParams | AutoRedactParams @@ -1525,6 +1606,7 @@ class Model( | PdfToPdfaParams | PdfToPresentationParams | PdfToTextParams + | PdfToUaParams | PdfToVectorParams | PdfToWordParams | PdfToXlsxParams @@ -1569,6 +1651,7 @@ class Model( | ScannerEffectParams | UnlockPdfFormsParams | UpdateMetadataParams + | AccessibilityReportParams | AddPasswordParams | AddWatermarkParams | AutoRedactParams @@ -1600,6 +1683,7 @@ type ParamToolModel = ( | PdfToPdfaParams | PdfToPresentationParams | PdfToTextParams + | PdfToUaParams | PdfToVectorParams | PdfToWordParams | PdfToXlsxParams @@ -1644,6 +1728,7 @@ type ParamToolModel = ( | ScannerEffectParams | UnlockPdfFormsParams | UpdateMetadataParams + | AccessibilityReportParams | AddPasswordParams | AddWatermarkParams | AutoRedactParams @@ -1676,6 +1761,7 @@ class ToolEndpoint(StrEnum): PDF_TO_PDFA = "/api/v1/convert/pdf/pdfa" PDF_TO_PRESENTATION = "/api/v1/convert/pdf/presentation" PDF_TO_TEXT = "/api/v1/convert/pdf/text" + PDF_TO_UA = "/api/v1/convert/pdf/ua" PDF_TO_VECTOR = "/api/v1/convert/pdf/vector" PDF_TO_WORD = "/api/v1/convert/pdf/word" PDF_TO_XLSX = "/api/v1/convert/pdf/xlsx" @@ -1720,6 +1806,7 @@ class ToolEndpoint(StrEnum): SCANNER_EFFECT = "/api/v1/misc/scanner-effect" UNLOCK_PDF_FORMS = "/api/v1/misc/unlock-pdf-forms" UPDATE_METADATA = "/api/v1/misc/update-metadata" + ACCESSIBILITY_REPORT = "/api/v1/security/accessibility-report" ADD_PASSWORD = "/api/v1/security/add-password" ADD_WATERMARK = "/api/v1/security/add-watermark" AUTO_REDACT = "/api/v1/security/auto-redact" @@ -1750,6 +1837,7 @@ OPERATIONS: dict[ToolEndpoint, ParamToolModelType] = { ToolEndpoint.PDF_TO_PDFA: PdfToPdfaParams, ToolEndpoint.PDF_TO_PRESENTATION: PdfToPresentationParams, ToolEndpoint.PDF_TO_TEXT: PdfToTextParams, + ToolEndpoint.PDF_TO_UA: PdfToUaParams, ToolEndpoint.PDF_TO_VECTOR: PdfToVectorParams, ToolEndpoint.PDF_TO_WORD: PdfToWordParams, ToolEndpoint.PDF_TO_XLSX: PdfToXlsxParams, @@ -1794,6 +1882,7 @@ OPERATIONS: dict[ToolEndpoint, ParamToolModelType] = { ToolEndpoint.SCANNER_EFFECT: ScannerEffectParams, ToolEndpoint.UNLOCK_PDF_FORMS: UnlockPdfFormsParams, ToolEndpoint.UPDATE_METADATA: UpdateMetadataParams, + ToolEndpoint.ACCESSIBILITY_REPORT: AccessibilityReportParams, ToolEndpoint.ADD_PASSWORD: AddPasswordParams, ToolEndpoint.ADD_WATERMARK: AddWatermarkParams, ToolEndpoint.AUTO_REDACT: AutoRedactParams, diff --git a/frontend/editor/public/locales/en-US/translation.toml b/frontend/editor/public/locales/en-US/translation.toml index c2287ace9c..2424b8e2c5 100644 --- a/frontend/editor/public/locales/en-US/translation.toml +++ b/frontend/editor/public/locales/en-US/translation.toml @@ -3402,6 +3402,24 @@ pdfOptions = "PDF Options" pdfToCbr = "PDF → CBR" pdfToCbz = "PDF → CBZ" pdfToEpub = "PDF → EPUB" +pdfUaAltTextNotice = "Images need a written description before a document can be certified. Descriptions are never generated automatically, because an invented one passes the checker while telling a screen-reader user nothing. Any image left without one is reported, and the file comes back tagged but not certified." +pdfUaAltTextScanFailed = "The images could not be listed. Convert anyway and the response reports what is missing." +pdfUaAltTextSingleFileOnly = "Descriptions belong to one document: an image is identified by its position, which is a different image in every file. Convert these {{fileCount}} files to tag them, then convert one at a time to describe its images." +pdfUaEmbedFonts = "Embed missing fonts" +pdfUaEmbedFontsHelp = "PDF/UA requires every font to be embedded. Turning this off is faster but usually prevents conformance." +pdfUaFigureLabel = "Page {{page}} {{kind}}" +pdfUaFigurePlaceholder = "What this image tells the reader" +pdfUaFindImages = "Find images needing a description" +pdfUaLanguage = "Document language" +pdfUaLanguageHelp = "A BCP-47 tag such as en-GB. Used only when the document does not already declare its own language." +pdfUaNoImagesNeedingText = "No image is missing a description." +pdfUaOptions = "PDF/UA Options" +pdfUaOverrideLanguage = "Replace the document's own language" +pdfUaOverrideLanguageHelp = "Only tick this if the language above is right and the document's own is wrong. Relabelling a document into a language it is not written in makes a screen reader unintelligible." +pdfUaProfile = "Conformance level" +pdfUaSignatureWarning = "This PDF is digitally signed. Tagging rewrites the page content the signature covers, so the signature will stop verifying. Convert first, then re-sign." +pdfUaTitle = "Document title" +pdfUaTitleHelp = "Shown by a reader instead of the filename. Left blank, the first heading is used." selectSourceFormatFirst = "Choose a source format first" settings = "Settings" single = "Single" @@ -6053,6 +6071,11 @@ header = "PDF To PDF/A" tags = "archive,long-term,standard,conversion,storage,preservation" title = "PDF To PDF/A" +[pdfToPDFUA] +header = "PDF To PDF/UA" +tags = "accessibility,accessible,tagged,screen reader,wcag,eaa,section 508,conversion" +title = "PDF To PDF/UA" + [pdfToPDFX] tags = "print,standard,conversion,production,prepress,archive" title = "PDF To PDF/X" diff --git a/frontend/editor/src/core/components/tools/convert/ConvertSettings.tsx b/frontend/editor/src/core/components/tools/convert/ConvertSettings.tsx index c3ee0758f6..c550832c78 100644 --- a/frontend/editor/src/core/components/tools/convert/ConvertSettings.tsx +++ b/frontend/editor/src/core/components/tools/convert/ConvertSettings.tsx @@ -20,6 +20,7 @@ import ConvertFromEmailSettings from "@app/components/tools/convert/ConvertFromE import ConvertFromCbzSettings from "@app/components/tools/convert/ConvertFromCbzSettings"; import ConvertToCbzSettings from "@app/components/tools/convert/ConvertToCbzSettings"; import ConvertToPdfaSettings from "@app/components/tools/convert/ConvertToPdfaSettings"; +import ConvertToPdfUaSettings from "@app/components/tools/convert/ConvertToPdfUaSettings"; import ConvertToPdfxSettings from "@app/components/tools/convert/ConvertToPdfxSettings"; import ConvertFromCbrSettings from "@app/components/tools/convert/ConvertFromCbrSettings"; import ConvertToCbrSettings from "@app/components/tools/convert/ConvertToCbrSettings"; @@ -456,6 +457,20 @@ const ConvertSettings = ({ )} + {/* PDF to PDF/UA options */} + {parameters.fromExtension === "pdf" && + parameters.toExtension === "pdfua" && ( + <> + + + + )} + {/* PDF to PDF/X options */} {parameters.fromExtension === "pdf" && parameters.toExtension === "pdfx" && ( diff --git a/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.selection.test.tsx b/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.selection.test.tsx new file mode 100644 index 0000000000..5fbeace1b5 --- /dev/null +++ b/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.selection.test.tsx @@ -0,0 +1,149 @@ +/** + * Which document the PDF/UA descriptions belong to. + * + * A description is keyed by an image's position inside one file, so it is only meaningful for the + * file it was written against. The panel therefore offers the description fields for a single + * selection only, and forgets what was typed as soon as the selection changes. + */ + +import { beforeEach, describe, expect, test, vi } from "vitest"; +import { render, screen, waitFor } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; +import { MantineProvider } from "@mantine/core"; +import ConvertToPdfUaSettings from "@app/components/tools/convert/ConvertToPdfUaSettings"; +import { defaultParameters } from "@app/hooks/tools/convert/useConvertParameters"; +import type { ConvertParameters } from "@app/hooks/tools/convert/useConvertParameters"; +import type { StirlingFile } from "@app/types/fileContext"; + +// Render the English fallbacks (the test i18n instance has no loaded locale). +vi.mock("react-i18next", () => ({ + useTranslation: () => ({ + t: (key: string, fallback?: unknown, options?: Record) => { + const text = typeof fallback === "string" ? fallback : key; + return options + ? text.replace(/\{\{(\w+)\}\}/g, (_, name) => String(options[name])) + : text; + }, + }), +})); + +const api = vi.hoisted(() => ({ post: vi.fn() })); +vi.mock("@app/services/apiClient", () => ({ default: { post: api.post } })); + +// The real hook parses the PDF in a worker, which is not what this file is about. +vi.mock("@app/hooks/usePdfSignatureDetection", () => ({ + usePdfSignatureDetection: () => ({ + hasDigitalSignatures: false, + isChecking: false, + }), +})); + +const file = (name: string, content = "%PDF-1.7") => + new File([content], name, { type: "application/pdf" }) as StirlingFile; + +const parametersWith = (altText: string): ConvertParameters => ({ + ...defaultParameters, + fromExtension: "pdf", + toExtension: "pdfua", + pdfUaOptions: { ...defaultParameters.pdfUaOptions, altText }, +}); + +function renderPanel(selectedFiles: StirlingFile[], altText = "") { + const onParameterChange = vi.fn(); + const view = render( + + + , + ); + const rerenderWith = (files: StirlingFile[], text = altText) => + view.rerender( + + + , + ); + return { onParameterChange, rerenderWith }; +} + +beforeEach(() => { + vi.clearAllMocks(); + api.post.mockResolvedValue({ + data: { + figuresNeedingDescription: [{ key: "0:1", page: 1, kind: "image" }], + }, + }); +}); + +describe("PDF/UA descriptions are scoped to one document", () => { + test("one file: the images can be listed and described", async () => { + const { onParameterChange } = renderPanel([file("report.pdf")]); + + await userEvent.click(screen.getByTestId("pdfua-find-figures")); + const field = await screen.findByTestId("pdfua-alt-text-0:1"); + await userEvent.type(field, "B"); + + expect(onParameterChange).toHaveBeenCalledWith( + "pdfUaOptions", + expect.objectContaining({ altText: "0:1=B" }), + ); + }); + + test("several files: no description fields, and a reason why", () => { + renderPanel([file("report.pdf"), file("appendix.pdf")]); + + expect( + screen.getByTestId("pdfua-alt-text-single-file-only"), + ).toHaveTextContent( + /Convert these 2 files to tag them, then convert one at a time/, + ); + expect(screen.queryByTestId("pdfua-find-figures")).toBeNull(); + }); + + test("several files: descriptions already typed are dropped, not carried over", async () => { + const { onParameterChange, rerenderWith } = renderPanel( + [file("report.pdf")], + "0:1=Bar chart of revenue", + ); + + rerenderWith([file("report.pdf"), file("appendix.pdf")]); + + await waitFor(() => + expect(onParameterChange).toHaveBeenCalledWith( + "pdfUaOptions", + expect.objectContaining({ altText: "" }), + ), + ); + }); + + test("swapping the single file clears the descriptions written for the old one", async () => { + const { onParameterChange, rerenderWith } = renderPanel( + [file("report.pdf")], + "0:1=Bar chart of revenue", + ); + + rerenderWith([file("other.pdf")]); + + await waitFor(() => + expect(onParameterChange).toHaveBeenCalledWith( + "pdfUaOptions", + expect.objectContaining({ altText: "" }), + ), + ); + }); + + test("mounting with stored descriptions keeps them, so an automation step survives editing", () => { + const { onParameterChange } = renderPanel( + [file("report.pdf")], + "0:1=Bar chart of revenue", + ); + + expect(onParameterChange).not.toHaveBeenCalled(); + }); +}); diff --git a/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.test.ts b/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.test.ts new file mode 100644 index 0000000000..ccef5b5142 --- /dev/null +++ b/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, test } from "vitest"; +import { + formatAltText, + parseAltText, +} from "@app/components/tools/convert/ConvertToPdfUaSettings"; + +describe("PDF/UA alt-text wire format", () => { + test("reads the key=description lines the report's keys produce", () => { + expect(parseAltText("0:12=Bar chart\n1:3=Company logo")).toEqual({ + "0:12": "Bar chart", + "1:3": "Company logo", + }); + }); + + test("keeps a description containing an equals sign whole", () => { + expect(parseAltText("2:7=Flow: approval = sign-off")).toEqual({ + "2:7": "Flow: approval = sign-off", + }); + }); + + test("skips blank and malformed lines rather than inventing keys", () => { + expect(parseAltText("\nnot-a-pair\n3:1= \n")).toEqual({}); + }); + + test("round-trips a half-typed description, spaces and all", () => { + // Trimming here would eat the space the moment it is typed, blocking the next word. + const typed = { "0:1": "Bar chart " }; + expect(parseAltText(formatAltText(typed))).toEqual(typed); + }); + + test("drops a description the user cleared", () => { + expect(formatAltText({ "0:1": "Kept", "0:2": " " })).toBe("0:1=Kept"); + }); +}); diff --git a/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.tsx b/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.tsx new file mode 100644 index 0000000000..b1e5e4e5fe --- /dev/null +++ b/frontend/editor/src/core/components/tools/convert/ConvertToPdfUaSettings.tsx @@ -0,0 +1,280 @@ +import { useEffect, useRef, useState } from "react"; +import { Stack, Text, Select, Alert, Checkbox, TextInput } from "@mantine/core"; +import { useTranslation } from "react-i18next"; +import apiClient from "@app/services/apiClient"; +import { Button } from "@app/ui/Button"; +import { ConvertParameters } from "@app/hooks/tools/convert/useConvertParameters"; +import { usePdfSignatureDetection } from "@app/hooks/usePdfSignatureDetection"; +import { StirlingFile } from "@app/types/fileContext"; +import { Z_INDEX_AUTOMATE_DROPDOWN } from "@app/styles/zIndex"; + +interface ConvertToPdfUaSettingsProps { + parameters: ConvertParameters; + onParameterChange: ( + key: K, + value: ConvertParameters[K], + ) => void; + selectedFiles: StirlingFile[]; + disabled?: boolean; +} + +/** One image the backend says has no description yet, keyed as the conversion expects it back. */ +interface FigureNeedingDescription { + key: string; + page: number; + kind: string; +} + +/** + * The wire form the endpoint parses: one `pageIndex:ordinal=description` per line. Descriptions are + * kept verbatim so that typing a space does not fight the field; the backend trims them. + */ +export const parseAltText = (raw: string): Record => { + const parsed: Record = {}; + raw.split(/\r?\n/).forEach((line) => { + const split = line.indexOf("="); + if (split <= 0) return; + const key = line.slice(0, split).trim(); + const description = line.slice(split + 1); + if (key && description.trim()) parsed[key] = description; + }); + return parsed; +}; + +export const formatAltText = (descriptions: Record): string => + Object.entries(descriptions) + .filter(([, description]) => description.trim()) + .map(([key, description]) => `${key}=${description}`) + .join("\n"); + +/** PDF/UA conversion options; copy is deliberate - conformance is not guaranteed by one click. */ +const ConvertToPdfUaSettings = ({ + parameters, + onParameterChange, + selectedFiles, + disabled = false, +}: ConvertToPdfUaSettingsProps) => { + const { t } = useTranslation(); + const { hasDigitalSignatures } = usePdfSignatureDetection(selectedFiles); + const [figures, setFigures] = useState( + null, + ); + const [isScanning, setIsScanning] = useState(false); + const [scanError, setScanError] = useState(null); + + const profileOptions = [ + { value: "ua1", label: "PDF/UA-1" }, + { value: "ua2", label: "PDF/UA-2 (PDF 2.0)" }, + ]; + + const update = (patch: Partial) => + onParameterChange("pdfUaOptions", { ...parameters.pdfUaOptions, ...patch }); + + const descriptions = parseAltText(parameters.pdfUaOptions.altText); + // A key is a position inside one document, so descriptions only mean anything for one file. + const scannableFile = selectedFiles.length === 1 ? selectedFiles[0] : null; + const tooManyFiles = selectedFiles.length > 1; + const fileKey = selectedFiles + .map((file) => `${file.name}:${file.size}`) + .join("|"); + const describedFileKey = useRef(fileKey); + + // The same key names a different image in the next document, so descriptions must not outlive the + // selection. Mount is skipped so a stored automation step keeps the text it was saved with. + useEffect(() => { + if (describedFileKey.current === fileKey) return; + describedFileKey.current = fileKey; + setFigures(null); + if (parameters.pdfUaOptions.altText) update({ altText: "" }); + }, [fileKey]); + + // The keys are opaque, so they have to come from the backend's own analysis of this file. + const findFigures = async () => { + const file = scannableFile; + if (!file) return; + setIsScanning(true); + setScanError(null); + try { + const formData = new FormData(); + formData.append("fileInput", file); + formData.append("profile", parameters.pdfUaOptions.profile); + const { data } = await apiClient.post<{ + figuresNeedingDescription?: FigureNeedingDescription[]; + }>("/api/v1/security/accessibility-report", formData); + setFigures(data.figuresNeedingDescription ?? []); + } catch { + setScanError( + t( + "convert.pdfUaAltTextScanFailed", + "The images could not be listed. Convert anyway and the response reports what is missing.", + ), + ); + } finally { + setIsScanning(false); + } + }; + + return ( + + + {t("convert.pdfUaOptions", "PDF/UA Options")}: + + + {hasDigitalSignatures && ( + + + {t( + "convert.pdfUaSignatureWarning", + "This PDF is digitally signed. Tagging rewrites the page content the signature covers, so the signature will stop verifying. Convert first, then re-sign.", + )} + + + )} + + + + {t("convert.pdfUaProfile", "Conformance level")}: + +