Stirling-Tools/Stirling-PDF
1
0
Fork 0
mirror of https://github.com/Stirling-Tools/Stirling-PDF.git synced 2026-09-03 05:10:16 +03:00
Code Issues Packages Projects Releases Wiki Activity

Cucumber concurrency validation plus fix (#7379)

Browse Source 
# Description of Changes

cucumber tests to run multiple threads of commands at same time 

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
This commit is contained in:
Anthony Stirling
2026-08-14 14:01:45 +01:00
committed by GitHub
parent 55087313b7
commit 0be10b2dff
46 changed files with 2422 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
testing/cucumber/behave.ini
+1
View File
@@ -3,6 +3,7 @@
# Run either directly, e.g. `python -m behave features/enterprise`.
exclude_re = features/(enterprise|multinode)
tags = ~@manual
~@nightly
[behave.formatters]
# Registers the html report formatter (behave-html-formatter) used by run-multinode-regression.sh.
testing/cucumber/features/analysis.feature
+8
View File
@@ -15,6 +15,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains <pages> pages
When I send the API request to the endpoint "/api/v1/analysis/page-count"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -35,6 +36,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 4 pages
When I send the API request to the endpoint "/api/v1/analysis/basic-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -66,6 +68,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/document-properties"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -104,6 +107,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages with random text
When I send the API request to the endpoint "/api/v1/analysis/page-dimensions"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -118,6 +122,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/form-fields"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -141,6 +146,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/annotation-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -164,6 +170,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/font-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -187,6 +194,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/security-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
testing/cucumber/features/async_jobs.feature
+143
View File
@@ -0,0 +1,143 @@
@jobs
Feature: Asynchronous job API
# Any tool endpoint accepts ?async=true and returns a jobId instead of the file.
# No parallel step here: every step after the submit depends on that one jobId.
@positive
Scenario: An async job runs to completion and returns its result
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the request data includes
| parameter | value |
| angle | 90 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
Then the response status code should be 200
And the response JSON field "async" should be true
When I store the job id from the response
And I wait for the job to complete
Then the job should be reported complete
When I request the job result
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
@positive
Scenario: Async job results are listed and downloadable as individual files
Given I generate a PDF file as "fileInput"
And the pdf contains 4 pages
And the request data includes
| parameter | value |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/general/split-pages?async=true"
Then the response status code should be 200
When I store the job id from the response
And I wait for the job to complete
And I request the job result file list
Then the response status code should be 200
And the job result file list should contain at least 4 file(s)
When I request the first job result file metadata
Then the response status code should be 200
And the response JSON field "fileName" should not be empty
When I download the first job result file
Then the response status code should be 200
And the response file should have size greater than 100
@negative
Scenario: Cancelling an already-finished job is rejected
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| angle | 180 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
And I store the job id from the response
And I wait for the job to complete
And I cancel the job
Then the response status code should be 400
@negative
Scenario: Polling a job id the caller does not own is forbidden
When I send a GET request to "/api/v1/general/job/does-not-exist-1234"
Then the response status code should be 403
@negative
Scenario: Downloading an unknown file id is rejected
When I send a GET request to "/api/v1/general/files/does-not-exist-1234"
Then the response status code should be 404
# An async submit persists a copy of the upload and its results under the server's
# file store. Those outlive the request by design, so the only proof they are not a
# leak is that cleanup actually removes them. These scenarios exercise that, and
# environment.after_all sweeps the rest so the post-run temp-file diff stays honest.
@positive @cleanup
Scenario: Downloading a result does not consume it
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| angle | 90 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
And I store the job id from the response
And I wait for the job to complete
And I request the job result file list
Then the response status code should be 200
When I download the first job result file
Then the response status code should be 200
And the response file should have size greater than 100
# A download is a read, not a take: the file survives for a retry or a second client.
And the job result file should still be downloadable
@positive @cleanup
Scenario: Cleanup releases a finished job and deletes its stored files
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the request data includes
| parameter | value |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/general/split-pages?async=true"
And I store the job id from the response
And I wait for the job to complete
And I request the job result file list
Then the response status code should be 200
And the job result file list should contain at least 3 file(s)
When I trigger the async job cleanup
Then the response status code should be 200
# 3 split results plus the persisted copy of the upload.
And the cleanup should report at least 1 job(s) removed
And the cleanup should report at least 4 file(s) deleted
And the job should no longer exist
And the job result file should no longer be downloadable
@positive @cleanup
Scenario: A second cleanup finds nothing left behind
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| angle | 180 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
And I store the job id from the response
And I wait for the job to complete
And I trigger the async job cleanup
Then the response status code should be 200
And the cleanup should report at least 1 job(s) removed
When I trigger the async job cleanup
Then the response status code should be 200
And the cleanup should report nothing left to remove
testing/cucumber/features/attachments.feature
+5
View File
@@ -7,6 +7,7 @@ Feature: Attachments API Validation
And the pdf contains 2 pages with random text
And I also generate a PDF file as "attachments"
When I send the API request to the endpoint "/api/v1/misc/add-attachments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -30,6 +31,7 @@ Feature: Attachments API Validation
And the pdf contains 2 pages
And the pdf has an attachment named "test_doc.txt"
When I send the API request to the endpoint "/api/v1/misc/list-attachments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -49,6 +51,7 @@ Feature: Attachments API Validation
And the pdf contains 2 pages
And the pdf has an attachment named "report.txt"
When I send the API request to the endpoint "/api/v1/misc/extract-attachments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 0
@@ -62,6 +65,7 @@ Feature: Attachments API Validation
| attachmentName | original.txt |
| newName | renamed.txt |
When I send the API request to the endpoint "/api/v1/misc/rename-attachment"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -76,6 +80,7 @@ Feature: Attachments API Validation
| parameter | value |
| attachmentName | to_delete.txt |
When I send the API request to the endpoint "/api/v1/misc/delete-attachment"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/bookmarks_chapters.feature
+2
View File
@@ -7,6 +7,7 @@ Feature: Bookmarks and Chapter Splitting API Validation
And the pdf contains 3 pages with random text
And the pdf has bookmarks
When I send the API request to the endpoint "/api/v1/general/extract-bookmarks"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -31,6 +32,7 @@ Feature: Bookmarks and Chapter Splitting API Validation
| includeMetadata | false |
| allowDuplicates | false |
When I send the API request to the endpoint "/api/v1/general/split-pdf-by-chapters"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 0
testing/cucumber/features/convert_nightly.feature
+53
View File
@@ -0,0 +1,53 @@
@nightly @convert
Feature: Heavy conversion endpoints
# Too slow for every PR: these shell out to LibreOffice, Calibre or Ghostscript.
# behave.ini excludes @nightly; the nightly job opts back in with --tags=@nightly.
@pdf-to-xlsx @positive
Scenario: Convert a PDF containing tables to XLSX
Given I use an example file at "exampleFiles/tables.pdf" as parameter "fileInput"
When I send the API request to the endpoint "/api/v1/convert/pdf/xlsx"
And this operation is run 3 times in parallel
Then the response status code should be 200
And the response file should have size greater than 1000
And the response file should have extension ".xlsx"
@pdf-to-xlsx @positive
Scenario: A PDF with no tables converts without producing a spreadsheet
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages with random text
When I send the API request to the endpoint "/api/v1/convert/pdf/xlsx"
Then the response status code should be 204
@text-editor @positive
Scenario: text-editor metadata describes the document for the editor
Given I use an example file at "exampleFiles/tables.pdf" as parameter "fileInput"
When I send the API request to the endpoint "/api/v1/convert/pdf/text-editor/metadata"
And this operation is run 3 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response JSON field "fonts" should not be empty
@vector @negative
Scenario: vector conversion rejects an input format Ghostscript cannot read
Given I generate an SVG file as "fileInput"
When I send the API request to the endpoint "/api/v1/convert/vector/pdf"
Then the response status code should be 400
And the response JSON error should contain "Unsupported"
@ebook @positive
Scenario: An EPUB produced by Stirling converts back to PDF
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages with random text
And the request data includes
| parameter | value |
| outputFormat | EPUB |
When I send the API request to the endpoint "/api/v1/convert/pdf/epub"
And this operation is run 3 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
testing/cucumber/features/environment.py
+60 -1
View File
@@ -1,8 +1,13 @@
import os
import subprocess
import sys
import requests
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "steps"))
import job_support # noqa: E402
import parallel_support # noqa: E402
_BASE_URL = "http://localhost:8080"
_CONTAINER_NAME = os.environ.get("TEST_CONTAINER_NAME", "")
_REPORT_DIR = os.environ.get("TEST_REPORT_DIR", "")
@@ -142,8 +147,17 @@ def before_all(context):
def before_scenario(context, scenario):
"""Reset all per-scenario state before each scenario runs."""
# Skip scenarios that require JWT Bearer auth when it is not functional.
scenario_tags = set(scenario.effective_tags)
# Concurrency is opted into by a step in the feature, never by configuration.
context.parallel_repeat = 1
context.parallel_decoy = False
context.parallel_validated = False
context.parallel_ran_at = 0
context.parallel_request = None
context.parallel_get = None
# Skip scenarios that require JWT Bearer auth when it is not functional.
if _JWT_DEPENDENT_TAGS & scenario_tags and not context.jwt_available:
scenario.skip(
"JWT Bearer authentication not available in this environment (V2 disabled). "
@@ -217,3 +231,48 @@ def after_scenario(context, scenario):
context.jwt_token = None
context.original_jwt_token = None
context._status_ok = False
context.parallel_request = None
context.parallel_get = None
def _cleanup_async_job_files():
"""Release every async job result the run left on the server.
An async submit persists a copy of the upload plus its results, and both are held
for the job retention window (30 minutes by default) - far longer than a test run.
The regression check that diffs the container filesystem before and after this suite
would otherwise flag them as leaked temp files. Sweeping them here keeps that check
strict: anything it still reports afterwards is a genuine leak.
"""
try:
response = job_support.trigger_cleanup()
except Exception as exc:
print(f"\n[CLEANUP] Async job cleanup request failed: {exc}")
return
if response.status_code == 404:
print(
"\n[CLEANUP] Async job cleanup endpoint not available on this build; "
"async job files will age out on their own."
)
return
if response.status_code != 200:
print(
f"\n[CLEANUP] Async job cleanup returned {response.status_code}: "
f"{response.text[:200]}"
)
return
try:
summary = response.json()
except ValueError:
print("\n[CLEANUP] Async job cleanup returned a non-JSON body")
return
print(
f"\n[CLEANUP] Released {summary.get('jobsRemoved', '?')} async job(s) and "
f"{summary.get('filesDeleted', '?')} stored file(s); "
f"{summary.get('jobsRetained', '?')} retained."
)
def after_all(context):
_cleanup_async_job_files()
parallel_support.print_summary()
testing/cucumber/features/examples.feature
+7
View File
@@ -10,6 +10,7 @@ Feature: API Validation
| parameter | value |
| password | password123 |
When I send the API request to the endpoint "/api/v1/security/remove-password"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 0
And the response PDF is not passworded
@@ -31,6 +32,7 @@ Feature: API Validation
Scenario: Get info
Given I generate a PDF file as "fileInput"
When I send the API request to the endpoint "/api/v1/security/get-info-on-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/json"
And the response file should have size greater than 100
And the response status code should be 200
@@ -43,6 +45,7 @@ Feature: API Validation
| parameter | value |
| password | password123 |
When I send the API request to the endpoint "/api/v1/security/add-password"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 100
And the response PDF is passworded
@@ -81,6 +84,7 @@ Feature: API Validation
| alphabet | roman |
| customColor | #d3d3d3 |
When I send the API request to the endpoint "/api/v1/security/add-watermark"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 100
And the response status code should be 200
@@ -94,6 +98,7 @@ Feature: API Validation
| threshold | 90 |
| whitePercent | 99.9 |
When I send the API request to the endpoint "/api/v1/misc/remove-blanks"
And this operation is run 5 times in parallel
Then the response content type should be "application/octet-stream"
And the response file should have extension ".zip"
And the response ZIP should contain 1 files
@@ -106,6 +111,7 @@ Feature: API Validation
| parameter | value |
| flattenOnlyForms | false |
When I send the API request to the endpoint "/api/v1/misc/flatten"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 0
And the response status code should be 200
@@ -121,6 +127,7 @@ Feature: API Validation
| keywords | sample, test |
| producer | Test Producer |
When I send the API request to the endpoint "/api/v1/misc/update-metadata"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 0
And the response PDF metadata should include "Author" as "John Doe"
testing/cucumber/features/external.feature
+3
View File
@@ -174,6 +174,7 @@ Feature: API Validation
| dpi | 300 |
| imageFormat | <format> |
When I send the API request to the endpoint "/api/v1/convert/pdf/img"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 100
And the response file should have extension ".zip"
@@ -231,6 +232,7 @@ Feature: API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages with random text
When I send the API request to the endpoint "/api/v1/convert/pdf/markdown"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 100
And the response file should have extension ".md"
@@ -244,6 +246,7 @@ Feature: API Validation
| outputFormat | csv |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/convert/pdf/csv"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
And the response file should have extension ".zip"
testing/cucumber/features/filter.feature
+6
View File
@@ -18,6 +18,7 @@ Feature: Filter API Endpoints
| pageCount | <pageCount> |
| comparator | <comparator> |
When I send the API request to the endpoint "/api/v1/filter/filter-page-count"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -59,6 +60,7 @@ Feature: Filter API Endpoints
| fileSize | <fileSize> |
| comparator | <comparator> |
When I send the API request to the endpoint "/api/v1/filter/filter-file-size"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -93,6 +95,7 @@ Feature: Filter API Endpoints
| rotation | 0 |
| comparator | Equal |
When I send the API request to the endpoint "/api/v1/filter/filter-page-rotation"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -170,6 +173,7 @@ Feature: Filter API Endpoints
| standardPageSize | LETTER |
| comparator | Equal |
When I send the API request to the endpoint "/api/v1/filter/filter-page-size"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -242,6 +246,7 @@ Feature: Filter API Endpoints
| text | FINDME |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/filter/filter-contains-text"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -281,6 +286,7 @@ Feature: Filter API Endpoints
| parameter | value |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/filter/filter-contains-image"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/form_advanced.feature
+1
View File
@@ -8,6 +8,7 @@ Feature: Advanced Forms API Validation (JSON data parts)
And the pdf has form fields
And the request includes a JSON part "data" with content "{}"
When I send the API request to the endpoint "/api/v1/form/fill"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/forms.feature
+35
View File
@@ -6,6 +6,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/fields"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -24,6 +25,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/fields-with-coordinates"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -67,6 +69,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/modify-fields"
And this operation is run 5 times in parallel
Then the response status code should be 400
@modify-fields @negative
@@ -81,6 +84,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/delete-fields"
And this operation is run 5 times in parallel
Then the response status code should be 400
@delete-fields @negative
@@ -89,3 +93,34 @@ Feature: Forms API Validation
And the pdf contains 4 pages
When I send the API request to the endpoint "/api/v1/form/delete-fields"
Then the response status code should be 400
@extract-csv @positive
Scenario: extract-csv returns CSV for a form PDF
Given I generate a PDF file as "file"
And the pdf contains 2 pages
And the pdf has form fields
When I send the API request to the endpoint "/api/v1/form/extract-csv"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "text/csv"
@extract-xlsx @positive
Scenario: extract-xlsx returns a spreadsheet for a form PDF
Given I generate a PDF file as "file"
And the pdf contains 2 pages
And the pdf has form fields
When I send the API request to the endpoint "/api/v1/form/extract-xlsx"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
@extract-csv @negative
Scenario: extract-csv rejects a upload sent under the wrong part name
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/extract-csv"
Then the response status code should be 400
And the response JSON error should contain "file"
testing/cucumber/features/general.feature
+3
View File
@@ -12,6 +12,7 @@ Feature: API Validation
| verticalDivisions | <verticalDivisions> |
| merge | true |
When I send the API request to the endpoint "/api/v1/general/split-pdf-by-sections"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 200
And the response status code should be 200
@@ -34,6 +35,7 @@ Feature: API Validation
| fileInput | fileInput |
| pageNumbers | <pageNumbers> |
When I send the API request to the endpoint "/api/v1/general/split-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/octet-stream"
And the response status code should be 200
And the response file should have size greater than 200
@@ -57,6 +59,7 @@ Feature: API Validation
| splitType | <splitType> |
| splitValue | <splitValue> |
When I send the API request to the endpoint "/api/v1/general/split-by-size-or-count"
And this operation is run 5 times in parallel
Then the response content type should be "application/octet-stream"
And the response status code should be 200
And the response file should have size greater than 200
testing/cucumber/features/general_new.feature
+60
View File
@@ -10,6 +10,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| angle | <angle> |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -41,6 +42,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pageNumbers | 3 |
When I send the API request to the endpoint "/api/v1/general/remove-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -76,6 +78,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| customMode | <customMode> |
When I send the API request to the endpoint "/api/v1/general/rearrange-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -103,6 +106,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pageSize | <pageSize> |
When I send the API request to the endpoint "/api/v1/general/scale-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -127,6 +131,7 @@ Feature: General PDF Operations API Validation
| width | 50 |
| height | 50 |
When I send the API request to the endpoint "/api/v1/general/crop"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -155,6 +160,7 @@ Feature: General PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 5 pages
When I send the API request to the endpoint "/api/v1/general/pdf-to-single-page"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -199,6 +205,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pagesPerSheet | 9 |
When I send the API request to the endpoint "/api/v1/general/multi-page-layout"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -213,6 +220,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pagesPerSheet | 2 |
When I send the API request to the endpoint "/api/v1/general/booklet-imposition"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -249,3 +257,55 @@ Feature: General PDF Operations API Validation
# Then the response content type should be "application/pdf"
# And the response status code should be 200
# And the response file should have size greater than 0
@edit-table-of-contents @positive
Scenario: edit-table-of-contents rewrites the outline
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the pdf has bookmarks
And the request data includes
| parameter | value |
| bookmarkData | [{"title":"Intro","pageNumber":1,"children":[]}] |
When I send the API request to the endpoint "/api/v1/general/edit-table-of-contents"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
@edit-text @positive
Scenario: edit-text applies a find and replace across the document
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the pdf pages all contain the text "Hello world"
And the request data includes
| parameter | value |
| edits | [{"find":"Hello","replace":"Goodbye"}] |
When I send the API request to the endpoint "/api/v1/general/edit-text"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
@edit-text @negative
Scenario: edit-text without any operations returns 400
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/general/edit-text"
Then the response status code should be 400
And the response JSON error should contain "find/replace"
@split-for-poster-print @positive
Scenario: split-for-poster-print tiles each page into an archive
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| horizontalDivisions | 1 |
| verticalDivisions | 1 |
When I send the API request to the endpoint "/api/v1/general/split-for-poster-print"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
testing/cucumber/features/info_config.feature
+134
View File
@@ -0,0 +1,134 @@
@info @config
Feature: Config, info and UI-data read APIs
# Cheap read-only JSON endpoints behind the frontend and admin surfaces.
@app-config @positive
Scenario: app-config returns the frontend configuration
When I send a GET request to "/api/v1/config/app-config"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
@endpoints-availability @positive
Scenario: endpoints-availability reports per-endpoint availability
When I send a GET request to "/api/v1/config/endpoints-availability"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
@endpoints-enabled @positive
Scenario: endpoints-enabled reports the status of a named endpoint
When I send a GET request to "/api/v1/config/endpoints-enabled" with parameters
| parameter | value |
| endpoints | merge-pdfs |
Then the response status code should be 200
And the response content type should be "application/json"
@endpoints-enabled @negative
Scenario: endpoints-enabled without the endpoints parameter returns 400
When I send a GET request to "/api/v1/config/endpoints-enabled"
Then the response status code should be 400
@login-disclaimer @positive
Scenario: login-disclaimer returns its configuration
When I send a GET request to "/api/v1/config/login-disclaimer"
Then the response status code should be 200
And the response JSON field "enabled" should be false
@health @positive
Scenario: info health reports the running version
When I send a GET request to "/api/v1/info/health"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON field "status" should equal "UP"
And the response JSON field "version" should not be empty
@metrics @positive
Scenario Outline: metrics endpoints return a numeric aggregate
When I send a GET request to "<endpoint>"
Then the response status code should be 200
And the response content type should be "application/json"
And the response should match the regex "^[0-9.]+$"
Examples:
| endpoint |
| /api/v1/info/load/unique |
| /api/v1/info/requests/unique |
@metrics @positive
Scenario Outline: metrics list endpoints return a JSON list
When I send a GET request to "<endpoint>"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON should be a list
Examples:
| endpoint |
| /api/v1/info/load/all/unique |
| /api/v1/info/requests/all/unique |
@metrics @positive
Scenario: weekly active users reports tracking metadata
When I send a GET request to "/api/v1/info/wau"
Then the response status code should be 200
And the response JSON field "trackingSince" should not be empty
@settings @positive
Scenario: get-endpoints-status returns the endpoint toggle map
When I send a GET request to "/api/v1/settings/get-endpoints-status"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
@ui-data @positive
Scenario Outline: UI data endpoints return JSON for the frontend
When I send a GET request to "<endpoint>"
Then the response status code should be 200
And the response content type should be "application/json"
Examples:
| endpoint |
| /api/v1/ui-data/footer-info |
| /api/v1/ui-data/home |
| /api/v1/ui-data/licenses |
| /api/v1/ui-data/pipeline |
@ui-data @positive
Scenario: OCR UI data lists the installed tesseract languages
When I send a GET request to "/api/v1/ui-data/ocr-pdf"
Then the response status code should be 200
And the response JSON field "languages" should be a list
@ui-data @positive
Scenario: Sign UI data lists the available signature fonts
When I send a GET request to "/api/v1/ui-data/sign"
Then the response status code should be 200
And the response JSON field "fonts" should be a list
@hardware-signing @positive
Scenario: Hardware signing capabilities are reported for the server build
When I send a GET request to "/api/v1/security/cert-sign/hardware/capabilities"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON field "desktop" should be false
@hardware-signing @negative
Scenario: Windows certificate store is rejected outside the desktop app
When I send a GET request to "/api/v1/security/cert-sign/hardware/windows-certificates"
Then the response status code should be 400
And the response JSON error should contain "desktop"
testing/cucumber/features/merge_overlay.feature
+2
View File
@@ -7,6 +7,7 @@ Feature: Merge and Overlay PDF API Validation
And the pdf contains 2 pages with random text
And I also generate a PDF file as "fileInput"
When I send the API request to the endpoint "/api/v1/general/merge-pdfs"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -51,6 +52,7 @@ Feature: Merge and Overlay PDF API Validation
| overlayMode | SequentialOverlay |
| overlayPosition | 0 |
When I send the API request to the endpoint "/api/v1/general/overlay-pdfs"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/misc_new.feature
+32
View File
@@ -48,6 +48,7 @@ Feature: Miscellaneous PDF Operations API Validation
| overrideX | -1 |
| overrideY | -1 |
When I send the API request to the endpoint "/api/v1/misc/add-stamp"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -92,6 +93,7 @@ Feature: Miscellaneous PDF Operations API Validation
| position | 2 |
| fontSize | 14 |
When I send the API request to the endpoint "/api/v1/misc/add-page-numbers"
And this operation is run 5 times in parallel against decoy traffic
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -103,6 +105,7 @@ Feature: Miscellaneous PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/misc/unlock-pdf-forms"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -138,6 +141,7 @@ Feature: Miscellaneous PDF Operations API Validation
| replaceAndInvertOption | HIGH_CONTRAST_COLOR |
| highContrastColorCombination | WHITE_TEXT_ON_BLACK |
When I send the API request to the endpoint "/api/v1/misc/replace-invert-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -163,6 +167,7 @@ Feature: Miscellaneous PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
When I send the API request to the endpoint "/api/v1/misc/decompress-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -188,6 +193,7 @@ Feature: Miscellaneous PDF Operations API Validation
| parameter | value |
| useFirstTextAsFallback | true |
When I send the API request to the endpoint "/api/v1/misc/auto-rename"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -212,6 +218,7 @@ Feature: Miscellaneous PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/misc/show-javascript"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 0
@@ -222,3 +229,28 @@ Feature: Miscellaneous PDF Operations API Validation
And the pdf contains 5 pages with random text
When I send the API request to the endpoint "/api/v1/misc/show-javascript"
Then the response status code should be 200
@auto-rotate-pdf @positive
Scenario: auto-rotate-pdf returns a PDF with the page count preserved
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages with random text
When I send the API request to the endpoint "/api/v1/misc/auto-rotate-pdf"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
@add-comments @positive
Scenario: add-comments annotates a page without changing the page count
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the request data includes
| parameter | value |
| comments | [{"pageNumber":1,"x":100,"y":100,"text":"review me","author":"qa"}] |
When I send the API request to the endpoint "/api/v1/misc/add-comments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
testing/cucumber/features/security_new.feature
+37
View File
@@ -14,6 +14,7 @@ Feature: Security API Validation
| removeLinks | true |
| removeFonts | false |
When I send the API request to the endpoint "/api/v1/security/sanitize-pdf"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -54,6 +55,7 @@ Feature: Security API Validation
| wholeWordSearch| true |
| convertPDFToImage | false |
When I send the API request to the endpoint "/api/v1/security/auto-redact"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -96,6 +98,7 @@ Feature: Security API Validation
| pageNumbers | 2,4 |
| pageRedactionColor | #000000 |
When I send the API request to the endpoint "/api/v1/security/redact"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -130,6 +133,7 @@ Feature: Security API Validation
Scenario: Verify PDF-A compliance
Given I use an example file at "exampleFiles/pdfa1.pdf" as parameter "fileInput"
When I send the API request to the endpoint "/api/v1/security/verify-pdf"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 2
@@ -148,6 +152,7 @@ Feature: Security API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/security/remove-cert-sign"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -161,3 +166,35 @@ Feature: Security API Validation
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@validate-signature @positive
Scenario: validate-signature reports no signatures on an unsigned PDF
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
When I send the API request to the endpoint "/api/v1/security/validate-signature"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON should be a list
@redact-execute @positive
Scenario: redact-execute removes the targeted text
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the pdf pages all contain the text "Hello world"
And the request data includes
| parameter | value |
| textValues | Hello |
When I send the API request to the endpoint "/api/v1/security/redact-execute"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
@redact-execute @negative
Scenario: redact-execute without any targets returns 400
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/security/redact-execute"
Then the response status code should be 400
testing/cucumber/features/steps/job_step_definitions.py
+171
View File
@@ -0,0 +1,171 @@
"""Steps for the async job API. DELETE is a cancel, so it 400s once the job finishes."""
import time
import requests
from behave import then, when
from job_support import API_HEADERS, BASE_URL, trigger_cleanup
POLL_TIMEOUT_SECONDS = 60
@when("I store the job id from the response")
def step_store_job_id(context):
payload = context.response.json()
context.job_id = payload.get("jobId")
assert context.job_id, f"No jobId in async submit response: {payload}"
@when("I wait for the job to complete")
def step_wait_for_job(context):
deadline = time.time() + POLL_TIMEOUT_SECONDS
while time.time() < deadline:
response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}",
headers=API_HEADERS, timeout=30,
)
assert response.status_code == 200, (
f"Job status returned {response.status_code}: {response.text}"
)
context.response = response
payload = response.json()
if payload.get("complete"):
context.job_status = payload
return
time.sleep(0.2)
raise AssertionError(
f"Job {context.job_id} did not complete within {POLL_TIMEOUT_SECONDS}s"
)
@when("I request the job result")
def step_request_job_result(context):
context.response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}/result",
headers=API_HEADERS, timeout=60,
)
@when("I request the job result file list")
def step_request_job_result_files(context):
context.response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}/result/files",
headers=API_HEADERS, timeout=60,
)
files = context.response.json().get("files") or []
context.job_files = files
if files:
context.job_file_id = files[0].get("fileId")
@when("I download the first job result file")
def step_download_job_file(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
context.response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}",
headers=API_HEADERS, timeout=60,
)
@when("I request the first job result file metadata")
def step_job_file_metadata(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
context.response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}/metadata",
headers=API_HEADERS, timeout=60,
)
@when("I cancel the job")
def step_cancel_job(context):
context.response = requests.delete(
f"{BASE_URL}/api/v1/general/job/{context.job_id}",
headers=API_HEADERS, timeout=30,
)
@then("the job result file list should contain at least {count:d} file(s)")
def step_check_job_file_count(context, count):
files = context.response.json().get("files") or []
assert len(files) >= count, f"Expected at least {count} result file(s), got {len(files)}"
@then("the job should be reported complete")
def step_check_job_complete(context):
payload = context.response.json()
assert payload.get("complete") is True, f"Job not complete: {payload}"
assert not payload.get("error"), f"Job reported an error: {payload.get('error')}"
# --- Cleanup: the async job files must actually go away, not just age out ---
@when("I trigger the async job cleanup")
def step_trigger_cleanup(context):
context.response = trigger_cleanup()
try:
context.cleanup_summary = context.response.json()
except ValueError:
context.cleanup_summary = {}
@then("the cleanup should report at least {count:d} job(s) removed")
def step_check_cleanup_jobs(context, count):
removed = context.cleanup_summary.get("jobsRemoved")
assert removed is not None, f"No jobsRemoved in cleanup response: {context.cleanup_summary}"
assert removed >= count, f"Expected at least {count} job(s) removed, got {removed}"
@then("the cleanup should report at least {count:d} file(s) deleted")
def step_check_cleanup_files(context, count):
deleted = context.cleanup_summary.get("filesDeleted")
assert deleted is not None, f"No filesDeleted in cleanup response: {context.cleanup_summary}"
assert deleted >= count, f"Expected at least {count} file(s) deleted, got {deleted}"
@then("the cleanup should report nothing left to remove")
def step_check_cleanup_idempotent(context):
removed = context.cleanup_summary.get("jobsRemoved")
deleted = context.cleanup_summary.get("filesDeleted")
assert removed == 0 and deleted == 0, (
"A repeat cleanup still found work to do, so the first pass did not fully clean up: "
f"{context.cleanup_summary}"
)
@then("the job should no longer exist")
def step_check_job_gone(context):
response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}",
headers=API_HEADERS, timeout=30,
)
assert response.status_code == 404, (
f"Job {context.job_id} still exists after cleanup: "
f"{response.status_code} {response.text[:200]}"
)
@then("the job result file should no longer be downloadable")
def step_check_job_file_gone(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}",
headers=API_HEADERS, timeout=30,
)
assert response.status_code == 404, (
f"File {context.job_file_id} still downloadable after cleanup: "
f"{response.status_code} {response.text[:200]}"
)
@then("the job result file should still be downloadable")
def step_check_job_file_still_there(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}",
headers=API_HEADERS, timeout=60,
)
assert response.status_code == 200, (
f"File {context.job_file_id} was not retrievable a second time: "
f"{response.status_code} {response.text[:200]}"
)
assert len(response.content) > 0, "Second download returned an empty body"
testing/cucumber/features/steps/job_support.py
+22
View File
@@ -0,0 +1,22 @@
"""Shared helpers for the async job API.
Support module, not a step module: behave execs everything under features/steps as
step definitions, so anything environment.py needs to import has to live apart from
the @when/@then decorators or they would register twice.
"""
import requests
BASE_URL = "http://localhost:8080"
API_HEADERS = {"X-API-KEY": "123456789"}
CLEANUP_URL = f"{BASE_URL}/api/v1/general/jobs/cleanup"
def trigger_cleanup():
"""Force-expire finished jobs so their stored files are released now.
An async submit persists a copy of the upload plus its results and holds them for
the job retention window (30 minutes by default), which far outlasts a test run.
Calling this keeps the post-run temp-file check strict: anything it still reports
afterwards is a genuine leak rather than a file that simply had not aged out.
"""
return requests.post(CLEANUP_URL, headers=API_HEADERS, timeout=60)
testing/cucumber/features/steps/parallel_step_definitions.py
+47
View File
@@ -0,0 +1,47 @@
"""Concurrency steps, usable either before the request or after it."""
from behave import given, then, when
import parallel_support
def _set_repeat(context, count, decoy=False):
context.parallel_decoy = decoy
if count < 2:
context.parallel_repeat = 1
return
context.parallel_repeat = count
# An asked-for level higher than one already run wins.
if count > getattr(context, "parallel_ran_at", 0):
context.parallel_validated = False
if getattr(context, "parallel_validated", False):
return
# Placed after the request: replay whichever request was just sent.
pending = getattr(context, "parallel_request", None)
if pending is not None:
url, spec, headers, label = pending
parallel_support.validate(context, url, spec, headers, context.response, label)
return
pending_get = getattr(context, "parallel_get", None)
if pending_get is not None:
url, params, headers, label = pending_get
parallel_support.validate_get(context, url, params, headers, context.response, label)
@given("this operation is run {count:d} times in parallel")
@when("this operation is run {count:d} times in parallel")
@then("this operation is run {count:d} times in parallel")
def step_operation_run_in_parallel(context, count):
_set_repeat(context, count)
# Decoy traffic is an equal number of concurrent requests carrying different page
# text, so a response that came from the wrong request becomes visible.
@given("this operation is run {count:d} times in parallel against decoy traffic")
@when("this operation is run {count:d} times in parallel against decoy traffic")
@then("this operation is run {count:d} times in parallel against decoy traffic")
def step_operation_run_in_parallel_with_decoy(context, count):
_set_repeat(context, count, decoy=True)
testing/cucumber/features/steps/parallel_support.py
+478
View File
@@ -0,0 +1,478 @@
"""Re-issues a request concurrently and asserts every response matches the baseline."""
import io
import json as json_module
import os
import re
import sys
import zipfile
from concurrent.futures import ThreadPoolExecutor
from hashlib import sha256
import requests
from pypdf import PdfReader
_UUID_RE = re.compile(r"[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}")
_LONG_NUM_RE = re.compile(r"\d{10,}")
_DATE_RE = re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?")
_VOLATILE_KEY_RE = re.compile(
r"^(id|uuid|.*Id|.*_id|.*[Tt]ime|.*[Dd]ate|timestamp|created.*|modified.*|"
r"updated.*|expires.*|token|traceId|requestId|duration.*|elapsed.*)$"
)
# Only hash text for reasonably sized documents so the check stays cheap.
_MAX_TEXT_PAGES = 25
# Sequential samples taken to separate inherent nondeterminism from a concurrency bug.
NOISE_PROBE_SAMPLES = 3
# Response size is allowed to drift this far before it counts as a difference.
SIZE_TOLERANCE = 0.05
# Collected per-scenario results, printed as a summary in after_all.
VALIDATIONS = []
# A spec replaces open file handles with bytes so it can be replayed from many threads.
def materialize(spec):
"""Turn a captured spec into a fresh `files=` list for one request."""
parts = []
for key, filename, payload, mime in spec:
if filename is None:
parts.append((key, (None, payload) if mime is None else (None, payload, mime)))
else:
parts.append((key, (filename, io.BytesIO(payload), mime)))
return parts
def send(url, spec, headers, timeout=300):
return requests.post(url, files=materialize(spec), headers=headers, timeout=timeout)
def _normalize_name(name):
name = _UUID_RE.sub("<uuid>", name)
return _LONG_NUM_RE.sub("<num>", name)
def _strip_volatile(value):
"""Drop keys whose values legitimately differ between two identical requests."""
if isinstance(value, dict):
return {
k: _strip_volatile(v)
for k, v in sorted(value.items())
if not _VOLATILE_KEY_RE.match(k)
}
if isinstance(value, list):
return [_strip_volatile(v) for v in value]
if isinstance(value, str):
return _normalize_name(value)
return value
def _pdf_fingerprint(body, prefix=""):
parts = {}
try:
reader = PdfReader(io.BytesIO(body))
except Exception as exc:
parts[prefix + "pdf"] = f"unreadable: {type(exc).__name__}"
return parts
parts[prefix + "encrypted"] = reader.is_encrypted
if reader.is_encrypted:
return parts
try:
pages = reader.pages
parts[prefix + "pages"] = len(pages)
except Exception as exc:
parts[prefix + "pdf"] = f"unreadable pages: {type(exc).__name__}"
return parts
if len(pages) <= _MAX_TEXT_PAGES:
try:
text = "\n".join((page.extract_text() or "") for page in pages)
parts[prefix + "text_sha"] = sha256(text.encode("utf-8")).hexdigest()[:16]
except Exception:
pass
return parts
def _entry_sha(entry):
"""Hash an archive entry, normalizing the ids and timestamps EPUB/ODF restamp each run."""
try:
text = entry.decode("utf-8")
if "\x00" not in text:
entry = _DATE_RE.sub("<date>", _normalize_name(text)).encode("utf-8")
except UnicodeDecodeError:
pass
return sha256(entry).hexdigest()[:16]
def _zip_fingerprint(body):
parts = {}
try:
with zipfile.ZipFile(io.BytesIO(body)) as archive:
names = sorted(_normalize_name(n) for n in archive.namelist())
parts["zip_entries"] = len(names)
parts["zip_names"] = names
for index, name in enumerate(sorted(archive.namelist())):
entry = archive.read(name)
if entry[:5] == b"%PDF-":
parts.update(_pdf_fingerprint(entry, prefix=f"zip[{index}]."))
else:
parts[f"zip[{index}].sha"] = _entry_sha(entry)
except Exception as exc:
parts["zip"] = f"unreadable: {type(exc).__name__}"
return parts
def fingerprint(response):
"""Structural signature of a response, ignoring benign per-request variance."""
body = response.content
content_type = (response.headers.get("Content-Type") or "").split(";")[0].strip()
parts = {"status": response.status_code, "content_type": content_type, "size": len(body)}
if "json" in content_type:
try:
parts["json"] = _strip_volatile(json_module.loads(body.decode("utf-8")))
except Exception:
parts["body_sha"] = sha256(body).hexdigest()[:16]
elif body[:5] == b"%PDF-":
parts.update(_pdf_fingerprint(body))
elif body[:2] == b"PK":
parts.update(_zip_fingerprint(body))
elif content_type.startswith("text/") or response.status_code >= 400:
try:
parts["text"] = _normalize_name(body.decode("utf-8", "replace"))[:2000]
except Exception:
parts["body_sha"] = sha256(body).hexdigest()[:16]
else:
parts["body_sha"] = sha256(body).hexdigest()[:16]
return parts
def differing_keys(baseline, other):
return {
key
for key in set(baseline) | set(other)
if key != "size" and baseline.get(key) != other.get(key)
}
def size_differs(baseline, other):
base_size, other_size = baseline.get("size", 0), other.get("size", 0)
return abs(other_size - base_size) > max(64, base_size * SIZE_TOLERANCE)
def compare(baseline, other, ignore=frozenset(), ignore_size=False):
"""Return a list of human-readable differences between two fingerprints."""
diffs = []
for key in sorted(differing_keys(baseline, other) - set(ignore)):
diffs.append(
f"{key}: baseline={_short(baseline.get(key))} parallel={_short(other.get(key))}"
)
if not ignore_size and size_differs(baseline, other):
diffs.append(
f"size: baseline={baseline.get('size', 0)} parallel={other.get('size', 0)} "
f"(differs by more than {SIZE_TOLERANCE:.0%})"
)
return diffs
def _short(value):
text = repr(value)
return text if len(text) <= 160 else text[:157] + "..."
def build_decoy_spec(spec):
"""Clone a spec with each PDF stamped with unique text, or None if not possible.
Structure is preserved so parameters stay valid; only the text differs, which
is what makes bleed visible.
"""
try:
from pypdf import PdfWriter
from reportlab.pdfgen import canvas
except ImportError:
return None
decoy = []
stamped_any = False
for key, filename, payload, mime in spec:
if filename is None or not isinstance(payload, bytes) or payload[:5] != b"%PDF-":
decoy.append((key, filename, payload, mime))
continue
try:
reader = PdfReader(io.BytesIO(payload))
if reader.is_encrypted:
return None
writer = PdfWriter()
for index, page in enumerate(reader.pages):
box = page.mediabox
width, height = float(box.width), float(box.height)
overlay_buffer = io.BytesIO()
overlay_canvas = canvas.Canvas(overlay_buffer, pagesize=(width, height))
overlay_canvas.drawString(
20, max(20.0, height - 20), f"DECOY-MARKER-{index}-do-not-mix"
)
overlay_canvas.showPage()
overlay_canvas.save()
overlay_buffer.seek(0)
page.merge_page(PdfReader(overlay_buffer).pages[0])
writer.add_page(page)
out = io.BytesIO()
writer.write(out)
decoy.append((key, filename, out.getvalue(), mime))
stamped_any = True
except Exception:
return None
return decoy if stamped_any else None
def _run_concurrently(url, specs, headers, timeout):
"""Fire every spec at once and return (response, error) in submission order."""
results = [None] * len(specs)
def _worker(index):
try:
results[index] = (send(url, specs[index], headers, timeout=timeout), None)
except Exception as exc:
results[index] = (None, exc)
with ThreadPoolExecutor(max_workers=len(specs)) as pool:
list(pool.map(_worker, range(len(specs))))
return results
def validate(context, url, spec, headers, baseline, label, timeout=300):
"""Re-issue the request concurrently; no-op unless the repeat count is above 1."""
repeat = getattr(context, "parallel_repeat", 1)
if repeat < 2 or getattr(context, "parallel_validated", False):
return
context.parallel_validated = True
context.parallel_ran_at = repeat
decoy_spec = build_decoy_spec(spec) if getattr(context, "parallel_decoy", False) else None
specs = [spec] * repeat + ([decoy_spec] * repeat if decoy_spec else [])
results = _run_concurrently(url, specs, headers, timeout)
main_results = results[:repeat]
decoy_results = results[repeat:]
baseline_fp = fingerprint(baseline)
noise, noisy_size = frozenset(), False
failures = _collect_failures(
main_results, decoy_results, baseline_fp, repeat, noise, noisy_size
)
if failures:
# Some endpoints are inherently nondeterministic (embedded ids, timestamps,
# deliberate randomness). Re-run sequentially to tell that apart from a real bug.
noise, noisy_size = _probe_noise(url, spec, headers, baseline_fp, timeout)
failures = _collect_failures(
main_results, decoy_results, baseline_fp, repeat, noise, noisy_size
)
VALIDATIONS.append(
{
"label": label,
"repeat": repeat,
"decoy": bool(decoy_spec),
"failed": bool(failures),
"noise": sorted(noise) + (["size"] if noisy_size else []),
}
)
if failures:
raise AssertionError(
f"Parallel consistency failed for {label} at concurrency {repeat}.\n"
f"Two sequential runs agreed on these fields, so the differences below are "
f"caused by running the same operation concurrently.\n"
f"Baseline fingerprint: {_short(baseline_fp)}\n - " + "\n - ".join(failures)
)
def _collect_failures(main_results, decoy_results, baseline_fp, repeat, noise, noisy_size):
decoy_fp = _decoy_reference(decoy_results, baseline_fp, noise, noisy_size)
failures = []
for index, (response, error) in enumerate(main_results):
if error is not None:
failures.append(f"copy {index + 1}/{repeat} raised {type(error).__name__}: {error}")
continue
actual_fp = fingerprint(response)
diffs = compare(baseline_fp, actual_fp, noise, noisy_size)
if not diffs:
continue
if decoy_fp is not None and not compare(
decoy_fp, actual_fp, noise, noisy_size
):
failures.append(
f"copy {index + 1}/{repeat} returned the CONCURRENT DECOY REQUEST'S response "
f"(cross-request bleed)"
)
else:
failures.append(f"copy {index + 1}/{repeat} diverged: " + "; ".join(diffs))
failures.extend(_check_decoys(decoy_results, repeat, noise, noisy_size))
return failures
def _probe_noise(url, spec, headers, baseline_fp, timeout, samples=NOISE_PROBE_SAMPLES):
"""Fields that already vary between uncontended runs, so they prove nothing.
Several samples: high-variance output can look stable across any single pair.
"""
probes = []
for _ in range(samples):
try:
probes.append(fingerprint(send(url, spec, headers, timeout=timeout)))
except Exception:
break
return _noise_from_samples(baseline_fp, probes)
def _noise_from_samples(baseline_fp, probes):
noise = set()
noisy_size = False
for index, probe in enumerate(probes):
for other in [baseline_fp] + probes[:index]:
noise |= differing_keys(other, probe)
noisy_size = noisy_size or size_differs(other, probe)
return frozenset(noise), noisy_size
def validate_get(context, url, params, headers, baseline, label, timeout=60):
"""Concurrency check for read-only GET endpoints."""
repeat = getattr(context, "parallel_repeat", 1)
if repeat < 2 or getattr(context, "parallel_validated", False):
return
context.parallel_validated = True
context.parallel_ran_at = repeat
results = [None] * repeat
def _worker(index):
try:
results[index] = (
requests.get(url, params=params, headers=headers, timeout=timeout),
None,
)
except Exception as exc:
results[index] = (None, exc)
with ThreadPoolExecutor(max_workers=repeat) as pool:
list(pool.map(_worker, range(repeat)))
baseline_fp = fingerprint(baseline)
def _failures(noise, noisy_size):
found = []
for index, (response, error) in enumerate(results):
if error is not None:
found.append(f"copy {index + 1}/{repeat} raised {type(error).__name__}: {error}")
continue
diffs = compare(
baseline_fp,
fingerprint(response),
noise,
noisy_size,
)
if diffs:
found.append(f"copy {index + 1}/{repeat} diverged: " + "; ".join(diffs))
return found
noise, noisy_size = frozenset(), False
failures = _failures(noise, noisy_size)
if failures:
probes = []
for _ in range(NOISE_PROBE_SAMPLES):
try:
probes.append(
fingerprint(
requests.get(url, params=params, headers=headers, timeout=timeout)
)
)
except Exception:
break
noise, noisy_size = _noise_from_samples(baseline_fp, probes)
failures = _failures(noise, noisy_size)
VALIDATIONS.append(
{
"label": label,
"repeat": repeat,
"decoy": False,
"failed": bool(failures),
"noise": sorted(noise) + (["size"] if noisy_size else []),
}
)
if failures:
raise AssertionError(
f"Parallel consistency failed for GET {label} at concurrency {repeat}.\n - "
+ "\n - ".join(failures)
)
def _decoy_reference(decoy_results, baseline_fp, noise, noisy_size):
"""Fingerprint of the decoy response, or None when it is not distinguishable."""
live = [r for r, _e in decoy_results if r is not None]
if not live:
return None
decoy_fp = fingerprint(live[0])
# Some endpoints ignore page content, so the decoy cannot prove anything there.
if not compare(baseline_fp, decoy_fp, noise, noisy_size):
return None
return decoy_fp
def _check_decoys(decoy_results, repeat, noise, noisy_size):
"""Assert the decoy load stayed self-consistent while contending with the main copies."""
if not decoy_results:
return []
live = [(i, r) for i, (r, _e) in enumerate(decoy_results) if r is not None]
if not live:
return ["every decoy request failed to complete"]
failures = []
reference_fp = fingerprint(live[0][1])
for index, response in live[1:]:
diffs = compare(
reference_fp,
fingerprint(response),
noise,
noisy_size,
)
if diffs:
failures.append(f"decoy {index + 1}/{repeat} diverged: " + "; ".join(diffs))
return failures
def print_summary():
if not VALIDATIONS:
return
total = len(VALIDATIONS)
failed = sum(1 for v in VALIDATIONS if v["failed"])
max_repeat = max(v["repeat"] for v in VALIDATIONS)
requests_sent = sum(v["repeat"] * (2 if v["decoy"] else 1) for v in VALIDATIONS)
lines = [
f"\n[PARALLEL] {total - failed}/{total} operations stayed consistent under "
f"concurrency (up to {max_repeat} at once, {requests_sent} concurrent requests sent)."
]
noisy = {}
for entry in VALIDATIONS:
if entry["noise"]:
noisy.setdefault(entry["label"], set()).update(entry["noise"])
if noisy:
lines.append(
f"[PARALLEL] {len(noisy)} endpoint(s) produce nondeterministic output. Those "
f"fields were excluded only after confirming they also vary across "
f"{NOISE_PROBE_SAMPLES} sequential runs:"
)
for label, fields in sorted(noisy.items()):
lines.append(f" {label} -> {', '.join(sorted(fields))}")
# behave's --junit reporter swallows after_all stdout, so bypass any capture.
stream = getattr(sys, "__stderr__", None) or sys.stderr
stream.write("\n".join(lines) + "\n")
stream.flush()
testing/cucumber/features/steps/step_definitions.py
+33 -10
View File
@@ -15,6 +15,8 @@ import zipfile
import re
from PIL import Image, ImageDraw
import parallel_support
API_HEADERS = {"X-API-KEY": "123456789"}
#########
@@ -585,6 +587,8 @@ def step_send_get_request(context, endpoint):
full_url = f"{base_url}{endpoint}"
response = requests.get(full_url, headers=API_HEADERS, timeout=60)
context.response = response
context.parallel_get = (full_url, None, API_HEADERS, endpoint)
parallel_support.validate_get(context, full_url, None, API_HEADERS, response, endpoint)
@when('I send a GET request to "{endpoint}" with parameters')
@@ -594,17 +598,18 @@ def step_send_get_request_with_params(context, endpoint):
full_url = f"{base_url}{endpoint}"
response = requests.get(full_url, params=params, headers=API_HEADERS, timeout=60)
context.response = response
context.parallel_get = (full_url, params, API_HEADERS, endpoint)
parallel_support.validate_get(context, full_url, params, API_HEADERS, response, endpoint)
@when('I send the API request to the endpoint "{endpoint}"')
def step_send_api_request(context, endpoint):
url = f"http://localhost:8080{endpoint}"
def _build_request_spec(context):
"""Capture the multipart payload as replayable bytes rather than file handles."""
files = context.files if hasattr(context, "files") else {}
if not hasattr(context, "request_data") or context.request_data is None:
context.request_data = {}
form_data = []
spec = []
for key, value in context.request_data.items():
# Handle list parameters (like 'languages') - send multiple form fields
# Split comma-separated values or treat single values as single-item lists
@@ -612,32 +617,50 @@ def step_send_api_request(context, endpoint):
# Split by comma if present, otherwise treat as single value
values = [v.strip() for v in value.split(",")] if "," in value else [value]
for val in values:
form_data.append((key, (None, val)))
spec.append((key, None, val, None))
else:
form_data.append((key, (None, value)))
spec.append((key, None, value, None))
def _read(file):
file.seek(0)
payload = file.read()
file.seek(0)
return payload
for key, file in files.items():
mime_type, _ = mimetypes.guess_type(file.name)
mime_type = mime_type or "application/octet-stream"
print(f"form_data {file.name} with {mime_type}")
form_data.append((key, (file.name, file, mime_type)))
spec.append((key, file.name, _read(file), mime_type))
# Multi-file entries (duplicate keys for MultipartFile[] endpoints, e.g. merge-pdfs)
for key, file in getattr(context, "multi_files", []):
mime_type, _ = mimetypes.guess_type(file.name)
mime_type = mime_type or "application/octet-stream"
print(f"form_data (multi) {file.name} with {mime_type}")
form_data.append((key, (file.name, file, mime_type)))
spec.append((key, file.name, _read(file), mime_type))
# JSON multipart parts for @RequestPart endpoints (e.g. /form/fill)
for part_name, json_content in getattr(context, "json_parts", {}).items():
form_data.append((part_name, (None, json_content, "application/json")))
spec.append((part_name, None, json_content, "application/json"))
return spec
@when('I send the API request to the endpoint "{endpoint}"')
def step_send_api_request(context, endpoint):
url = f"http://localhost:8080{endpoint}"
spec = _build_request_spec(context)
# Set timeout to 300 seconds (5 minutes) to prevent infinite hangs
print(f"Sending POST request to {endpoint} with timeout=300s")
response = requests.post(url, files=form_data, headers=API_HEADERS, timeout=300)
response = parallel_support.send(url, spec, API_HEADERS, timeout=300)
context.response = response
# Remembered so a later "run N times in parallel" step can replay this request.
context.parallel_request = (url, spec, API_HEADERS, endpoint)
parallel_support.validate(context, url, spec, API_HEADERS, response, endpoint)
########
# THEN #
testing/cucumber/run-parallel.sh
+143
View File
@@ -0,0 +1,143 @@
#!/bin/bash
# Run the behave suite as N concurrent shards against one server.
# Usage: ./run-parallel.sh [SHARDS] [-- behave args] Env: BASE_URL
set -uo pipefail
SHARDS="${1:-10}"
if [[ "$SHARDS" =~ ^[0-9]+$ ]]; then
shift
else
SHARDS=10
fi
[[ "${1:-}" == "--" ]] && shift
BASE_URL="${BASE_URL:-http://localhost:8080}"
CUCUMBER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WORK_ROOT="$CUCUMBER_DIR/.parallel"
REPORT_DIR="${PARALLEL_REPORT_DIR:-$WORK_ROOT/reports}"
cd "$CUCUMBER_DIR" || exit 1
if ! curl -sf --retry 30 --retry-delay 2 --retry-connrefused --retry-all-errors \
"$BASE_URL/api/v1/info/status" >/dev/null; then
echo "ERROR: no server responding at $BASE_URL"
exit 1
fi
mapfile -t ALL_FEATURES < <(
find features -name '*.feature' \
-not -path 'features/enterprise/*' \
-not -path 'features/multinode/*' | sort
)
if [ "${#ALL_FEATURES[@]}" -eq 0 ]; then
echo "ERROR: no feature files found under $CUCUMBER_DIR/features"
exit 1
fi
# user_management.feature changes the admin password mid-scenario, so a concurrent
# shard logging in then gets a 401. Keep all admin-auth features in one shard.
AUTH_RE='logged in as admin|I login with username|JWT authentication|stored JWT token'
declare -a AUTH_FEATURES=() FEATURES=()
for feature in "${ALL_FEATURES[@]}"; do
if grep -qE "$AUTH_RE" "$feature"; then
AUTH_FEATURES+=("$feature")
else
FEATURES+=("$feature")
fi
done
if [ "${#AUTH_FEATURES[@]}" -gt 0 ]; then
echo "Pinning ${#AUTH_FEATURES[@]} auth-coupled feature(s) to a single shard:"
printf ' %s\n' "${AUTH_FEATURES[@]##*/}"
SHARDS=$((SHARDS - 1))
fi
# The pin above can take SHARDS to 0 or below, which makes the shard loop run zero
# times: every shardable feature is skipped and the run still reports success.
[ "$SHARDS" -lt 0 ] && SHARDS=0
[ "${#FEATURES[@]}" -gt 0 ] && [ "$SHARDS" -lt 1 ] && SHARDS=1
if [ "$SHARDS" -gt "${#FEATURES[@]}" ]; then
echo "Only ${#FEATURES[@]} shardable feature files, reducing shards from $SHARDS"
SHARDS="${#FEATURES[@]}"
fi
rm -rf "$WORK_ROOT"
mkdir -p "$WORK_ROOT" "$REPORT_DIR"
TOTAL_SHARDS=$SHARDS
[ "${#AUTH_FEATURES[@]}" -gt 0 ] && TOTAL_SHARDS=$((SHARDS + 1))
echo "Running ${#ALL_FEATURES[@]} feature files across $TOTAL_SHARDS concurrent shards against $BASE_URL"
declare -a PIDS=()
start_shard() {
local shard=$1
shift
local SHARD_DIR="$WORK_ROOT/shard-$shard"
mkdir -p "$SHARD_DIR"
# Feature files reference exampleFiles/ and behave.ini relative to the CWD.
cp -r "$CUCUMBER_DIR/exampleFiles" "$SHARD_DIR/exampleFiles"
cp "$CUCUMBER_DIR/behave.ini" "$SHARD_DIR/behave.ini"
local assigned=("$@")
(
cd "$SHARD_DIR" || exit 1
uv run --project "$CUCUMBER_DIR/../../engine" --locked --group cucumber \
python -m behave "${assigned[@]}" \
--junit --junit-directory "$REPORT_DIR/shard-$shard" \
--no-capture -f plain "${BEHAVE_EXTRA[@]}" \
>"$REPORT_DIR/shard-$shard.log" 2>&1
) &
PIDS+=($!)
}
declare -a BEHAVE_EXTRA=("$@")
for ((shard = 0; shard < SHARDS; shard++)); do
assigned=()
for ((i = shard; i < ${#FEATURES[@]}; i += SHARDS)); do
assigned+=("$CUCUMBER_DIR/${FEATURES[$i]}")
done
start_shard "$shard" "${assigned[@]}"
done
if [ "${#AUTH_FEATURES[@]}" -gt 0 ]; then
assigned=()
for feature in "${AUTH_FEATURES[@]}"; do
assigned+=("$CUCUMBER_DIR/$feature")
done
start_shard "$SHARDS" "${assigned[@]}"
fi
FAILED=0
for ((shard = 0; shard < TOTAL_SHARDS; shard++)); do
if wait "${PIDS[$shard]}"; then
echo "shard $shard PASSED"
else
echo "shard $shard FAILED"
FAILED=$((FAILED + 1))
fi
done
echo ""
echo "=== Parallel shard summary ==="
grep -h '^[0-9]* scenarios passed' "$REPORT_DIR"/shard-*.log 2>/dev/null || true
grep -h '^\[PARALLEL\] [0-9]*/' "$REPORT_DIR"/shard-*.log 2>/dev/null || true
if [ "$FAILED" -gt 0 ]; then
echo ""
echo "$FAILED of $TOTAL_SHARDS shards failed. Failing scenarios:"
sed -n '/^Failing scenarios:/,/^[0-9]* features/p' "$REPORT_DIR"/shard-*.log 2>/dev/null |
grep 'feature:' | sort -u
echo ""
echo "Parallel consistency failures:"
grep -h -A4 'Parallel consistency failed' "$REPORT_DIR"/shard-*.log 2>/dev/null | head -40
echo ""
echo "Full logs: $REPORT_DIR/shard-*.log"
exit 1
fi
echo "All $TOTAL_SHARDS shards passed."