Stirling-Tools/Stirling-PDF
1
0
Fork 0
mirror of https://github.com/Stirling-Tools/Stirling-PDF.git synced 2026-09-02 21:03:34 +03:00
Code Issues Packages Projects Releases Wiki Activity

Cucumber concurrency validation plus fix (#7379)

Browse Source 
# Description of Changes

cucumber tests to run multiple threads of commands at same time 

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
This commit is contained in:
Anthony Stirling
2026-08-14 14:01:45 +01:00
committed by GitHub
parent 55087313b7
commit 0be10b2dff
46 changed files with 2422 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
testing/cucumber/behave.ini
+1
View File
@@ -3,6 +3,7 @@
# Run either directly, e.g. `python -m behave features/enterprise`.
exclude_re = features/(enterprise|multinode)
tags = ~@manual
~@nightly
[behave.formatters]
# Registers the html report formatter (behave-html-formatter) used by run-multinode-regression.sh.
testing/cucumber/features/analysis.feature
+8
View File
@@ -15,6 +15,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains <pages> pages
When I send the API request to the endpoint "/api/v1/analysis/page-count"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -35,6 +36,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 4 pages
When I send the API request to the endpoint "/api/v1/analysis/basic-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -66,6 +68,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/document-properties"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -104,6 +107,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages with random text
When I send the API request to the endpoint "/api/v1/analysis/page-dimensions"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -118,6 +122,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/form-fields"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -141,6 +146,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/annotation-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -164,6 +170,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/font-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -187,6 +194,7 @@ Feature: Analysis API Endpoints
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/analysis/security-info"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
testing/cucumber/features/async_jobs.feature
+143
View File
@@ -0,0 +1,143 @@
@jobs
Feature: Asynchronous job API
# Any tool endpoint accepts ?async=true and returns a jobId instead of the file.
# No parallel step here: every step after the submit depends on that one jobId.
@positive
Scenario: An async job runs to completion and returns its result
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the request data includes
| parameter | value |
| angle | 90 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
Then the response status code should be 200
And the response JSON field "async" should be true
When I store the job id from the response
And I wait for the job to complete
Then the job should be reported complete
When I request the job result
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
@positive
Scenario: Async job results are listed and downloadable as individual files
Given I generate a PDF file as "fileInput"
And the pdf contains 4 pages
And the request data includes
| parameter | value |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/general/split-pages?async=true"
Then the response status code should be 200
When I store the job id from the response
And I wait for the job to complete
And I request the job result file list
Then the response status code should be 200
And the job result file list should contain at least 4 file(s)
When I request the first job result file metadata
Then the response status code should be 200
And the response JSON field "fileName" should not be empty
When I download the first job result file
Then the response status code should be 200
And the response file should have size greater than 100
@negative
Scenario: Cancelling an already-finished job is rejected
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| angle | 180 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
And I store the job id from the response
And I wait for the job to complete
And I cancel the job
Then the response status code should be 400
@negative
Scenario: Polling a job id the caller does not own is forbidden
When I send a GET request to "/api/v1/general/job/does-not-exist-1234"
Then the response status code should be 403
@negative
Scenario: Downloading an unknown file id is rejected
When I send a GET request to "/api/v1/general/files/does-not-exist-1234"
Then the response status code should be 404
# An async submit persists a copy of the upload and its results under the server's
# file store. Those outlive the request by design, so the only proof they are not a
# leak is that cleanup actually removes them. These scenarios exercise that, and
# environment.after_all sweeps the rest so the post-run temp-file diff stays honest.
@positive @cleanup
Scenario: Downloading a result does not consume it
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| angle | 90 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
And I store the job id from the response
And I wait for the job to complete
And I request the job result file list
Then the response status code should be 200
When I download the first job result file
Then the response status code should be 200
And the response file should have size greater than 100
# A download is a read, not a take: the file survives for a retry or a second client.
And the job result file should still be downloadable
@positive @cleanup
Scenario: Cleanup releases a finished job and deletes its stored files
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the request data includes
| parameter | value |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/general/split-pages?async=true"
And I store the job id from the response
And I wait for the job to complete
And I request the job result file list
Then the response status code should be 200
And the job result file list should contain at least 3 file(s)
When I trigger the async job cleanup
Then the response status code should be 200
# 3 split results plus the persisted copy of the upload.
And the cleanup should report at least 1 job(s) removed
And the cleanup should report at least 4 file(s) deleted
And the job should no longer exist
And the job result file should no longer be downloadable
@positive @cleanup
Scenario: A second cleanup finds nothing left behind
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| angle | 180 |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf?async=true"
And I store the job id from the response
And I wait for the job to complete
And I trigger the async job cleanup
Then the response status code should be 200
And the cleanup should report at least 1 job(s) removed
When I trigger the async job cleanup
Then the response status code should be 200
And the cleanup should report nothing left to remove
testing/cucumber/features/attachments.feature
+5
View File
@@ -7,6 +7,7 @@ Feature: Attachments API Validation
And the pdf contains 2 pages with random text
And I also generate a PDF file as "attachments"
When I send the API request to the endpoint "/api/v1/misc/add-attachments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -30,6 +31,7 @@ Feature: Attachments API Validation
And the pdf contains 2 pages
And the pdf has an attachment named "test_doc.txt"
When I send the API request to the endpoint "/api/v1/misc/list-attachments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -49,6 +51,7 @@ Feature: Attachments API Validation
And the pdf contains 2 pages
And the pdf has an attachment named "report.txt"
When I send the API request to the endpoint "/api/v1/misc/extract-attachments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 0
@@ -62,6 +65,7 @@ Feature: Attachments API Validation
| attachmentName | original.txt |
| newName | renamed.txt |
When I send the API request to the endpoint "/api/v1/misc/rename-attachment"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -76,6 +80,7 @@ Feature: Attachments API Validation
| parameter | value |
| attachmentName | to_delete.txt |
When I send the API request to the endpoint "/api/v1/misc/delete-attachment"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/bookmarks_chapters.feature
+2
View File
@@ -7,6 +7,7 @@ Feature: Bookmarks and Chapter Splitting API Validation
And the pdf contains 3 pages with random text
And the pdf has bookmarks
When I send the API request to the endpoint "/api/v1/general/extract-bookmarks"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -31,6 +32,7 @@ Feature: Bookmarks and Chapter Splitting API Validation
| includeMetadata | false |
| allowDuplicates | false |
When I send the API request to the endpoint "/api/v1/general/split-pdf-by-chapters"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 0
testing/cucumber/features/convert_nightly.feature
+53
View File
@@ -0,0 +1,53 @@
@nightly @convert
Feature: Heavy conversion endpoints
# Too slow for every PR: these shell out to LibreOffice, Calibre or Ghostscript.
# behave.ini excludes @nightly; the nightly job opts back in with --tags=@nightly.
@pdf-to-xlsx @positive
Scenario: Convert a PDF containing tables to XLSX
Given I use an example file at "exampleFiles/tables.pdf" as parameter "fileInput"
When I send the API request to the endpoint "/api/v1/convert/pdf/xlsx"
And this operation is run 3 times in parallel
Then the response status code should be 200
And the response file should have size greater than 1000
And the response file should have extension ".xlsx"
@pdf-to-xlsx @positive
Scenario: A PDF with no tables converts without producing a spreadsheet
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages with random text
When I send the API request to the endpoint "/api/v1/convert/pdf/xlsx"
Then the response status code should be 204
@text-editor @positive
Scenario: text-editor metadata describes the document for the editor
Given I use an example file at "exampleFiles/tables.pdf" as parameter "fileInput"
When I send the API request to the endpoint "/api/v1/convert/pdf/text-editor/metadata"
And this operation is run 3 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response JSON field "fonts" should not be empty
@vector @negative
Scenario: vector conversion rejects an input format Ghostscript cannot read
Given I generate an SVG file as "fileInput"
When I send the API request to the endpoint "/api/v1/convert/vector/pdf"
Then the response status code should be 400
And the response JSON error should contain "Unsupported"
@ebook @positive
Scenario: An EPUB produced by Stirling converts back to PDF
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages with random text
And the request data includes
| parameter | value |
| outputFormat | EPUB |
When I send the API request to the endpoint "/api/v1/convert/pdf/epub"
And this operation is run 3 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
testing/cucumber/features/environment.py
+60 -1
View File
@@ -1,8 +1,13 @@
import os
import subprocess
import sys
import requests
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "steps"))
import job_support # noqa: E402
import parallel_support # noqa: E402
_BASE_URL = "http://localhost:8080"
_CONTAINER_NAME = os.environ.get("TEST_CONTAINER_NAME", "")
_REPORT_DIR = os.environ.get("TEST_REPORT_DIR", "")
@@ -142,8 +147,17 @@ def before_all(context):
def before_scenario(context, scenario):
"""Reset all per-scenario state before each scenario runs."""
# Skip scenarios that require JWT Bearer auth when it is not functional.
scenario_tags = set(scenario.effective_tags)
# Concurrency is opted into by a step in the feature, never by configuration.
context.parallel_repeat = 1
context.parallel_decoy = False
context.parallel_validated = False
context.parallel_ran_at = 0
context.parallel_request = None
context.parallel_get = None
# Skip scenarios that require JWT Bearer auth when it is not functional.
if _JWT_DEPENDENT_TAGS & scenario_tags and not context.jwt_available:
scenario.skip(
"JWT Bearer authentication not available in this environment (V2 disabled). "
@@ -217,3 +231,48 @@ def after_scenario(context, scenario):
context.jwt_token = None
context.original_jwt_token = None
context._status_ok = False
context.parallel_request = None
context.parallel_get = None
def _cleanup_async_job_files():
"""Release every async job result the run left on the server.
An async submit persists a copy of the upload plus its results, and both are held
for the job retention window (30 minutes by default) - far longer than a test run.
The regression check that diffs the container filesystem before and after this suite
would otherwise flag them as leaked temp files. Sweeping them here keeps that check
strict: anything it still reports afterwards is a genuine leak.
"""
try:
response = job_support.trigger_cleanup()
except Exception as exc:
print(f"\n[CLEANUP] Async job cleanup request failed: {exc}")
return
if response.status_code == 404:
print(
"\n[CLEANUP] Async job cleanup endpoint not available on this build; "
"async job files will age out on their own."
)
return
if response.status_code != 200:
print(
f"\n[CLEANUP] Async job cleanup returned {response.status_code}: "
f"{response.text[:200]}"
)
return
try:
summary = response.json()
except ValueError:
print("\n[CLEANUP] Async job cleanup returned a non-JSON body")
return
print(
f"\n[CLEANUP] Released {summary.get('jobsRemoved', '?')} async job(s) and "
f"{summary.get('filesDeleted', '?')} stored file(s); "
f"{summary.get('jobsRetained', '?')} retained."
)
def after_all(context):
_cleanup_async_job_files()
parallel_support.print_summary()
testing/cucumber/features/examples.feature
+7
View File
@@ -10,6 +10,7 @@ Feature: API Validation
| parameter | value |
| password | password123 |
When I send the API request to the endpoint "/api/v1/security/remove-password"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 0
And the response PDF is not passworded
@@ -31,6 +32,7 @@ Feature: API Validation
Scenario: Get info
Given I generate a PDF file as "fileInput"
When I send the API request to the endpoint "/api/v1/security/get-info-on-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/json"
And the response file should have size greater than 100
And the response status code should be 200
@@ -43,6 +45,7 @@ Feature: API Validation
| parameter | value |
| password | password123 |
When I send the API request to the endpoint "/api/v1/security/add-password"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 100
And the response PDF is passworded
@@ -81,6 +84,7 @@ Feature: API Validation
| alphabet | roman |
| customColor | #d3d3d3 |
When I send the API request to the endpoint "/api/v1/security/add-watermark"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 100
And the response status code should be 200
@@ -94,6 +98,7 @@ Feature: API Validation
| threshold | 90 |
| whitePercent | 99.9 |
When I send the API request to the endpoint "/api/v1/misc/remove-blanks"
And this operation is run 5 times in parallel
Then the response content type should be "application/octet-stream"
And the response file should have extension ".zip"
And the response ZIP should contain 1 files
@@ -106,6 +111,7 @@ Feature: API Validation
| parameter | value |
| flattenOnlyForms | false |
When I send the API request to the endpoint "/api/v1/misc/flatten"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 0
And the response status code should be 200
@@ -121,6 +127,7 @@ Feature: API Validation
| keywords | sample, test |
| producer | Test Producer |
When I send the API request to the endpoint "/api/v1/misc/update-metadata"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 0
And the response PDF metadata should include "Author" as "John Doe"
testing/cucumber/features/external.feature
+3
View File
@@ -174,6 +174,7 @@ Feature: API Validation
| dpi | 300 |
| imageFormat | <format> |
When I send the API request to the endpoint "/api/v1/convert/pdf/img"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 100
And the response file should have extension ".zip"
@@ -231,6 +232,7 @@ Feature: API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages with random text
When I send the API request to the endpoint "/api/v1/convert/pdf/markdown"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 100
And the response file should have extension ".md"
@@ -244,6 +246,7 @@ Feature: API Validation
| outputFormat | csv |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/convert/pdf/csv"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
And the response file should have extension ".zip"
testing/cucumber/features/filter.feature
+6
View File
@@ -18,6 +18,7 @@ Feature: Filter API Endpoints
| pageCount | <pageCount> |
| comparator | <comparator> |
When I send the API request to the endpoint "/api/v1/filter/filter-page-count"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -59,6 +60,7 @@ Feature: Filter API Endpoints
| fileSize | <fileSize> |
| comparator | <comparator> |
When I send the API request to the endpoint "/api/v1/filter/filter-file-size"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -93,6 +95,7 @@ Feature: Filter API Endpoints
| rotation | 0 |
| comparator | Equal |
When I send the API request to the endpoint "/api/v1/filter/filter-page-rotation"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -170,6 +173,7 @@ Feature: Filter API Endpoints
| standardPageSize | LETTER |
| comparator | Equal |
When I send the API request to the endpoint "/api/v1/filter/filter-page-size"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -242,6 +246,7 @@ Feature: Filter API Endpoints
| text | FINDME |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/filter/filter-contains-text"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -281,6 +286,7 @@ Feature: Filter API Endpoints
| parameter | value |
| pageNumbers | all |
When I send the API request to the endpoint "/api/v1/filter/filter-contains-image"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/form_advanced.feature
+1
View File
@@ -8,6 +8,7 @@ Feature: Advanced Forms API Validation (JSON data parts)
And the pdf has form fields
And the request includes a JSON part "data" with content "{}"
When I send the API request to the endpoint "/api/v1/form/fill"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/forms.feature
+35
View File
@@ -6,6 +6,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/fields"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -24,6 +25,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/fields-with-coordinates"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 0
@@ -67,6 +69,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/modify-fields"
And this operation is run 5 times in parallel
Then the response status code should be 400
@modify-fields @negative
@@ -81,6 +84,7 @@ Feature: Forms API Validation
Given I generate a PDF file as "file"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/delete-fields"
And this operation is run 5 times in parallel
Then the response status code should be 400
@delete-fields @negative
@@ -89,3 +93,34 @@ Feature: Forms API Validation
And the pdf contains 4 pages
When I send the API request to the endpoint "/api/v1/form/delete-fields"
Then the response status code should be 400
@extract-csv @positive
Scenario: extract-csv returns CSV for a form PDF
Given I generate a PDF file as "file"
And the pdf contains 2 pages
And the pdf has form fields
When I send the API request to the endpoint "/api/v1/form/extract-csv"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "text/csv"
@extract-xlsx @positive
Scenario: extract-xlsx returns a spreadsheet for a form PDF
Given I generate a PDF file as "file"
And the pdf contains 2 pages
And the pdf has form fields
When I send the API request to the endpoint "/api/v1/form/extract-xlsx"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
@extract-csv @negative
Scenario: extract-csv rejects a upload sent under the wrong part name
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/form/extract-csv"
Then the response status code should be 400
And the response JSON error should contain "file"
testing/cucumber/features/general.feature
+3
View File
@@ -12,6 +12,7 @@ Feature: API Validation
| verticalDivisions | <verticalDivisions> |
| merge | true |
When I send the API request to the endpoint "/api/v1/general/split-pdf-by-sections"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response file should have size greater than 200
And the response status code should be 200
@@ -34,6 +35,7 @@ Feature: API Validation
| fileInput | fileInput |
| pageNumbers | <pageNumbers> |
When I send the API request to the endpoint "/api/v1/general/split-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/octet-stream"
And the response status code should be 200
And the response file should have size greater than 200
@@ -57,6 +59,7 @@ Feature: API Validation
| splitType | <splitType> |
| splitValue | <splitValue> |
When I send the API request to the endpoint "/api/v1/general/split-by-size-or-count"
And this operation is run 5 times in parallel
Then the response content type should be "application/octet-stream"
And the response status code should be 200
And the response file should have size greater than 200
testing/cucumber/features/general_new.feature
+60
View File
@@ -10,6 +10,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| angle | <angle> |
When I send the API request to the endpoint "/api/v1/general/rotate-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -41,6 +42,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pageNumbers | 3 |
When I send the API request to the endpoint "/api/v1/general/remove-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -76,6 +78,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| customMode | <customMode> |
When I send the API request to the endpoint "/api/v1/general/rearrange-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -103,6 +106,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pageSize | <pageSize> |
When I send the API request to the endpoint "/api/v1/general/scale-pages"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -127,6 +131,7 @@ Feature: General PDF Operations API Validation
| width | 50 |
| height | 50 |
When I send the API request to the endpoint "/api/v1/general/crop"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -155,6 +160,7 @@ Feature: General PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 5 pages
When I send the API request to the endpoint "/api/v1/general/pdf-to-single-page"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -199,6 +205,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pagesPerSheet | 9 |
When I send the API request to the endpoint "/api/v1/general/multi-page-layout"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -213,6 +220,7 @@ Feature: General PDF Operations API Validation
| parameter | value |
| pagesPerSheet | 2 |
When I send the API request to the endpoint "/api/v1/general/booklet-imposition"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -249,3 +257,55 @@ Feature: General PDF Operations API Validation
# Then the response content type should be "application/pdf"
# And the response status code should be 200
# And the response file should have size greater than 0
@edit-table-of-contents @positive
Scenario: edit-table-of-contents rewrites the outline
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the pdf has bookmarks
And the request data includes
| parameter | value |
| bookmarkData | [{"title":"Intro","pageNumber":1,"children":[]}] |
When I send the API request to the endpoint "/api/v1/general/edit-table-of-contents"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
@edit-text @positive
Scenario: edit-text applies a find and replace across the document
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the pdf pages all contain the text "Hello world"
And the request data includes
| parameter | value |
| edits | [{"find":"Hello","replace":"Goodbye"}] |
When I send the API request to the endpoint "/api/v1/general/edit-text"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
@edit-text @negative
Scenario: edit-text without any operations returns 400
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/general/edit-text"
Then the response status code should be 400
And the response JSON error should contain "find/replace"
@split-for-poster-print @positive
Scenario: split-for-poster-print tiles each page into an archive
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
And the request data includes
| parameter | value |
| horizontalDivisions | 1 |
| verticalDivisions | 1 |
When I send the API request to the endpoint "/api/v1/general/split-for-poster-print"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 200
testing/cucumber/features/info_config.feature
+134
View File
@@ -0,0 +1,134 @@
@info @config
Feature: Config, info and UI-data read APIs
# Cheap read-only JSON endpoints behind the frontend and admin surfaces.
@app-config @positive
Scenario: app-config returns the frontend configuration
When I send a GET request to "/api/v1/config/app-config"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
@endpoints-availability @positive
Scenario: endpoints-availability reports per-endpoint availability
When I send a GET request to "/api/v1/config/endpoints-availability"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
@endpoints-enabled @positive
Scenario: endpoints-enabled reports the status of a named endpoint
When I send a GET request to "/api/v1/config/endpoints-enabled" with parameters
| parameter | value |
| endpoints | merge-pdfs |
Then the response status code should be 200
And the response content type should be "application/json"
@endpoints-enabled @negative
Scenario: endpoints-enabled without the endpoints parameter returns 400
When I send a GET request to "/api/v1/config/endpoints-enabled"
Then the response status code should be 400
@login-disclaimer @positive
Scenario: login-disclaimer returns its configuration
When I send a GET request to "/api/v1/config/login-disclaimer"
Then the response status code should be 200
And the response JSON field "enabled" should be false
@health @positive
Scenario: info health reports the running version
When I send a GET request to "/api/v1/info/health"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON field "status" should equal "UP"
And the response JSON field "version" should not be empty
@metrics @positive
Scenario Outline: metrics endpoints return a numeric aggregate
When I send a GET request to "<endpoint>"
Then the response status code should be 200
And the response content type should be "application/json"
And the response should match the regex "^[0-9.]+$"
Examples:
| endpoint |
| /api/v1/info/load/unique |
| /api/v1/info/requests/unique |
@metrics @positive
Scenario Outline: metrics list endpoints return a JSON list
When I send a GET request to "<endpoint>"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON should be a list
Examples:
| endpoint |
| /api/v1/info/load/all/unique |
| /api/v1/info/requests/all/unique |
@metrics @positive
Scenario: weekly active users reports tracking metadata
When I send a GET request to "/api/v1/info/wau"
Then the response status code should be 200
And the response JSON field "trackingSince" should not be empty
@settings @positive
Scenario: get-endpoints-status returns the endpoint toggle map
When I send a GET request to "/api/v1/settings/get-endpoints-status"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
@ui-data @positive
Scenario Outline: UI data endpoints return JSON for the frontend
When I send a GET request to "<endpoint>"
Then the response status code should be 200
And the response content type should be "application/json"
Examples:
| endpoint |
| /api/v1/ui-data/footer-info |
| /api/v1/ui-data/home |
| /api/v1/ui-data/licenses |
| /api/v1/ui-data/pipeline |
@ui-data @positive
Scenario: OCR UI data lists the installed tesseract languages
When I send a GET request to "/api/v1/ui-data/ocr-pdf"
Then the response status code should be 200
And the response JSON field "languages" should be a list
@ui-data @positive
Scenario: Sign UI data lists the available signature fonts
When I send a GET request to "/api/v1/ui-data/sign"
Then the response status code should be 200
And the response JSON field "fonts" should be a list
@hardware-signing @positive
Scenario: Hardware signing capabilities are reported for the server build
When I send a GET request to "/api/v1/security/cert-sign/hardware/capabilities"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON field "desktop" should be false
@hardware-signing @negative
Scenario: Windows certificate store is rejected outside the desktop app
When I send a GET request to "/api/v1/security/cert-sign/hardware/windows-certificates"
Then the response status code should be 400
And the response JSON error should contain "desktop"
testing/cucumber/features/merge_overlay.feature
+2
View File
@@ -7,6 +7,7 @@ Feature: Merge and Overlay PDF API Validation
And the pdf contains 2 pages with random text
And I also generate a PDF file as "fileInput"
When I send the API request to the endpoint "/api/v1/general/merge-pdfs"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -51,6 +52,7 @@ Feature: Merge and Overlay PDF API Validation
| overlayMode | SequentialOverlay |
| overlayPosition | 0 |
When I send the API request to the endpoint "/api/v1/general/overlay-pdfs"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
testing/cucumber/features/misc_new.feature
+32
View File
@@ -48,6 +48,7 @@ Feature: Miscellaneous PDF Operations API Validation
| overrideX | -1 |
| overrideY | -1 |
When I send the API request to the endpoint "/api/v1/misc/add-stamp"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -92,6 +93,7 @@ Feature: Miscellaneous PDF Operations API Validation
| position | 2 |
| fontSize | 14 |
When I send the API request to the endpoint "/api/v1/misc/add-page-numbers"
And this operation is run 5 times in parallel against decoy traffic
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 200
@@ -103,6 +105,7 @@ Feature: Miscellaneous PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/misc/unlock-pdf-forms"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -138,6 +141,7 @@ Feature: Miscellaneous PDF Operations API Validation
| replaceAndInvertOption | HIGH_CONTRAST_COLOR |
| highContrastColorCombination | WHITE_TEXT_ON_BLACK |
When I send the API request to the endpoint "/api/v1/misc/replace-invert-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -163,6 +167,7 @@ Feature: Miscellaneous PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
When I send the API request to the endpoint "/api/v1/misc/decompress-pdf"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -188,6 +193,7 @@ Feature: Miscellaneous PDF Operations API Validation
| parameter | value |
| useFirstTextAsFallback | true |
When I send the API request to the endpoint "/api/v1/misc/auto-rename"
And this operation is run 5 times in parallel
Then the response content type should be "application/pdf"
And the response status code should be 200
And the response file should have size greater than 0
@@ -212,6 +218,7 @@ Feature: Miscellaneous PDF Operations API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/misc/show-javascript"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response file should have size greater than 0
@@ -222,3 +229,28 @@ Feature: Miscellaneous PDF Operations API Validation
And the pdf contains 5 pages with random text
When I send the API request to the endpoint "/api/v1/misc/show-javascript"
Then the response status code should be 200
@auto-rotate-pdf @positive
Scenario: auto-rotate-pdf returns a PDF with the page count preserved
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages with random text
When I send the API request to the endpoint "/api/v1/misc/auto-rotate-pdf"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
@add-comments @positive
Scenario: add-comments annotates a page without changing the page count
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the request data includes
| parameter | value |
| comments | [{"pageNumber":1,"x":100,"y":100,"text":"review me","author":"qa"}] |
When I send the API request to the endpoint "/api/v1/misc/add-comments"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response PDF should contain 3 pages
testing/cucumber/features/security_new.feature
+37
View File
@@ -14,6 +14,7 @@ Feature: Security API Validation
| removeLinks | true |
| removeFonts | false |
When I send the API request to the endpoint "/api/v1/security/sanitize-pdf"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -54,6 +55,7 @@ Feature: Security API Validation
| wholeWordSearch| true |
| convertPDFToImage | false |
When I send the API request to the endpoint "/api/v1/security/auto-redact"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -96,6 +98,7 @@ Feature: Security API Validation
| pageNumbers | 2,4 |
| pageRedactionColor | #000000 |
When I send the API request to the endpoint "/api/v1/security/redact"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -130,6 +133,7 @@ Feature: Security API Validation
Scenario: Verify PDF-A compliance
Given I use an example file at "exampleFiles/pdfa1.pdf" as parameter "fileInput"
When I send the API request to the endpoint "/api/v1/security/verify-pdf"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/json"
And the response file should have size greater than 2
@@ -148,6 +152,7 @@ Feature: Security API Validation
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/security/remove-cert-sign"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@@ -161,3 +166,35 @@ Feature: Security API Validation
Then the response status code should be 200
And the response content type should be "application/pdf"
And the response file should have size greater than 0
@validate-signature @positive
Scenario: validate-signature reports no signatures on an unsigned PDF
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
When I send the API request to the endpoint "/api/v1/security/validate-signature"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response JSON should be a list
@redact-execute @positive
Scenario: redact-execute removes the targeted text
Given I generate a PDF file as "fileInput"
And the pdf contains 3 pages
And the pdf pages all contain the text "Hello world"
And the request data includes
| parameter | value |
| textValues | Hello |
When I send the API request to the endpoint "/api/v1/security/redact-execute"
And this operation is run 5 times in parallel
Then the response status code should be 200
And the response content type should be "application/pdf"
@redact-execute @negative
Scenario: redact-execute without any targets returns 400
Given I generate a PDF file as "fileInput"
And the pdf contains 2 pages
When I send the API request to the endpoint "/api/v1/security/redact-execute"
Then the response status code should be 400
testing/cucumber/features/steps/job_step_definitions.py
+171
View File
@@ -0,0 +1,171 @@
"""Steps for the async job API. DELETE is a cancel, so it 400s once the job finishes."""
import time
import requests
from behave import then, when
from job_support import API_HEADERS, BASE_URL, trigger_cleanup
POLL_TIMEOUT_SECONDS = 60
@when("I store the job id from the response")
def step_store_job_id(context):
payload = context.response.json()
context.job_id = payload.get("jobId")
assert context.job_id, f"No jobId in async submit response: {payload}"
@when("I wait for the job to complete")
def step_wait_for_job(context):
deadline = time.time() + POLL_TIMEOUT_SECONDS
while time.time() < deadline:
response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}",
headers=API_HEADERS, timeout=30,
)
assert response.status_code == 200, (
f"Job status returned {response.status_code}: {response.text}"
)
context.response = response
payload = response.json()
if payload.get("complete"):
context.job_status = payload
return
time.sleep(0.2)
raise AssertionError(
f"Job {context.job_id} did not complete within {POLL_TIMEOUT_SECONDS}s"
)
@when("I request the job result")
def step_request_job_result(context):
context.response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}/result",
headers=API_HEADERS, timeout=60,
)
@when("I request the job result file list")
def step_request_job_result_files(context):
context.response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}/result/files",
headers=API_HEADERS, timeout=60,
)
files = context.response.json().get("files") or []
context.job_files = files
if files:
context.job_file_id = files[0].get("fileId")
@when("I download the first job result file")
def step_download_job_file(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
context.response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}",
headers=API_HEADERS, timeout=60,
)
@when("I request the first job result file metadata")
def step_job_file_metadata(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
context.response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}/metadata",
headers=API_HEADERS, timeout=60,
)
@when("I cancel the job")
def step_cancel_job(context):
context.response = requests.delete(
f"{BASE_URL}/api/v1/general/job/{context.job_id}",
headers=API_HEADERS, timeout=30,
)
@then("the job result file list should contain at least {count:d} file(s)")
def step_check_job_file_count(context, count):
files = context.response.json().get("files") or []
assert len(files) >= count, f"Expected at least {count} result file(s), got {len(files)}"
@then("the job should be reported complete")
def step_check_job_complete(context):
payload = context.response.json()
assert payload.get("complete") is True, f"Job not complete: {payload}"
assert not payload.get("error"), f"Job reported an error: {payload.get('error')}"
# --- Cleanup: the async job files must actually go away, not just age out ---
@when("I trigger the async job cleanup")
def step_trigger_cleanup(context):
context.response = trigger_cleanup()
try:
context.cleanup_summary = context.response.json()
except ValueError:
context.cleanup_summary = {}
@then("the cleanup should report at least {count:d} job(s) removed")
def step_check_cleanup_jobs(context, count):
removed = context.cleanup_summary.get("jobsRemoved")
assert removed is not None, f"No jobsRemoved in cleanup response: {context.cleanup_summary}"
assert removed >= count, f"Expected at least {count} job(s) removed, got {removed}"
@then("the cleanup should report at least {count:d} file(s) deleted")
def step_check_cleanup_files(context, count):
deleted = context.cleanup_summary.get("filesDeleted")
assert deleted is not None, f"No filesDeleted in cleanup response: {context.cleanup_summary}"
assert deleted >= count, f"Expected at least {count} file(s) deleted, got {deleted}"
@then("the cleanup should report nothing left to remove")
def step_check_cleanup_idempotent(context):
removed = context.cleanup_summary.get("jobsRemoved")
deleted = context.cleanup_summary.get("filesDeleted")
assert removed == 0 and deleted == 0, (
"A repeat cleanup still found work to do, so the first pass did not fully clean up: "
f"{context.cleanup_summary}"
)
@then("the job should no longer exist")
def step_check_job_gone(context):
response = requests.get(
f"{BASE_URL}/api/v1/general/job/{context.job_id}",
headers=API_HEADERS, timeout=30,
)
assert response.status_code == 404, (
f"Job {context.job_id} still exists after cleanup: "
f"{response.status_code} {response.text[:200]}"
)
@then("the job result file should no longer be downloadable")
def step_check_job_file_gone(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}",
headers=API_HEADERS, timeout=30,
)
assert response.status_code == 404, (
f"File {context.job_file_id} still downloadable after cleanup: "
f"{response.status_code} {response.text[:200]}"
)
@then("the job result file should still be downloadable")
def step_check_job_file_still_there(context):
assert getattr(context, "job_file_id", None), "No fileId captured from the result file list"
response = requests.get(
f"{BASE_URL}/api/v1/general/files/{context.job_file_id}",
headers=API_HEADERS, timeout=60,
)
assert response.status_code == 200, (
f"File {context.job_file_id} was not retrievable a second time: "
f"{response.status_code} {response.text[:200]}"
)
assert len(response.content) > 0, "Second download returned an empty body"
testing/cucumber/features/steps/job_support.py
+22
View File
@@ -0,0 +1,22 @@
"""Shared helpers for the async job API.
Support module, not a step module: behave execs everything under features/steps as
step definitions, so anything environment.py needs to import has to live apart from
the @when/@then decorators or they would register twice.
"""
import requests
BASE_URL = "http://localhost:8080"
API_HEADERS = {"X-API-KEY": "123456789"}
CLEANUP_URL = f"{BASE_URL}/api/v1/general/jobs/cleanup"
def trigger_cleanup():
"""Force-expire finished jobs so their stored files are released now.
An async submit persists a copy of the upload plus its results and holds them for
the job retention window (30 minutes by default), which far outlasts a test run.
Calling this keeps the post-run temp-file check strict: anything it still reports
afterwards is a genuine leak rather than a file that simply had not aged out.
"""
return requests.post(CLEANUP_URL, headers=API_HEADERS, timeout=60)
testing/cucumber/features/steps/parallel_step_definitions.py
+47
View File
@@ -0,0 +1,47 @@
"""Concurrency steps, usable either before the request or after it."""
from behave import given, then, when
import parallel_support
def _set_repeat(context, count, decoy=False):
context.parallel_decoy = decoy
if count < 2:
context.parallel_repeat = 1
return
context.parallel_repeat = count
# An asked-for level higher than one already run wins.
if count > getattr(context, "parallel_ran_at", 0):
context.parallel_validated = False
if getattr(context, "parallel_validated", False):
return
# Placed after the request: replay whichever request was just sent.
pending = getattr(context, "parallel_request", None)
if pending is not None:
url, spec, headers, label = pending
parallel_support.validate(context, url, spec, headers, context.response, label)
return
pending_get = getattr(context, "parallel_get", None)
if pending_get is not None:
url, params, headers, label = pending_get
parallel_support.validate_get(context, url, params, headers, context.response, label)
@given("this operation is run {count:d} times in parallel")
@when("this operation is run {count:d} times in parallel")
@then("this operation is run {count:d} times in parallel")
def step_operation_run_in_parallel(context, count):
_set_repeat(context, count)
# Decoy traffic is an equal number of concurrent requests carrying different page
# text, so a response that came from the wrong request becomes visible.
@given("this operation is run {count:d} times in parallel against decoy traffic")
@when("this operation is run {count:d} times in parallel against decoy traffic")
@then("this operation is run {count:d} times in parallel against decoy traffic")
def step_operation_run_in_parallel_with_decoy(context, count):
_set_repeat(context, count, decoy=True)
testing/cucumber/features/steps/parallel_support.py
+478
View File
@@ -0,0 +1,478 @@
"""Re-issues a request concurrently and asserts every response matches the baseline."""
import io
import json as json_module
import os
import re
import sys
import zipfile
from concurrent.futures import ThreadPoolExecutor
from hashlib import sha256
import requests
from pypdf import PdfReader
_UUID_RE = re.compile(r"[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}")
_LONG_NUM_RE = re.compile(r"\d{10,}")
_DATE_RE = re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?")
_VOLATILE_KEY_RE = re.compile(
r"^(id|uuid|.*Id|.*_id|.*[Tt]ime|.*[Dd]ate|timestamp|created.*|modified.*|"
r"updated.*|expires.*|token|traceId|requestId|duration.*|elapsed.*)$"
)
# Only hash text for reasonably sized documents so the check stays cheap.
_MAX_TEXT_PAGES = 25
# Sequential samples taken to separate inherent nondeterminism from a concurrency bug.
NOISE_PROBE_SAMPLES = 3
# Response size is allowed to drift this far before it counts as a difference.
SIZE_TOLERANCE = 0.05
# Collected per-scenario results, printed as a summary in after_all.
VALIDATIONS = []
# A spec replaces open file handles with bytes so it can be replayed from many threads.
def materialize(spec):
"""Turn a captured spec into a fresh `files=` list for one request."""
parts = []
for key, filename, payload, mime in spec:
if filename is None:
parts.append((key, (None, payload) if mime is None else (None, payload, mime)))
else:
parts.append((key, (filename, io.BytesIO(payload), mime)))
return parts
def send(url, spec, headers, timeout=300):
return requests.post(url, files=materialize(spec), headers=headers, timeout=timeout)
def _normalize_name(name):
name = _UUID_RE.sub("<uuid>", name)
return _LONG_NUM_RE.sub("<num>", name)
def _strip_volatile(value):
"""Drop keys whose values legitimately differ between two identical requests."""
if isinstance(value, dict):
return {
k: _strip_volatile(v)
for k, v in sorted(value.items())
if not _VOLATILE_KEY_RE.match(k)
}
if isinstance(value, list):
return [_strip_volatile(v) for v in value]
if isinstance(value, str):
return _normalize_name(value)
return value
def _pdf_fingerprint(body, prefix=""):
parts = {}
try:
reader = PdfReader(io.BytesIO(body))
except Exception as exc:
parts[prefix + "pdf"] = f"unreadable: {type(exc).__name__}"
return parts
parts[prefix + "encrypted"] = reader.is_encrypted
if reader.is_encrypted:
return parts
try:
pages = reader.pages
parts[prefix + "pages"] = len(pages)
except Exception as exc:
parts[prefix + "pdf"] = f"unreadable pages: {type(exc).__name__}"
return parts
if len(pages) <= _MAX_TEXT_PAGES:
try:
text = "\n".join((page.extract_text() or "") for page in pages)
parts[prefix + "text_sha"] = sha256(text.encode("utf-8")).hexdigest()[:16]
except Exception:
pass
return parts
def _entry_sha(entry):
"""Hash an archive entry, normalizing the ids and timestamps EPUB/ODF restamp each run."""
try:
text = entry.decode("utf-8")
if "\x00" not in text:
entry = _DATE_RE.sub("<date>", _normalize_name(text)).encode("utf-8")
except UnicodeDecodeError:
pass
return sha256(entry).hexdigest()[:16]
def _zip_fingerprint(body):
parts = {}
try:
with zipfile.ZipFile(io.BytesIO(body)) as archive:
names = sorted(_normalize_name(n) for n in archive.namelist())
parts["zip_entries"] = len(names)
parts["zip_names"] = names
for index, name in enumerate(sorted(archive.namelist())):
entry = archive.read(name)
if entry[:5] == b"%PDF-":
parts.update(_pdf_fingerprint(entry, prefix=f"zip[{index}]."))
else:
parts[f"zip[{index}].sha"] = _entry_sha(entry)
except Exception as exc:
parts["zip"] = f"unreadable: {type(exc).__name__}"
return parts
def fingerprint(response):
"""Structural signature of a response, ignoring benign per-request variance."""
body = response.content
content_type = (response.headers.get("Content-Type") or "").split(";")[0].strip()
parts = {"status": response.status_code, "content_type": content_type, "size": len(body)}
if "json" in content_type:
try:
parts["json"] = _strip_volatile(json_module.loads(body.decode("utf-8")))
except Exception:
parts["body_sha"] = sha256(body).hexdigest()[:16]
elif body[:5] == b"%PDF-":
parts.update(_pdf_fingerprint(body))
elif body[:2] == b"PK":
parts.update(_zip_fingerprint(body))
elif content_type.startswith("text/") or response.status_code >= 400:
try:
parts["text"] = _normalize_name(body.decode("utf-8", "replace"))[:2000]
except Exception:
parts["body_sha"] = sha256(body).hexdigest()[:16]
else:
parts["body_sha"] = sha256(body).hexdigest()[:16]
return parts
def differing_keys(baseline, other):
return {
key
for key in set(baseline) | set(other)
if key != "size" and baseline.get(key) != other.get(key)
}
def size_differs(baseline, other):
base_size, other_size = baseline.get("size", 0), other.get("size", 0)
return abs(other_size - base_size) > max(64, base_size * SIZE_TOLERANCE)
def compare(baseline, other, ignore=frozenset(), ignore_size=False):
"""Return a list of human-readable differences between two fingerprints."""
diffs = []
for key in sorted(differing_keys(baseline, other) - set(ignore)):
diffs.append(
f"{key}: baseline={_short(baseline.get(key))} parallel={_short(other.get(key))}"
)
if not ignore_size and size_differs(baseline, other):
diffs.append(
f"size: baseline={baseline.get('size', 0)} parallel={other.get('size', 0)} "
f"(differs by more than {SIZE_TOLERANCE:.0%})"
)
return diffs
def _short(value):
text = repr(value)
return text if len(text) <= 160 else text[:157] + "..."
def build_decoy_spec(spec):
"""Clone a spec with each PDF stamped with unique text, or None if not possible.
Structure is preserved so parameters stay valid; only the text differs, which
is what makes bleed visible.
"""
try:
from pypdf import PdfWriter
from reportlab.pdfgen import canvas
except ImportError:
return None
decoy = []
stamped_any = False
for key, filename, payload, mime in spec:
if filename is None or not isinstance(payload, bytes) or payload[:5] != b"%PDF-":
decoy.append((key, filename, payload, mime))
continue
try:
reader = PdfReader(io.BytesIO(payload))
if reader.is_encrypted:
return None
writer = PdfWriter()
for index, page in enumerate(reader.pages):
box = page.mediabox
width, height = float(box.width), float(box.height)
overlay_buffer = io.BytesIO()
overlay_canvas = canvas.Canvas(overlay_buffer, pagesize=(width, height))
overlay_canvas.drawString(
20, max(20.0, height - 20), f"DECOY-MARKER-{index}-do-not-mix"
)
overlay_canvas.showPage()
overlay_canvas.save()
overlay_buffer.seek(0)
page.merge_page(PdfReader(overlay_buffer).pages[0])
writer.add_page(page)
out = io.BytesIO()
writer.write(out)
decoy.append((key, filename, out.getvalue(), mime))
stamped_any = True
except Exception:
return None
return decoy if stamped_any else None
def _run_concurrently(url, specs, headers, timeout):
"""Fire every spec at once and return (response, error) in submission order."""
results = [None] * len(specs)
def _worker(index):
try:
results[index] = (send(url, specs[index], headers, timeout=timeout), None)
except Exception as exc:
results[index] = (None, exc)
with ThreadPoolExecutor(max_workers=len(specs)) as pool:
list(pool.map(_worker, range(len(specs))))
return results
def validate(context, url, spec, headers, baseline, label, timeout=300):
"""Re-issue the request concurrently; no-op unless the repeat count is above 1."""
repeat = getattr(context, "parallel_repeat", 1)
if repeat < 2 or getattr(context, "parallel_validated", False):
return
context.parallel_validated = True
context.parallel_ran_at = repeat
decoy_spec = build_decoy_spec(spec) if getattr(context, "parallel_decoy", False) else None
specs = [spec] * repeat + ([decoy_spec] * repeat if decoy_spec else [])
results = _run_concurrently(url, specs, headers, timeout)
main_results = results[:repeat]
decoy_results = results[repeat:]
baseline_fp = fingerprint(baseline)
noise, noisy_size = frozenset(), False
failures = _collect_failures(
main_results, decoy_results, baseline_fp, repeat, noise, noisy_size
)
if failures:
# Some endpoints are inherently nondeterministic (embedded ids, timestamps,
# deliberate randomness). Re-run sequentially to tell that apart from a real bug.
noise, noisy_size = _probe_noise(url, spec, headers, baseline_fp, timeout)
failures = _collect_failures(
main_results, decoy_results, baseline_fp, repeat, noise, noisy_size
)
VALIDATIONS.append(
{
"label": label,
"repeat": repeat,
"decoy": bool(decoy_spec),
"failed": bool(failures),
"noise": sorted(noise) + (["size"] if noisy_size else []),
}
)
if failures:
raise AssertionError(
f"Parallel consistency failed for {label} at concurrency {repeat}.\n"
f"Two sequential runs agreed on these fields, so the differences below are "
f"caused by running the same operation concurrently.\n"
f"Baseline fingerprint: {_short(baseline_fp)}\n - " + "\n - ".join(failures)
)
def _collect_failures(main_results, decoy_results, baseline_fp, repeat, noise, noisy_size):
decoy_fp = _decoy_reference(decoy_results, baseline_fp, noise, noisy_size)
failures = []
for index, (response, error) in enumerate(main_results):
if error is not None:
failures.append(f"copy {index + 1}/{repeat} raised {type(error).__name__}: {error}")
continue
actual_fp = fingerprint(response)
diffs = compare(baseline_fp, actual_fp, noise, noisy_size)
if not diffs:
continue
if decoy_fp is not None and not compare(
decoy_fp, actual_fp, noise, noisy_size
):
failures.append(
f"copy {index + 1}/{repeat} returned the CONCURRENT DECOY REQUEST'S response "
f"(cross-request bleed)"
)
else:
failures.append(f"copy {index + 1}/{repeat} diverged: " + "; ".join(diffs))
failures.extend(_check_decoys(decoy_results, repeat, noise, noisy_size))
return failures
def _probe_noise(url, spec, headers, baseline_fp, timeout, samples=NOISE_PROBE_SAMPLES):
"""Fields that already vary between uncontended runs, so they prove nothing.
Several samples: high-variance output can look stable across any single pair.
"""
probes = []
for _ in range(samples):
try:
probes.append(fingerprint(send(url, spec, headers, timeout=timeout)))
except Exception:
break
return _noise_from_samples(baseline_fp, probes)
def _noise_from_samples(baseline_fp, probes):
noise = set()
noisy_size = False
for index, probe in enumerate(probes):
for other in [baseline_fp] + probes[:index]:
noise |= differing_keys(other, probe)
noisy_size = noisy_size or size_differs(other, probe)
return frozenset(noise), noisy_size
def validate_get(context, url, params, headers, baseline, label, timeout=60):
"""Concurrency check for read-only GET endpoints."""
repeat = getattr(context, "parallel_repeat", 1)
if repeat < 2 or getattr(context, "parallel_validated", False):
return
context.parallel_validated = True
context.parallel_ran_at = repeat
results = [None] * repeat
def _worker(index):
try:
results[index] = (
requests.get(url, params=params, headers=headers, timeout=timeout),
None,
)
except Exception as exc:
results[index] = (None, exc)
with ThreadPoolExecutor(max_workers=repeat) as pool:
list(pool.map(_worker, range(repeat)))
baseline_fp = fingerprint(baseline)
def _failures(noise, noisy_size):
found = []
for index, (response, error) in enumerate(results):
if error is not None:
found.append(f"copy {index + 1}/{repeat} raised {type(error).__name__}: {error}")
continue
diffs = compare(
baseline_fp,
fingerprint(response),
noise,
noisy_size,
)
if diffs:
found.append(f"copy {index + 1}/{repeat} diverged: " + "; ".join(diffs))
return found
noise, noisy_size = frozenset(), False
failures = _failures(noise, noisy_size)
if failures:
probes = []
for _ in range(NOISE_PROBE_SAMPLES):
try:
probes.append(
fingerprint(
requests.get(url, params=params, headers=headers, timeout=timeout)
)
)
except Exception:
break
noise, noisy_size = _noise_from_samples(baseline_fp, probes)
failures = _failures(noise, noisy_size)
VALIDATIONS.append(
{
"label": label,
"repeat": repeat,
"decoy": False,
"failed": bool(failures),
"noise": sorted(noise) + (["size"] if noisy_size else []),
}
)
if failures:
raise AssertionError(
f"Parallel consistency failed for GET {label} at concurrency {repeat}.\n - "
+ "\n - ".join(failures)
)
def _decoy_reference(decoy_results, baseline_fp, noise, noisy_size):
"""Fingerprint of the decoy response, or None when it is not distinguishable."""
live = [r for r, _e in decoy_results if r is not None]
if not live:
return None
decoy_fp = fingerprint(live[0])
# Some endpoints ignore page content, so the decoy cannot prove anything there.
if not compare(baseline_fp, decoy_fp, noise, noisy_size):
return None
return decoy_fp
def _check_decoys(decoy_results, repeat, noise, noisy_size):
"""Assert the decoy load stayed self-consistent while contending with the main copies."""
if not decoy_results:
return []
live = [(i, r) for i, (r, _e) in enumerate(decoy_results) if r is not None]
if not live:
return ["every decoy request failed to complete"]
failures = []
reference_fp = fingerprint(live[0][1])
for index, response in live[1:]:
diffs = compare(
reference_fp,
fingerprint(response),
noise,
noisy_size,
)
if diffs:
failures.append(f"decoy {index + 1}/{repeat} diverged: " + "; ".join(diffs))
return failures
def print_summary():
if not VALIDATIONS:
return
total = len(VALIDATIONS)
failed = sum(1 for v in VALIDATIONS if v["failed"])
max_repeat = max(v["repeat"] for v in VALIDATIONS)
requests_sent = sum(v["repeat"] * (2 if v["decoy"] else 1) for v in VALIDATIONS)
lines = [
f"\n[PARALLEL] {total - failed}/{total} operations stayed consistent under "
f"concurrency (up to {max_repeat} at once, {requests_sent} concurrent requests sent)."
]
noisy = {}
for entry in VALIDATIONS:
if entry["noise"]:
noisy.setdefault(entry["label"], set()).update(entry["noise"])
if noisy:
lines.append(
f"[PARALLEL] {len(noisy)} endpoint(s) produce nondeterministic output. Those "
f"fields were excluded only after confirming they also vary across "
f"{NOISE_PROBE_SAMPLES} sequential runs:"
)
for label, fields in sorted(noisy.items()):
lines.append(f" {label} -> {', '.join(sorted(fields))}")
# behave's --junit reporter swallows after_all stdout, so bypass any capture.
stream = getattr(sys, "__stderr__", None) or sys.stderr
stream.write("\n".join(lines) + "\n")
stream.flush()
testing/cucumber/features/steps/step_definitions.py
+33 -10
View File
@@ -15,6 +15,8 @@ import zipfile
import re
from PIL import Image, ImageDraw
import parallel_support
API_HEADERS = {"X-API-KEY": "123456789"}
#########
@@ -585,6 +587,8 @@ def step_send_get_request(context, endpoint):
full_url = f"{base_url}{endpoint}"
response = requests.get(full_url, headers=API_HEADERS, timeout=60)
context.response = response
context.parallel_get = (full_url, None, API_HEADERS, endpoint)
parallel_support.validate_get(context, full_url, None, API_HEADERS, response, endpoint)
@when('I send a GET request to "{endpoint}" with parameters')
@@ -594,17 +598,18 @@ def step_send_get_request_with_params(context, endpoint):
full_url = f"{base_url}{endpoint}"
response = requests.get(full_url, params=params, headers=API_HEADERS, timeout=60)
context.response = response
context.parallel_get = (full_url, params, API_HEADERS, endpoint)
parallel_support.validate_get(context, full_url, params, API_HEADERS, response, endpoint)
@when('I send the API request to the endpoint "{endpoint}"')
def step_send_api_request(context, endpoint):
url = f"http://localhost:8080{endpoint}"
def _build_request_spec(context):
"""Capture the multipart payload as replayable bytes rather than file handles."""
files = context.files if hasattr(context, "files") else {}
if not hasattr(context, "request_data") or context.request_data is None:
context.request_data = {}
form_data = []
spec = []
for key, value in context.request_data.items():
# Handle list parameters (like 'languages') - send multiple form fields
# Split comma-separated values or treat single values as single-item lists
@@ -612,32 +617,50 @@ def step_send_api_request(context, endpoint):
# Split by comma if present, otherwise treat as single value
values = [v.strip() for v in value.split(",")] if "," in value else [value]
for val in values:
form_data.append((key, (None, val)))
spec.append((key, None, val, None))
else:
form_data.append((key, (None, value)))
spec.append((key, None, value, None))
def _read(file):
file.seek(0)
payload = file.read()
file.seek(0)
return payload
for key, file in files.items():
mime_type, _ = mimetypes.guess_type(file.name)
mime_type = mime_type or "application/octet-stream"
print(f"form_data {file.name} with {mime_type}")
form_data.append((key, (file.name, file, mime_type)))
spec.append((key, file.name, _read(file), mime_type))
# Multi-file entries (duplicate keys for MultipartFile[] endpoints, e.g. merge-pdfs)
for key, file in getattr(context, "multi_files", []):
mime_type, _ = mimetypes.guess_type(file.name)
mime_type = mime_type or "application/octet-stream"
print(f"form_data (multi) {file.name} with {mime_type}")
form_data.append((key, (file.name, file, mime_type)))
spec.append((key, file.name, _read(file), mime_type))
# JSON multipart parts for @RequestPart endpoints (e.g. /form/fill)
for part_name, json_content in getattr(context, "json_parts", {}).items():
form_data.append((part_name, (None, json_content, "application/json")))
spec.append((part_name, None, json_content, "application/json"))
return spec
@when('I send the API request to the endpoint "{endpoint}"')
def step_send_api_request(context, endpoint):
url = f"http://localhost:8080{endpoint}"
spec = _build_request_spec(context)
# Set timeout to 300 seconds (5 minutes) to prevent infinite hangs
print(f"Sending POST request to {endpoint} with timeout=300s")
response = requests.post(url, files=form_data, headers=API_HEADERS, timeout=300)
response = parallel_support.send(url, spec, API_HEADERS, timeout=300)
context.response = response
# Remembered so a later "run N times in parallel" step can replay this request.
context.parallel_request = (url, spec, API_HEADERS, endpoint)
parallel_support.validate(context, url, spec, API_HEADERS, response, endpoint)
########
# THEN #
testing/cucumber/run-parallel.sh
+143
View File
@@ -0,0 +1,143 @@
#!/bin/bash
# Run the behave suite as N concurrent shards against one server.
# Usage: ./run-parallel.sh [SHARDS] [-- behave args] Env: BASE_URL
set -uo pipefail
SHARDS="${1:-10}"
if [[ "$SHARDS" =~ ^[0-9]+$ ]]; then
shift
else
SHARDS=10
fi
[[ "${1:-}" == "--" ]] && shift
BASE_URL="${BASE_URL:-http://localhost:8080}"
CUCUMBER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WORK_ROOT="$CUCUMBER_DIR/.parallel"
REPORT_DIR="${PARALLEL_REPORT_DIR:-$WORK_ROOT/reports}"
cd "$CUCUMBER_DIR" || exit 1
if ! curl -sf --retry 30 --retry-delay 2 --retry-connrefused --retry-all-errors \
"$BASE_URL/api/v1/info/status" >/dev/null; then
echo "ERROR: no server responding at $BASE_URL"
exit 1
fi
mapfile -t ALL_FEATURES < <(
find features -name '*.feature' \
-not -path 'features/enterprise/*' \
-not -path 'features/multinode/*' | sort
)
if [ "${#ALL_FEATURES[@]}" -eq 0 ]; then
echo "ERROR: no feature files found under $CUCUMBER_DIR/features"
exit 1
fi
# user_management.feature changes the admin password mid-scenario, so a concurrent
# shard logging in then gets a 401. Keep all admin-auth features in one shard.
AUTH_RE='logged in as admin|I login with username|JWT authentication|stored JWT token'
declare -a AUTH_FEATURES=() FEATURES=()
for feature in "${ALL_FEATURES[@]}"; do
if grep -qE "$AUTH_RE" "$feature"; then
AUTH_FEATURES+=("$feature")
else
FEATURES+=("$feature")
fi
done
if [ "${#AUTH_FEATURES[@]}" -gt 0 ]; then
echo "Pinning ${#AUTH_FEATURES[@]} auth-coupled feature(s) to a single shard:"
printf ' %s\n' "${AUTH_FEATURES[@]##*/}"
SHARDS=$((SHARDS - 1))
fi
# The pin above can take SHARDS to 0 or below, which makes the shard loop run zero
# times: every shardable feature is skipped and the run still reports success.
[ "$SHARDS" -lt 0 ] && SHARDS=0
[ "${#FEATURES[@]}" -gt 0 ] && [ "$SHARDS" -lt 1 ] && SHARDS=1
if [ "$SHARDS" -gt "${#FEATURES[@]}" ]; then
echo "Only ${#FEATURES[@]} shardable feature files, reducing shards from $SHARDS"
SHARDS="${#FEATURES[@]}"
fi
rm -rf "$WORK_ROOT"
mkdir -p "$WORK_ROOT" "$REPORT_DIR"
TOTAL_SHARDS=$SHARDS
[ "${#AUTH_FEATURES[@]}" -gt 0 ] && TOTAL_SHARDS=$((SHARDS + 1))
echo "Running ${#ALL_FEATURES[@]} feature files across $TOTAL_SHARDS concurrent shards against $BASE_URL"
declare -a PIDS=()
start_shard() {
local shard=$1
shift
local SHARD_DIR="$WORK_ROOT/shard-$shard"
mkdir -p "$SHARD_DIR"
# Feature files reference exampleFiles/ and behave.ini relative to the CWD.
cp -r "$CUCUMBER_DIR/exampleFiles" "$SHARD_DIR/exampleFiles"
cp "$CUCUMBER_DIR/behave.ini" "$SHARD_DIR/behave.ini"
local assigned=("$@")
(
cd "$SHARD_DIR" || exit 1
uv run --project "$CUCUMBER_DIR/../../engine" --locked --group cucumber \
python -m behave "${assigned[@]}" \
--junit --junit-directory "$REPORT_DIR/shard-$shard" \
--no-capture -f plain "${BEHAVE_EXTRA[@]}" \
>"$REPORT_DIR/shard-$shard.log" 2>&1
) &
PIDS+=($!)
}
declare -a BEHAVE_EXTRA=("$@")
for ((shard = 0; shard < SHARDS; shard++)); do
assigned=()
for ((i = shard; i < ${#FEATURES[@]}; i += SHARDS)); do
assigned+=("$CUCUMBER_DIR/${FEATURES[$i]}")
done
start_shard "$shard" "${assigned[@]}"
done
if [ "${#AUTH_FEATURES[@]}" -gt 0 ]; then
assigned=()
for feature in "${AUTH_FEATURES[@]}"; do
assigned+=("$CUCUMBER_DIR/$feature")
done
start_shard "$SHARDS" "${assigned[@]}"
fi
FAILED=0
for ((shard = 0; shard < TOTAL_SHARDS; shard++)); do
if wait "${PIDS[$shard]}"; then
echo "shard $shard PASSED"
else
echo "shard $shard FAILED"
FAILED=$((FAILED + 1))
fi
done
echo ""
echo "=== Parallel shard summary ==="
grep -h '^[0-9]* scenarios passed' "$REPORT_DIR"/shard-*.log 2>/dev/null || true
grep -h '^\[PARALLEL\] [0-9]*/' "$REPORT_DIR"/shard-*.log 2>/dev/null || true
if [ "$FAILED" -gt 0 ]; then
echo ""
echo "$FAILED of $TOTAL_SHARDS shards failed. Failing scenarios:"
sed -n '/^Failing scenarios:/,/^[0-9]* features/p' "$REPORT_DIR"/shard-*.log 2>/dev/null |
grep 'feature:' | sort -u
echo ""
echo "Parallel consistency failures:"
grep -h -A4 'Parallel consistency failed' "$REPORT_DIR"/shard-*.log 2>/dev/null | head -40
echo ""
echo "Full logs: $REPORT_DIR/shard-*.log"
exit 1
fi
echo "All $TOTAL_SHARDS shards passed."
testing/test.sh
+17 -4
View File
@@ -391,6 +391,17 @@ capture_file_list() {
gha_endgroup
}
# Tail of container logs shown when a leak is detected. The full log is captured as a
# job artifact anyway, and dumping all of it here buried the actual finding.
TEMP_FILE_LOG_TAIL=${TEMP_FILE_LOG_TAIL:-200}
# Print the log context for a detected temp-file leak, bounded so the finding stays readable.
print_temp_file_leak_logs() {
local container_name=$1
echo "Last $TEMP_FILE_LOG_TAIL lines of container logs (full log is in the uploaded artifacts):"
docker logs --tail "$TEMP_FILE_LOG_TAIL" "$container_name" 2>&1 || true
}
# Function to compare before and after file lists
compare_file_lists() {
local before_file=$1
@@ -417,8 +428,7 @@ compare_file_lists() {
if [ -s "${diff_file}.tmp" ]; then
echo "WARNING: Temporary files found:"
cat "${diff_file}.tmp"
echo "Printing docker logs due to temporary file detection:"
docker logs "$container_name" # Print logs when temp files are found
print_temp_file_leak_logs "$container_name"
gha_endgroup
return 1
else
@@ -450,8 +460,11 @@ compare_file_lists() {
if [ -s "${diff_file}.tmp" ]; then
echo "WARNING: Temporary files detected:"
cat "${diff_file}.tmp"
echo "Printing docker logs due to temporary file detection:"
docker logs "$container_name" # Print logs when temp files are found
echo "These files were still present after the suite finished. Async job results"
echo "and their input copies live under the server's file store and are released"
echo "by the cleanup that features/environment.py runs in after_all - if they show"
echo "up here, that cleanup did not cover them."
print_temp_file_leak_logs "$container_name"
return 1
fi
fi