mirror of
https://github.com/Stirling-Tools/Stirling-PDF.git
synced 2026-09-03 05:10:16 +03:00
Add multi-node cluster regression suite (compose stack + behave e2e) (#7026)
# Description of Changes - The multi-node compose stack + behave suite (11 features) - The nightly multinode-e2e job in build-enterprise.yml cuke features are cluster_health - both nodes boot healthy and join the Valkey backplane load_balancing - traffic spreads across nodes; no spurious 401 when bounced cross_node_auth - a token from one node validates on all nodes (shared DB keys) shared_state - teams/sources/org visible from every node policy_management - create/rename/delete a policy on any node, reflected everywhere source_management - source CRUD cross-node; referenced source can't be deleted anywhere connections - S3 connection resolves (secret masked) and deletes cluster-wide processor_ledger - files processed exactly once even when both nodes trigger together policy_run_coordination - a run on one node is visible from every node rate_limiting - rate-limit counters shared via Valkey, not per node failover - LB keeps serving when a node dies; recovered node accepts existing tokens can now start a full node system with export PREMIUM_KEY=<your licence key> ./start-multinode-test.sh starts a 40 person org DB install with multi node and database (--no-seed to have without DB on startup) 4 teams 1 s3 connection 1 policy --- ## Checklist ### General - [ ] I have read the [Contribution Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md) - [ ] I have read the [Stirling-PDF Developer Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md) (if applicable) - [ ] I have read the [How to add new languages to Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md) (if applicable) - [ ] I have performed a self-review of my own code - [ ] My changes generate no new warnings ### Documentation - [ ] I have updated relevant docs on [Stirling-PDF's doc repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/) (if functionality has heavily changed) - [ ] I have read the section [Add New Translation Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags) (for new translation tags only) ### Translations (if applicable) - [ ] I ran [`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md) ### UI Changes (if applicable) - [ ] Screenshots or videos demonstrating the UI changes are attached (e.g., as comments or direct attachments in the PR) ### Testing (if applicable) - [ ] I have run `task check` to verify linters, typechecks, and tests pass - [ ] I have tested my changes locally. Refer to the [Testing Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing) for more details.
This commit is contained in:
Anthony Stirling
@@ -0,0 +1,225 @@
|
||||
# Multi-node Stirling-PDF processor test stack: shared Postgres/MinIO/Valkey behind an nginx LB fronting N app nodes, with enterprise features unlocked via a local test licence key.
|
||||
# Bring up: ./start-multinode-test.sh Validate: ./validate-multinode-test.sh Access: http://localhost:8080 (admin / stirling)
|
||||
|
||||
x-stirling-node: &stirling-node
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: docker/embedded/Dockerfile
|
||||
image: stirling-pdf-multinode:local
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
minio-init:
|
||||
condition: service_completed_successfully
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/api/v1/info/status | grep -q UP"]
|
||||
interval: 10s
|
||||
timeout: 10s
|
||||
retries: 30
|
||||
start_period: 90s
|
||||
environment: &stirling-env
|
||||
# --- Licensing: unlock pro/enterprise (custom DB + cluster mode) ---
|
||||
# Cluster mode is licence-gated - set PREMIUM_KEY (locally: export PREMIUM_KEY=...; CI: the PREMIUM_KEY_ENTERPRISE secret); the default below is a non-functional placeholder, never a real key.
|
||||
PREMIUM_ENABLED: "true"
|
||||
PREMIUM_KEY: "${PREMIUM_KEY:-00000000-0000-0000-0000-000000000000}"
|
||||
DISABLE_ADDITIONAL_FEATURES: "false"
|
||||
|
||||
# --- Login + teams ---
|
||||
DOCKER_ENABLE_SECURITY: "true"
|
||||
SECURITY_ENABLELOGIN: "true"
|
||||
SECURITY_INITIALLOGIN_USERNAME: "admin"
|
||||
SECURITY_INITIALLOGIN_PASSWORD: "stirling"
|
||||
# Global API key: lets the seed/validate scripts call admin APIs without a login flow.
|
||||
SECURITY_CUSTOMGLOBALAPIKEY: "multinode-test-key"
|
||||
|
||||
# --- Shared Postgres (the single DB every node coordinates through) ---
|
||||
# enableCustomDatabase is an enterprise feature, unlocked by PREMIUM_* above.
|
||||
SYSTEM_DATASOURCE_ENABLECUSTOMDATABASE: "true"
|
||||
SYSTEM_DATASOURCE_TYPE: "postgresql"
|
||||
SYSTEM_DATASOURCE_HOSTNAME: "postgres"
|
||||
SYSTEM_DATASOURCE_PORT: "5432"
|
||||
SYSTEM_DATASOURCE_NAME: "stirling"
|
||||
SYSTEM_DATASOURCE_USERNAME: "stirling"
|
||||
SYSTEM_DATASOURCE_PASSWORD: "stirling"
|
||||
# User.settings has no @Lob annotation - with one, Postgres treats it as a large object and every login 500s ("Large Objects may not be used in auto-commit mode"); see multinode/README.md.
|
||||
|
||||
# --- Shared object storage (persistent user uploads + share/store feature) ---
|
||||
STORAGE_ENABLED: "true"
|
||||
STORAGE_PROVIDER: "s3"
|
||||
STORAGE_S3_ENDPOINT: "http://minio:9000"
|
||||
STORAGE_S3_BUCKET: "stirling-storage"
|
||||
STORAGE_S3_REGION: "us-east-1"
|
||||
STORAGE_S3_ACCESSKEY: "minioadmin"
|
||||
STORAGE_S3_SECRETKEY: "minioadmin"
|
||||
STORAGE_S3_PATHSTYLEACCESS: "true"
|
||||
STORAGE_S3_ALLOWPRIVATEENDPOINTS: "true"
|
||||
|
||||
# --- Cluster mode: Valkey backplane + shared S3 job-artifact store ---
|
||||
# artifactStore=s3 is REQUIRED for multi-node (transient job artifacts must be shared).
|
||||
CLUSTER_ENABLED: "true"
|
||||
CLUSTER_BACKPLANE: "valkey"
|
||||
CLUSTER_ARTIFACTSTORE: "s3"
|
||||
CLUSTER_VALKEY_URL: "redis://valkey:6379"
|
||||
# SPRING_DATA_REDIS_REPOSITORIES_ENABLED is not needed: DataRedisRepositoriesAutoConfiguration is excluded (see application.properties, multinode/README.md).
|
||||
|
||||
# --- Shared credential-encryption key (REQUIRED in cluster mode) ---
|
||||
# AES-256 key that encrypts stored integration/S3 secrets - must match on every node or secrets encrypted on one can't decrypt on another; boot fails if unset with cluster.enabled=true (test-only value; JWT keys persist separately in the shared DB).
|
||||
STIRLING_CREDENTIAL_ENCRYPTION_KEY: "dMobekyUEnEV7WHBah2FkbboP4Coqifd3JRXB00LiIY="
|
||||
|
||||
# --- Policy / processor subsystem (the thing under test) ---
|
||||
POLICIES_ENABLED: "true"
|
||||
# Let policy S3 sources/webhook-staging connections point at the in-cluster MinIO.
|
||||
POLICIES_ALLOWPRIVATES3ENDPOINTS: "true"
|
||||
|
||||
# --- Misc features on ---
|
||||
METRICS_ENABLED: "true"
|
||||
SYSTEM_DEFAULTLOCALE: "en-US"
|
||||
SYSTEM_MAXFILESIZE: "100"
|
||||
UI_APPNAME: "Stirling-PDF Multi-Node"
|
||||
UI_APPNAMENAVBAR: "Stirling Multi-Node"
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
services:
|
||||
# ---- Shared Postgres: the single DB all nodes coordinate through ----------
|
||||
postgres:
|
||||
image: postgres:17-alpine
|
||||
container_name: multinode-postgres
|
||||
environment:
|
||||
POSTGRES_USER: stirling
|
||||
POSTGRES_PASSWORD: stirling
|
||||
POSTGRES_DB: stirling
|
||||
ports:
|
||||
- "5434:5432" # host access for inspection (psql -h localhost -p 5434 -U stirling)
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U stirling -d stirling"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
# ---- Valkey: cluster backplane (shared job state / rate limiting) ---------
|
||||
valkey:
|
||||
image: valkey/valkey:8-alpine
|
||||
container_name: multinode-valkey
|
||||
command: ["valkey-server", "--save", "", "--appendonly", "no"]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "valkey-cli ping | grep -q PONG"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
# ---- MinIO: shared S3 (platform storage + policy connections + webhook staging) ----
|
||||
minio:
|
||||
image: minio/minio:latest
|
||||
container_name: multinode-minio
|
||||
command: ["server", "/data", "--console-address", ":9001"]
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minioadmin
|
||||
ports:
|
||||
- "9000:9000" # S3 API
|
||||
- "9001:9001" # web console (minioadmin / minioadmin)
|
||||
volumes:
|
||||
- minio-data:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "mc ready local || exit 1"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
# ---- One-shot: create the buckets the stack needs -------------------------
|
||||
minio-init:
|
||||
image: minio/mc:latest
|
||||
container_name: multinode-minio-init
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_healthy
|
||||
entrypoint: >
|
||||
/bin/sh -c "
|
||||
mc alias set local http://minio:9000 minioadmin minioadmin &&
|
||||
mc mb --ignore-existing local/stirling-storage &&
|
||||
mc mb --ignore-existing local/policy-data &&
|
||||
echo 'buckets ready: stirling-storage, policy-data'
|
||||
"
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
# ---- App node 1 -----------------------------------------------------------
|
||||
stirling-1:
|
||||
<<: *stirling-node
|
||||
container_name: multinode-stirling-1
|
||||
environment:
|
||||
<<: *stirling-env
|
||||
SYSTEM_NODEID: "node-1"
|
||||
|
||||
# ---- App node 2 -----------------------------------------------------------
|
||||
# No ordering vs node-1 needed - each node mints its own signing key at boot and publishes the public half to the shared DB, so both verify each other's tokens.
|
||||
stirling-2:
|
||||
<<: *stirling-node
|
||||
container_name: multinode-stirling-2
|
||||
environment:
|
||||
<<: *stirling-env
|
||||
SYSTEM_NODEID: "node-2"
|
||||
|
||||
# ---- One-shot seeder: teams, ~40 users, S3 connection, policies -----------
|
||||
# Profile-gated so plain `docker compose up` skips it; the start script runs it once, using postgres:alpine for psql with curl+jq added for the HTTP calls.
|
||||
seed:
|
||||
image: postgres:17-alpine
|
||||
container_name: multinode-seed
|
||||
profiles: ["seed"]
|
||||
depends_on:
|
||||
stirling-1:
|
||||
condition: service_healthy
|
||||
stirling-2:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
BASE_URL: "http://nginx:8080"
|
||||
PGHOST: "postgres"
|
||||
PGUSER: "stirling"
|
||||
PGPASSWORD: "stirling"
|
||||
PGDATABASE: "stirling"
|
||||
USER_COUNT: "40"
|
||||
volumes:
|
||||
- ./multinode/seed.sh:/seed.sh:ro
|
||||
entrypoint: ["/bin/sh", "-c", "apk add --no-cache curl jq >/dev/null && sh /seed.sh"]
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
# ---- nginx load balancer: single entrypoint round-robining the nodes ------
|
||||
nginx:
|
||||
image: nginx:1.27-alpine
|
||||
container_name: multinode-nginx
|
||||
depends_on:
|
||||
stirling-1:
|
||||
condition: service_healthy
|
||||
stirling-2:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./multinode/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
networks:
|
||||
- stirling-multinode
|
||||
|
||||
networks:
|
||||
stirling-multinode:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
minio-data:
|
||||
@@ -0,0 +1,49 @@
|
||||
worker_processes 1;
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
# Round-robin across the app nodes. Add a node here to scale out.
|
||||
# max_fails=1 marks a node down after a single failure so a dead node drains fast.
|
||||
upstream stirling_nodes {
|
||||
server stirling-1:8080 max_fails=1 fail_timeout=10s;
|
||||
server stirling-2:8080 max_fails=1 fail_timeout=10s;
|
||||
}
|
||||
|
||||
# Large uploads (SYSTEM_MAXFILESIZE=100MB) plus headroom.
|
||||
client_max_body_size 200m;
|
||||
|
||||
map $http_upgrade $connection_upgrade {
|
||||
default upgrade;
|
||||
'' close;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
|
||||
location / {
|
||||
proxy_pass http://stirling_nodes;
|
||||
|
||||
# Graceful failover: retries the other node only on connection-level failures (unreachable/timeout), never on 5xx, so a POST a node already started is never re-sent.
|
||||
proxy_next_upstream error timeout;
|
||||
proxy_next_upstream_tries 2;
|
||||
proxy_connect_timeout 3s;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# WebSocket / SSE upgrade support (policy run streaming).
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
|
||||
# Stream server-sent events straight through, don't buffer.
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 3600s;
|
||||
|
||||
# Surface which app node served the request, so the validate script can prove the LB is spreading load.
|
||||
add_header X-Served-By $upstream_addr always;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
#!/bin/sh
|
||||
# Seeds a running multi-node stack: 4 teams, ~40 users, an S3 connection, a scheduled S3 policy, and a webhook source if the build supports it.
|
||||
# Auth uses the Bearer JWT from the login response body (not a cookie) since the global API key can't create teams.
|
||||
# Idempotent-ish: re-running skips existing teams/users; each step is best-effort and logs failures without aborting.
|
||||
set -u
|
||||
|
||||
BASE_URL="${BASE_URL:-http://localhost:8080}"
|
||||
ADMIN_USER="${ADMIN_USER:-admin}"
|
||||
ADMIN_PASS="${ADMIN_PASS:-stirling}"
|
||||
USER_COUNT="${USER_COUNT:-40}"
|
||||
USER_PASS="${USER_PASS:-Password123!}"
|
||||
PGHOST="${PGHOST:-postgres}"
|
||||
PGUSER="${PGUSER:-stirling}"
|
||||
PGPASSWORD="${PGPASSWORD:-stirling}"
|
||||
PGDATABASE="${PGDATABASE:-stirling}"
|
||||
export PGPASSWORD
|
||||
|
||||
TEAMS="Engineering Finance Legal Operations"
|
||||
|
||||
log() { echo "[seed] $*"; }
|
||||
psqlq() { psql -h "$PGHOST" -U "$PGUSER" -d "$PGDATABASE" -tAc "$1" 2>/dev/null | tr -d '[:space:]'; }
|
||||
|
||||
# --- wait for the load balancer to serve a healthy app -----------------------
|
||||
log "waiting for $BASE_URL ..."
|
||||
i=0
|
||||
until curl -fsS "$BASE_URL/api/v1/info/status" 2>/dev/null | grep -q UP; do
|
||||
i=$((i+1)); [ "$i" -gt 120 ] && { log "timed out waiting for API"; exit 1; }
|
||||
sleep 3
|
||||
done
|
||||
log "API is up"
|
||||
|
||||
# --- admin login -> Bearer token ---------------------------------------------
|
||||
code=$(curl -sS -o /tmp/login.json -w '%{http_code}' \
|
||||
-X POST "$BASE_URL/api/v1/auth/login" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"username\":\"$ADMIN_USER\",\"password\":\"$ADMIN_PASS\"}")
|
||||
log "admin login: HTTP $code"
|
||||
[ "$code" = "200" ] || { log "login failed: $(cat /tmp/login.json)"; exit 1; }
|
||||
TOKEN=$(jq -r '.session.access_token' </tmp/login.json)
|
||||
[ -n "$TOKEN" ] && [ "$TOKEN" != "null" ] || { log "no access_token in login response"; exit 1; }
|
||||
|
||||
auth() { curl -sS -H "Authorization: Bearer $TOKEN" "$@"; }
|
||||
|
||||
# --- teams -------------------------------------------------------------------
|
||||
for t in $TEAMS; do
|
||||
code=$(auth -o /dev/null -w '%{http_code}' -X POST "$BASE_URL/api/v1/team/create" \
|
||||
--data-urlencode "name=$t")
|
||||
log "team '$t': HTTP $code"
|
||||
done
|
||||
|
||||
# Resolve team ids from the DB (no admin list endpoint self-hosted).
|
||||
seed_team_ids=""
|
||||
for t in $TEAMS; do
|
||||
id=$(psqlq "select team_id from teams where name='$t' limit 1")
|
||||
[ -n "$id" ] && seed_team_ids="$seed_team_ids $id"
|
||||
done
|
||||
set -- $seed_team_ids
|
||||
team_count=$#
|
||||
log "seedable team ids:$seed_team_ids (count=$team_count)"
|
||||
|
||||
# --- users: spread across teams, first two are admins ------------------------
|
||||
created=0; failed=0
|
||||
n=1
|
||||
while [ "$n" -le "$USER_COUNT" ]; do
|
||||
uname=$(printf "user%02d@stirling.test" "$n")
|
||||
role="ROLE_USER"; [ "$n" -le 2 ] && role="ROLE_ADMIN"
|
||||
team_id=""
|
||||
if [ "$team_count" -gt 0 ]; then
|
||||
idx=$(( (n % team_count) + 1 )); team_id=$(eval echo "\${$idx}")
|
||||
fi
|
||||
code=$(auth -o /tmp/user.json -w '%{http_code}' -X POST "$BASE_URL/api/v1/user/admin/saveUser" \
|
||||
--data-urlencode "username=$uname" \
|
||||
--data-urlencode "password=$USER_PASS" \
|
||||
--data-urlencode "role=$role" \
|
||||
${team_id:+--data-urlencode "teamId=$team_id"} \
|
||||
--data-urlencode "authType=WEB" \
|
||||
--data-urlencode "forceChange=false")
|
||||
case "$code" in
|
||||
200|201) created=$((created+1));;
|
||||
409) log "user $uname already exists";;
|
||||
*) failed=$((failed+1)); [ "$failed" -le 3 ] && log "user $uname failed HTTP $code: $(cat /tmp/user.json)";;
|
||||
esac
|
||||
n=$((n+1))
|
||||
done
|
||||
log "users created: $created (failed: $failed, requested: $USER_COUNT)"
|
||||
|
||||
# --- S3 connection -> the in-cluster MinIO 'policy-data' bucket ---------------
|
||||
conn_body=$(cat <<JSON
|
||||
{"integrationType":"S3","name":"MinIO policy bucket","scope":"SERVER","enabled":true,"locked":false,"defaultAccess":"ORG_ALL",
|
||||
"config":{"bucket":"policy-data","region":"us-east-1","endpoint":"http://minio:9000","accessKeyId":"minioadmin","secretAccessKey":"minioadmin","pathStyleAccess":true}}
|
||||
JSON
|
||||
)
|
||||
conn_id=$(auth -X POST "$BASE_URL/api/v1/integrations" -H 'Content-Type: application/json' -d "$conn_body" \
|
||||
| jq -r '.id // empty' 2>/dev/null)
|
||||
log "S3 connection id: ${conn_id:-<none>}"
|
||||
|
||||
# --- a scheduled S3 -> compress -> S3 policy ---------------------------------
|
||||
if [ -n "${conn_id:-}" ]; then
|
||||
src_body=$(cat <<JSON
|
||||
{"name":"Incoming S3","type":"s3","enabled":true,
|
||||
"options":{"connectionId":$conn_id,"prefix":"incoming/","mode":"consume"}}
|
||||
JSON
|
||||
)
|
||||
src_id=$(auth -X POST "$BASE_URL/api/v1/sources" -H 'Content-Type: application/json' -d "$src_body" \
|
||||
| jq -r '.id // empty' 2>/dev/null)
|
||||
log "S3 source id: ${src_id:-<none>}"
|
||||
|
||||
if [ -n "${src_id:-}" ]; then
|
||||
pol_body=$(cat <<JSON
|
||||
{"name":"Compress incoming PDFs","enabled":true,
|
||||
"trigger":{"type":"schedule","options":{"schedule":{"type":"every","count":5,"unit":"MINUTES"}}},
|
||||
"sourceIds":["$src_id"],
|
||||
"steps":[{"operation":"/api/v1/misc/compress-pdf","parameters":{}}],
|
||||
"output":{"type":"s3","options":{"connectionId":$conn_id,"prefix":"processed/"}}}
|
||||
JSON
|
||||
)
|
||||
code=$(auth -o /tmp/pol.json -w '%{http_code}' -X POST "$BASE_URL/api/v1/policies" \
|
||||
-H 'Content-Type: application/json' -d "$pol_body")
|
||||
log "policy create: HTTP $code $( [ "$code" != 200 ] && head -c 160 /tmp/pol.json )"
|
||||
fi
|
||||
|
||||
# --- webhook source + policy (only if this build has the webhook type) -----
|
||||
wh_body=$(cat <<JSON
|
||||
{"name":"Partner webhook","type":"webhook","enabled":true,
|
||||
"options":{"connectionId":$conn_id,"mode":"consume"}}
|
||||
JSON
|
||||
)
|
||||
wh=$(auth -o /tmp/wh.json -w '%{http_code}' -X POST "$BASE_URL/api/v1/sources" \
|
||||
-H 'Content-Type: application/json' -d "$wh_body")
|
||||
if [ "$wh" = "200" ] || [ "$wh" = "201" ]; then
|
||||
wh_url=$(jq -r '.options.webhookId // empty' </tmp/wh.json 2>/dev/null)
|
||||
log "webhook source created (deliver to /api/v1/webhooks/$wh_url)"
|
||||
else
|
||||
log "webhook source not created (HTTP $wh) - expected on builds without the webhook branch"
|
||||
fi
|
||||
fi
|
||||
|
||||
log "seed complete."
|
||||
log " login: $ADMIN_USER / $ADMIN_PASS at $BASE_URL"
|
||||
log " users: user01..$(printf '%02d' "$USER_COUNT")@stirling.test / $USER_PASS"
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs the multi-node regression suite (behave features/multinode) against the clustered stack: brings it up if needed, runs non-destructive scenarios then @destructive failover ones, and restores any killed node.
|
||||
# Usage: ./run-multinode-regression.sh [--no-failover] [--no-seed]
|
||||
# @known_gap scenarios are expected to fail - they mark work not yet done, so a non-zero exit is fine while those are open.
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
COMPOSE="docker compose -f docker-compose-multinode.yml"
|
||||
CUKE_DIR="../cucumber"
|
||||
RUN_FAILOVER=1
|
||||
SEED=1
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--no-failover) RUN_FAILOVER=0 ;;
|
||||
--no-seed) SEED=0 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo "==> Ensuring the multi-node stack is up..."
|
||||
if ! docker inspect -f '{{.State.Health.Status}}' multinode-stirling-1 2>/dev/null | grep -q healthy; then
|
||||
./start-multinode-test.sh $([ "$SEED" = 0 ] && echo --no-seed) || exit 1
|
||||
elif [ "$SEED" = 1 ]; then
|
||||
echo " stack already up; seeding (idempotent)..."
|
||||
$COMPOSE --profile seed run --rm seed >/dev/null 2>&1 || echo " (seed reported issues, continuing)"
|
||||
fi
|
||||
|
||||
echo "==> Checking Python + behave..."
|
||||
PY="${PYTHON:-python}"
|
||||
command -v "$PY" >/dev/null || PY=python3
|
||||
if ! "$PY" -c "import behave" 2>/dev/null; then
|
||||
echo " installing test deps..."
|
||||
"$PY" -m pip install -q -r "$CUKE_DIR/requirements.txt" || {
|
||||
echo " could not install behave; install $CUKE_DIR/requirements.txt manually"; exit 1; }
|
||||
fi
|
||||
|
||||
REPORT_DIR="$(pwd)/multinode/regression-report"
|
||||
mkdir -p "$REPORT_DIR"
|
||||
|
||||
run_behave() { # $1=tags $2=label
|
||||
echo "==> behave features/multinode --tags='$1' ($2)"
|
||||
# behave.ini excludes features/multinode by default; -e here overrides that while still excluding the licence-gated enterprise suite.
|
||||
( cd "$CUKE_DIR" && "$PY" -m behave features/multinode -e "features/enterprise" \
|
||||
--tags="$1" --no-capture --format plain --format html --outfile "$REPORT_DIR/$2.html" )
|
||||
return $?
|
||||
}
|
||||
|
||||
rc=0
|
||||
run_behave "~@destructive" "core" || rc=1
|
||||
|
||||
if [ "$RUN_FAILOVER" = 1 ]; then
|
||||
run_behave "@destructive" "failover" || rc=1
|
||||
echo "==> Restoring any killed nodes..."
|
||||
$COMPOSE up -d >/dev/null 2>&1
|
||||
for n in multinode-stirling-1 multinode-stirling-2; do
|
||||
for i in $(seq 1 24); do
|
||||
[ "$(docker inspect -f '{{.State.Health.Status}}' "$n" 2>/dev/null)" = "healthy" ] && break
|
||||
sleep 5
|
||||
done
|
||||
done
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "============================================================"
|
||||
echo " Regression run complete. Reports: $REPORT_DIR"
|
||||
echo " Exit $rc (non-zero = at least one scenario failed;"
|
||||
echo " @known_gap scenarios are expected to fail - see the report)."
|
||||
echo " Stack left running: http://localhost:8080 (admin / stirling)"
|
||||
echo "============================================================"
|
||||
exit $rc
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
# Brings up the multi-node stack (Postgres/Valkey/MinIO/2 app nodes/nginx LB), seeds teams/users/an S3 connection/policies, then leaves it running for manual testing.
|
||||
# Usage: ./start-multinode-test.sh [--no-seed | --down]
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
COMPOSE="docker compose -f docker-compose-multinode.yml"
|
||||
|
||||
if [ "${1:-}" = "--down" ]; then
|
||||
echo "Tearing down multi-node stack + volumes..."
|
||||
$COMPOSE --profile seed down -v --remove-orphans
|
||||
exit 0
|
||||
fi
|
||||
|
||||
SEED=1
|
||||
[ "${1:-}" = "--no-seed" ] && SEED=0
|
||||
|
||||
# Cluster mode is licence-gated. Without a valid key the nodes fail the cluster licence gate at boot.
|
||||
if [ -z "${PREMIUM_KEY:-}" ]; then
|
||||
echo "WARNING: PREMIUM_KEY is not set - cluster mode needs a valid enterprise/pro licence key."
|
||||
echo " Run: export PREMIUM_KEY=<your test licence key> before starting."
|
||||
fi
|
||||
|
||||
echo "==> Building the Stirling image (first run compiles the app; be patient)..."
|
||||
$COMPOSE build
|
||||
|
||||
echo "==> Starting Postgres + Valkey + MinIO + 2 app nodes + nginx..."
|
||||
$COMPOSE up -d
|
||||
|
||||
echo "==> Waiting for both app nodes to report healthy..."
|
||||
for node in multinode-stirling-1 multinode-stirling-2; do
|
||||
for i in $(seq 1 60); do
|
||||
status=$(docker inspect -f '{{.State.Health.Status}}' "$node" 2>/dev/null || echo "starting")
|
||||
[ "$status" = "healthy" ] && { echo " $node: healthy"; break; }
|
||||
[ "$i" = "60" ] && { echo " $node did not become healthy; see: $COMPOSE logs $node"; exit 1; }
|
||||
sleep 5
|
||||
done
|
||||
done
|
||||
|
||||
if [ "$SEED" = "1" ]; then
|
||||
echo "==> Seeding teams / users / S3 connection / policies..."
|
||||
$COMPOSE --profile seed run --rm seed || echo " (seed reported issues; check output above)"
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
|
||||
============================================================================
|
||||
Multi-node Stirling is UP.
|
||||
|
||||
App (via load balancer): http://localhost:8080 (admin / stirling)
|
||||
MinIO console: http://localhost:9001 (minioadmin / minioadmin)
|
||||
Postgres: localhost:5434 (stirling / stirling, db 'stirling')
|
||||
|
||||
Seeded users: user01..user40@stirling.test / Password123!
|
||||
Global API key: multinode-test-key (header: X-API-KEY)
|
||||
|
||||
Try it:
|
||||
./validate-multinode-test.sh # multi-node smoke tests (optional)
|
||||
$COMPOSE logs -f stirling-1 # tail a node
|
||||
./start-multinode-test.sh --down # stop + wipe
|
||||
|
||||
Nodes are reachable directly for cross-node checks:
|
||||
docker compose -f docker-compose-multinode.yml exec stirling-1 curl -s localhost:8080/api/v1/info/status
|
||||
============================================================================
|
||||
EOF
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bash
|
||||
# Multi-node smoke tests against a running stack (start-multinode-test.sh): load-balancer spread, cross-node JWT validation (signing keys persist in the shared DB), and processor state visible from every node.
|
||||
# Auth: extracts the Bearer JWT from the login body via sed (no jq needed host-side) and hits nodes directly with docker exec.
|
||||
# Non-destructive - safe to re-run against the stack at http://localhost:8080.
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
LB="http://localhost:8080"
|
||||
ADMIN_USER="admin"; ADMIN_PASS="stirling"
|
||||
NODES="multinode-stirling-1 multinode-stirling-2"
|
||||
# An authed, admin-visible endpoint that returns 200 with a valid token, 401 without.
|
||||
PROBE="/api/v1/sources"
|
||||
pass=0; fail=0
|
||||
ok() { echo " PASS - $*"; pass=$((pass+1)); }
|
||||
bad() { echo " FAIL - $*"; fail=$((fail+1)); }
|
||||
|
||||
login() { # -> prints the bearer token
|
||||
curl -s -X POST "$LB/api/v1/auth/login" -H 'Content-Type: application/json' \
|
||||
-d "{\"username\":\"$ADMIN_USER\",\"password\":\"$ADMIN_PASS\"}" \
|
||||
| sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p'
|
||||
}
|
||||
|
||||
echo "== 1. Load balancer spreads requests across nodes =="
|
||||
served=$(for i in $(seq 1 12); do
|
||||
curl -s -D - -o /dev/null "$LB/api/v1/info/status" | tr -d '\r' | awk -F': ' '/^X-Served-By/{print $2}'
|
||||
done | sort -u)
|
||||
distinct=$(printf '%s\n' "$served" | grep -c .)
|
||||
echo " upstreams seen: $(printf '%s' "$served" | paste -sd, -)"
|
||||
[ "$distinct" -ge 2 ] && ok "LB round-robined across $distinct nodes" \
|
||||
|| bad "only $distinct node(s) served (expected >=2; is X-Served-By enabled?)"
|
||||
|
||||
echo "== 2. A JWT from the LB is accepted by BOTH nodes directly (shared signing key) =="
|
||||
jwt=$(login)
|
||||
if [ -z "$jwt" ]; then
|
||||
bad "admin login via LB failed - cannot test cross-node JWT"
|
||||
else
|
||||
ok "logged in via LB, got a JWT (${#jwt} chars)"
|
||||
for n in $NODES; do
|
||||
hc=$(docker exec "$n" curl -s -o /dev/null -w '%{http_code}' \
|
||||
-H "Authorization: Bearer $jwt" "http://localhost:8080$PROBE" 2>/dev/null)
|
||||
[ "$hc" = "200" ] && ok "$n accepted the foreign-minted JWT (HTTP 200)" \
|
||||
|| bad "$n rejected the JWT (HTTP $hc) - keys not shared across nodes"
|
||||
done
|
||||
fi
|
||||
|
||||
echo "== 3. Processor state is shared: each node sees the same sources =="
|
||||
count_of() { # $1=node -> number of sources that node reports
|
||||
docker exec "$1" curl -s -H "Authorization: Bearer $jwt" "http://localhost:8080$PROBE" 2>/dev/null \
|
||||
| grep -o '"id"' | grep -c .
|
||||
}
|
||||
a=$(count_of multinode-stirling-1); b=$(count_of multinode-stirling-2)
|
||||
echo " stirling-1 sources: $a stirling-2 sources: $b"
|
||||
if [ "$a" -gt 0 ] && [ "$a" = "$b" ]; then
|
||||
ok "both nodes report the same $a sources (shared DB)"
|
||||
else
|
||||
bad "source counts differ or zero across nodes ($a vs $b)"
|
||||
fi
|
||||
|
||||
echo "== 4. Seeded org is in the shared DB =="
|
||||
users=$(docker exec multinode-postgres psql -U stirling -d stirling -tAc "select count(*) from users" 2>/dev/null | tr -d '[:space:]')
|
||||
teams=$(docker exec multinode-postgres psql -U stirling -d stirling -tAc "select count(*) from teams" 2>/dev/null | tr -d '[:space:]')
|
||||
conns=$(docker exec multinode-postgres psql -U stirling -d stirling -tAc "select count(*) from integration_configs" 2>/dev/null | tr -d '[:space:]')
|
||||
echo " users=$users teams=$teams integration_configs=$conns"
|
||||
[ "${users:-0}" -ge 40 ] && ok "$users users present" || bad "only ${users:-0} users (did the seed run?)"
|
||||
[ "${conns:-0}" -ge 1 ] && ok "$conns S3/integration connection(s) present" || bad "no integration connections"
|
||||
|
||||
echo "== 5. Cross-node encrypted-secret read (shared credential key) =="
|
||||
# The seed's S3 secret was encrypted by whichever node handled it; fetching it via the LB (either node) and getting a masked, non-error view proves the credential key is shared, not per-node.
|
||||
lc=$(curl -s -o /tmp/mn_conns.json -w '%{http_code}' -H "Authorization: Bearer $jwt" "$LB/api/v1/integrations")
|
||||
if [ "$lc" = "200" ] && grep -q '"integrationType"' /tmp/mn_conns.json; then
|
||||
ok "integration list decrypts through the LB (HTTP 200) - credential key is shared"
|
||||
else
|
||||
bad "integration list failed (HTTP $lc) - credential key may not be shared across nodes"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "============================================================"
|
||||
echo " Multi-node validation: $pass passed, $fail failed."
|
||||
echo " Stack left running: $LB (admin / stirling)"
|
||||
echo "============================================================"
|
||||
[ "$fail" -eq 0 ]
|
||||
Reference in New Issue
Block a user