mirror of
https://github.com/J3vb/OwnCord.git
synced 2026-09-03 03:50:00 +03:00
feat(bughunt): single-finder hunt with graph-fed targeting, telemetry, and defect fixes (#1364)
* chore: ignore graphify-out * fix(bughunt): assemble the report in-script - the report agent dropped findings * fix(bughunt): retry only unverified candidates and catch garbage-verdict batches * fix(bughunt): retune the round budget floor and require a budget directive * feat(bughunt): per-round telemetry and a runStats aggregate Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(bughunt-run): record run telemetry and validate coordinates after each hunt * feat(bughunt): drop the sonnet finder slot Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(bughunt): rebuild adaptive targeting - directory clusters, cooldown, graph-fed explore lenses * fix(bughunt): rewind a dead finder's explore draw so unread files are never marked clean * docs(bughunt-run): pre-hunt graph ranking checklist and offline test roster * fix(bughunt): rewind thrown-stage explore draws and correct log/doc wording Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(bughunt): disambiguate absorb() drop log and retire dead sonnet label alternation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -15,14 +15,29 @@ reach a commit, an issue, or a PR body.
|
||||
|
||||
## 1. Hunt
|
||||
|
||||
Read the ledger and pass every record in as `known`, so the hunt does not
|
||||
re-derive anything already found, fixed, declined, or refuted:
|
||||
**Launch the hunt from a turn that carries a token-budget directive** (recommended:
|
||||
`+25M`, comfortably above a full 8-round run). The workflow's cost ceiling is gated
|
||||
on `budget.total`, which is null without a directive — a directive-less run has **no
|
||||
ceiling at all**. The workflow's first log line echoes the state: `budget=25M` means
|
||||
armed; `budget=NONE - cost ceiling disarmed` means stop the run and relaunch with a
|
||||
directive.
|
||||
|
||||
Before launching, in order:
|
||||
|
||||
1. **Rebuild the graph** (stale coordinates aim the explore lens at moved code):
|
||||
`graphify update . --no-cluster` — local tree-sitter, zero LLM cost, ~10.7k nodes.
|
||||
2. **Rank explore targets**: `node .superpowers/rank-explore.mjs` — writes
|
||||
`.superpowers/explore-ranking.json`, deprioritizing files recorded clean in
|
||||
`.superpowers/explored-clean.json` and dropping files that no longer exist.
|
||||
3. Read the ledger and pass every record in as `known`, so the hunt does not
|
||||
re-derive anything already found, fixed, declined, or refuted.
|
||||
|
||||
```
|
||||
Workflow({
|
||||
name: "bughunt",
|
||||
args: {
|
||||
known: <every record from findings-ledger.json, as {file, line, title, status}>,
|
||||
graph: <the rows of .superpowers/explore-ranking.json>,
|
||||
lenses: [ {key, prompt}, ... ], // optional: scope the hunt to one subsystem
|
||||
maxRounds: 8,
|
||||
dryThreshold: 2,
|
||||
@@ -30,6 +45,10 @@ Workflow({
|
||||
})
|
||||
```
|
||||
|
||||
If `graph` is omitted or empty the hunt logs
|
||||
`explore: args.graph absent/empty - falling back to churn-based fresh eyes` and
|
||||
still runs — degraded targeting, never a smaller lens family.
|
||||
|
||||
Omit `lenses` for a general hunt across the rotating families.
|
||||
|
||||
Each lens object is `{key, prompt}`. `key` must match `^[a-z0-9-]+$` —
|
||||
@@ -40,7 +59,21 @@ validated — it reaches the finder prompt as the literal string `undefined`,
|
||||
silently degrading that lens instead of failing loudly. Check your lens
|
||||
objects before passing them.
|
||||
|
||||
When it returns, append each entry of `result.confirmed` to the ledger with
|
||||
When it returns, first save the raw result verbatim to
|
||||
`.superpowers/hunts/<YYYY-MM-DD>-raw.json`, then:
|
||||
|
||||
```bash
|
||||
node .superpowers/render-run-stats.mjs .superpowers/hunts/<YYYY-MM-DD>-raw.json <hunt-name>
|
||||
```
|
||||
|
||||
This validates the result shape, appends the run's telemetry to
|
||||
`.superpowers/run-history.json`, updates `.superpowers/explored-clean.json`, and
|
||||
checks **every** confirmed finding's coordinates against the working tree (file
|
||||
exists, line within length — the report agent that used to spot-check two findings
|
||||
is gone). Resolve any `COORD` warnings before appending to the ledger: stale
|
||||
coordinates poison `bughunt-fix`.
|
||||
|
||||
Then append each entry of `result.confirmed` to the ledger with
|
||||
`status: "open"`, an id from `nextId`, and today's date. Bump `nextId`. The
|
||||
incoming record carries a prose `fix` field (bughunt's suggested remedy) —
|
||||
rename it to `suggestedFix` when appending, so the ledger's `fix` field starts
|
||||
@@ -51,15 +84,12 @@ revertProof}` once something is actually fixed. Then:
|
||||
node .superpowers/render-ledger.mjs
|
||||
```
|
||||
|
||||
Each confirmed record carries `finder` — which model in the dual-model panel
|
||||
produced it. The run also logs one `panel:` line with the split. The two finders
|
||||
are unioned, not voted, so the second model's entire value is what it finds
|
||||
alone; because duplicates collapse to the opus-slot record, a `sonnet` tag means
|
||||
opus missed it. Watch that count across a few hunts. Consistently zero is the
|
||||
evidence for dropping to a single finder — but note that would also weaken
|
||||
convergence, since a round is only allowed to count as dry when the full panel
|
||||
reported, so a lone finder having a bad day would read as "clean" instead of
|
||||
"we didn't fully look".
|
||||
Each confirmed record carries `finder: "opus"`. The dual-model finder panel was
|
||||
retired 2026-08-12: attribution over the only measured run priced sonnet's unique
|
||||
yield (1 high, 4 medium, 9 low) at roughly a third of the run's agents. The known
|
||||
cost: with one finder, a lazy-but-non-null finder round can read as "clean" where
|
||||
the panel required both models to agree it was. Watch `runStats` — per-lens
|
||||
candidate counts make an anomalously empty lens visible after the fact.
|
||||
|
||||
## 2. Gate (human)
|
||||
|
||||
@@ -159,7 +189,9 @@ node .claude/workflows/bughunt.harness.mjs
|
||||
node .claude/workflows/bughunt-fix.harness.mjs
|
||||
node .superpowers/render-ledger.mjs --selftest
|
||||
node .superpowers/verify-fixes.mjs --selftest
|
||||
node .superpowers/rank-explore.mjs --selftest
|
||||
node .superpowers/render-run-stats.mjs --selftest
|
||||
```
|
||||
|
||||
All four run offline with zero API calls. Run them after any edit to the
|
||||
All six run offline with zero API calls. Run them after any edit to the
|
||||
relevant script.
|
||||
|
||||
@@ -43,18 +43,17 @@ export async function run({ agentStub, args = undefined, budget = undefined }) {
|
||||
}
|
||||
|
||||
// ---------- stub kit (used from Task 2 onward; harmless now) ----------
|
||||
export function makeStub({ hunt, verify, report = () => 'REPORT_MD', recon = defaultRecon }) {
|
||||
export function makeStub({ hunt, verify, recon = defaultRecon }) {
|
||||
return (prompt, opts) => {
|
||||
const label = opts.label || ''
|
||||
if (label.startsWith('recon:')) return recon(label)
|
||||
let m = /^r(\d+):hunt:([a-z0-9-]+):(opus|sonnet)$/.exec(label)
|
||||
if (m) return hunt(Number(m[1]), m[2], m[3], prompt)
|
||||
let m = /^r(\d+):hunt:([a-z0-9-]+):opus$/.exec(label)
|
||||
if (m) return hunt(Number(m[1]), m[2], 'opus', prompt)
|
||||
m = /^r(\d+):verify:([a-z0-9-]+?)(:retry)?$/.exec(label)
|
||||
if (m) {
|
||||
const candidates = JSON.parse(prompt.split('--- CANDIDATES ---')[1])
|
||||
return verify(Number(m[1]), m[2], candidates, Boolean(m[3]), prompt)
|
||||
}
|
||||
if (label === 'report') return report(prompt)
|
||||
throw new Error(`unexpected agent label: ${label}`)
|
||||
}
|
||||
}
|
||||
@@ -72,6 +71,8 @@ export const finding = (n, over = {}) => ({
|
||||
evidence: 'e',
|
||||
...over,
|
||||
})
|
||||
export const graphRows = (n) =>
|
||||
Array.from({ length: n }, (_, i) => ({ file: `Server/gen/g${i}.go`, score: 1 - i / (n + 1), degree: 10, cited: 5 }))
|
||||
export const confirmAll = (cands) => ({
|
||||
verdicts: cands.map((c) => ({
|
||||
title: c.title, file: c.file, line: c.line,
|
||||
@@ -92,16 +93,11 @@ const scenarios = {}
|
||||
|
||||
// S1: happy convergence - one bug in round 1, rounds 2-3 dry -> converged.
|
||||
scenarios.s1_convergence = async () => {
|
||||
const reportPrompts = []
|
||||
const { result, calls } = await run({
|
||||
const { result, calls, logs } = await run({
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) =>
|
||||
round === 1 && key === 'ws-hub' && model === 'opus' ? { findings: [finding(1)] } : none,
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
report: (prompt) => {
|
||||
reportPrompts.push(prompt)
|
||||
return 'REPORT_MD'
|
||||
},
|
||||
}),
|
||||
})
|
||||
for (const k of ['converged', 'stoppedOnBudget', 'rounds', 'confirmed', 'unverified', 'report'])
|
||||
@@ -111,23 +107,23 @@ scenarios.s1_convergence = async () => {
|
||||
assert.deepEqual(result.rounds.map((r) => r.dryAfter), [0, 1, 2])
|
||||
assert.deepEqual(result.rounds.map((r) => r.family), ['surfaces', 'bug-classes', 'flows'])
|
||||
assert.equal(result.confirmed.length, 1)
|
||||
assert.equal(result.report, 'REPORT_MD')
|
||||
assert.ok(!calls.some((c) => (c.opts.label || '').startsWith('r4:')), 'no round 4 after convergence')
|
||||
assert.match(reportPrompts[0], /CONVERGED after 3 round\(s\)/)
|
||||
assert.match(reportPrompts[0], /\| 1 \| surfaces \|/)
|
||||
assert.ok(!calls.some((c) => c.opts.label === 'report'), 'the report is built in-script')
|
||||
assert.match(result.report, /CONVERGED after 3 round\(s\)/)
|
||||
assert.match(result.report, /### high - distinct bug alpha1 omega1/)
|
||||
assert.match(result.report, /\| 1 \| surfaces \|/)
|
||||
assert.ok(logs.some((l) => /budget=NONE - cost ceiling disarmed/.test(l)), 'a directive-less run must announce the dead ceiling')
|
||||
}
|
||||
|
||||
// S2: panel dedupe - opus and sonnet report the same bug -> one candidate, one verify call.
|
||||
scenarios.s2_panel_dedupe = async () => {
|
||||
// S2: near-duplicate findings from a single finder collapse - one candidate, one verify call.
|
||||
scenarios.s2_finder_dedupe = async () => {
|
||||
const verifyBatches = []
|
||||
const { result } = await run({
|
||||
const { result, calls } = await run({
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (round !== 1 || key !== 'ws-hub') return none
|
||||
return model === 'opus'
|
||||
? { findings: [finding(1, { line: 100 })] }
|
||||
: { findings: [finding(1, { line: 105, title: 'distinct bug alpha1 omega1 variant' })] }
|
||||
},
|
||||
hunt: (round, key) =>
|
||||
round === 1 && key === 'ws-hub'
|
||||
? { findings: [finding(1, { line: 100 }), finding(1, { line: 105, title: 'distinct bug alpha1 omega1 variant' })] }
|
||||
: none,
|
||||
verify: (round, key, cands) => {
|
||||
verifyBatches.push(cands)
|
||||
return confirmAll(cands)
|
||||
@@ -137,6 +133,9 @@ scenarios.s2_panel_dedupe = async () => {
|
||||
assert.equal(verifyBatches.length, 1)
|
||||
assert.equal(verifyBatches[0].length, 1)
|
||||
assert.equal(result.confirmed.length, 1)
|
||||
const hunts = calls.filter((c) => /^r1:hunt:ws-hub:/.test(c.opts.label || ''))
|
||||
assert.equal(hunts.length, 1, 'exactly one finder call per lens - the sonnet slot is gone')
|
||||
assert.match(hunts[0].opts.label, /:opus$/, 'the label keeps the :opus suffix the harness parses')
|
||||
}
|
||||
|
||||
// S3: refuted findings stay dead - re-reported next round, never re-verified; refutes count toward dry.
|
||||
@@ -237,15 +236,132 @@ scenarios.s6b_verifier_double_failure = async () => {
|
||||
assert.equal(result.unverified.length, 0, 'later-confirmed candidate must leave the unverified list')
|
||||
}
|
||||
|
||||
// S7: rounds 1-3 each confirm a bug -> round 4 runs adaptive lenses built from the stats.
|
||||
// N1 (spec #1): the top-ranked hotspot cluster sits out exactly the next round, then returns.
|
||||
// The producing cluster keeps running when eligible (the old s7b lock, restated under cooldown).
|
||||
scenarios.s_cluster_cooldown = async () => {
|
||||
const { calls } = await run({
|
||||
args: { maxRounds: 6, dryThreshold: 9, graph: graphRows(60) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => {
|
||||
if (round === 1 && key === 'ws-hub')
|
||||
return { findings: [
|
||||
finding(1, { file: 'Server/ws/hub.go', title: 'ws bug alpha one' }),
|
||||
finding(2, { file: 'Server/ws/pubsub.go', line: 300, title: 'ws bug beta two' }),
|
||||
] }
|
||||
if (round === 2 && key === 'concurrency')
|
||||
return { findings: [
|
||||
finding(3, { file: 'Server/ws/emit.go', title: 'ws bug gamma three' }),
|
||||
finding(4, { file: 'Server/api/user.go', title: 'api bug delta four' }),
|
||||
] }
|
||||
if (round === 4 && key === 'hotspot-server-ws')
|
||||
return { findings: [finding(9, { file: 'Server/ws/late.go', title: 'late ws bug nine' })] }
|
||||
return none
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const hunted = (rnd, key) => calls.some((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`)
|
||||
assert.ok(hunted(4, 'hotspot-server-ws'), 'top cluster hunts in r4')
|
||||
assert.ok(!hunted(5, 'hotspot-server-ws'), 'the r4 top cluster must sit out r5 even though it produced')
|
||||
assert.ok(hunted(5, 'hotspot-server-api'), 'the next cluster takes the top slot in r5')
|
||||
assert.ok(hunted(6, 'hotspot-server-ws'), 'cooldown lasts exactly one round')
|
||||
}
|
||||
|
||||
// N2 (spec #2): a cooldown gap FREEZES cleanStreak - neither increments nor resets - so
|
||||
// demotion still means two consecutive clean APPEARANCES. If the gap incremented, ws would
|
||||
// be demoted before r6; if demotion broke, ws would still run in r8.
|
||||
scenarios.s_cooldown_freezes_streak = async () => {
|
||||
const { result, calls } = await run({
|
||||
args: { maxRounds: 8, dryThreshold: 9, graph: graphRows(100) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => {
|
||||
if (round === 1 && key === 'ws-hub')
|
||||
return { findings: [
|
||||
finding(1, { file: 'Server/ws/hub.go', title: 'ws bug alpha one' }),
|
||||
finding(2, { file: 'Server/ws/pubsub.go', line: 300, title: 'ws bug beta two' }),
|
||||
] }
|
||||
if (round === 2 && key === 'concurrency')
|
||||
return { findings: [finding(3, { file: 'Server/api/user.go', title: 'api bug delta three' })] }
|
||||
return none
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const hunted = (rnd, key) => calls.some((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`)
|
||||
assert.ok(hunted(4, 'hotspot-server-ws'), 'clean appearance #1 in r4')
|
||||
assert.ok(!hunted(5, 'hotspot-server-ws'), 'cooldown in r5')
|
||||
assert.ok(hunted(6, 'hotspot-server-ws'), 'the gap must freeze the streak at 1, not increment it')
|
||||
assert.equal(result.rounds.length, 8, 'the run must reach r8 for the demotion assert to mean anything')
|
||||
assert.ok(!hunted(8, 'hotspot-server-ws'), 'two clean appearances (r4, r6) demote the lens')
|
||||
}
|
||||
|
||||
// N3 (spec #3): cooldown+demotion emptying the hotspot pool must backfill from explore and
|
||||
// log it - silent family shrinkage is the exact freshEyesLens() defect this rebuild removes.
|
||||
scenarios.s_hotspot_backfill = async () => {
|
||||
const { calls, logs } = await run({
|
||||
args: { maxRounds: 5, dryThreshold: 9, graph: graphRows(100) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) =>
|
||||
round === 1 && key === 'ws-hub'
|
||||
? { findings: [finding(1, { file: 'Server/ws/hub.go', title: 'lone ws bug one' })] }
|
||||
: none,
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const r5 = calls.filter((c) => /^r5:hunt:/.test(c.opts.label || '')).map((c) => c.opts.label.split(':')[2])
|
||||
assert.ok(!r5.some((k) => k.startsWith('hotspot-')), 'the sole cluster is on cooldown in r5')
|
||||
assert.deepEqual([...r5].sort(), ['explore-1', 'explore-2', 'explore-3', 'explore-4'], 'the family backfills to full size from explore')
|
||||
assert.ok(logs.some((l) => /hotspot pool short/.test(l)), 'backfill must be logged, never silent')
|
||||
}
|
||||
|
||||
// N4 (spec #4): within-run consumption - later rounds draw the NEXT chunk of the ranking,
|
||||
// never re-offering files already handed to an explore lens this run.
|
||||
scenarios.s_explore_consumption = async () => {
|
||||
const { calls } = await run({
|
||||
args: { maxRounds: 5, dryThreshold: 9, graph: graphRows(80) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) =>
|
||||
round === 1 && key === 'ws-hub'
|
||||
? { findings: [finding(1, { file: 'Server/ws/hub.go', title: 'seed bug one' })] }
|
||||
: none,
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const promptOf = (rnd, key) => (calls.find((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`) || {}).prompt || ''
|
||||
assert.match(promptOf(4, 'explore-1'), /Server\/gen\/g0\.go/)
|
||||
assert.match(promptOf(4, 'explore-2'), /Server\/gen\/g10\.go/)
|
||||
assert.match(promptOf(4, 'explore-3'), /Server\/gen\/g20\.go/, 'r4 backfills a third explore lens (single cluster)')
|
||||
assert.match(promptOf(5, 'explore-1'), /Server\/gen\/g30\.go/, 'r5 draws the next chunk')
|
||||
assert.doesNotMatch(promptOf(5, 'explore-1'), /Server\/gen\/g0\.go/, 'r5 must not re-offer r4 files')
|
||||
}
|
||||
|
||||
// N6 (spec #6): args.graph absent -> churn-based fresh-eyes fallback, logged, family intact.
|
||||
scenarios.s_graph_missing_fallback = async () => {
|
||||
const { calls, logs } = await run({
|
||||
args: { maxRounds: 4, dryThreshold: 9 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) =>
|
||||
round === 1 && key === 'ws-hub'
|
||||
? { findings: [finding(1, { file: 'Server/ws/hub.go', title: 'seed bug one' })] }
|
||||
: none,
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
assert.ok(logs.some((l) => /falling back to churn/.test(l)), 'the fallback must be logged')
|
||||
const e1 = calls.find((c) => (c.opts.label || '') === 'r4:hunt:explore-1:opus')
|
||||
assert.ok(e1, 'an explore lens must still run from churn')
|
||||
assert.match(e1.prompt, /Server\/api\/user\.go/, 'churned file with no findings feeds the fallback')
|
||||
}
|
||||
|
||||
// S7: rounds 1-3 each confirm a bug -> round 4 runs adaptive lenses: directory-granularity
|
||||
// hotspots plus explore (churn fallback here - no args.graph is passed).
|
||||
scenarios.s7_adaptive_lenses = async () => {
|
||||
const A = finding(1, { file: 'Server/ws/hub.go', line: 120, title: 'alpha race window one' })
|
||||
const B = finding(2, { file: 'Server/ws/pubsub.go', line: 60, title: 'beta subscription leak two' })
|
||||
const C = finding(3, { file: 'Client/tauri-client/src/lib/livekitE2EE.ts', line: 200, title: 'gamma epoch desync three' })
|
||||
const { result, calls } = await run({
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (model !== 'opus') return none
|
||||
hunt: (round, key) => {
|
||||
if (round === 1 && key === 'ws-hub') return { findings: [A] }
|
||||
if (round === 2 && key === 'concurrency') return { findings: [B] }
|
||||
if (round === 3 && key === 'flow-voice') return { findings: [C] }
|
||||
@@ -257,49 +373,26 @@ scenarios.s7_adaptive_lenses = async () => {
|
||||
assert.equal(result.converged, true)
|
||||
assert.equal(result.rounds.length, 5) // r4, r5 adaptive + dry
|
||||
assert.equal(result.rounds[3].family, 'adaptive')
|
||||
const r4Hunts = calls.filter((c) => /^r4:hunt:/.test(c.opts.label || ''))
|
||||
const r4Keys = [...new Set(r4Hunts.map((c) => c.opts.label.split(':')[2]))]
|
||||
const r4Keys = [...new Set(calls.filter((c) => /^r4:hunt:/.test(c.opts.label || '')).map((c) => c.opts.label.split(':')[2]))]
|
||||
assert.ok(r4Keys.includes('hotspot-server-ws'), `r4 keys: ${r4Keys}`)
|
||||
assert.ok(r4Keys.includes('fresh-eyes'), `r4 keys: ${r4Keys}`)
|
||||
const hotspot = r4Hunts.find((c) => c.opts.label.includes('hotspot-server-ws'))
|
||||
assert.ok(r4Keys.includes('hotspot-client-tauri-client-src-lib'), `r4 keys: ${r4Keys}`)
|
||||
assert.ok(r4Keys.includes('explore-1'), `r4 keys: ${r4Keys}`)
|
||||
const hotspot = calls.find((c) => (c.opts.label || '').includes('hotspot-server-ws'))
|
||||
assert.match(hotspot.prompt, /Server\/ws\/hub\.go/)
|
||||
assert.match(hotspot.prompt, /alpha race window one/)
|
||||
const freshEyes = r4Hunts.find((c) => c.opts.label.includes('fresh-eyes'))
|
||||
assert.match(freshEyes.prompt, /Server\/api\/user\.go/) // churned, never a finding
|
||||
const explore = calls.find((c) => (c.opts.label || '') === 'r4:hunt:explore-1:opus')
|
||||
assert.match(explore.prompt, /Server\/api\/user\.go/) // churned, never a finding
|
||||
assert.equal(result.confirmed.length, 3)
|
||||
}
|
||||
|
||||
// S7b: a lens with 2 consecutive clean rounds is demoted from later rounds.
|
||||
scenarios.s7b_demotion = async () => {
|
||||
const { result, calls } = await run({
|
||||
args: { maxRounds: 6 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (model !== 'opus') return none
|
||||
const src = { 1: 'ws-hub', 2: 'concurrency', 3: 'flow-reconnect', 4: 'hotspot-server-ws', 5: 'hotspot-server-ws' }
|
||||
if (key === src[round])
|
||||
return { findings: [finding(round, { file: `Server/ws/a${round}.go`, title: `unique bug number${round} zeta${round}` })] }
|
||||
return none
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const labels = calls.map((c) => c.opts.label || '')
|
||||
assert.ok(labels.some((l) => /^r5:hunt:fresh-eyes:/.test(l)), 'fresh-eyes still runs in r5 (streak 1)')
|
||||
assert.ok(!labels.some((l) => /^r6:hunt:fresh-eyes:/.test(l)), 'fresh-eyes demoted in r6 (streak 2)')
|
||||
assert.ok(labels.some((l) => /^r6:hunt:hotspot-server-ws:/.test(l)), 'producing hotspot keeps running')
|
||||
assert.equal(result.converged, false)
|
||||
assert.equal(result.confirmed.length, 5)
|
||||
}
|
||||
|
||||
// S7c: a lens whose VERIFIER died is not demoted; a zero-candidate lens still is.
|
||||
// S7c: a lens whose VERIFIER died is not demoted (its cluster returns after cooldown);
|
||||
// a zero-candidate explore lens still accrues streak and demotes.
|
||||
scenarios.s7c_verifier_failure_not_demoted = async () => {
|
||||
const early = { 1: 'ws-hub', 2: 'concurrency', 3: 'flow-reconnect' }
|
||||
const { result, calls } = await run({
|
||||
args: { maxRounds: 6 },
|
||||
args: { maxRounds: 6, dryThreshold: 9, graph: graphRows(100) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (model !== 'opus') return none
|
||||
hunt: (round, key) => {
|
||||
if (round <= 3 && key === early[round])
|
||||
return { findings: [finding(round, { file: `Server/ws/a${round}.go`, title: `early bug item${round} kappa${round}` })] }
|
||||
if (round >= 4 && key === 'hotspot-server-ws')
|
||||
@@ -310,12 +403,15 @@ scenarios.s7c_verifier_failure_not_demoted = async () => {
|
||||
}),
|
||||
})
|
||||
const labels = calls.map((c) => c.opts.label || '')
|
||||
assert.ok(labels.some((l) => /^r6:hunt:hotspot-server-ws:/.test(l)), 'verifier-dead lens must NOT be demoted')
|
||||
assert.ok(!labels.some((l) => /^r6:hunt:fresh-eyes:/.test(l)), 'zero-candidate lens still accrues streak and demotes')
|
||||
assert.ok(labels.some((l) => /^r4:hunt:hotspot-server-ws:/.test(l)))
|
||||
assert.ok(!labels.some((l) => /^r5:hunt:hotspot-server-ws:/.test(l)), 'cooldown after topping r4')
|
||||
assert.ok(labels.some((l) => /^r6:hunt:hotspot-server-ws:/.test(l)), 'a verifier-dead lens must NOT be demoted')
|
||||
assert.ok(!labels.some((l) => /^r6:hunt:explore-1:/.test(l)), 'a zero-candidate explore lens still demotes')
|
||||
assert.equal(result.confirmed.length, 3)
|
||||
assert.equal(result.unverified.length, 3)
|
||||
assert.equal(result.unverified.length, 2) // b4 and b6, each denied a verdict twice
|
||||
assert.equal(result.converged, false)
|
||||
assert.ok(result.rounds.slice(3).every((r) => r.dryEligible === false))
|
||||
assert.equal(result.rounds[3].dryEligible, false)
|
||||
assert.equal(result.rounds[5].dryEligible, false)
|
||||
}
|
||||
|
||||
// S12: empty adaptive family (no confirms, no churn) must break honestly, not count dry rounds.
|
||||
@@ -354,7 +450,7 @@ scenarios.s8_budget_floor = async () => {
|
||||
scenarios.s8b_budget_midrun = async () => {
|
||||
let n = 0
|
||||
const { result } = await run({
|
||||
budget: { total: 1000000, spent: () => 0, remaining: () => (n++ === 0 ? 200000 : 100000) },
|
||||
budget: { total: 10000000, spent: () => 0, remaining: () => (n++ === 0 ? 3000000 : 1000000) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) =>
|
||||
round === 1 && key === 'ws-hub' && model === 'opus' ? { findings: [finding(1)] } : none,
|
||||
@@ -496,7 +592,7 @@ scenarios.s_custom_lenses = async () => {
|
||||
agentStub: makeStub({ hunt: () => none, verify: (r, k, c) => confirmAll(c) }),
|
||||
})
|
||||
const keys = calls
|
||||
.map((c) => /^r1:hunt:([a-z0-9-]+):(opus|sonnet)$/.exec(c.opts.label || ''))
|
||||
.map((c) => /^r1:hunt:([a-z0-9-]+):opus$/.exec(c.opts.label || ''))
|
||||
.filter(Boolean)
|
||||
.map((m) => m[1])
|
||||
assert.deepEqual([...new Set(keys)].sort(), ['voice-e2ee-keyholder', 'voice-e2ee-rotation'])
|
||||
@@ -549,50 +645,7 @@ scenarios.s_confirmed_carries_finder_detail = async () => {
|
||||
assert.equal(r.fix, 'FIX_TEXT')
|
||||
assert.equal(r.lens, 'ws-hub')
|
||||
assert.equal(r.round, 1)
|
||||
}
|
||||
|
||||
// S_FINDER_ATTRIBUTION: confirmed records name which model found them. The panel unions rather
|
||||
// than votes, so this is the only way to tell whether the second finder earns its cost. Because
|
||||
// dedupe keeps the first occurrence and opus is index 0, `finder: 'sonnet'` means opus did NOT
|
||||
// report it - i.e. a sonnet-unique find. `finder: 'opus'` says nothing about sonnet either way.
|
||||
scenarios.s_finder_attribution = async () => {
|
||||
const shared = finding(1)
|
||||
// Different file, not just a different line: isDup treats same-file findings within
|
||||
// TITLE_MATCH_WINDOW as duplicates on title-word overlap, and the finding() fixtures share
|
||||
// enough words to collapse into one another.
|
||||
const sonnetOnly = finding(2, { file: 'Server/api/user.go' })
|
||||
const { result } = await run({
|
||||
args: { maxRounds: 1, dryThreshold: 9 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (round !== 1 || key !== 'ws-hub') return none
|
||||
return model === 'opus' ? { findings: [shared] } : { findings: [shared, sonnetOnly] }
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const byTitle = Object.fromEntries(result.confirmed.map((r) => [r.title, r.finder]))
|
||||
assert.equal(byTitle[shared.title], 'opus', 'a find both models made keeps the opus-first record')
|
||||
assert.equal(byTitle[sonnetOnly.title], 'sonnet', 'a find only sonnet made must be attributed to sonnet')
|
||||
}
|
||||
|
||||
// S_FINDER_ATTRIBUTION_SURVIVES_DEAD_OPUS: the tag must be taken from the panel slot, not from the
|
||||
// position in the surviving list. Filtering the nulls out BEFORE reading the index shifts sonnet
|
||||
// into slot 0 and mislabels every one of its finds as opus - exactly when attribution matters most.
|
||||
scenarios.s_finder_attribution_survives_dead_opus = async () => {
|
||||
const only = finding(3)
|
||||
const { result } = await run({
|
||||
args: { maxRounds: 1, dryThreshold: 9 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (round !== 1 || key !== 'ws-hub') return none
|
||||
return model === 'opus' ? null : { findings: [only] }
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
assert.equal(result.confirmed.length, 1)
|
||||
assert.equal(result.confirmed[0].finder, 'sonnet', 'a dead opus must not relabel sonnet finds as opus')
|
||||
assert.equal(r.finder, 'opus', 'the finder tag is constant now but the ledger still expects it')
|
||||
}
|
||||
|
||||
// S_VERIFIER_IS_NOT_TOLD_THE_FINDER: the verifier prompt deliberately says "another model" and
|
||||
@@ -619,26 +672,162 @@ scenarios.s_verifier_is_not_told_the_finder = async () => {
|
||||
}
|
||||
}
|
||||
|
||||
// S_PANEL_SPLIT_IS_REPORTED: a tag nobody reads is not a measurement. The run must say out loud
|
||||
// how many confirmed findings only the second finder produced, which is the number that decides
|
||||
// whether the second model is worth its cost.
|
||||
scenarios.s_panel_split_is_reported = async () => {
|
||||
const { logs } = await run({
|
||||
// New (spec Testing #8): the report is built in-script. Section count must equal the confirmed
|
||||
// count at 82 (the agent version emitted 79 for 82), and the unverified section must survive
|
||||
// the agent's removal - it used to exist only inside the report agent's prompt.
|
||||
scenarios.s_report_deterministic = async () => {
|
||||
const many = Array.from({ length: 82 }, (_, i) =>
|
||||
finding(i, { file: `Server/ws/f${i}.go`, line: 10, title: `unique bug row${i} tag${i}` }))
|
||||
const stuck = finding(999, { file: 'Server/api/stuck.go', line: 40, title: 'stuck bug never verified' })
|
||||
const { result, calls } = await run({
|
||||
args: { maxRounds: 1, dryThreshold: 9 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) =>
|
||||
round === 1 && key === 'ws-hub' && model === 'opus' ? { findings: [...many, stuck] } : none,
|
||||
verify: (round, key, cands) => confirmAll(cands.filter((c) => c.file !== 'Server/api/stuck.go')),
|
||||
}),
|
||||
})
|
||||
assert.equal(result.confirmed.length, 82)
|
||||
assert.equal(result.unverified.length, 1)
|
||||
assert.ok(!calls.some((c) => c.opts.label === 'report'), 'no report agent may run')
|
||||
const sections = (result.report.match(/^### /gm) || []).length
|
||||
assert.equal(sections, 82, 'one section per confirmed finding, none dropped')
|
||||
assert.match(result.report, /## Unverified - re-run/)
|
||||
assert.match(result.report, /stuck bug never verified/)
|
||||
assert.match(result.report, /## Convergence/)
|
||||
}
|
||||
|
||||
// New (spec Testing #7): the retry re-sends ONLY unmatched candidates, and N garbage verdicts
|
||||
// (count == candidate count, zero of them matching) must still trigger it - the hole S10 misses
|
||||
// because S10's verdict list is empty rather than full of junk.
|
||||
scenarios.s_targeted_retry = async () => {
|
||||
const a = finding(1, { file: 'Server/ws/a.go', title: 'alpha bug one paired' })
|
||||
const b = finding(2, { file: 'Server/api/b.go', title: 'beta bug two orphaned' })
|
||||
const retryBatches = []
|
||||
const { result } = await run({
|
||||
args: { maxRounds: 1, dryThreshold: 9 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) =>
|
||||
round === 1 && key === 'ws-hub' && model === 'opus' ? { findings: [a, b] } : none,
|
||||
verify: (round, key, cands, isRetry) => {
|
||||
if (isRetry) {
|
||||
retryBatches.push(cands)
|
||||
return confirmAll(cands)
|
||||
}
|
||||
// one real verdict for a, one garbage verdict pointing nowhere: count matches, content doesn't
|
||||
return {
|
||||
verdicts: [
|
||||
{ title: a.title, file: a.file, line: a.line, refuted: false, reason: 'ok', confidence: 'high', severity: 'high', fix: 'f' },
|
||||
{ title: 'hallucinated', file: 'Server/nowhere.go', line: 1, refuted: false, reason: 'x', confidence: 'low', severity: 'low' },
|
||||
],
|
||||
}
|
||||
},
|
||||
}),
|
||||
})
|
||||
assert.equal(retryBatches.length, 1, 'retry must fire despite verdict count == candidate count')
|
||||
assert.deepEqual(retryBatches[0].map((c) => c.file), ['Server/api/b.go'], 'only the unmatched candidate is re-sent')
|
||||
assert.equal(result.confirmed.length, 2)
|
||||
assert.equal(result.unverified.length, 0)
|
||||
}
|
||||
|
||||
// New (spec Testing #9): the retuned floor must stop a run the old 150k floor let through.
|
||||
// 1M remaining is under the ~2M measured per-round cost - starting a round would overshoot.
|
||||
scenarios.s9_budget_ceiling_retuned = async () => {
|
||||
const { result, logs } = await run({
|
||||
budget: { total: 10000000, spent: () => 9000000, remaining: () => 1000000 },
|
||||
agentStub: makeStub({ hunt: () => none, verify: (r, k, c) => confirmAll(c) }),
|
||||
})
|
||||
assert.equal(result.rounds.length, 0, '1M remaining must not start a ~2M round')
|
||||
assert.equal(result.stoppedOnBudget, true)
|
||||
assert.ok(logs.some((l) => /Budget floor/.test(l)))
|
||||
}
|
||||
|
||||
// New: telemetry. Per-round suppression split (ledger vs same-run), spend sampling, file
|
||||
// coverage, severity mix, per-lens precision, and the top-level runStats aggregate. Without
|
||||
// this every cost figure from a run is eyewitness-only - the 2026-08-12 problem.
|
||||
scenarios.s_telemetry = async () => {
|
||||
let spent = 0
|
||||
const known = [{ file: 'Server/ws/hub.go', line: 100, title: 'known bug from ledger prior', status: 'fixed' }]
|
||||
const { result } = await run({
|
||||
args: { maxRounds: 1, dryThreshold: 9, known },
|
||||
budget: { total: 50000000, spent: () => (spent += 500000), remaining: () => 40000000 },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key, model) => {
|
||||
if (round !== 1 || key !== 'ws-hub') return none
|
||||
return model === 'opus'
|
||||
? { findings: [finding(1)] }
|
||||
: { findings: [finding(1), finding(2, { file: 'Server/api/user.go' })] }
|
||||
if (round !== 1 || key !== 'ws-hub' || model !== 'opus') return none
|
||||
return { findings: [
|
||||
finding(1, { file: 'Server/ws/hub.go', line: 102, title: 'known bug from ledger prior' }),
|
||||
finding(2, { file: 'Server/api/fresh.go', severity: 'medium', title: 'fresh bug beta gamma' }),
|
||||
] }
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const line = logs.find((l) => /panel:/.test(l))
|
||||
assert.ok(line, 'the run must report the panel split')
|
||||
assert.match(line, /sonnet-only/, 'the split must name the sonnet-only count explicitly')
|
||||
assert.match(line, /\b1\b/, 'exactly one confirmed finding here was sonnet-only')
|
||||
const r1 = result.rounds[0]
|
||||
assert.equal(r1.suppressedLedger, 1, 'the ledger-known duplicate must be counted as ledger suppression')
|
||||
assert.equal(r1.suppressedRun, 0)
|
||||
assert.ok(r1.spentAfter > r1.spentBefore, 'per-round spend must be sampled')
|
||||
assert.equal(r1.filesTouched, 1)
|
||||
assert.equal(r1.filesNew, 1)
|
||||
assert.deepEqual(r1.severity, { critical: 0, high: 0, medium: 1, low: 0 })
|
||||
assert.equal(r1.perLens['ws-hub'].confirmed, 1)
|
||||
assert.equal(r1.perLens['ws-hub'].fresh, 1)
|
||||
assert.ok(result.runStats, 'runStats missing from the result')
|
||||
assert.equal(result.runStats.confirmed, 1)
|
||||
assert.equal(result.runStats.suppressedLedger, 1)
|
||||
assert.equal(result.runStats.config.maxRounds, 1)
|
||||
assert.match(result.report, /## Run stats/)
|
||||
}
|
||||
|
||||
// N7 (Task 9 review finding): a dead finder on an explore lens read nothing - its draw is
|
||||
// rewound so the files never reach exploredFiles, where the session would record them clean
|
||||
// and deprioritize them in every future hunt. maxRounds caps at 4 on purpose: a live round-5
|
||||
// lens would legitimately re-read the rewound files and they would CORRECTLY re-enter
|
||||
// exploredFiles - the poison-prevention property is only assertable when the run ends here.
|
||||
// Re-offering in later rounds follows from the same exploreConsumed state drawExploreFiles
|
||||
// filters on, so this one scenario locks the mechanism.
|
||||
scenarios.s_explore_rewind_on_dead_finder = async () => {
|
||||
const { result, calls } = await run({
|
||||
args: { maxRounds: 4, dryThreshold: 9, graph: graphRows(80) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => {
|
||||
if (round === 1 && key === 'ws-hub')
|
||||
return { findings: [finding(1, { file: 'Server/ws/hub.go', title: 'seed bug one' })] }
|
||||
if (round === 4 && key === 'explore-1') return null // dead finder: read nothing
|
||||
return none
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const promptOf = (rnd, key) => (calls.find((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`) || {}).prompt || ''
|
||||
assert.match(promptOf(4, 'explore-1'), /Server\/gen\/g0\.go/, 'r4 explore-1 drew the head of the ranking')
|
||||
for (let i = 0; i < 10; i++)
|
||||
assert.ok(!result.exploredFiles.includes(`Server/gen/g${i}.go`), `g${i} was never read - must not be reported explored`)
|
||||
assert.ok(result.exploredFiles.includes('Server/gen/g10.go'), 'files a LIVE lens drew stay reported')
|
||||
assert.ok(result.exploredFiles.includes('Server/gen/g20.go'), 'backfilled live lens files stay reported too')
|
||||
}
|
||||
|
||||
// N8 (final-review finding): a THROWN stage nulls the whole lens result - the second
|
||||
// finder-failure mode the code documents. Its explore draw must rewind exactly like the
|
||||
// null-finder case, or never-read files reach exploredFiles and poison explored-clean.
|
||||
scenarios.s_explore_rewind_on_thrown_stage = async () => {
|
||||
const { result, calls } = await run({
|
||||
args: { maxRounds: 4, dryThreshold: 9, graph: graphRows(80) },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => {
|
||||
if (round === 1 && key === 'ws-hub')
|
||||
return { findings: [finding(1, { file: 'Server/ws/hub.go', title: 'seed bug one' })] }
|
||||
if (round === 4 && key === 'explore-1') throw new Error('finder infrastructure blew up')
|
||||
return none
|
||||
},
|
||||
verify: (round, key, cands) => confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const promptOf = (rnd, key) => (calls.find((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`) || {}).prompt || ''
|
||||
assert.match(promptOf(4, 'explore-1'), /Server\/gen\/g0\.go/, 'r4 explore-1 drew the head of the ranking')
|
||||
for (let i = 0; i < 10; i++)
|
||||
assert.ok(!result.exploredFiles.includes(`Server/gen/g${i}.go`), `g${i} was never read - must not be reported explored`)
|
||||
assert.ok(result.exploredFiles.includes('Server/gen/g10.go'), 'files a LIVE lens drew stay reported')
|
||||
assert.equal(result.rounds[3].dryEligible, false, 'a nulled lens result still makes the round ineligible')
|
||||
}
|
||||
|
||||
// ---------- runner ----------
|
||||
|
||||
+229
-106
@@ -1,10 +1,9 @@
|
||||
export const meta = {
|
||||
name: 'bughunt',
|
||||
description: 'Converging multi-round bug hunt: rotating lens families, dual-model panels, fable refute-by-default verification, dry-threshold stop',
|
||||
description: 'Converging multi-round bug hunt: rotating lens families, single opus finder, fable refute-by-default verification, dry-threshold stop',
|
||||
whenToUse: 'Hunting real bugs across the Go server, Tauri Rust backend, and TS client until consecutive rounds go dry. Not a security-only scan.',
|
||||
phases: [
|
||||
{ title: 'Recon', detail: 'haiku: churn + concurrency-surface inventory' },
|
||||
{ title: 'Report', detail: 'fable: ranked findings + convergence table' },
|
||||
],
|
||||
}
|
||||
|
||||
@@ -19,14 +18,15 @@ const ARGS = (() => {
|
||||
const MAX_ROUNDS = ARGS.maxRounds || 8
|
||||
const DRY_THRESHOLD = ARGS.dryThreshold || 2
|
||||
// A scoped hunt (args.lenses) replaces the round-1 family outright; later rounds still go
|
||||
// adaptive, so hotspot and fresh-eyes coverage - and therefore convergence - still work.
|
||||
// adaptive, so hotspot and explore coverage - and therefore convergence - still work.
|
||||
const CUSTOM_LENSES = Array.isArray(ARGS.lenses) && ARGS.lenses.length ? ARGS.lenses : null
|
||||
// ponytail: rough floor for one round (up to 12 high-effort finders + verifiers); tune after live runs
|
||||
const ROUND_BUDGET_FLOOR = 150000
|
||||
// Floor for one round, tuned from the 2026-08-12 run: ~2.6M output tokens per round measured.
|
||||
// The old 150k floor would overshoot the ceiling by nearly a full round.
|
||||
const ROUND_BUDGET_FLOOR = 2000000
|
||||
// The args channel has already been observed delivering something the script
|
||||
// could not read; an unnoticed fallback here is an 8x cost surprise, so say out
|
||||
// loud what the run is actually going to do.
|
||||
log(`config: maxRounds=${MAX_ROUNDS} dryThreshold=${DRY_THRESHOLD}${CUSTOM_LENSES ? ` lenses=custom(${CUSTOM_LENSES.length})` : ''}`)
|
||||
log(`config: maxRounds=${MAX_ROUNDS} dryThreshold=${DRY_THRESHOLD}${CUSTOM_LENSES ? ` lenses=custom(${CUSTOM_LENSES.length})` : ''} budget=${budget.total ? Math.round(budget.total / 1e6) + 'M' : 'NONE - cost ceiling disarmed'}`)
|
||||
|
||||
// ---------- schemas: copied VERBATIM from the current bughunt.js ----------
|
||||
const FINDINGS = {
|
||||
@@ -93,14 +93,16 @@ Out of scope, do not report: naming, formatting, missing tests, "consider adding
|
||||
performance that is not a hang, anything you cannot point at specific lines for.
|
||||
|
||||
Method:
|
||||
1. Read the actual files. Never report from a filename or a grep hit alone.
|
||||
1. Read the actual files. Never report from a filename, a grep hit, or a graph edge alone - a
|
||||
graphify edge is structural evidence of coupling, not of a bug; open the cited file and confirm.
|
||||
2. For every candidate, grep for ALL callers before judging - a guard may already live upstream.
|
||||
3. Check whether an existing test already locks the behavior you think is wrong. If a test asserts it,
|
||||
it is intended behavior, not a bug. Test files are *_test.go and tests/unit/*.test.ts.
|
||||
4. Report EVERY finding you can prove - there is no cap. The quality bar stays: zero findings is a
|
||||
valid, respectable answer, and each finding needs file, line, and a concrete repro.
|
||||
|
||||
You may run read-only shell commands (grep, git log, go doc). Do not modify any file. Do not run the test suite.
|
||||
You may run read-only shell commands (grep, git log, go doc, graphify path, graphify explain).
|
||||
Do not modify any file. Do not run the test suite.
|
||||
`
|
||||
|
||||
// ---------- lens catalog ----------
|
||||
@@ -271,46 +273,83 @@ const FLOW_LENSES = [
|
||||
]
|
||||
|
||||
function lensesForRound(round) {
|
||||
if (CUSTOM_LENSES) return round === 1 ? CUSTOM_LENSES : buildAdaptiveLenses()
|
||||
if (CUSTOM_LENSES) return round === 1 ? CUSTOM_LENSES : buildAdaptiveLenses(round)
|
||||
if (round === 1) return SURFACE_LENSES
|
||||
if (round === 2) return BUGCLASS_LENSES
|
||||
if (round === 3) return FLOW_LENSES
|
||||
return buildAdaptiveLenses()
|
||||
return buildAdaptiveLenses(round)
|
||||
}
|
||||
function familyName(round) {
|
||||
if (CUSTOM_LENSES) return round === 1 ? 'custom' : 'adaptive'
|
||||
return ['surfaces', 'bug-classes', 'flows'][round - 1] || 'adaptive'
|
||||
}
|
||||
// Directory granularity: the old two/three-segment cluster collapsed the whole TS client into
|
||||
// one bucket (35 of 82 findings), so the "top cluster" never changed for five straight rounds.
|
||||
function clusterOf(file) {
|
||||
const parts = String(file).split('/')
|
||||
return parts.slice(0, parts[0] === 'Client' ? 3 : 2).join('/')
|
||||
return parts.length > 1 ? parts.slice(0, -1).join('/') : parts[0]
|
||||
}
|
||||
function freshEyesLens() {
|
||||
const files = churnFiles.filter((f) => !seen.some((s) => s.file === f)).slice(0, 10)
|
||||
if (!files.length) return []
|
||||
return [
|
||||
{
|
||||
key: 'fresh-eyes',
|
||||
prompt:
|
||||
`These files churned heavily in the last 8 weeks, yet no hunt round has confirmed or refuted a ` +
|
||||
`single finding in them - either they are clean or every lens so far walked past them. Read each ` +
|
||||
`one IN FULL with fresh eyes and hunt for real bugs of any class:\n` +
|
||||
files.map((f) => ` - ${f}`).join('\n'),
|
||||
},
|
||||
]
|
||||
|
||||
// ---------- explore targeting ----------
|
||||
// args.graph: session-computed coupling ranking (rank-explore.mjs). The workflow only reads
|
||||
// .file - scoring already happened outside, where the filesystem is.
|
||||
const GRAPH_ROWS = (Array.isArray(ARGS.graph) ? ARGS.graph : []).filter((r) => r && typeof r.file === 'string')
|
||||
const EXPLORE_FILES_PER_LENS = 10
|
||||
const exploreConsumed = new Set() // within-run consumption: never re-offer a file to a later round
|
||||
let exploreFallbackLogged = false
|
||||
function drawExploreFiles() {
|
||||
let pool
|
||||
if (GRAPH_ROWS.length) pool = GRAPH_ROWS.map((r) => r.file)
|
||||
else {
|
||||
if (!exploreFallbackLogged) {
|
||||
log('explore: args.graph absent/empty - falling back to churn-based fresh eyes')
|
||||
exploreFallbackLogged = true
|
||||
}
|
||||
pool = churnFiles
|
||||
}
|
||||
const files = pool
|
||||
.filter((f) => !exploreConsumed.has(f) && !seen.some((s) => s.file === f))
|
||||
.slice(0, EXPLORE_FILES_PER_LENS)
|
||||
for (const f of files) exploreConsumed.add(f)
|
||||
return files
|
||||
}
|
||||
function buildAdaptiveLenses() {
|
||||
function exploreLens(i) {
|
||||
const files = drawExploreFiles()
|
||||
if (!files.length) return null
|
||||
const src = GRAPH_ROWS.length
|
||||
? `These files are heavily coupled (per the code graph) to files where confirmed bugs live, yet no ` +
|
||||
`hunt has confirmed or refuted a single finding in them - either they are clean or every lens so ` +
|
||||
`far walked past them.`
|
||||
: `These files churned heavily in the last 8 weeks, yet no hunt round has confirmed or refuted a ` +
|
||||
`single finding in them - either they are clean or every lens so far walked past them.`
|
||||
return {
|
||||
key: `explore-${i}`,
|
||||
prompt: `${src} Read each one IN FULL with fresh eyes and hunt for real bugs of any class:\n` +
|
||||
files.map((f) => ` - ${f}`).join('\n'),
|
||||
files,
|
||||
}
|
||||
}
|
||||
|
||||
let cooldownCluster = null // the top-ranked cluster hunted in round N sits out round N+1
|
||||
function buildAdaptiveLenses(round) {
|
||||
const byCluster = {}
|
||||
for (const c of confirmedAll) {
|
||||
const cl = clusterOf(c.file)
|
||||
if (!byCluster[cl]) byCluster[cl] = []
|
||||
byCluster[cl].push(c)
|
||||
}
|
||||
const top = Object.entries(byCluster)
|
||||
// Explore-heavy schedule: measured hotspot yield flattened to 0.25 high+med/agent by round 6.
|
||||
const hotspotQuota = round <= 5 ? 2 : 1
|
||||
const exploreQuota = round <= 5 ? 2 : 3
|
||||
const hotKey = (cl) => ('hotspot ' + cl).toLowerCase().replace(/[^a-z0-9]+/g, '-')
|
||||
const picked = Object.entries(byCluster)
|
||||
.sort((a, b) => b[1].length - a[1].length)
|
||||
.slice(0, 3)
|
||||
const hotspots = top.map(([cluster, items]) => ({
|
||||
key: ('hotspot ' + cluster).toLowerCase().replace(/[^a-z0-9]+/g, '-'),
|
||||
.filter(([cl]) => cl !== cooldownCluster)
|
||||
.filter(([cl]) => (cleanStreak[hotKey(cl)] || 0) < 2) // pre-filter so backfill sees the real shortfall
|
||||
.slice(0, hotspotQuota)
|
||||
cooldownCluster = picked.length ? picked[0][0] : null
|
||||
const hotspots = picked.map(([cluster, items]) => ({
|
||||
key: hotKey(cluster),
|
||||
prompt:
|
||||
`Bugs cluster. Confirmed findings so far in ${cluster}:\n` +
|
||||
items.map((i) => ` - ${i.file}:${i.line} ${i.title}`).join('\n') +
|
||||
@@ -318,7 +357,19 @@ function buildAdaptiveLenses() {
|
||||
`(subscribe/unsubscribe, open/close, register/transfer, acquire/release), and the paths a past fix ` +
|
||||
`here did NOT cover. Do not re-report the findings listed above - they are already known.`,
|
||||
}))
|
||||
return [...hotspots, ...freshEyesLens()]
|
||||
const shortfall = hotspotQuota - hotspots.length
|
||||
if (shortfall > 0) log(`adaptive: hotspot pool short by ${shortfall} - trying explore backfill`)
|
||||
const explores = []
|
||||
for (let i = 1; i <= exploreQuota + shortfall; i++) {
|
||||
if ((cleanStreak[`explore-${i}`] || 0) >= 2) continue // demoted slot: no substitution, that IS demotion
|
||||
const lens = exploreLens(i)
|
||||
if (!lens) {
|
||||
log(`adaptive: explore pool exhausted after ${explores.length} lens(es)`)
|
||||
break
|
||||
}
|
||||
explores.push(lens)
|
||||
}
|
||||
return [...hotspots, ...explores]
|
||||
}
|
||||
|
||||
// ---------- dedupe + ledger helpers ----------
|
||||
@@ -343,10 +394,18 @@ function isDup(a, b) {
|
||||
const hits = aw.filter((w) => bw.has(w)).length
|
||||
return hits * 2 >= aw.length
|
||||
}
|
||||
function dedupe(cands, priors) {
|
||||
function dedupe(cands, priors, counts) {
|
||||
const kept = []
|
||||
for (const c of cands) {
|
||||
if (priors.some((p) => isDup(c, p)) || kept.some((k) => isDup(c, k))) continue
|
||||
const prior = priors.find((p) => isDup(c, p))
|
||||
if (prior) {
|
||||
if (counts) counts[prior.fromLedger ? 'suppressedLedger' : 'suppressedRun']++
|
||||
continue
|
||||
}
|
||||
if (kept.some((k) => isDup(c, k))) {
|
||||
if (counts) counts.suppressedRun++
|
||||
continue
|
||||
}
|
||||
kept.push(c)
|
||||
}
|
||||
return kept
|
||||
@@ -417,6 +476,7 @@ const seen = (ARGS.known || []).map((k) => ({
|
||||
line: k.line,
|
||||
title: k.title,
|
||||
status: k.status || 'known',
|
||||
fromLedger: true, // telemetry: distinguishes ledger suppression from same-run suppression
|
||||
}))
|
||||
const confirmedAll = []
|
||||
const unverified = []
|
||||
@@ -464,10 +524,12 @@ while (dry < DRY_THRESHOLD && round < MAX_ROUNDS) {
|
||||
const family = lensesForRound(round + 1)
|
||||
if (!family || !family.length) break // nothing to hunt != everything demoted
|
||||
round++
|
||||
const spentBefore = budget.spent()
|
||||
const counts = { suppressedLedger: 0, suppressedRun: 0, finderNull: 0, finderEmpty: 0, verifierNull: 0 }
|
||||
const lenses = family.filter((l) => (cleanStreak[l.key] || 0) < 2)
|
||||
if (!lenses.length) {
|
||||
dry++
|
||||
roundStats.push({ round, family: familyName(round), lenses: 0, candidates: 0, fresh: 0, confirmed: 0, refuted: 0, dryEligible: true, dryAfter: dry })
|
||||
roundStats.push({ round, family: familyName(round), lenses: 0, candidates: 0, fresh: 0, confirmed: 0, refuted: 0, dryEligible: true, dryAfter: dry, severity: { critical: 0, high: 0, medium: 0, low: 0 }, perLens: {}, filesTouched: 0, filesNew: 0, ...counts, spentBefore, spentAfter: budget.spent() })
|
||||
log(`Round ${round}: every lens demoted - counts as a dry round (dry=${dry})`)
|
||||
continue
|
||||
}
|
||||
@@ -476,123 +538,184 @@ while (dry < DRY_THRESHOLD && round < MAX_ROUNDS) {
|
||||
const lensResults = await pipeline(
|
||||
lenses,
|
||||
(lens) =>
|
||||
parallel([
|
||||
() => agent(finderPrompt(lens, rnd), { label: `r${rnd}:hunt:${lens.key}:opus`, phase: `Round ${rnd}`, model: 'opus', effort: 'high', schema: FINDINGS }),
|
||||
() => agent(finderPrompt(lens, rnd), { label: `r${rnd}:hunt:${lens.key}:sonnet`, phase: `Round ${rnd}`, model: 'sonnet', effort: 'high', schema: FINDINGS }),
|
||||
]).then((pair) => ({ lens, pair })),
|
||||
agent(finderPrompt(lens, rnd), { label: `r${rnd}:hunt:${lens.key}:opus`, phase: `Round ${rnd}`, model: 'opus', effort: 'high', schema: FINDINGS })
|
||||
.then((res) => ({ lens, res })),
|
||||
async (r) => {
|
||||
const { lens, pair } = r
|
||||
const finderFailed = pair.some((p) => p === null)
|
||||
// Tag by panel slot, not by position in the surviving list: filtering the nulls out first
|
||||
// would shift sonnet into slot 0 whenever opus dies and mislabel its finds. The panel unions
|
||||
// rather than votes, so this is the only signal for whether the second finder earns its cost
|
||||
// - and since dedupe keeps the first occurrence and opus is slot 0, `finder: 'sonnet'` means
|
||||
// opus did not report it. `finder: 'opus'` says nothing about sonnet either way.
|
||||
const union = pair.flatMap((p, i) =>
|
||||
p ? (p.findings || []).map((f) => ({ ...f, finder: i ? 'sonnet' : 'opus' })) : [],
|
||||
)
|
||||
const fresh = dedupe(union, seenAtStart)
|
||||
if (!fresh.length) return { lens, finderFailed, unionCount: union.length, fresh: [], verdicts: [] }
|
||||
const { lens, res } = r
|
||||
// agent() returns null on failure; a thrown stage instead nulls the whole lens result,
|
||||
// which the eligibility check catches separately. Both checks are needed.
|
||||
const finderFailed = res === null
|
||||
if (finderFailed) counts.finderNull++
|
||||
else if (!(res.findings || []).length) counts.finderEmpty++
|
||||
// finder is constant now; kept on the record for ledger continuity across hunts
|
||||
const union = res ? (res.findings || []).map((f) => ({ ...f, finder: 'opus' })) : []
|
||||
const fresh = dedupe(union, seenAtStart, counts)
|
||||
if (!fresh.length) return { lens, finderFailed, unionCount: union.length, fresh: [], matched: [], unmatched: [] }
|
||||
log(`r${rnd} ${lens.key}: ${fresh.length} fresh candidate(s) -> verification`)
|
||||
const vopts = { phase: `Round ${rnd}`, model: 'fable', effort: 'high', schema: VERDICTS }
|
||||
let v = await agent(verifyPrompt(lens.key, fresh), { ...vopts, label: `r${rnd}:verify:${lens.key}` })
|
||||
if (!v || (v.verdicts || []).length < fresh.length) {
|
||||
const retry = await agent(verifyPrompt(lens.key, fresh), { ...vopts, label: `r${rnd}:verify:${lens.key}:retry` })
|
||||
if (((retry && retry.verdicts) || []).length > ((v && v.verdicts) || []).length) v = retry
|
||||
// Pair verdicts to candidates as they arrive, then retry ONLY what got no usable verdict.
|
||||
// Retrying the whole batch re-burned every verdict on a partial return, and the old
|
||||
// count-based trigger let N unmatched garbage verdicts skip the retry entirely.
|
||||
const matched = []
|
||||
const unmatched = fresh.slice()
|
||||
const absorb = (vs) => {
|
||||
for (const v of vs || []) {
|
||||
const vRec = { file: v.file, line: v.line, title: v.title }
|
||||
const idx = unmatched.findIndex((f) => isDup(vRec, f) || isDup(f, vRec))
|
||||
if (idx === -1) {
|
||||
const claimed = matched.some(({ cand }) => isDup(vRec, cand) || isDup(cand, vRec))
|
||||
log(`r${rnd} ${lens.key}: verifier verdict "${v.title}" (${v.file}:${v.line}) ${claimed ? 'duplicates an already-claimed candidate' : 'matched no candidate'} - dropped`)
|
||||
continue
|
||||
}
|
||||
const [cand] = unmatched.splice(idx, 1)
|
||||
matched.push({ v, cand })
|
||||
}
|
||||
}
|
||||
return { lens, finderFailed, unionCount: union.length, fresh, verdicts: (v && v.verdicts) || [] }
|
||||
const v1 = await agent(verifyPrompt(lens.key, fresh), { ...vopts, label: `r${rnd}:verify:${lens.key}` })
|
||||
absorb(v1 && v1.verdicts)
|
||||
if (!v1) counts.verifierNull++
|
||||
if (unmatched.length) {
|
||||
const v2 = await agent(verifyPrompt(lens.key, unmatched.slice()), { ...vopts, label: `r${rnd}:verify:${lens.key}:retry` })
|
||||
absorb(v2 && v2.verdicts)
|
||||
if (!v2) counts.verifierNull++
|
||||
}
|
||||
return { lens, finderFailed, unionCount: union.length, fresh, matched, unmatched }
|
||||
},
|
||||
)
|
||||
|
||||
// a thrown stage nulls the whole lens result - rewind its explore draw too, or the
|
||||
// session records never-read files as explored-clean (the same poison as a null finder)
|
||||
lensResults.forEach((r, i) => {
|
||||
if (!r && lenses[i].files) for (const f of lenses[i].files) exploreConsumed.delete(f)
|
||||
})
|
||||
|
||||
let eligible = !lensResults.some((r) => !r)
|
||||
let newConfirmed = 0
|
||||
let newRefuted = 0
|
||||
let candCount = 0
|
||||
let freshCount = 0
|
||||
const perLens = {}
|
||||
const sevMix = { critical: 0, high: 0, medium: 0, low: 0 }
|
||||
const filesTouched = new Set()
|
||||
const filesNew = new Set()
|
||||
for (const r of lensResults.filter(Boolean)) {
|
||||
candCount += r.unionCount
|
||||
freshCount += r.fresh.length
|
||||
if (r.finderFailed) eligible = false
|
||||
if (r.finderFailed && r.lens.files) {
|
||||
// a dead finder read nothing: un-consume its draw so later rounds can re-offer the
|
||||
// files and the session does not record never-examined files as explored-clean
|
||||
for (const f of r.lens.files) exploreConsumed.delete(f)
|
||||
}
|
||||
let lensConfirmed = 0
|
||||
const unmatched = r.fresh.slice()
|
||||
for (const v of r.verdicts) {
|
||||
const vRec = { file: v.file, line: v.line, title: v.title }
|
||||
const idx = unmatched.findIndex((f) => isDup(vRec, f) || isDup(f, vRec))
|
||||
if (idx === -1) {
|
||||
log(`r${round} ${r.lens.key}: verifier verdict "${v.title}" (${v.file}:${v.line}) matched no candidate - dropped`)
|
||||
continue
|
||||
}
|
||||
let lensRefuted = 0
|
||||
for (const { v, cand } of r.matched) {
|
||||
// Keep the matched candidate: the verdict schema has no why/repro/evidence, and the
|
||||
// ledger needs them. Verdict fields are spread last so the verifier's re-rated severity
|
||||
// and its corrected title/file/line win over the finder's.
|
||||
const [cand] = unmatched.splice(idx, 1)
|
||||
const rec = { file: v.file, line: v.line, title: v.title, status: v.refuted ? 'refuted' : 'confirmed' }
|
||||
if (seen.some((p) => isDup(rec, p))) continue // cross-lens same-round duplicate
|
||||
if (seen.some((p) => isDup(rec, p))) { counts.suppressedRun++; continue } // cross-lens same-round duplicate
|
||||
seen.push(rec)
|
||||
if (v.refuted) newRefuted++
|
||||
if (v.refuted) { newRefuted++; lensRefuted++ }
|
||||
else {
|
||||
newConfirmed++
|
||||
lensConfirmed++
|
||||
sevMix[v.severity] = (sevMix[v.severity] || 0) + 1
|
||||
confirmedAll.push({ ...cand, ...v, lens: r.lens.key, round })
|
||||
}
|
||||
}
|
||||
if (unmatched.length) {
|
||||
if (r.unmatched.length) {
|
||||
eligible = false // partial verifier failure: some candidates got no verdict at all
|
||||
for (const f of unmatched) unverified.push({ ...f, lens: r.lens.key, round })
|
||||
for (const f of r.unmatched) unverified.push({ ...f, lens: r.lens.key, round })
|
||||
}
|
||||
// a lens hunted at partial panel strength, or whose candidates never got a verdict, is not evidence of cleanliness
|
||||
if (!r.finderFailed && !unmatched.length) cleanStreak[r.lens.key] = lensConfirmed > 0 ? 0 : (cleanStreak[r.lens.key] || 0) + 1
|
||||
if (!r.finderFailed && !r.unmatched.length) cleanStreak[r.lens.key] = lensConfirmed > 0 ? 0 : (cleanStreak[r.lens.key] || 0) + 1
|
||||
perLens[r.lens.key] = { candidates: r.unionCount, fresh: r.fresh.length, confirmed: lensConfirmed, refuted: lensRefuted, unverified: r.unmatched.length }
|
||||
// coverage proxy: files that produced fresh candidates this round (finder reading is unobservable)
|
||||
for (const f of r.fresh) {
|
||||
filesTouched.add(f.file)
|
||||
if (!seenAtStart.some((s) => s.file === f.file)) filesNew.add(f.file)
|
||||
}
|
||||
}
|
||||
|
||||
if (newConfirmed > 0) dry = 0
|
||||
else if (eligible) dry++
|
||||
// ineligible zero-confirm round: dry unchanged - "we didn't fully look" is not "it's clean"
|
||||
roundStats.push({ round, family: familyName(round), lenses: lenses.length, candidates: candCount, fresh: freshCount, confirmed: newConfirmed, refuted: newRefuted, dryEligible: eligible, dryAfter: dry })
|
||||
roundStats.push({ round, family: familyName(round), lenses: lenses.length, candidates: candCount, fresh: freshCount, confirmed: newConfirmed, refuted: newRefuted, dryEligible: eligible, dryAfter: dry, severity: sevMix, perLens, filesTouched: filesTouched.size, filesNew: filesNew.size, ...counts, spentBefore, spentAfter: budget.spent() })
|
||||
log(`Round ${round} (${familyName(round)}): ${newConfirmed} confirmed, ${newRefuted} refuted, dry=${dry}${eligible ? '' : ' (ineligible)'}`)
|
||||
}
|
||||
|
||||
const converged = dry >= DRY_THRESHOLD
|
||||
|
||||
// The panel unions rather than votes, so the second finder's whole value is what it finds alone.
|
||||
// dedupe keeps the opus-slot record when both report the same bug, so a confirmed finding tagged
|
||||
// sonnet is one opus missed. A run where that count is 0 is the evidence for dropping the second
|
||||
// model; anything above 0 is what it bought.
|
||||
const sonnetOnly = confirmedAll.filter((f) => f.finder === 'sonnet').length
|
||||
log(`panel: ${confirmedAll.length} confirmed, ${sonnetOnly} sonnet-only (opus missed), ${confirmedAll.length - sonnetOnly} found by opus`)
|
||||
|
||||
// ---------- report ----------
|
||||
phase('Report')
|
||||
// ---------- report (deterministic) ----------
|
||||
// A report agent silently dropped findings (79 sections for 82 confirmed on 2026-08-12), so the
|
||||
// markdown is assembled in-script from confirmedSorted. Coordinate spot-checking moved to the
|
||||
// calling session, which validates EVERY finding's file/line after return (see bughunt-run).
|
||||
const RANK = { critical: 0, high: 1, medium: 2, low: 3 }
|
||||
const confirmedSorted = confirmedAll.slice().sort((a, b) => RANK[a.severity] - RANK[b.severity])
|
||||
const unverifiedFinal = unverified.filter((u) => !seen.some((p) => isDup(u, p)))
|
||||
const table = convergenceTable(roundStats, converged, stoppedOnBudget)
|
||||
|
||||
let report
|
||||
if (!confirmedSorted.length && !unverifiedFinal.length) {
|
||||
const outcome = converged ? 'Converged' : stoppedOnBudget ? 'Stopped on budget - NOT converged' : 'Hit the round backstop - NOT converged'
|
||||
report = `${outcome} after ${round} round(s) with zero confirmed findings.\n\n${table}`
|
||||
} else {
|
||||
report = await agent(
|
||||
`You are writing the final bug-hunt report for OwnCord (D:/Local-Lab/Repos/OwnCord).\n\n` +
|
||||
`The findings below already survived adversarial verification - do NOT re-litigate them, and do NOT add new ` +
|
||||
`ones. Your job is presentation and prioritization for a maintainer who will fix these today.\n\n` +
|
||||
`Spot-check the two highest-severity findings against the real files to make sure file paths and line numbers ` +
|
||||
`are accurate; correct them silently if they drifted.\n\n` +
|
||||
`Write markdown:\n` +
|
||||
` - Open with one paragraph: how many real bugs, in which subsystems, whether the hunt CONVERGED, and which ` +
|
||||
`finding to fix first and why.\n` +
|
||||
` - Then one section per finding, ordered by severity: a "### <severity> - <title>" heading, the ` +
|
||||
`\`file:line\` reference, what breaks and under exactly what conditions, and the smallest correct fix.\n` +
|
||||
(unverifiedFinal.length
|
||||
? ` - Then an "## Unverified - re-run" section listing these candidates whose verification failed twice: ` +
|
||||
`${JSON.stringify(unverifiedFinal)}\n`
|
||||
: '') +
|
||||
` - End with the convergence table below, VERBATIM.\n` +
|
||||
`Write in complete sentences. No emoji, no "consider" hedging.\n\n` +
|
||||
`--- CONFIRMED FINDINGS ---\n${JSON.stringify(confirmedSorted, null, 2)}\n\n` +
|
||||
`--- CONVERGENCE TABLE ---\n${table}`,
|
||||
{ label: 'report', phase: 'Report', model: 'fable', effort: 'high' },
|
||||
)
|
||||
const sum = (k) => roundStats.reduce((n, r) => n + (r[k] || 0), 0)
|
||||
const runStats = {
|
||||
config: { maxRounds: MAX_ROUNDS, dryThreshold: DRY_THRESHOLD, customLenses: !!CUSTOM_LENSES, knownCount: (ARGS.known || []).length, graphRows: GRAPH_ROWS.length, budgetTotal: budget.total },
|
||||
spentTotal: budget.spent(),
|
||||
rounds: roundStats.length,
|
||||
converged,
|
||||
stoppedOnBudget,
|
||||
confirmed: confirmedSorted.length,
|
||||
refuted: sum('refuted'),
|
||||
unverified: unverifiedFinal.length,
|
||||
suppressedLedger: sum('suppressedLedger'),
|
||||
suppressedRun: sum('suppressedRun'),
|
||||
finderNull: sum('finderNull'),
|
||||
finderEmpty: sum('finderEmpty'),
|
||||
verifierNull: sum('verifierNull'),
|
||||
}
|
||||
|
||||
return { converged, stoppedOnBudget, rounds: roundStats, confirmed: confirmedSorted, unverified: unverifiedFinal, report }
|
||||
function buildReport() {
|
||||
const outcome = converged
|
||||
? `CONVERGED after ${round} round(s).`
|
||||
: stoppedOnBudget
|
||||
? `NOT converged - stopped on budget after ${round} round(s).`
|
||||
: `NOT converged - hit the round backstop after ${round} round(s).`
|
||||
const sev = { critical: 0, high: 0, medium: 0, low: 0 }
|
||||
for (const f of confirmedSorted) sev[f.severity] = (sev[f.severity] || 0) + 1
|
||||
const lines = ['# Bug hunt report', '']
|
||||
lines.push(
|
||||
`${confirmedSorted.length} confirmed finding(s) - ${sev.critical} critical, ${sev.high} high, ` +
|
||||
`${sev.medium} medium, ${sev.low} low. ${outcome}` +
|
||||
(confirmedSorted.length
|
||||
? ` Fix first: ${confirmedSorted[0].title} (\`${confirmedSorted[0].file}:${confirmedSorted[0].line}\`).`
|
||||
: ''),
|
||||
'',
|
||||
)
|
||||
for (const f of confirmedSorted) {
|
||||
lines.push(`### ${f.severity} - ${f.title}`, '')
|
||||
lines.push(`\`${f.file}:${f.line}\` - lens \`${f.lens}\`, round ${f.round}, confidence ${f.confidence}`, '')
|
||||
if (f.why) lines.push(f.why, '')
|
||||
if (f.repro) lines.push(`**Repro:** ${f.repro}`, '')
|
||||
if (f.evidence) lines.push(`**Evidence:** ${f.evidence}`, '')
|
||||
if (f.fix) lines.push(`**Fix:** ${f.fix}`, '')
|
||||
}
|
||||
if (unverifiedFinal.length) {
|
||||
lines.push('## Unverified - re-run', '')
|
||||
for (const u of unverifiedFinal) lines.push(`- \`${u.file}:${u.line}\` ${u.title} (lens \`${u.lens}\`, round ${u.round})`)
|
||||
lines.push('')
|
||||
}
|
||||
lines.push(table)
|
||||
lines.push('', '## Run stats', '')
|
||||
lines.push(
|
||||
`Total spent: ${runStats.spentTotal} output tokens across ${runStats.rounds} round(s). ` +
|
||||
`Suppressed by dedupe: ${runStats.suppressedLedger} ledger-known, ${runStats.suppressedRun} same-run. ` +
|
||||
`Agent failures: ${runStats.finderNull} finder null, ${runStats.finderEmpty} finder empty, ${runStats.verifierNull} verifier null.`,
|
||||
'',
|
||||
)
|
||||
lines.push('| round | spent | files (new) | suppressed ledger/run | finder null/empty | verifier null |')
|
||||
lines.push('|---|---|---|---|---|---|')
|
||||
for (const s of roundStats)
|
||||
lines.push(`| ${s.round} | ${s.spentAfter - s.spentBefore} | ${s.filesTouched} (${s.filesNew}) | ${s.suppressedLedger}/${s.suppressedRun} | ${s.finderNull}/${s.finderEmpty} | ${s.verifierNull} |`)
|
||||
return lines.join('\n')
|
||||
}
|
||||
const report = buildReport()
|
||||
|
||||
return { converged, stoppedOnBudget, rounds: roundStats, confirmed: confirmedSorted, unverified: unverifiedFinal, runStats, exploredFiles: [...exploreConsumed], report }
|
||||
|
||||
@@ -98,3 +98,4 @@ Client/tauri-client/.env
|
||||
|
||||
# local server run logs
|
||||
server.log
|
||||
graphify-out/
|
||||
|
||||
Reference in New Issue
Block a user