mirror of
https://github.com/J3vb/OwnCord.git
synced 2026-09-02 19:43:10 +03:00
feat(bughunt): coverage-driven convergence (#1399)
* feat(bughunt): coverage-driven stop rule and directory-coherent sweep Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bughunt): return uncredited explore draws to the pool An explore lens denied coverage credit (dead finder or unverified candidates) now un-consumes its draw so later rounds re-offer the files; consumed-but-uncovered files could otherwise pin uncoveredCount above zero and block convergence. Directory grouping reuses clusterOf(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bughunt): stalled-coverage guard, risky-file class sweep, exhausted-dry convergence Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bughunt): stall guard never stops a still-confirming hunt A round with newConfirmed > 0 resets the coverage-stall counter instead of counting toward it; hotspot yield does not shrink the uncovered pool, and a stuck sweep must not cut off a hunt that is still finding bugs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bughunt): coverage telemetry in report and operator docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(bughunt): scoped-hunt coverage trap and current cost estimate Final-review fixes: warn that args.lenses plus an examined-armed inventory still sweeps the whole pool (pass a filtered inventory or legacy rows to truly scope), and align the budget note with the coverage-run estimate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -16,7 +16,7 @@ reach a commit, an issue, or a PR body.
|
||||
## 1. Hunt
|
||||
|
||||
**Launch the hunt from a turn that carries a token-budget directive** (recommended:
|
||||
`+25M`, comfortably above a full 8-round run). The workflow's cost ceiling is gated
|
||||
`+25M`, comfortably above a full coverage run's ~8-12M). The workflow's cost ceiling is gated
|
||||
on `budget.total`, which is null without a directive — a directive-less run has **no
|
||||
ceiling at all**. The workflow's first log line echoes the state: `budget=25M` means
|
||||
armed; `budget=NONE - cost ceiling disarmed` means stop the run and relaunch with a
|
||||
@@ -26,9 +26,19 @@ Before launching, in order:
|
||||
|
||||
1. **Rebuild the graph** (stale coordinates aim the explore lens at moved code):
|
||||
`graphify update . --no-cluster` — local tree-sitter, zero LLM cost, ~10.7k nodes.
|
||||
2. **Rank explore targets**: `node .superpowers/rank-explore.mjs` — writes
|
||||
`.superpowers/explore-ranking.json`, deprioritizing files recorded clean in
|
||||
`.superpowers/explored-clean.json` and dropping files that no longer exist.
|
||||
2. **Build the inventory**: `node .superpowers/rank-explore.mjs` — writes
|
||||
`.superpowers/explore-ranking.json`: EVERY non-test source file (~419 rows), each with
|
||||
`examined` (already carries a ledger finding or a LIVE explored-clean record → the hunt
|
||||
pre-seeds its covered set), `risky` (top coupling ∪ past-bug clusters ∪ top churn,
|
||||
capped at 40 → they get an extra pass through all 5 bug-class lenses), and `churn`.
|
||||
Explored-clean records carry content hashes: editing a file expires its clean record,
|
||||
so re-runs automatically re-hunt what changed. The hunt cannot stop while any inventory
|
||||
file is uncovered, so a full run now takes ~10-20 rounds and ~8-12M tokens — the `+25M`
|
||||
directive still covers it. Regenerate the inventory and read `known` from the ledger in
|
||||
the SAME session step: both derive from `findings-ledger.json`, and every `known` file
|
||||
must be `examined` in the inventory — a `known` file the inventory does not mark
|
||||
examined can never be drawn (the seen-filter blocks it) nor covered, which would
|
||||
strand `uncoveredCount()` above zero and block convergence.
|
||||
3. Read the ledger and pass every record in as `known`, so the hunt does not
|
||||
re-derive anything already found, fixed, declined, or refuted.
|
||||
|
||||
@@ -39,7 +49,7 @@ Workflow({
|
||||
known: <every record from findings-ledger.json, as {file, line, title, status}>,
|
||||
graph: <the rows of .superpowers/explore-ranking.json>,
|
||||
lenses: [ {key, prompt}, ... ], // optional: scope the hunt to one subsystem
|
||||
maxRounds: 8,
|
||||
maxRounds: 30, // safety backstop only - coverage + dry is the real stop
|
||||
dryThreshold: 2,
|
||||
},
|
||||
})
|
||||
@@ -49,8 +59,28 @@ If `graph` is omitted or empty the hunt logs
|
||||
`explore: args.graph absent/empty - falling back to churn-based fresh eyes` and
|
||||
still runs — degraded targeting, never a smaller lens family.
|
||||
|
||||
`converged: true` now means: every inventory file was covered by a completed
|
||||
explicit-file lens (or carries a verdict), the risky class sweep ran, and then
|
||||
`dryThreshold` consecutive eligible rounds confirmed nothing (a round where the
|
||||
lens family comes up empty with the pool drained counts as dry — family
|
||||
`exhausted`). Rows without the `examined` field fall back to the old
|
||||
quietness-only stop. Two new run outcomes: `stalledCoverage: true` means adaptive
|
||||
rounds stopped shrinking the uncovered pool (usually mass finder failures —
|
||||
investigate before re-running); a budget stop now reports
|
||||
`coverage.uncoveredAtStop` so the next run knows exactly what remains (re-run
|
||||
with the ledger as `known`; live explored-clean records pre-cover what was
|
||||
finished, so the sweep naturally continues where it stopped).
|
||||
|
||||
Omit `lenses` for a general hunt across the rotating families.
|
||||
|
||||
**Scoping a hunt while coverage mode is armed is a budget trap:** `lenses` only
|
||||
replaces round 1, and inventory rows with `examined` force the coverage stop
|
||||
rule — from round 2 the run sweeps the ENTIRE uncovered pool and the risky
|
||||
sweep before it may converge, at general-hunt cost. For a true scoped hunt,
|
||||
pass a subsystem-filtered inventory as `graph` (only the rows you want swept),
|
||||
or rows without the `examined` field to fall back to the legacy quietness-only
|
||||
stop.
|
||||
|
||||
Each lens object is `{key, prompt}`. `key` must match `^[a-z0-9-]+$` —
|
||||
lowercase letters, digits, and hyphens only. Keys get interpolated into agent
|
||||
labels of the form `r<N>:hunt:<key>:<model>`, and a key containing uppercase,
|
||||
|
||||
@@ -73,6 +73,11 @@ export const finding = (n, over = {}) => ({
|
||||
})
|
||||
export const graphRows = (n) =>
|
||||
Array.from({ length: n }, (_, i) => ({ file: `Server/gen/g${i}.go`, score: 1 - i / (n + 1), degree: 10, cited: 5 }))
|
||||
export const inventoryRows = (n, over = () => ({})) =>
|
||||
Array.from({ length: n }, (_, i) => ({
|
||||
file: `Server/gen/g${i}.go`, degree: 10, cited: 5, score: 1 - i / (n + 1),
|
||||
examined: false, risky: false, ...over(i),
|
||||
}))
|
||||
export const confirmAll = (cands) => ({
|
||||
verdicts: cands.map((c) => ({
|
||||
title: c.title, file: c.file, line: c.line,
|
||||
@@ -845,6 +850,193 @@ scenarios.s_explore_rewind_on_thrown_stage = async () => {
|
||||
assert.equal(result.rounds[3].dryEligible, false, 'a nulled lens result still makes the round ineligible')
|
||||
}
|
||||
|
||||
// COV1 (spec §3): coverage mode may NOT stop on quietness while inventory files are uncovered.
|
||||
// 60 rows, 10 pre-examined -> 50 to sweep. Nothing is ever found, so dry passes the threshold
|
||||
// at round 2 - the old stop rule would have converged there. The new rule keeps going until
|
||||
// round 4's explore lenses (quota 4 + backfill 2 slots; 5 draw files, the 6th comes up empty)
|
||||
// cover all 50, then exits. Also locks the enriched explore prompt (class checklist).
|
||||
scenarios.s_coverage_blocks_stop = async () => {
|
||||
const inv = inventoryRows(60, (i) => (i >= 50 ? { examined: true } : {}))
|
||||
const { result, calls } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({ hunt: () => none, verify: (r, k, c) => confirmAll(c) }),
|
||||
})
|
||||
assert.equal(result.rounds.length, 4, 'must run past the dry threshold (hit at r2) to sweep in r4')
|
||||
assert.equal(result.converged, true)
|
||||
assert.deepEqual(result.rounds.map((r) => r.dryAfter), [1, 2, 3, 4])
|
||||
assert.deepEqual(result.runStats.coverage, { inventory: 60, preCovered: 10, covered: 60, uncoveredAtStop: 0 })
|
||||
assert.match(result.report, /CONVERGED/)
|
||||
assert.match(result.report, /Coverage: 60\/60 files \(10 pre-covered/)
|
||||
const ep = (calls.find((c) => (c.opts.label || '') === 'r4:hunt:explore-1:opus') || {}).prompt || ''
|
||||
assert.match(ep, /error-path data loss/, 'explore lenses carry the distilled class checklist')
|
||||
}
|
||||
|
||||
// COV2 (spec §2+§4): a dead explore finder's files stay uncovered and get re-offered; the
|
||||
// run only converges after a LIVE lens covers them.
|
||||
scenarios.s_coverage_dead_finder = async () => {
|
||||
const inv = inventoryRows(20)
|
||||
const { result, calls } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => (round === 4 && key === 'explore-1' ? null : none),
|
||||
verify: (r, k, c) => confirmAll(c),
|
||||
}),
|
||||
})
|
||||
const promptOf = (rnd, key) => (calls.find((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`) || {}).prompt || ''
|
||||
assert.match(promptOf(4, 'explore-1'), /Server\/gen\/g0\.go/, 'r4 explore-1 drew the head of the pool')
|
||||
assert.match(promptOf(5, 'explore-1'), /Server\/gen\/g0\.go/, 'dead lens files are re-offered next round')
|
||||
assert.equal(result.rounds[3].dryEligible, false, 'dead finder keeps the round ineligible')
|
||||
assert.equal(result.converged, true)
|
||||
assert.deepEqual(result.runStats.coverage, { inventory: 20, preCovered: 0, covered: 20, uncoveredAtStop: 0 })
|
||||
}
|
||||
|
||||
// COV7 (amendment 4): explore draws are directory-coherent - one lens reads one module,
|
||||
// not ten strangers. Cross-file classes (state desync, acquire/release pairs) need siblings
|
||||
// in one agent's context.
|
||||
scenarios.s_directory_coherent_draws = async () => {
|
||||
const inv = inventoryRows(20, (i) => ({ file: i % 2 === 0 ? `Server/alpha/a${i}.go` : `Server/beta/b${i}.go` }))
|
||||
const { calls } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({ hunt: () => none, verify: (r, k, c) => confirmAll(c) }),
|
||||
})
|
||||
const p1 = (calls.find((c) => (c.opts.label || '') === 'r4:hunt:explore-1:opus') || {}).prompt || ''
|
||||
assert.match(p1, /Server\/alpha\/a0\.go/)
|
||||
assert.match(p1, /Server\/alpha\/a18\.go/, 'all ten alpha files ride in the first lens')
|
||||
assert.doesNotMatch(p1, /Server\/beta\//, 'no stranger directories in a coherent draw')
|
||||
}
|
||||
|
||||
// COV8 (Task 5 review finding): an explore lens whose candidates never get a verdict is
|
||||
// denied coverage credit - its draw must return to the pool and be re-offered, or the
|
||||
// consumed-but-uncovered files strand uncoveredCount() above zero and the run can never
|
||||
// converge (it would grind to the round backstop instead).
|
||||
scenarios.s_uncredited_draw_returns_to_pool = async () => {
|
||||
const inv = inventoryRows(20)
|
||||
const { result, calls } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) =>
|
||||
round === 4 && key === 'explore-1'
|
||||
? { findings: [finding(1, { file: 'Server/gen/g0.go', title: 'orphaned candidate one' })] }
|
||||
: none,
|
||||
verify: (round, key, cands) =>
|
||||
round === 4 && key === 'explore-1' ? { verdicts: [] } : confirmAll(cands),
|
||||
}),
|
||||
})
|
||||
const promptOf = (rnd, key) => (calls.find((c) => (c.opts.label || '') === `r${rnd}:hunt:${key}:opus`) || {}).prompt || ''
|
||||
assert.match(promptOf(4, 'explore-1'), /Server\/gen\/g0\.go/)
|
||||
assert.match(promptOf(5, 'explore-1'), /Server\/gen\/g0\.go/, 'uncredited draw is re-offered next round')
|
||||
assert.equal(result.rounds[3].dryEligible, false, 'unverified candidates keep the round ineligible')
|
||||
assert.equal(result.converged, true, 'the run must still converge once a later lens covers the files')
|
||||
assert.deepEqual(result.runStats.coverage, { inventory: 20, preCovered: 0, covered: 20, uncoveredAtStop: 0 })
|
||||
assert.equal(result.unverified.length, 1, 'the orphaned candidate stays reported unverified')
|
||||
}
|
||||
|
||||
// COV3 (spec §4): finders dying every adaptive round -> uncovered never shrinks -> stop with
|
||||
// stalledCoverage after 2 stalled adaptive rounds instead of burning 26 more rounds.
|
||||
scenarios.s_coverage_stall = async () => {
|
||||
const inv = inventoryRows(20)
|
||||
const { result, logs } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => (/^explore-/.test(key) ? null : none),
|
||||
verify: (r, k, c) => confirmAll(c),
|
||||
}),
|
||||
})
|
||||
assert.equal(result.stalledCoverage, true)
|
||||
assert.equal(result.converged, false)
|
||||
assert.equal(result.rounds.length, 5, 'r1-3 families, then exactly 2 stalled adaptive rounds')
|
||||
assert.equal(result.runStats.coverage.uncoveredAtStop, 20)
|
||||
assert.ok(logs.some((l) => /Coverage stalled/.test(l)))
|
||||
assert.match(result.report, /NOT converged - coverage stalled/)
|
||||
}
|
||||
|
||||
// COV4 (spec §2 smart depth): once the sweep completes, the 5 bug-class lenses run once more
|
||||
// scoped to the risky files - then the run may converge.
|
||||
scenarios.s_risky_sweep = async () => {
|
||||
const inv = inventoryRows(10, (i) => (i < 2 ? { risky: true } : {}))
|
||||
const { result, calls } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({ hunt: () => none, verify: (r, k, c) => confirmAll(c) }),
|
||||
})
|
||||
const r5keys = [...new Set(calls.filter((c) => /^r5:hunt:/.test(c.opts.label || '')).map((c) => c.opts.label.split(':')[2]))]
|
||||
assert.deepEqual(r5keys.sort(), ['risky-concurrency', 'risky-error-paths', 'risky-lifecycle', 'risky-ordering-boundary', 'risky-state-desync'])
|
||||
const p = calls.find((c) => (c.opts.label || '') === 'r5:hunt:risky-concurrency:opus').prompt
|
||||
assert.match(p, /Server\/gen\/g0\.go/)
|
||||
assert.match(p, /Server\/gen\/g1\.go/)
|
||||
assert.doesNotMatch(p, /Server\/gen\/g5\.go/, 'the sweep is scoped to risky files only')
|
||||
assert.equal(result.rounds[4].family, 'risky-sweep')
|
||||
assert.equal(result.converged, true)
|
||||
assert.equal(result.rounds.length, 5, 'dry was already past threshold - the run ends right after the risky sweep')
|
||||
assert.ok(!calls.some((c) => /^r6:hunt:risky-/.test(c.opts.label || '')), 'the risky sweep runs exactly once')
|
||||
}
|
||||
|
||||
// COV5: legacy rows (no `examined` key) leave every new mechanism inert - old stop rule,
|
||||
// no coverage stats, no risky sweep.
|
||||
scenarios.s_legacy_rows_inert = async () => {
|
||||
const { result, calls } = await run({
|
||||
args: { graph: graphRows(30) },
|
||||
agentStub: makeStub({ hunt: () => none, verify: (r, k, c) => confirmAll(c) }),
|
||||
})
|
||||
assert.equal(result.rounds.length, 2, 'legacy mode still converges on the plain dry threshold')
|
||||
assert.equal(result.converged, true)
|
||||
assert.equal(result.runStats.coverage, null)
|
||||
assert.equal(result.stalledCoverage, false)
|
||||
assert.ok(!calls.some((c) => /risky-/.test(c.opts.label || '')))
|
||||
}
|
||||
|
||||
// COV6 (amendment 1): a confirm on the sweep's last round resets dry to 0; hotspot rounds then
|
||||
// go clean, cooldown + an empty pool empty the family - in coverage mode that emptiness IS
|
||||
// quietness and must count dry rounds instead of stranding a fully-covered run at converged:false.
|
||||
scenarios.s_exhausted_counts_dry = async () => {
|
||||
const inv = inventoryRows(10)
|
||||
const { result } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) =>
|
||||
round === 4 && key === 'explore-1'
|
||||
? { findings: [finding(1, { file: 'Server/gen/g0.go', title: 'late sweep bug one' })] }
|
||||
: none,
|
||||
verify: (r, k, c) => confirmAll(c),
|
||||
}),
|
||||
})
|
||||
assert.equal(result.converged, true, 'a fully-covered, fully-quiet run must converge')
|
||||
assert.equal(result.rounds.length, 6)
|
||||
assert.equal(result.rounds[5].family, 'exhausted')
|
||||
assert.equal(result.rounds[5].dryAfter, 2)
|
||||
assert.equal(result.confirmed.length, 1)
|
||||
}
|
||||
|
||||
// COV9 (Task 6 review finding): a stuck pool must NOT stop a hunt that is still confirming.
|
||||
// Dead explore lenses pin uncovered at 20 while hotspot lenses confirm fresh bugs in rounds
|
||||
// 4 and 6 - each productive round resets the stall counter, so the run survives to round 8
|
||||
// and only then stops with stalledCoverage (without the productivity term it would have
|
||||
// stopped at round 5, mid-yield).
|
||||
scenarios.s_stall_deferred_while_productive = async () => {
|
||||
const inv = inventoryRows(20)
|
||||
const { result, logs } = await run({
|
||||
args: { graph: inv },
|
||||
agentStub: makeStub({
|
||||
hunt: (round, key) => {
|
||||
if (round === 1 && key === 'ws-hub')
|
||||
return { findings: [finding(1, { file: 'Server/ws/hub.go', title: 'seed bug alpha one' })] }
|
||||
if (round === 4 && key === 'hotspot-server-ws')
|
||||
return { findings: [finding(2, { file: 'Server/ws/emit.go', title: 'adjacent bug beta two' })] }
|
||||
if (round === 6 && key === 'hotspot-server-ws')
|
||||
return { findings: [finding(3, { file: 'Server/ws/pubsub.go', title: 'adjacent bug gamma three' })] }
|
||||
if (/^explore-/.test(key)) return null // dead explore finders: the pool never shrinks
|
||||
return none
|
||||
},
|
||||
verify: (r, k, c) => confirmAll(c),
|
||||
}),
|
||||
})
|
||||
assert.equal(result.rounds.length, 8, 'productive rounds 4 and 6 must defer the stall to round 8')
|
||||
assert.equal(result.stalledCoverage, true)
|
||||
assert.equal(result.converged, false)
|
||||
assert.equal(result.confirmed.length, 3)
|
||||
assert.equal(result.runStats.coverage.uncoveredAtStop, 20)
|
||||
assert.ok(logs.some((l) => /Coverage stalled/.test(l)))
|
||||
}
|
||||
|
||||
// ---------- runner ----------
|
||||
const only = process.argv[2]
|
||||
for (const [name, fn] of Object.entries(scenarios)) {
|
||||
|
||||
+125
-33
@@ -15,7 +15,7 @@ const ARGS = (() => {
|
||||
}
|
||||
return args || {}
|
||||
})()
|
||||
const MAX_ROUNDS = ARGS.maxRounds || 8
|
||||
const MAX_ROUNDS = ARGS.maxRounds || 30
|
||||
const DRY_THRESHOLD = ARGS.dryThreshold || 2
|
||||
// A scoped hunt (args.lenses) replaces the round-1 family outright; later rounds still go
|
||||
// adaptive, so hotspot and explore coverage - and therefore convergence - still work.
|
||||
@@ -281,17 +281,31 @@ const FLOW_LENSES = [
|
||||
},
|
||||
]
|
||||
|
||||
let riskySweepDone = false
|
||||
function riskySweepLenses() {
|
||||
if (riskySweepDone || !HAS_INVENTORY || !RISKY_FILES.length || uncoveredCount() > 0) return null
|
||||
riskySweepDone = true // consumed even if this round's finders die: same at-most-once semantics as cooldown
|
||||
const list = RISKY_FILES.map((f) => ` - ${f}`).join('\n')
|
||||
return BUGCLASS_LENSES.map((l) => ({
|
||||
key: `risky-${l.key}`,
|
||||
prompt: `${l.prompt}\n\nScope this sweep to ONLY these highest-risk files (read each one in full):\n${list}`,
|
||||
}))
|
||||
}
|
||||
let currentFamilyName = 'surfaces'
|
||||
function lensesForRound(round) {
|
||||
if (CUSTOM_LENSES) return round === 1 ? CUSTOM_LENSES : buildAdaptiveLenses(round)
|
||||
if (round === 1) return SURFACE_LENSES
|
||||
if (round === 2) return BUGCLASS_LENSES
|
||||
if (round === 3) return FLOW_LENSES
|
||||
return buildAdaptiveLenses(round)
|
||||
}
|
||||
function familyName(round) {
|
||||
if (CUSTOM_LENSES) return round === 1 ? 'custom' : 'adaptive'
|
||||
return ['surfaces', 'bug-classes', 'flows'][round - 1] || 'adaptive'
|
||||
const pick = (name, lenses) => { currentFamilyName = name; return lenses }
|
||||
if (CUSTOM_LENSES) {
|
||||
if (round === 1) return pick('custom', CUSTOM_LENSES)
|
||||
} else {
|
||||
if (round === 1) return pick('surfaces', SURFACE_LENSES)
|
||||
if (round === 2) return pick('bug-classes', BUGCLASS_LENSES)
|
||||
if (round === 3) return pick('flows', FLOW_LENSES)
|
||||
}
|
||||
const risky = riskySweepLenses()
|
||||
if (risky) return pick('risky-sweep', risky)
|
||||
return pick('adaptive', buildAdaptiveLenses(round))
|
||||
}
|
||||
function familyName() { return currentFamilyName }
|
||||
// Directory granularity: the old two/three-segment cluster collapsed the whole TS client into
|
||||
// one bucket (35 of 82 findings), so the "top cluster" never changed for five straight rounds.
|
||||
function clusterOf(file) {
|
||||
@@ -303,6 +317,17 @@ function clusterOf(file) {
|
||||
// args.graph: session-computed coupling ranking (rank-explore.mjs). The workflow only reads
|
||||
// .file - scoring already happened outside, where the filesystem is.
|
||||
const GRAPH_ROWS = (Array.isArray(ARGS.graph) ? ARGS.graph : []).filter((r) => r && typeof r.file === 'string')
|
||||
// ---------- coverage mode (spec 2026-08-20) ----------
|
||||
// Arms only when rows carry the `examined` flag (full inventory from rank-explore.mjs).
|
||||
// Legacy rows and the churn fallback leave all of this inert: covered stays empty,
|
||||
// uncoveredCount() is 0, and the loop condition reduces to the old dry-threshold rule.
|
||||
const HAS_INVENTORY = GRAPH_ROWS.some((r) => 'examined' in r)
|
||||
const INVENTORY = HAS_INVENTORY ? GRAPH_ROWS.map((r) => r.file) : []
|
||||
const covered = new Set(HAS_INVENTORY ? GRAPH_ROWS.filter((r) => r.examined).map((r) => r.file) : [])
|
||||
const PRE_COVERED = covered.size
|
||||
const RISKY_FILES = HAS_INVENTORY ? GRAPH_ROWS.filter((r) => r.risky).map((r) => r.file) : []
|
||||
const uncoveredCount = () => (HAS_INVENTORY ? INVENTORY.reduce((n, f) => n + (covered.has(f) ? 0 : 1), 0) : 0)
|
||||
if (HAS_INVENTORY) log(`coverage: inventory=${INVENTORY.length} preCovered=${PRE_COVERED} risky=${RISKY_FILES.length}`)
|
||||
const EXPLORE_FILES_PER_LENS = 10
|
||||
const exploreConsumed = new Set() // within-run consumption: never re-offer a file to a later round
|
||||
let exploreFallbackLogged = false
|
||||
@@ -316,9 +341,18 @@ function drawExploreFiles() {
|
||||
}
|
||||
pool = churnFiles
|
||||
}
|
||||
const files = pool
|
||||
.filter((f) => !exploreConsumed.has(f) && !seen.some((s) => s.file === f))
|
||||
.slice(0, EXPLORE_FILES_PER_LENS)
|
||||
const avail = pool.filter((f) => !exploreConsumed.has(f) && !covered.has(f) && !seen.some((s) => s.file === f))
|
||||
const files = []
|
||||
while (files.length < EXPLORE_FILES_PER_LENS && avail.length) {
|
||||
const head = avail.shift()
|
||||
files.push(head)
|
||||
const dir = clusterOf(head)
|
||||
// pull same-directory siblings forward: one lens reading one module beats ten strangers
|
||||
for (let i = 0; i < avail.length && files.length < EXPLORE_FILES_PER_LENS; ) {
|
||||
if (clusterOf(avail[i]) === dir) files.push(avail.splice(i, 1)[0])
|
||||
else i++
|
||||
}
|
||||
}
|
||||
for (const f of files) exploreConsumed.add(f)
|
||||
return files
|
||||
}
|
||||
@@ -333,7 +367,12 @@ function exploreLens(i) {
|
||||
`single finding in them - either they are clean or every lens so far walked past them.`
|
||||
return {
|
||||
key: `explore-${i}`,
|
||||
prompt: `${src} Read each one IN FULL with fresh eyes and hunt for real bugs of any class:\n` +
|
||||
prompt: `${src} Read each one IN FULL with fresh eyes. Hunt every class: concurrency and ` +
|
||||
`interleaving (races, TOCTOU, lock ordering, stale-closure writes after await); lifecycle and ` +
|
||||
`teardown (unreleased acquires, use-after-close, missing disposal on error paths); state desync ` +
|
||||
`(two sources of truth updated by different code paths); error-path data loss (swallowed errors, ` +
|
||||
`partial writes, silent fallbacks); ordering and boundaries (off-by-one, pagination truncation, ` +
|
||||
`sequence gaps, inclusive/exclusive disagreements).\n` +
|
||||
files.map((f) => ` - ${f}`).join('\n'),
|
||||
files,
|
||||
}
|
||||
@@ -348,8 +387,10 @@ function buildAdaptiveLenses(round) {
|
||||
byCluster[cl].push(c)
|
||||
}
|
||||
// Explore-heavy schedule: measured hotspot yield flattened to 0.25 high+med/agent by round 6.
|
||||
const sweeping = uncoveredCount() > 0
|
||||
const hotspotQuota = round <= 5 ? 2 : 1
|
||||
const exploreQuota = round <= 5 ? 2 : 3
|
||||
// sweep pace: 4 explore lenses x 10 files while inventory files remain uncovered
|
||||
const exploreQuota = sweeping ? 4 : round <= 5 ? 2 : 3
|
||||
const hotKey = (cl) => ('hotspot ' + cl).toLowerCase().replace(/[^a-z0-9]+/g, '-')
|
||||
const picked = Object.entries(byCluster)
|
||||
.sort((a, b) => b[1].length - a[1].length)
|
||||
@@ -370,7 +411,7 @@ function buildAdaptiveLenses(round) {
|
||||
if (shortfall > 0) log(`adaptive: hotspot pool short by ${shortfall} - trying explore backfill`)
|
||||
const explores = []
|
||||
for (let i = 1; i <= exploreQuota + shortfall; i++) {
|
||||
if ((cleanStreak[`explore-${i}`] || 0) >= 2) continue // demoted slot: no substitution, that IS demotion
|
||||
if (!sweeping && (cleanStreak[`explore-${i}`] || 0) >= 2) continue // demoted slot: no substitution, that IS demotion
|
||||
const lens = exploreLens(i)
|
||||
if (!lens) {
|
||||
log(`adaptive: explore pool exhausted after ${explores.length} lens(es)`)
|
||||
@@ -424,12 +465,14 @@ function seenBlock(seen) {
|
||||
const lines = seen.map((s) => ` - ${s.file}:${s.line} [${s.status}] ${s.title}`)
|
||||
return `\n--- KNOWN FINDINGS (already investigated - do NOT re-report; refuted means examined and rejected) ---\n${lines.join('\n')}\n`
|
||||
}
|
||||
function convergenceTable(stats, converged, stoppedOnBudget) {
|
||||
function convergenceTable(stats, converged, stoppedOnBudget, stalled) {
|
||||
const verdict = converged
|
||||
? `CONVERGED after ${stats.length} round(s).`
|
||||
: stoppedOnBudget
|
||||
? 'NOT converged - stopped on budget.'
|
||||
: 'NOT converged - hit the round backstop.'
|
||||
: stalled
|
||||
? 'NOT converged - coverage stalled.'
|
||||
: stoppedOnBudget
|
||||
? 'NOT converged - stopped on budget.'
|
||||
: 'NOT converged - hit the round backstop.'
|
||||
const rows = stats.map(
|
||||
(s) =>
|
||||
`| ${s.round} | ${s.family} | ${s.lenses} | ${s.candidates} | ${s.fresh} | ${s.confirmed} | ${s.refuted} | ${s.dryEligible ? 'yes' : 'NO'} | ${s.dryAfter} |`,
|
||||
@@ -494,6 +537,8 @@ const cleanStreak = {}
|
||||
let dry = 0
|
||||
let round = 0
|
||||
let stoppedOnBudget = false
|
||||
let coverageStall = 0
|
||||
let stalledCoverage = false
|
||||
|
||||
function finderPrompt(lens, rnd) {
|
||||
return (
|
||||
@@ -524,18 +569,35 @@ function verifyPrompt(lensKey, candidates) {
|
||||
)
|
||||
}
|
||||
|
||||
while (dry < DRY_THRESHOLD && round < MAX_ROUNDS) {
|
||||
const riskySweepPending = () => HAS_INVENTORY && RISKY_FILES.length > 0 && !riskySweepDone
|
||||
while ((uncoveredCount() > 0 || riskySweepPending() || dry < DRY_THRESHOLD) && round < MAX_ROUNDS) {
|
||||
if (BUDGET_TOTAL && remainingBudget() < ROUND_BUDGET_FLOOR) {
|
||||
stoppedOnBudget = true
|
||||
log(`Budget floor reached (${Math.round(remainingBudget() / 1000)}k left) - stopping before round ${round + 1}`)
|
||||
break
|
||||
}
|
||||
const family = lensesForRound(round + 1)
|
||||
if (!family || !family.length) break // nothing to hunt != everything demoted
|
||||
if (!family || !family.length) {
|
||||
// Coverage mode with the pool drained and the risky sweep done: an empty family means
|
||||
// hotspots are demoted/cooled and there is genuinely nothing left to hunt - that IS
|
||||
// quietness. Count it as a dry round so a late confirm cannot strand a fully-covered
|
||||
// run one dry round short of its earned convergence. Legacy mode keeps the hard stop:
|
||||
// an empty family there means "nothing targetable" (no churn, no graph), not "done".
|
||||
if (HAS_INVENTORY && uncoveredCount() === 0 && !riskySweepPending()) {
|
||||
round++
|
||||
dry++
|
||||
roundStats.push({ round, family: 'exhausted', lenses: 0, candidates: 0, fresh: 0, confirmed: 0, refuted: 0, dryEligible: true, dryAfter: dry, severity: { critical: 0, high: 0, medium: 0, low: 0 }, perLens: {}, filesTouched: 0, filesNew: 0, suppressedLedger: 0, suppressedRun: 0, finderNull: 0, finderEmpty: 0, verifierNull: 0, spentBefore: budget.spent(), spentAfter: budget.spent() })
|
||||
log(`Round ${round}: nothing left to hunt - counts as a dry round (dry=${dry})`)
|
||||
continue
|
||||
}
|
||||
break // nothing to hunt != everything demoted
|
||||
}
|
||||
round++
|
||||
const uncBefore = uncoveredCount()
|
||||
const spentBefore = budget.spent()
|
||||
const counts = { suppressedLedger: 0, suppressedRun: 0, finderNull: 0, finderEmpty: 0, verifierNull: 0 }
|
||||
const lenses = family.filter((l) => (cleanStreak[l.key] || 0) < 2)
|
||||
const sweepingNow = uncoveredCount() > 0
|
||||
const lenses = family.filter((l) => (sweepingNow && /^explore-/.test(l.key)) || (cleanStreak[l.key] || 0) < 2)
|
||||
if (!lenses.length) {
|
||||
dry++
|
||||
roundStats.push({ round, family: familyName(round), lenses: 0, candidates: 0, fresh: 0, confirmed: 0, refuted: 0, dryEligible: true, dryAfter: dry, severity: { critical: 0, high: 0, medium: 0, low: 0 }, perLens: {}, filesTouched: 0, filesNew: 0, ...counts, spentBefore, spentAfter: budget.spent() })
|
||||
@@ -613,10 +675,13 @@ while (dry < DRY_THRESHOLD && round < MAX_ROUNDS) {
|
||||
candCount += r.unionCount
|
||||
freshCount += r.fresh.length
|
||||
if (r.finderFailed) eligible = false
|
||||
if (r.finderFailed && r.lens.files) {
|
||||
// a dead finder read nothing: un-consume its draw so later rounds can re-offer the
|
||||
// files and the session does not record never-examined files as explored-clean
|
||||
for (const f of r.lens.files) exploreConsumed.delete(f)
|
||||
if (r.lens.files) {
|
||||
// coverage credit (spec: only explicit-file lenses that ran to completion). A lens
|
||||
// denied credit - dead finder OR candidates left unverified - returns its whole draw
|
||||
// to the pool: consumed-but-uncovered files would otherwise strand uncoveredCount()
|
||||
// above zero forever, and a partially-verified draw is not evidence of cleanliness.
|
||||
if (!r.finderFailed && !r.unmatched.length) for (const f of r.lens.files) covered.add(f)
|
||||
else for (const f of r.lens.files) exploreConsumed.delete(f)
|
||||
}
|
||||
let lensConfirmed = 0
|
||||
let lensRefuted = 0
|
||||
@@ -627,6 +692,7 @@ while (dry < DRY_THRESHOLD && round < MAX_ROUNDS) {
|
||||
const rec = { file: v.file, line: v.line, title: v.title, status: v.refuted ? 'refuted' : 'confirmed' }
|
||||
if (seen.some((p) => isDup(rec, p))) { counts.suppressedRun++; continue } // cross-lens same-round duplicate
|
||||
seen.push(rec)
|
||||
covered.add(rec.file) // any verdict proves the file was read (inert in legacy mode: seen already blocks re-draws)
|
||||
if (v.refuted) { newRefuted++; lensRefuted++ }
|
||||
else {
|
||||
newConfirmed++
|
||||
@@ -654,9 +720,24 @@ while (dry < DRY_THRESHOLD && round < MAX_ROUNDS) {
|
||||
// ineligible zero-confirm round: dry unchanged - "we didn't fully look" is not "it's clean"
|
||||
roundStats.push({ round, family: familyName(round), lenses: lenses.length, candidates: candCount, fresh: freshCount, confirmed: newConfirmed, refuted: newRefuted, dryEligible: eligible, dryAfter: dry, severity: sevMix, perLens, filesTouched: filesTouched.size, filesNew: filesNew.size, ...counts, spentBefore, spentAfter: budget.spent() })
|
||||
log(`Round ${round} (${familyName(round)}): ${newConfirmed} confirmed, ${newRefuted} refuted, dry=${dry}${eligible ? '' : ' (ineligible)'}`)
|
||||
|
||||
// stalled coverage: an adaptive round that failed to shrink a non-empty uncovered pool
|
||||
// AND confirmed nothing. Only adaptive rounds count - rounds 1-3 never draw explore
|
||||
// files by design - and a round that confirmed a bug is never a stall: hotspot yield
|
||||
// does not shrink the pool, and cutting off a still-productive hunt is the one thing
|
||||
// a bug-finding tool must not do.
|
||||
const uncAfter = uncoveredCount()
|
||||
if (HAS_INVENTORY && familyName() === 'adaptive' && uncAfter > 0 && newConfirmed === 0) {
|
||||
coverageStall = uncAfter < uncBefore ? 0 : coverageStall + 1
|
||||
if (coverageStall >= 2) {
|
||||
stalledCoverage = true
|
||||
log(`Coverage stalled: uncovered=${uncAfter} did not shrink for 2 adaptive rounds - stopping`)
|
||||
break
|
||||
}
|
||||
} else coverageStall = 0
|
||||
}
|
||||
|
||||
const converged = dry >= DRY_THRESHOLD
|
||||
const converged = uncoveredCount() === 0 && !riskySweepPending() && dry >= DRY_THRESHOLD
|
||||
|
||||
// ---------- report (deterministic) ----------
|
||||
// A report agent silently dropped findings (79 sections for 82 confirmed on 2026-08-12), so the
|
||||
@@ -665,14 +746,16 @@ const converged = dry >= DRY_THRESHOLD
|
||||
const RANK = { critical: 0, high: 1, medium: 2, low: 3 }
|
||||
const confirmedSorted = confirmedAll.slice().sort((a, b) => RANK[a.severity] - RANK[b.severity])
|
||||
const unverifiedFinal = unverified.filter((u) => !seen.some((p) => isDup(u, p)))
|
||||
const table = convergenceTable(roundStats, converged, stoppedOnBudget)
|
||||
const table = convergenceTable(roundStats, converged, stoppedOnBudget, stalledCoverage)
|
||||
const sum = (k) => roundStats.reduce((n, r) => n + (r[k] || 0), 0)
|
||||
const runStats = {
|
||||
config: { maxRounds: MAX_ROUNDS, dryThreshold: DRY_THRESHOLD, customLenses: !!CUSTOM_LENSES, knownCount: (ARGS.known || []).length, graphRows: GRAPH_ROWS.length, budgetTotal: BUDGET_TOTAL },
|
||||
coverage: HAS_INVENTORY ? { inventory: INVENTORY.length, preCovered: PRE_COVERED, covered: INVENTORY.length - uncoveredCount(), uncoveredAtStop: uncoveredCount() } : null,
|
||||
spentTotal: budget.spent(),
|
||||
rounds: roundStats.length,
|
||||
converged,
|
||||
stoppedOnBudget,
|
||||
stalledCoverage,
|
||||
confirmed: confirmedSorted.length,
|
||||
refuted: sum('refuted'),
|
||||
unverified: unverifiedFinal.length,
|
||||
@@ -686,9 +769,11 @@ const runStats = {
|
||||
function buildReport() {
|
||||
const outcome = converged
|
||||
? `CONVERGED after ${round} round(s).`
|
||||
: stoppedOnBudget
|
||||
? `NOT converged - stopped on budget after ${round} round(s).`
|
||||
: `NOT converged - hit the round backstop after ${round} round(s).`
|
||||
: stalledCoverage
|
||||
? `NOT converged - coverage stalled after ${round} round(s).`
|
||||
: stoppedOnBudget
|
||||
? `NOT converged - stopped on budget after ${round} round(s).`
|
||||
: `NOT converged - hit the round backstop after ${round} round(s).`
|
||||
const sev = { critical: 0, high: 0, medium: 0, low: 0 }
|
||||
for (const f of confirmedSorted) sev[f.severity] = (sev[f.severity] || 0) + 1
|
||||
const lines = ['# Bug hunt report', '']
|
||||
@@ -721,6 +806,13 @@ function buildReport() {
|
||||
`Agent failures: ${runStats.finderNull} finder null, ${runStats.finderEmpty} finder empty, ${runStats.verifierNull} verifier null.`,
|
||||
'',
|
||||
)
|
||||
if (runStats.coverage)
|
||||
lines.push(
|
||||
`Coverage: ${runStats.coverage.covered}/${runStats.coverage.inventory} files ` +
|
||||
`(${runStats.coverage.preCovered} pre-covered from ledger + live explored-clean); ` +
|
||||
`${runStats.coverage.uncoveredAtStop} uncovered at stop.`,
|
||||
'',
|
||||
)
|
||||
lines.push('| round | spent | files (new) | suppressed ledger/run | finder null/empty | verifier null |')
|
||||
lines.push('|---|---|---|---|---|---|')
|
||||
for (const s of roundStats)
|
||||
@@ -729,4 +821,4 @@ function buildReport() {
|
||||
}
|
||||
const report = buildReport()
|
||||
|
||||
return { converged, stoppedOnBudget, rounds: roundStats, confirmed: confirmedSorted, unverified: unverifiedFinal, runStats, exploredFiles: [...exploreConsumed], report }
|
||||
return { converged, stoppedOnBudget, stalledCoverage, rounds: roundStats, confirmed: confirmedSorted, unverified: unverifiedFinal, runStats, exploredFiles: [...exploreConsumed], report }
|
||||
|
||||
Reference in New Issue
Block a user