Skip to content

fix(rescue): the schema asks the Stage-1 vocabulary and the legacy INSUFFICIENT_CONTEXT value is one answer everywhere (#623) - #655

Open
gadievron wants to merge 1 commit into
masterfrom
fix/623-rescue-enum
Open

gadievron wants to merge 1 commit into
masterfrom
fix/623-rescue-enum

Conversation

@gadievron

@gadievron gadievron commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Fixes #623. Follow-ups named below (the rescue schema's shape mismatch; stage1_consistency's own 3-value adjudication enum) — recorded, not filed here.

What was wrong

Two defects, one root: the vocabulary sets diverged.

  • The JSON rescue schema offered a verdict enum the Stage-1 prompt never does (VULNERABLE | SAFE | INSUFFICIENT_CONTEXT vs the prompt's safe | protected | vulnerable | inconclusive). A malformed Stage-1 reply rescued under this schema was a forced choice from a mismatched enum — the rescue could change meaning, not merely repair JSON: a protected or inconclusive reply had no legal rescue destination, and the schema offered a verdict the pipeline's own prompt never asks for. The historical receipt: one rescued row carries INSUFFICIENT_CONTEXT whose own reasoning describes the finding as "inconclusive/latent" — the forced choice, in the wild.
  • The legacy INSUFFICIENT_CONTEXT value hit four consumers that disagreed (all re-derived at current master): the analyzer summary counted it completed (Progress/summary counters use a two-branch error vs else split over three states, so incomplete work is counted as completed (5 sites) #293's adjudication); resume adopted it as a legal verdict (STAGE1_VERDICTS); the metrics fold counted it errors (Unrecognized-but-effective Stage-1 VERDICT strings still fail open on every sink (#316's verdict-key twin) #427's catch-all — the issue's letter, confirmed); the verify recount silently dropped it (no bucket matched and the elif caught only ERROR). One row, three different answers inside one report — and no display category anywhere.

The fix

Evidence

  • RED on base 6dde272 (final bytes, copies byte-identical, diff-verified): the new file 13 failed / 5 passed — 13 value-RED rows (the schema/prompt census; the counting folds; the recount folds, the both-keys flip, the garbage else; the display folds; the Stage-1 consistency corrections are shadowed by a stale finding, so a corrected VULNERABLE is never verified and never disclosed #331 finding-first pin) and 5 GUARD rows green at base by design (the rescue-runtime flows — the acceptor already admitted the full vocabulary at master, so the schema change is a prompt-text property with no runtime RED; the bypassable exclusion; the Unrecognized-but-effective Stage-1 VERDICT strings still fail open on every sink (#316's verdict-key twin) #427 negative control; the resume-adoption guard). The 13/5 split is the file's own documented receipt.
  • GREEN at head: 18/18; the adjacent battery (293 / 316-324 / 427 / 331 / 215 / 284 / 621 / the reconciliation / schema-shape files) — 135 passed. Full suite, matched conditions (the go-parser binary built in both trees): head 4304 passed / 34 skipped / 2 failed (the pre-existing test_llm_sdk_contract_floor SDK-pin pair, reproduced identically on pristine master; CI's pinned env is authoritative); base 4291 / 34 / 15 — skipped 34=34, the passed delta exactly the 13 RED rows flipping, failures 15→2.
  • SAST: ruff clean; Semgrep 0/290 on the nine diff files; CodeQL (python, 642 files) — 166 tree findings, 9 in touched files, all pre-existing or intentional (located by SARIF line vs the diff hunks; the one taxonomy hit is the pre-existing _SEVERITIES frozenset).
  • The hunt loop to exhaustion (three fable rounds + the four-model panel): r1 caught my own mis-sectioned test labels and the stale docstring bullets; the panel caught the fold's 4× hand-inlining (fable MED — fixed with the taxonomy helper), the recount's unpinned both-keys flip (DeepSeek), the contradictory scope paragraph (Kimi), my stale line-refs (Kimi); r2 caught my false receipt (the docstring still said 11/5 after the rows grew to 18 — a live count-without-command catch) and the CSV second-read pointer; r3 verified all fixes and found one trivial double-docstring (merged). GLM approve; Kimi approve-with-lows (all folded); Gemini approve; fable SHIP after r1's fold-ins.

The de-balancing check (the intent trace)

serves: I-ana (Stage-1 candidates, recall-first — a rescue forced into wrong verdicts is a potential FN at the rescue boundary; the aligned enum recovers the true verdict), I-out (the counts no longer disagree with the resume contract). de-balances-checked: the analyze fingerprint inputs unchanged (the prompt text is never rendered from the constant); every acceptor map recover-only; the #427 garbage class still errors (the negative control pinned); the consistency pass's correctable set excludes the legacy value (cannot produce it); severity_display_verdict's outcome identical for both spellings; the research harness's own insufficient_context buckets deliberately untouched; the disclosure path never saw either value (both excluded from DISCLOSURE_ELIGIBLE and the confirmed filter).

Notes for the reviewer

  • Known residuals, scoped: rows rescued under the pre-The JSON rescue schema offers a verdict enum the analysis prompt never does, and INSUFFICIENT_CONTEXT hits four consumers that disagree (errors / completed / dropped / absent) #623 enum keep their old verdicts forever (the schema is outside the fingerprint — only future rescues are aligned; the fold makes their counting consistent regardless). The rescue schema's shape mismatch (it asks a verdict key; the prompt asks finding — the analyzer's bridge derives one from the other) and stage1_consistency's own 3-value adjudication enum are named follow-ups.
  • The recount's {verdict: "ERROR", finding: "insufficient_context"} shape flips errors→inconclusive (the fold runs first, finding-first like the analyzer's counter) while resume still retries it (verdict-first, ERROR wins) — the documented transient class, pinned on both sides.
  • The CLI/HTML aggregate wiring (two one-line calls to the pinned helper) is justified as unpinned-by-full-drive: the listing read IS pinned (prepare_findings_summary), the helper is pinned, and a full generate_html_report/cmd_report_data drive was judged out of proportion for a display fold.

CI note (the transient-CI protocol, recorded not laundered): the first push's Go build + integration (windows-latest) job failed one test — test_go_cli.py::TestParse::test_parse_js_repo, a 30 s subprocess timeout with zero file overlap with this PR (no Go/parser/fixture files touched; the job's other 22 tests passed, so the build/setup path was fine). Rerun-once per the protocol: the rerun is green — 19/19 SUCCESS, windows both legs (gh run rerun 35453369039 --failedconclusion: success).

…SUFFICIENT_CONTEXT value is one answer everywhere (#623)

The JSON rescue schema offered VULNERABLE|SAFE|INSUFFICIENT_CONTEXT while the
Stage-1 prompt offers safe|protected|vulnerable|inconclusive — a rescue was a
forced choice that could change meaning. The schema's enum now renders from
the shared STAGE1_PROMPT_FINDINGS constant (the alignment target; the prompt
text is never rendered from it — the analyze fingerprint hashes prompt texts
and the schema sits outside it). The legacy INSUFFICIENT_CONTEXT value —
which the four consumers answered four ways (completed / resume-adopted /
errors / silently dropped) — folds to inconclusive's legacy synonym at every
counting and display sink through ONE function (fold_legacy_finding); rows
are never rewritten (provenance, resume adoption unchanged). The verify
recount gains the #427-mirroring terminal else and the half-stamped error
twin. The CSV stays raw by design.

Receipts (all run at commit time, on this commit):
- RED on base 6dde272 (final bytes, copies byte-identical, diff-verified):
  python3 -m pytest tests/test_issue623_rescue_enum_alignment.py -q
  -> 13 failed (value-RED), 5 passed (the labeled GUARD rows) in 0.14s
  (per-test rows: red-receipt-623.log; the 13/5 split documented in the
  file's own docstring)
- GREEN at head: the file 18/18; the adjacent battery (293/316-324/427/
  331/215/284/621 + the reconciliation + schema-shape files) 135 passed
- Full suite, matched conditions (go-parser binary built in both trees):
  head: 2 failed, 4304 passed, 34 skipped in 121.35s
  base: 15 failed, 4291 passed, 34 skipped in 126.17s
  (the 2 failures are the pre-existing SDK-pin pair; skipped 34=34; the
  passed delta is exactly the 13 RED rows flipping)
  command: python3 -m pytest tests/ -q
- ruff check . -> All checks passed!
- semgrep --config auto --error <the 9 diff files> -> 0 findings, 290 rules
- Serves: I-ana (a rescue repairs structure, never vocabulary), I-out (the
  counts no longer disagree with the resume contract). De-balancing
  checked: the analyze fingerprint inputs unchanged; the acceptor maps
  recover-only; the #427 garbage class still errors (pinned); the CSV raw
  by design; the research harness's own buckets untouched.
not its RED. (The corrector's contract is verdict-only out; the finding
derivation is the analyzer's bridge — the verdict-only counting shape is
pinned separately below.)"""
import utilities.json_corrector as jc # noqa: PLC0415


def test_rescued_protected_verdict_flows(tmp_path, monkeypatch):
import utilities.json_corrector as jc # noqa: PLC0415
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant