Conversation
…SUFFICIENT_CONTEXT value is one answer everywhere (#623) The JSON rescue schema offered VULNERABLE|SAFE|INSUFFICIENT_CONTEXT while the Stage-1 prompt offers safe|protected|vulnerable|inconclusive — a rescue was a forced choice that could change meaning. The schema's enum now renders from the shared STAGE1_PROMPT_FINDINGS constant (the alignment target; the prompt text is never rendered from it — the analyze fingerprint hashes prompt texts and the schema sits outside it). The legacy INSUFFICIENT_CONTEXT value — which the four consumers answered four ways (completed / resume-adopted / errors / silently dropped) — folds to inconclusive's legacy synonym at every counting and display sink through ONE function (fold_legacy_finding); rows are never rewritten (provenance, resume adoption unchanged). The verify recount gains the #427-mirroring terminal else and the half-stamped error twin. The CSV stays raw by design. Receipts (all run at commit time, on this commit): - RED on base 6dde272 (final bytes, copies byte-identical, diff-verified): python3 -m pytest tests/test_issue623_rescue_enum_alignment.py -q -> 13 failed (value-RED), 5 passed (the labeled GUARD rows) in 0.14s (per-test rows: red-receipt-623.log; the 13/5 split documented in the file's own docstring) - GREEN at head: the file 18/18; the adjacent battery (293/316-324/427/ 331/215/284/621 + the reconciliation + schema-shape files) 135 passed - Full suite, matched conditions (go-parser binary built in both trees): head: 2 failed, 4304 passed, 34 skipped in 121.35s base: 15 failed, 4291 passed, 34 skipped in 126.17s (the 2 failures are the pre-existing SDK-pin pair; skipped 34=34; the passed delta is exactly the 13 RED rows flipping) command: python3 -m pytest tests/ -q - ruff check . -> All checks passed! - semgrep --config auto --error <the 9 diff files> -> 0 findings, 290 rules - Serves: I-ana (a rescue repairs structure, never vocabulary), I-out (the counts no longer disagree with the resume contract). De-balancing checked: the analyze fingerprint inputs unchanged; the acceptor maps recover-only; the #427 garbage class still errors (pinned); the CSV raw by design; the research harness's own buckets untouched.
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
September 19, 2026 15:56
| not its RED. (The corrector's contract is verdict-only out; the finding | ||
| derivation is the analyzer's bridge — the verdict-only counting shape is | ||
| pinned separately below.)""" | ||
| import utilities.json_corrector as jc # noqa: PLC0415 |
|
|
||
|
|
||
| def test_rescued_protected_verdict_flows(tmp_path, monkeypatch): | ||
| import utilities.json_corrector as jc # noqa: PLC0415 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #623. Follow-ups named below (the rescue schema's shape mismatch;
stage1_consistency's own 3-value adjudication enum) — recorded, not filed here.What was wrong
Two defects, one root: the vocabulary sets diverged.
VULNERABLE | SAFE | INSUFFICIENT_CONTEXTvs the prompt'ssafe | protected | vulnerable | inconclusive). A malformed Stage-1 reply rescued under this schema was a forced choice from a mismatched enum — the rescue could change meaning, not merely repair JSON: a protected or inconclusive reply had no legal rescue destination, and the schema offered a verdict the pipeline's own prompt never asks for. The historical receipt: one rescued row carriesINSUFFICIENT_CONTEXTwhose own reasoning describes the finding as "inconclusive/latent" — the forced choice, in the wild.INSUFFICIENT_CONTEXTvalue hit four consumers that disagreed (all re-derived at current master): the analyzer summary counted it completed (Progress/summary counters use a two-brancherrorvselsesplit over three states, so incomplete work is counted as completed (5 sites) #293's adjudication); resume adopted it as a legal verdict (STAGE1_VERDICTS); the metrics fold counted it errors (Unrecognized-but-effective Stage-1 VERDICT strings still fail open on every sink (#316's verdict-key twin) #427's catch-all — the issue's letter, confirmed); the verify recount silently dropped it (no bucket matched and the elif caught onlyERROR). One row, three different answers inside one report — and no display category anywhere.The fix
STAGE1_PROMPT_FINDINGSincore/verdict_taxonomy.py— the exact set the analysis prompt offers. The rescue schema's enum renders from it (utilities/json_corrector.py), so a rescue repairs structure, never vocabulary. The legacy value is no longer asked for but stays accepted downstream (every mapping is recover-only). The constant is the alignment target, never the prompt's text source — the analyze fingerprint hashes prompt texts (_analyze_fingerprint), and the schema sits outside it; a static census test pins both directions (the constant == the prompt's literal set; the schema == the constant uppercased;BYPASSABLEexcluded with the documented reason — it originates in the Stage-2 finish enum; offering it would let a rescue upgrade a reply into a verify-only verdict).fold_legacy_finding()(withLEGACY_FINDING_SYNONYMS) in the taxonomy — the pre-The JSON rescue schema offers a verdict enum the analysis prompt never does, and INSUFFICIENT_CONTEXT hits four consumers that disagree (errors / completed / dropped / absent) #623 value folds toinconclusive's legacy synonym at every counting and display sink: the metrics fold (_count_verdicts), the verify recount, the Stage-2 outcome counter's legacy arm, and the CLI/HTML display reads. The row's own stored values are never rewritten (provenance preserved; resume adoption unchanged; the Progress/summary counters use a two-brancherrorvselsesplit over three states, so incomplete work is counted as completed (5 sites) #293 completed-counting unchanged). Four hand-inlined copies of the fold was the drift shape the taxonomy module's own docstring warns against — hence the helper.elsemirrors_count_verdicts' Unrecognized-but-effective Stage-1 VERDICT strings still fail open on every sink (#316's verdict-key twin) #427 catch-all (an unrecognized non-empty finding is an error in the metrics, never a silent drop), and the half-stampedfinding == "error"twin (handled on the analyzer side since JSON corrector synthesises a verdict from free text and reports it as a successful correction, producing rows no bucket counts #316/A Stage-1 result with neitherverdictnorfindingis counted in no bucket and adopted as complete on resume, sounits_analyzedoverstates and the unit is never retried #324) joins it. The#284partition now sums for every shape.get_stage1_verdict's docstring with pointers at both raw reads); nothing programmatic recomputes buckets from it.Evidence
finding, so a corrected VULNERABLE is never verified and never disclosed #331 finding-first pin) and 5 GUARD rows green at base by design (the rescue-runtime flows — the acceptor already admitted the full vocabulary at master, so the schema change is a prompt-text property with no runtime RED; the bypassable exclusion; the Unrecognized-but-effective Stage-1 VERDICT strings still fail open on every sink (#316's verdict-key twin) #427 negative control; the resume-adoption guard). The 13/5 split is the file's own documented receipt.test_llm_sdk_contract_floorSDK-pin pair, reproduced identically on pristine master; CI's pinned env is authoritative); base 4291 / 34 / 15 — skipped 34=34, the passed delta exactly the 13 RED rows flipping, failures 15→2._SEVERITIESfrozenset).The de-balancing check (the intent trace)
serves: I-ana(Stage-1 candidates, recall-first — a rescue forced into wrong verdicts is a potential FN at the rescue boundary; the aligned enum recovers the true verdict),I-out(the counts no longer disagree with the resume contract).de-balances-checked: the analyze fingerprint inputs unchanged (the prompt text is never rendered from the constant); every acceptor map recover-only; the #427 garbage class still errors (the negative control pinned); the consistency pass's correctable set excludes the legacy value (cannot produce it);severity_display_verdict's outcome identical for both spellings; the research harness's owninsufficient_contextbuckets deliberately untouched; the disclosure path never saw either value (both excluded fromDISCLOSURE_ELIGIBLEand the confirmed filter).Notes for the reviewer
verdictkey; the prompt asksfinding— the analyzer's bridge derives one from the other) andstage1_consistency's own 3-value adjudication enum are named follow-ups.{verdict: "ERROR", finding: "insufficient_context"}shape flips errors→inconclusive (the fold runs first, finding-first like the analyzer's counter) while resume still retries it (verdict-first, ERROR wins) — the documented transient class, pinned on both sides.prepare_findings_summary), the helper is pinned, and a fullgenerate_html_report/cmd_report_datadrive was judged out of proportion for a display fold.CI note (the transient-CI protocol, recorded not laundered): the first push's
Go build + integration (windows-latest)job failed one test —test_go_cli.py::TestParse::test_parse_js_repo, a 30 s subprocess timeout with zero file overlap with this PR (no Go/parser/fixture files touched; the job's other 22 tests passed, so the build/setup path was fine). Rerun-once per the protocol: the rerun is green — 19/19 SUCCESS, windows both legs (gh run rerun 35453369039 --failed→conclusion: success).