fix(pr-review): refresh GitHub state when resuming keyed reviews

[neubig]

Problem

The GitHub PR reviewer reuses its keyed conversation for an explicit re-review, but the resumed agent can repeat stale mutable GitHub state instead of refreshing it.

Live example: OpenHands/software-agent-sdk#5039 was first reviewed at 14:05 UTC. Its linked issue #5038 was then re-triaged and labeled ready-for-dev at 14:11 UTC. An explicit re-review at 14:14 UTC reused the conversation and reposted the earlier verdict, still claiming #5038 had priority:medium and no ready-for-dev label.

Conversation reuse is desirable, but a new review request must evaluate the current PR, linked issues, comments, and checks rather than treating earlier observations as current.

Acceptance criteria

• Every explicit review request refreshes mutable GitHub state before deciding a verdict: exact head, PR body, review comments, linked issue body and labels, review requests, and current-head checks.
• A same-head re-review after a linked issue label/body change observes the new issue state and does not repeat the stale finding.
• A same-head re-review after check status or review/comment changes observes those updates.
• Existing keyed conversation reuse and context retention remain intact.
• Automated tests reproduce a completed first review, mutate linked issue/check/review state, resume the same conversation, and verify the second review uses the refreshed state.
• The reusable github-pr-reviewer extension and generated automation bundle remain synchronized.

---

OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

Bounded scope: the resume path of the reusable skills/github-pr-reviewer automation and the review prompt it sends when it reuses an existing keyed conversation. AgentConversationDispatcher.deliver() (skills/github/scripts/agent_conversation.py) resumes the stable per-PR conversation and, for a new delivery, sends one fresh turn; the review prompt is built by _build_review_prompt() in skills/github-pr-reviewer/scripts/main.py and wrapped by PullRequestReviewer._prompt() in scripts/worker.py. The existing prompt already requires re-reading the PR head and inspecting current-head Actions results, but it does not require re-reading the other mutable GitHub state, so a resumed agent can report earlier observations (for example a linked issue's earlier labels or priority) as current. The fix is to make a fresh explicit review request re-establish current state before it decides a verdict, using the automation's existing mechanism (the prompt instructs the agent, which reads GitHub itself through the configured token). The dispatcher's keyed reuse and delivery idempotency stay as they are.

Non-goals: no change to head-moved stale-result suppression, the labeled/review-request trigger selection, the deterministic maintainer handoff, the native review verdict format, or the checkout lifecycle; no change to AgentConversationDispatcher behavior for other automations; no runtime, agent-server, SDK, or automation-service changes, and nothing outside this repository.

Priority: priority:high. This is a correctness defect in the live review pipeline: #648 describes a reproduced stale verdict on OpenHands/software-agent-sdk#5039, and its sibling #646 (delayed submitted events restarting completed reviews) is already priority:high. A stale verdict is externally visible, can wrongly report a linked issue as not ready for development, and undermines the automation's purpose, so it outranks the catalog-addition and skill-authoring backlog items.

Acceptance Criteria

• On every explicit review request that reuses an existing keyed conversation (a re-applied trigger label in scheduled mode, or a new review_requested event in event mode), the prompt requires the reviewer to re-fetch current mutable GitHub state before deciding its verdict: the exact head, the PR body, review comments and threads, the linked issue body and labels, current review requests, and current-head check results.
• A same-head re-review following a linked issue label or body change (for example the linked issue gains ready-for-dev after the first review) resumes the same conversation and produces a review that reflects the new issue state; it does not repeat the earlier finding that the issue was not ready or carried a different priority.
• A same-head re-review following a check-status, review, or comment change reflects that update rather than the earlier observation.
• Keyed conversation reuse and context retention are unchanged: a redelivered identical delivery is still deduplicated without sending a new turn, and a new delivery resumes the same conversation id, sends exactly one new turn, and retains the conversation's prior context (no new conversation is created for an existing subject).
• Automated tests reproduce a completed first review, then mutate linked-issue label and body, check-status, and review/comment state, resume the same conversation, and verify the second review uses the refreshed state; they pass with uv run pytest -q.
• skills/github-pr-reviewer/SKILL.md (and README.md where it describes re-review) document that an explicit re-review refreshes mutable GitHub state rather than trusting earlier observations.
• The reusable extension and the generated automation bundle remain synchronized: python scripts/sync_extensions.py --check and npm run build:automations report no drift, and if shipped bundle files change, automations/catalog/github-pr-reviewer/manifest.json setup.bundle.version is bumped with the matching template.version in tests/fixtures/automations/github-pr-reviewer.json.

[neubig]

@all-hands-bot Please implement this ready-for-dev issue. Keep the change focused on refreshing mutable GitHub state when a keyed review conversation resumes, preserve useful repository analysis, and add regression coverage for a readiness/CI change between review requests.

[all-hands-bot]

Implemented the fix, scoped to refreshing mutable GitHub state when a keyed review conversation resumes.

Root cause

Conversation reuse is by design: AgentConversationDispatcher.deliver() resumes the stable per-PR conversation and, for a new delivery, sends exactly one fresh turn. But the review prompt built by _build_review_prompt() only required the reviewer to re-read the PR head and inspect current-head Actions results. The other mutable GitHub state - PR body, review comments and threads, review requests, and, most importantly, the linked issues' bodies and labels - was never required to be re-read, so a resumed agent could report an earlier turn's observation as current (the reported case: an issue that gained ready-for-dev and changed priority between two review requests at the same head).

Change

skills/github-pr-reviewer/scripts/main.py (shared prompt, so both scheduled label mode and event mode inherit it):

• New workflow step 2, CURRENT STATE, runs before the scope gate and before any verdict: re-fetch the exact head (and whether it still matches), the current PR title and body, review comments and threads, review requests, current-head Actions results, and the current body and labels of every linked issue the PR references.
• It explicitly states that an earlier turn's analysis and the repository guidance already read (AGENTS.md and friends) remain useful background, but every mutable fact must be re-established now, and an earlier finding, verdict, or label/priority claim that the current state does not support must not be repeated. This preserves useful repository analysis instead of discarding the conversation's context.
• Remaining steps renumbered; the step-8 fallback reference updated.

skills/github-pr-reviewer/scripts/worker.py:

• The event-mode acceptance block now also requires confirming the PR's current body, labels, review threads, and head check results, and re-reading every linked issue's current body and labels, immediately before reporting.

No dispatcher change: keyed reuse, delivery dedup, and the no-new-conversation-for-an-existing-subject contract are untouched.

Documentation (skills/github-pr-reviewer/SKILL.md, README.md) now states that an explicit re-review refreshes mutable GitHub state rather than trusting earlier observations, and that repository analysis already done is retained.

Tests

tests/test_github_reviewer_delivery.py:

• test_same_head_re_review_resumes_keyed_conversation_with_refreshed_state reproduces the reported scenario with the real dispatcher (SDK/KV boundaries stubbed): a first review completes, the linked issue gains readiness and the head's checks/reviews move, then an explicit re-review at the same head. It asserts the second delivery attaches the same keyed conversation id, sends exactly one new turn, and that the resumed prompt carries the refreshed PR state plus the requirement to re-read every mutable surface. The test fails on the unpatched code.

skills/github-pr-reviewer/tests/test_main.py:

• test_prompt_requires_refreshing_mutable_github_state pins each surface, including ready-for-dev.
• test_prompt_forbids_repeating_an_earlier_observation pins the no-stale-claims wording.
• test_prompt_refreshes_state_before_the_scope_gate_and_keeps_analysis pins that the refresh precedes the scope gate while AGENTS.md guidance remains part of the workflow.

All three prompt tests fail without the fix.

Verification

• uv run pytest -q - 984 passed, 23 skipped.
• uv run pytest skills/github-pr-reviewer/tests -q - 57 passed.
• uv run python scripts/sync_extensions.py --check - clean (the pre-existing non-blocking issue-duplicate-checker coverage warning is unrelated).
• npm run build - regenerated automations/bundle-index.js and skills/index.js; no drift.
• Bundle files changed, so automations/catalog/github-pr-reviewer/manifest.json setup.bundle.version (and the entry version) bumped 1.4.0 -> 1.5.0, with the matching template.version in tests/fixtures/automations/github-pr-reviewer.json.

The changes are in the local checkout only - I have not pushed a branch or opened a PR.

Generated by OpenHands AI on behalf of the user.

[neubig]

@all-hands-bot Please publish the completed implementation: push a branch to OpenHands/extensions, open a focused PR against main that closes #648, include the verification results in the PR description, and request an appropriate human reviewer because the PR will be authored by all-hands-bot.

[all-hands-bot]

Published the completed implementation.

Pull request: #649 — fix(pr-review): refresh GitHub state when resuming keyed reviews, based on main, closes #648.

Branch: fix/648-refresh-github-state-on-resume (pushed to OpenHands/extensions).

Reviewer: @neubig was requested, since the PR is authored by all-hands-bot and needs a human reviewer.

Contents (same change discussed above, now on a branch):

• skills/github-pr-reviewer/scripts/main.py — new workflow step 2, CURRENT STATE, running before the scope gate and before any verdict: re-fetch the exact head (and whether it still matches), the current PR title and body, review comments and threads, review requests, current-head Actions results, and the current body and labels of every linked issue. Earlier analysis and already-read repository guidance stay as useful background, but every mutable fact must be re-established and a stale finding, verdict, or label/priority claim must not be repeated.
• skills/github-pr-reviewer/scripts/worker.py — event-mode acceptance block also confirms the PR's current body, labels, review threads, and head check results, and re-reads every linked issue's current body and labels, immediately before reporting.
• skills/github-pr-reviewer/SKILL.md, README.md — document that an explicit re-review refreshes mutable GitHub state.
• No dispatcher change: keyed reuse, delivery dedup, and the no-new-conversation contract are untouched.
• Regression tests reproduce a completed first review, mutate linked-issue and head check/review state, resume the same conversation, and assert the second review uses the refreshed state.

Verification re-run on the branch:

• uv run pytest -q — 984 passed, 23 skipped.
• uv run pytest skills/github-pr-reviewer/tests -q — 57 passed.
• uv run python scripts/sync_extensions.py --check — clean (the pre-existing non-blocking issue-duplicate-checker coverage warning is unrelated).
• npm run build — regenerated automations/bundle-index.js and skills/index.js, no drift.
• Bundle files changed, so automations/catalog/github-pr-reviewer/manifest.json bundle version (and the entry version) was bumped 1.4.0 -> 1.5.0, with the matching template.version in tests/fixtures/automations/github-pr-reviewer.json.

Generated by OpenHands AI on behalf of the user.