Skip to content

ci: open drift issues on job-level sensor failures - #170

Open
fullstackjam wants to merge 1 commit into
mainfrom
ci/drift-to-issue-job-failures
Open

fullstackjam wants to merge 1 commit into
mainfrom
ci/drift-to-issue-job-failures

Conversation

@fullstackjam

Copy link
Copy Markdown
Member

What does this PR do?

Makes drift-to-issue.yml open or update tracking issues when individual Harness drift jobs fail. Before this, it only acted when the whole Harness run failed, which never happens.

Why?

The sensor-to-issue loop has never fired:

  1. Every Harness job sets continue-on-error: true, so the Harness run concludes success even when a sensor fails.
  2. drift-to-issue.yml gated its only step on github.event.workflow_run.conclusion == 'failure', so the step was always skipped. It did nothing, and its runs still showed success.
  3. As a result, the repo has zero harness-drift issues, even though govulncheck (drift) has failed on every recent nightly run.

Evidence from the Actions API (scheduled runs on main):

Harness run run conclusion govulncheck (drift) job conclusion
35860790347 (09-23) success failure
35726222588 (09-22) success failure
35606250669 (09-21) success failure
35509243702 (09-20) success failure

I got these values with gh api repos/openbootdotdev/openboot/actions/runs/<id>/jobs. Under continue-on-error, the failed job reports conclusion: failure in the jobs API. Only the run-level conclusion is masked. The script already fetched that jobs JSON and filtered on job["conclusion"] != "failure", so the only thing needed was to stop skipping it.

Changes

  • Drop the run-level conclusion == 'failure' gate. The Python loop's per-job filter is now the real gate. Jobs with any other conclusion (success, cancelled, skipped) are ignored, so a run with no failed jobs is a no-op.
  • Exclude PR runs explicitly, with a job-level if: github.event.workflow_run.event != 'pull_request'. The branches: [main, master] filter matches the triggering run's head_branch, and a fork PR from <fork>:main also has head_branch == main. So the branch filter alone doesn't keep PRs out. push, schedule and workflow_dispatch runs on main still go through. This is the permissive form: if Harness ever adds a pull_request_target trigger, those runs would get through too, so that change would need this gate updated.
  • Create the harness-drift label before opening an issue. This is a second root cause: the label doesn't exist in the repo, and gh issue create --label harness-drift fails on a missing label. In cli/cli, LabelsToIDs returns 'harness-drift' not found, which issue create wraps as could not add label: …. Dropping the gate alone would have turned a silent no-op into a failing step, and still no issue. The fix is gh label create harness-drift --force, an idempotent create-or-update that only runs when a new issue is being opened. issues: write already covers labels.

One issue per sensor is unchanged. It's still the [harness-drift] <job name> title, looked up among open issues with the exact-title check, and a repeat failure adds a comment instead of opening a new issue.

Testing

  • Job-conclusion values: queried the jobs API for the four runs above (table). Checked the run object fields the new gate reads:
    • {"event":"schedule","head_branch":"main","conclusion":"success"} for run 35860790347
    • {"event":"pull_request","head_branch":"claude/fix-review-bot-allowlist","conclusion":"success"} for PR run 30735002314
  • Offline run of the actual step script: pulled the run: block out of the workflow YAML and ran it with bash -eo pipefail. It used the real jobs JSON from run 35860790347 and a stub gh on PATH that logs its arguments:
    • A. No existing issue: gh label create harness-drift --force … then gh issue create --title "[harness-drift] govulncheck (drift)" …. That's exactly one issue, and nothing for the four passing sensors.
    • B. Stubbed existing open issue refactor(progress): replace byte-tracking bar with spinner + elapsed #99: only gh issue comment 99 --body "Sensor fired again. Run: …".
    • C. Same payload with all jobs set to success: only the gh api …/jobs call, no issue calls.
  • go run github.com/rhysd/actionlint/cmd/actionlint@latest (v1.7.12): exit 0 for drift-to-issue.yml and for all workflows. actionlint doesn't type-check github.event.*, so the workflow_run.event field is backed by the API output above, not by actionlint.
  • git merge-tree against ci/bump-actions (ci: bump GitHub Actions off the deprecated Node 20 runtime #167, checkout v4→v7 in this file): merges cleanly. This diff leaves the actions/checkout line alone, so the two PRs can land in either order.
  • N/A: go vet, Go tests, and openboot install --dry-run don't apply; this is a workflow-only change.

Not verified live:

  • The update path (scenario B) was only tested against a stub. It will first run for real on the second failure after the first issue exists.
  • The PR-exclusion if: is backed by the API field values, not by an end-to-end PR-triggered run.

workflow_run workflows always run from the default branch's copy of the file, so none of this takes effect until merge.

Cross-repo checklist

  • Does this need a docs/content update in openboot.dev? No.
  • Does this change the CLI ↔ server API contract? No.

Notes for reviewer

  • Expected right after merge: the Harness run for the merge commit on main should open [harness-drift] govulncheck (drift) and create the harness-drift label. If chore: bump Go toolchain to 1.26.8 #166 (Go 1.26.8) lands first and clears govulncheck, no issue opens. To see it sooner, run Harness via workflow_dispatch on main.
  • Comment volume: each later push to main plus each nightly run adds a comment to the open issue while the sensor keeps failing. This matches the existing "repeat failures update this same issue" design; I didn't change it.
  • Pre-existing, not changed here: docs/MERGE_POLICY.md:36 lists four drift sensors and leaves out required-checks alignment.

Every Harness job sets continue-on-error: true, so the Harness run
concludes success even when a sensor fails. drift-to-issue gated its
step on workflow_run.conclusion == 'failure', so it never ran and no
drift issue was ever opened. Drop that gate; the script already filters
on per-job conclusions from the jobs API, where a failed sensor reports
failure.

Exclude PR-triggered Harness runs with an explicit event check, since
the branches filter also matches a fork PR whose head branch is main.

Create the harness-drift label before opening an issue: it does not
exist in the repo, and gh issue create --label fails on a missing label.
@github-actions github-actions Bot added the ci CI/CD changes label Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI/CD changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant