Deleted instance records leak permanently when the DB watcher drops the final status event under burst deletions

[KT-Doan]

Summary

When many scale-set runners are deleted in one burst, the watcher event that carries an instance's final deleted status write can be dropped. The scaleset worker removes an instance's DB row only when it receives that event, so every dropped event leaks a permanent row at status deleted. The leaked rows survive indefinitely: the delete API refuses records in that status, nothing reconciles them at runtime, and only a controller restart clears them. After a cancelled workflow batch on 2026-08-24 we found 47 such rows, 18 hours old.

This is a sibling of #854. That issue tracks records stranded in pending_delete by a dropped update inside the provider worker's instance manager. This one is a second, independent drop point in database/watcher, and it produces a different terminal state that the fixes in #855 do not cover: with those fixes applied, deletion-lane records no longer wedge autoscaling, but the leaked deleted rows still accumulate.

Observed on a production deployment built from main at afda4e76f1808e8b41d72edd0c17f99a51d84758 with the three fixes from #855 applied, the GCP provider v0.1.5, and agent mode enabled.

The chain, verified in source and controller logs

1. A workflow batch is cancelled. About 47 runners across two scale sets go through deletion within seconds of each other. The provider deletes every VM. 31 records reach runner_status=terminated; 16 die mid-bootstrap at runner_status=pending.

2. Under the burst the provider worker's update channel saturates, the same starvation Runner records stranded in pending_delete permanently wedge scale set autoscaling #854 describes. One instance from the batch, garm-itlbwyalbr2q, shows the provider delete being driven twice and then silence, with the send timeouts nearby (times UTC):

   13:47:22.196 INFO  runner was terminated
   13:47:46.098 INFO  deleting instance in provider
   13:48:43.561 INFO  deleting instance in provider
   13:48:45.168 INFO  updates channel closed
   14:05:45.233 ERROR failed to update instance | timeout while sending update to instance manager
   14:06:54.645 ERROR failed to update instance | timeout while sending update to instance manager
   

3. Each instance's final deleted write lands in the DB and the watcher fans it out. database/watcher/consumer.go gives a consumer 1 second, then silently drops the payload; the only trace is a DEBUG line:

   timer := time.NewTimer(1 * time.Second)
   ...
   case <-timer.C:
       slog.DebugContext(w.ctx, "timeout trying to send payload", "payload", payload)

4. The scaleset worker removes rows only in handleInstanceCleanup(), from exactly two triggers: the watcher update event with Status == InstanceDeleted in handleInstanceEntityEvent(), and the startup sweep in Start(). Nothing reconciles deleted rows periodically. A dropped event therefore leaks the row until the next daemon restart.

5. The leaked rows cannot be removed by hand. garm-cli runner delete -f fails, because deleted is not an accepted source state:

   [DELETE /instances/{instanceName}][400] DeleteInstance default {"error":"Bad Request","details":"runner must be in one of the following states: \"running, error, pending_force_delete, pending_delete\""}
   

6. A restart clears all of them at once through the Start() sweep. Verified live: 47 rows at deleted before the restart, 0 within 20 seconds after it, with running records untouched.

The leak is easy to spot on the metrics side: the garm_runner_status gauge for status="deleted" sits at a constant non-zero value between restarts.

Suggested fixes

Any one of these breaks the chain:

1. Reconcile periodically: purge rows already at deleted on a timer in the scaleset worker, not only in Start().
2. Let the delete API accept records at deleted and remove the row directly. No state transition is needed for a record whose provider resource and GitHub registration are both gone.
3. Treat lifecycle-terminal events as undroppable: block or retry the watcher send for deleted writes instead of dropping after 1 second.
4. At minimum, log dropped watcher payloads at WARN with the entity identity. Today the only trace of a dropped lifecycle event is a DEBUG line.

[gabriel-samfira]

Could you give the latest main branch a shot and see if this is fixed?