Forgejo support

[Maks1mS]

I found various comments about Forgejo support by searching (#323, #697 (comment)), but I think it's a good idea to pin this as a separate issue.

[Maks1mS]

@gabriel-samfira, could you describe what is currently missing from Forgejo? You mentioned that Forgejo has a couple of PRs that still need to be merged.

[gabriel-samfira]

Hi @Maks1mS,

TL;DR;

Essentially, this one: https://codeberg.org/forgejo/forgejo/pulls/9803

And about 2-3 weeks of actual time to try and implement it.

TS;NM;

(I apologize for the wall of text and hold no grudge if you value your time and want to skip the following)

---

I had a brief look over the KEDA endpoint in the hopes that I might be able to use that as a source of truth for jobs, but it's hard to sync state using only that endpoint. More on that a bit later. For now some context:

The context

GARM currently expects to be able to receive - or at least consume - information of the following workflow job transitions:

• queued - a new job exists and needs to be serviced
• in_progress - job was picked up by a runner somewhere
• completed - the job has finished, regardless of outcome (success/failure/canceled)

Jobs that are in queued state are what GARM uses to decide if it needs to spin up runners and where.

Jobs that are in in_progress carry information about which runners are currently servicing a job. If we own that runner, it becomes active which in turn lets GARM know that it has one fewer idle runner. As a result, it might need to spin one up to maintain min_idle_runners if max_runners has not been reached yet.

Jobs that are in completed state are used by GARM to know when to remove a runner from the IaaS it was spun up in.

GARM currently has two modes of operating:

• Webhooks
• GitHub scale sets

Both these modes of operation offer information about the 3 job transitions mentioned above. The first one and the basis for both the initial GitHub but also Gitea support, are the webhook mode of operation.

Webhook mode

When operating in webhook mode, GARM waits for a notification from the source forge about jobs and their current state. You basically add the controller specific webhook endpoint in your repo, org or enterprise. When a workflow job transition into one of the above mentioned state, GARM is notified.

This mode of operation is not without downsides though. If the source forge has an incident or if GARM is unreachable for whatever reason, we may lose notifications and our internal state may drift from reality. To work around this, we have a thread that runs a consolidation loop for each managed entity (repo, org, enterprise). Every 5 minutes or so, we check:

• IaaS runners are still there - we do this for 2 reasons:
  • it's not uncommon for cloud admins to manually remove a VM/container and there is no way for GARM to know without an agent it checks the heartbeat of. So if the compute instance goes the way of the Dodo bird, the only indicator that it happened is that the runner is offline in the forge (if it was idle), or a completed (with the outcome of failed/canceled) hook is sent by the forge, if the runner was active.
  • We were offline while the runner we spun up serviced a job to completion and was removed by the forge, orphaning the compute instance in the IaaS. And because we were offline, we missed the webhook that was supposed to let us know of the event.
• The runners are still there in the forge - we need to account for periods in which GARM was offline or periods in which the forge is unable to send webhooks (maintenance, an incident, etc).

Any drift is consolidated once every 5 minutes. If compute instances are removed, we clean up forge runners. If forge runners are removed but compute resources linger, we remove those. All resources are tagged using the GARM controller ID and pool/scaleset ID.

We hope for the best but account for the worst. This is one of the major downsides of webhooks in general. It's not the only downside (out of order webhooks are another, for example), but it's the only one relevant to this discussion.

Scale sets

The second mode of operation is github scale sets. This one is github specific. It essentially implements most of what we had to implement in GARM via pools to schedule the creation of runners for an entity, but the scheduling is done server side. In this mode of operation, an autoscaler such as GARM or ARC, will subscribe to a message queue via HTTP longpoll. Github holds an internal state of job notifications it delivered. After we subscribe to the queue and start listening on the endpoint, we get a message every time something relevant happens (job transitions to one of the above states). When a message is emitted by github, it comes with a message ID. On the next iteration over the blocking HTTP longpoll, you supply that message ID. That way, github will give you a delta of every actionable (from the perspective of the autoscaler) event that was emitted from the moment of that last message. These events are de-duplicated.

For example, if you were online for a while and received an in_progress event for a runner you manage, but were taken offline before a completed event was emitted, you supply the lastMessageID and you will be notified that the job was completed. You can then remove the runner. However, if several new jobs were started and completed/canceled while you were offline, those notifications will be skipped, as they are no longer actionable by the autoscaler. So the heavy lifting is done by GitHub in this case. There is less need for the consolidation loop in this mode of operation. We still do it in GARM because we're paranoid and cloud costs is not something that should soar due to an assumption that things will just work, but in theory it should happen less with a mechanism lie scale sets. Scale sets are also more reliable than webhooks and there is no out-of-order events emitted.

The KEDA attempt

Now that we got that bit of (probably boring) background out of the way, we return to the KEDA. The current implementation in Forgejo has the following characteristics:

• will only give you information about jobs that are waiting (the equivalent of queued) or running (the equivalent of in_progress).
• It will not show you information about completed jobs.
• it's non-blocking. This means we must poll the endpoint, potentially becoming bad netizens in the process. I had a look at the runner code, and it seems to do the same, with an exception for Codeberg in terms of how often it polls (5 seconds for self-hosted instances, about 30 or 60 seconds for Codeberg - I can't remember exactly).
• the run_id was missing from the job payload when I last looked. This is easy to add if we really need it. Without it it's hard to correlate a running job with what you see in the UI. But not without a round trip through the API.

Besides the above mentioned, there doesn't seem to be a way to get a job by ID in Forgejo. This is not a huge blocker, but it does mean that we need to list all jobs and iterate over them to find one we care about. This may have changed since the last time I checked.

Now, I did add an "agent mode" to GARM which could be used to help with the completed update, by having the agent phone home to GARM when it notices the runner is busy. But it can only tell GARM that a job is running, not which job. The forgejo runner does not expose that info. At least not in a way we can easily identify it by what we get from the KEDA jobs endpoint. So our state is somewhat in sync, but only partially. Also, leaning on the agent mode thing would mean that all forgejo endpoint would be obliged to use the agent and would not be able to turn it off (by default, GARM is agent-less).

All of that being said, if the PR mentioned at the very top of this absurd wall of text lands, then we get what we need to get the full list of job status transitions to make state work in the most complete, less hacky way possible. The only wrinkle will be the get job by ID API endpoint, which we use to periodically consolidate, bu I suspect that would be a simple addition I can propose upstream as I implement Forgejo support in GARM. The bulk is in that last PR mentioned.