feat(eris): kb-whisper large on the gpu via whisper.cpp

[johnae]

kb-whisper large on the 7900 XTX through whisper.cpp's Vulkan backend, fronted by
a small wyoming bridge, replacing Qwen3-ASR on eris.

Speed. ~0.2s per command end to end through wyoming, with the model resident.
On the CPU the same model takes ~8s; faster-whisper has no AMD GPU backend and a
rocm torch is not cached anywhere, while whisper.cpp with Vulkan is.

Memory. gemma4 ~14 GiB + kb-whisper-large ~4 GiB of 24, so both stay
loaded. OLLAMA_KEEP_ALIVE=-1 keeps the LLM from unloading between questions.

Prompt. The bridge reuses wyoming-faster-whisper's HassNameCache, so Home
Assistant names are tiered, budgeted and refreshed exactly as on icarus, counted
with kb-whisper's own tokenizer. Ahead of them sits a Swedish sentence of the
foreign artist names that recur in the Spotify library (89 tokens). In testing
it turned "Miss Kitty in", "Sharon-vanetten" and "hans Zimmer" into the right
names; Swedish commands were correct with or without it.

Before this builds: eris needs the HA token rekeyed —
agenix rekey -a and commit secrets/rekeyed/eris/. It reuses icarus's
existing read-only whisper-hass-token.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XfYTVcQVjCevVeoJofu8FF

[claude]

Claude finished @johnae's task —— View job

---

Review

• Read repo conventions (DEVELOPMENT.md)
• Review packages/wyoming-whisper-cpp/ (package + bridge)
• Review profiles/whisper-cpp.nix
• Review configurations/nixos/x86_64-linux/eris.nix
• Post feedback

Nicely shaped change — the comments carry the reasoning (why Vulkan, why not turbo, why HOME, why .part+mv, why LoadCredential), the module mirrors qwen-asr.nix closely, and reusing HassNameCache instead of reimplementing the budget is the right call. Notes below, roughly by severity.

Blocking

1. secrets/rekeyed/eris/ has no whisper-hass-token — confirmed on this branch: secrets/rekeyed/eris/ contains 14 entries, none of them whisper-hass-token.age (it only exists under secrets/rekeyed/icarus/). As the PR body says, agenix rekey -a + commit is needed or the eris build fails.

Bugs / robustness

2. Prompt building sits outside the failure net — wyoming_whisper_cpp.py:112 awaits self._prompt() before the try at line 118. If icarus is down or slow, HassNameCache.initial_prompt() raises, the exception escapes handle_event, no Transcript is ever written and Home Assistant just waits out its own timeout. The comment at line 126 ("a failed transcription must not wedge the pipeline") applies equally here — the failure mode of the biasing path shouldn't be worse than the failure mode of the transcription path. Wrap _prompt() in its own try/except falling back to cli_args.initial_prompt.

3. body.get("text", "").strip() (line 131) assumes a well-formed success body — it's outside the try too, so a response that parses as JSON but carries {"error": ...} (or a null text) becomes an AttributeError with the same wedge as above. (body.get("text") or "") plus a check that body is a dict is enough.

4. The bridge needs the network at startup but isn't ordered against it — hf_hub_download runs at line 197 before AsyncServer.from_uri, and systemd.services.whisper-cpp (profiles/whisper-cpp.nix:139-143) has no after/wants on network-online.target, unlike the server unit. On a cold boot it'll crash-loop on RestartSec=5 until the network is up. Worth adding the ordering; also worth deciding what should happen when HF is unreachable but HF_HOME is warm (hf_hub_download does fall back to cache, but only after its own retries).

5. Missing token degrades silently — _read_token returning None (unset/empty file) with --hass-api set means names stays None and all HA biasing disappears with no log line saying why (line 202). Either _LOGGER.warning in that branch, or a module-level assertion that hassApi != null -> hassTokenFile != null.

6. Model cache key ignores the repo — modelPath (profiles/whisper-cpp.nix:9) is built from cfg.modelFile only. Switch services.whisper-cpp.model to, say, KBLab/kb-whisper-medium and preStart's [ ! -s ... ] finds the old ggml-model.bin, so the server keeps serving large while the bridge counts the prompt budget with medium's tokenizer — precisely the mismatch the model option's docstring promises can't happen. Put the repo in the filename, or drop a stamp file alongside the weights and re-download when it disagrees.

7. No integrity check on the 3 GB download — resolve/main is a mutable ref and the only guarantee is TLS. The .part+mv dance already handles truncation; an optional modelSha256 verified before the mv would close the rest.

8. Unbounded audio buffer — self._audio.extend(...) (line 79) has no cap and the socket listens on 0.0.0.0. Firewall-closed + trusted tailscale0 keeps this low risk, but a client that never sends AudioStop grows the bytearray until the OOM killer arrives. A max-seconds guard is a couple of lines.

Performance

9. A fresh aiohttp.ClientSession per utterance (line 119) — for a change whose headline is ~0.2s end to end, a session created once per process (or per handler) avoids connector/DNS setup on every command. Small, but free.

Quality / maintainability

10. Zero build-time verification — format = "other", dontBuild = true, no pythonImportsCheck, no installCheck. The bridge imports two modules that aren't part of wyoming-faster-whisper's advertised surface (wyoming_faster_whisper.hass_api, .name_cache) and duck-types count_prompt_tokens against HassNameCache's expectations, while nixpkgs-unstable bumps that package under you. A rename upstream is currently a runtime crash on eris, not a failed build. installCheckPhase = "$out/bin/wyoming-whisper-cpp --help >/dev/null" costs nothing (argparse exits before hf_hub_download) and catches import-level drift; asserting the HassNameCache(prefix=, max_tokens=, refresh_seconds=) signature would catch the rest. This is the one thing I'd most like to see added, given there are no tests in the diff.

11. Prompt file is read once, not per request — the qwen bridge deliberately re-read its vocabulary file (_read_vocabulary, "so it can be regenerated without a restart"); here --initial-prompt-file is consumed once in main (line 192). Fine if intentional (the file is a Nix store path now, so it can't change without a restart anyway) — maybe just say so in a comment.

12. --hass-refresh-seconds is unreachable from Nix — the flag exists (line 184) but the module never passes it, so the 300s default is hardcoded twice. Same for --timeout, --debug and whisper-server's -bs 5. Probably fine, just noting the asymmetry with how promptMaxTokens is exposed.

13. Shell quoting in preStart — [ ! -s ${modelPath} ] and -o ${modelPath}.part are bare while the URL gets escapeShellArg. No spaces in practice; quoting all three reads more consistently.

14. promptFile is realised even when initialPrompt = "" — unreferenced then, harmless, but it could live inside the optional.

15. Dead code after this PR — nothing imports profiles/qwen-asr.nix and nothing consumes packages/wyoming-qwen-asr any more; CI will keep building the package. Intentional (kept for comparison against icarus's sttLibrary = "qwen3-asr" server) or should both go? world dead may have an opinion.

Things I liked

• Token never in argv or the environment — LoadCredential + %d/hass-token matches the existing home-assistant-voice.nix pattern exactly.
• RequiresMountsFor on the /var/lib/private/... paths, carried over from the qwen profile's hard-won lesson.
• openFirewall = false by default with tailscale0 already trusted, so 0.0.0.0 isn't actually LAN-exposed (profiles/defaults.nix:141 + profiles/tailscale.nix:18).
• Downloading to .part and renaming, and TimeoutStartSec = "30min" covering the first-boot fetch.

I did not run world lint/world check/nix build — no Nix evaluation available in this job, and the eris config can't evaluate until the rekey in (1) lands anyway.
· branch eris-whisper-cpp

[claude]

Claude finished @johnae's task —— View job

---

Claude finished @johnae's task —— View job

Re-review (after synchronize)

• Read repo conventions (DEVELOPMENT.md)
• Re-read all four changed files on the updated branch
• Diff against the previously reviewed state
• Post feedback

The tree is byte-identical to what I reviewed earlier. git log still shows a single commit (00545f1) and git diff HEAD~1 HEAD --stat is the same 473/-4 across the same four files — so the synchronize looks like a rebase/force-push of identical content rather than new work. None of the previous points have been addressed yet, so they all still stand. Condensed below, plus three things I noticed on the second pass.

Blocking

1. secrets/rekeyed/eris/whisper-hass-token.age is still missing. Re-confirmed: 14 entries under secrets/rekeyed/eris/, none of them the token. configurations/nixos/x86_64-linux/eris.nix:65 declares age.secrets.whisper-hass-token.rekeyFile = ../../../secrets/icarus/whisper-hass-token.age, and profiles/whisper-cpp.nix:165 turns that into a LoadCredential, so eris won't build until agenix rekey -a + commit lands. (Reusing icarus's read-only token is the right call — agenix-rekey re-encrypts from the master identity, so the cross-host rekeyFile is fine.)

Bugs / robustness

2. Prompt building sits outside the failure net — wyoming_whisper_cpp.py:112 awaits self._prompt() before the try at line 118. If icarus is down, HassNameCache.initial_prompt() raises, the exception escapes handle_event, no Transcript is written, and HA waits out its own timeout. The comment at line 126 ("a failed transcription must not wedge the pipeline") applies at least as strongly to the biasing path. Wrap _prompt() in its own try/except falling back to cli_args.initial_prompt.

3. body.get("text", "").strip() (line 131) assumes a well-formed success body, and it too is outside the try. A JSON response carrying {"error": ...} or a null text becomes an AttributeError with the same wedge. (body.get("text") or "").strip() plus an isinstance(body, dict) check is enough.

4. The bridge needs the network at startup but isn't ordered against it — hf_hub_download (line 197) runs before AsyncServer.from_uri, yet systemd.services.whisper-cpp (profiles/whisper-cpp.nix:139-143) has no after/wants on network-online.target, unlike the server unit at :100-101. Cold boot means crash-looping on RestartSec=5 until DNS is up.

5. Missing token degrades silently — _read_token returning None with --hass-api set leaves names = None and all HA biasing gone, with no log line (line 202). A _LOGGER.warning in the else branch, or a module assertion that hassApi != null -> hassTokenFile != null, makes the failure visible.

6. Model cache key ignores the repo — modelPath (profiles/whisper-cpp.nix:9) is derived from cfg.modelFile alone. Switch model to KBLab/kb-whisper-medium and preStart's [ ! -s … ] finds the stale ggml-model.bin: the server keeps serving large while the bridge budgets with medium's tokenizer — exactly the mismatch the option's docstring (:32-35) promises can't happen. Put the repo in the filename, or write a stamp file next to the weights and re-download on disagreement.

7. No integrity check on the 3 GB download — resolve/main is a mutable ref; TLS is the only guarantee. The .part + mv already covers truncation; an optional modelSha256 verified before the mv closes the rest.

8. Unbounded audio buffer — self._audio.extend(...) (line 79) has no cap and the socket listens on 0.0.0.0. Firewall-closed plus trusted tailscale0 keeps this low risk, but a client that never sends AudioStop grows the bytearray until the OOM killer arrives. A max-seconds guard is two lines.

New on this pass

16. Transcribe.language is dropped — line 65-66 acknowledges the event and returns, so the per-request language HA sends is ignored in favour of cli_args.language (line 109). Fine while everything is Swedish, and arguably deliberate given kb-whisper is Swedish-only, but it's a silent divergence from what wyoming-faster-whisper does; one comment would settle it.

17. after=whisper-cpp-server.service only orders against process spawn, not readiness — the server unit is Type=simple with TimeoutStartSec = "30min" covering a 3 GB first-boot fetch in preStart. So on a fresh eris the bridge is up and answering while the server is still downloading; every utterance in that window hits connection-refused, returns "" (line 129) and HA says it didn't catch that. Handled gracefully, just worth knowing that first boot looks broken for several minutes.

18. The wyoming port moved 10402 → 10403 — HA's wyoming integration entry for eris is UI-managed, not in this repo, so it has to be re-pointed by hand after deploy or STT silently disappears from the pipeline. Worth a line in the PR body / commit message next to the rekey note.

Performance

9. A fresh aiohttp.ClientSession per utterance (line 119) — for a change whose headline is ~0.2s end to end, hoisting the session to process (or handler) scope avoids connector/DNS setup on every command. Small, but free.

Quality / maintainability

10. Zero build-time verification (the one I'd most like to see addressed) — format = "other", dontBuild = true, no pythonImportsCheck, no installCheck. The bridge imports wyoming_faster_whisper.hass_api and .name_cache — internal modules of an application package (toPythonModule at default.nix:25 is the tell), from plain nixos-unstable (flake.nix:70), with no pin. It also duck-types count_prompt_tokens against HassNameCache's expectations. An upstream rename is a runtime crash on eris today, not a failed build. installCheckPhase = "$out/bin/wyoming-whisper-cpp --help >/dev/null" costs nothing — argparse exits before hf_hub_download — and catches import-level drift; asserting the HassNameCache(prefix=, max_tokens=, refresh_seconds=) signature catches the rest. There are no tests in the diff, so this is the only safety net available.

11. Prompt file read once, not per request — the qwen bridge deliberately re-reads its vocabulary (wyoming_qwen_asr.py:124-134, "so it can be regenerated without a restart"); here --initial-prompt-file is consumed once in main (line 192). Defensible — it's a store path now, so it can't change without a restart — but the asymmetry with the sibling bridge deserves a comment.

12. --hass-refresh-seconds, --timeout, --debug and -bs 5 are unreachable from Nix — the flags exist (lines 184-187) but the module never passes them, so 300s is hardcoded in two places. Fine, just noting it against how promptMaxTokens is exposed.

13. Shell quoting in preStart — [ ! -s ${modelPath} ] and -o ${modelPath}.part are bare while the URL gets escapeShellArg. No spaces in practice; quoting all three reads more consistently.

14. promptFile is realised even when initialPrompt = "" — unreferenced, harmless; could move inside the optional.

15. Dead code — nothing imports profiles/qwen-asr.nix and nothing consumes packages/wyoming-qwen-asr after this; CI keeps building the package. Intentional (kept to compare against icarus's sttLibrary = "qwen3-asr" server) or should both go? world dead likely has an opinion.

Still liked

• Token never in argv or the environment — LoadCredential + %d/hass-token matches profiles/home-assistant-voice.nix:125-133 exactly.
• RequiresMountsFor on both /var/lib/private/... paths, carried over from the qwen profile's hard-won lesson.
• openFirewall = false by default with tailscale0 already trusted, so 0.0.0.0 isn't LAN-exposed.
• HOME=/var/lib/whisper-cpp for the Vulkan shader cache, and .part + mv for the download — both comments explain the why, which is the house style here.

I still can't run world lint / world dead / world check / nix build — no Nix evaluation in this job — and the eris config can't evaluate until the rekey in (1) lands regardless.

· branch eris-whisper-cpp

[claude]

Claude finished @johnae's task —— View job

---

Re-review (third pass, after synchronize)

• Read repo conventions (DEVELOPMENT.md)
• Diff branch against previously reviewed state
• Re-read all changed files
• Post feedback

What changed: exactly one thing — secrets/rekeyed/eris/5375028bef50f797c4cf8c7cfe3739e7-whisper-hass-token.age now exists (426 bytes). git diff HEAD~1 HEAD shows the other four files are byte-identical to the previous pass. So my earlier blocking point is resolved, and everything else still stands.

Resolved

1. Missing rekeyed secret — now present, and the hash differs from icarus's 1debee04…-whisper-hass-token.age exactly as it should (agenix-rekey mixes the host pubkey into the name). Reusing icarus's read-only token via a cross-host rekeyFile (eris.nix:65) is the right call — one token to issue and revoke.

Bugs / robustness (unchanged, still open)

2. Prompt building sits outside the failure net — wyoming_whisper_cpp.py:112 awaits self._prompt() before the try at :118. If icarus is down, HassNameCache.initial_prompt() raises, the exception escapes handle_event, no Transcript is written, and HA waits out its own timeout. The comment at :126 ("a failed transcription must not wedge the pipeline") applies at least as strongly to the biasing path — losing name biasing should degrade to a plain transcript, not to silence. Wrap _prompt() in its own try/except falling back to cli_args.initial_prompt.

3. body.get("text", "").strip() (:131) assumes a well-formed success body — also outside the try. A 200 carrying {"error": …} or a null text becomes an AttributeError with the same wedge. (body.get("text") or "").strip() plus an isinstance(body, dict) guard is enough.

4. The bridge needs the network at startup but isn't ordered against it — hf_hub_download (:197) runs before AsyncServer.from_uri, yet systemd.services.whisper-cpp (profiles/whisper-cpp.nix:139-143) has no after/wants on network-online.target, unlike the server unit at :100-101. Cold boot means crash-looping on RestartSec=5 until DNS is up.

5. Missing token degrades silently — _read_token returning None with --hass-api set leaves names = None and all HA biasing gone, with no log line (:202). A _LOGGER.warning in the else branch, or a module assertion that hassApi != null -> hassTokenFile != null, makes it visible.

6. Model cache key ignores the repo — modelPath (whisper-cpp.nix:9) is derived from cfg.modelFile alone. Switch model to KBLab/kb-whisper-medium and preStart's [ ! -s … ] finds the stale ggml-model.bin: the server keeps serving large while the bridge budgets with medium's tokenizer — exactly the mismatch the option's docstring (:32-35) promises can't happen. Put the repo in the filename, or write a stamp file next to the weights and re-download on disagreement.

7. No integrity check on the 3 GB download — resolve/main is a mutable ref; TLS is the only guarantee. .part + mv covers truncation; an optional modelSha256 verified before the mv closes the rest.

8. Unbounded audio buffer — self._audio.extend(...) (:79) has no cap and the socket listens on 0.0.0.0. Firewall-closed plus trusted tailscale0 keeps this low risk, but a client that never sends AudioStop grows the bytearray until the OOM killer arrives. A max-seconds guard is two lines.

9. aiohttp.ClientSession per utterance (:119) — for a change headlined at ~0.2s end to end, hoisting the session to process scope avoids connector/DNS setup per command. Small, but free.

10. Zero build-time verification (still the one I'd most like to see addressed) — format = "other", dontBuild = true, no pythonImportsCheck, no installCheck. The bridge imports wyoming_faster_whisper.hass_api and .name_cache — internal modules of an application package (toPythonModule at default.nix:25 is the tell) from unpinned nixos-unstable — and duck-types count_prompt_tokens against HassNameCache's expectations. An upstream rename is a runtime crash on eris today, not a failed build. installCheckPhase = "$out/bin/wyoming-whisper-cpp --help >/dev/null" costs nothing (argparse exits before hf_hub_download) and catches import drift; asserting the HassNameCache(prefix=, max_tokens=, refresh_seconds=) signature catches the rest. There are no tests in the diff, so it is the only safety net available.

11-15. Prompt file read once rather than per request (asymmetric with the qwen bridge's deliberate re-read, probably fine now it's a store path — worth a comment); --hass-refresh-seconds/--timeout/--debug/-bs 5 unreachable from Nix; bare ${modelPath} in preStart while the URL gets escapeShellArg; promptFile realised even when initialPrompt = ""; and the dead qwen code — see below, it got sharper.

16. Transcribe.language is dropped (:65-66) — HA's per-request language is ignored in favour of cli_args.language (:109). Defensible for a Swedish-only model, but a silent divergence from wyoming-faster-whisper; one comment settles it.

17. after=whisper-cpp-server.service orders against spawn, not readiness — Type=simple with a 3 GB fetch in preStart means that on a fresh eris the bridge answers while the server is still downloading; every utterance in that window hits connection-refused, returns "" and HA says it didn't catch that. Handled gracefully, just worth knowing first boot looks broken for several minutes.

18. The wyoming port moved 10402 → 10403 — HA's wyoming entry for eris is UI-managed, so it needs re-pointing by hand after deploy or STT silently drops out of the pipeline. Worth a line in the commit message beside the rekey note.

New on this pass

19. VRAM is now permanently committed on a machine that also runs Steam. eris.nix:212 has programs.steam.enable = true, and this PR adds OLLAMA_KEEP_ALIVE = "-1" (:79) pinning ~14 GiB while whisper-cpp-server holds ~4 GiB for the lifetime of the unit. The arithmetic in the comment is about the two services coexisting, which it does — but before this PR ollama released the card when idle, so a game got the full 24 GiB. Now it gets ~6. If eris is ever used for gaming that's a real regression hiding inside a comment that reads like pure upside; if it isn't, the comment could say so. Either way systemctl stop whisper-cpp-server becomes part of the pre-game ritual, which is the kind of thing that wants writing down.

20. The artist-prompt token count disagrees with itself. The PR body says the Swedish artist sentence is 89 tokens; eris.nix:58 says ~107, and the promptMaxTokens = 210 (of whisper's 223 hard limit) is derived from the 107 figure. One of the two measurements is stale. At 210 there are only 13 tokens of headroom, so it's worth being sure which.

21. The 210 budget assumes max_tokens is inclusive of prefix. HassNameCache(prefix=…, max_tokens=210) (:203-208) is only safe if the prefix counts against the budget rather than being prepended on top of it — otherwise the worst case is 107 + 210 = 317 and whisper truncates. The module docstring ("Kept ahead of Home Assistant's names within the token budget", :56) asserts the inclusive reading, and the upstream code presumably backs it, but nothing in this diff proves it and the consequence of being wrong is silent prompt truncation at the far end. This is another thing an installCheck against the upstream API would pin down.

22. Point 15 sharpened: the qwen path is now two PRs old and fully orphaned. profiles/qwen-asr.nix and packages/wyoming-qwen-asr/ landed in #1779 and were fixed in #1781; after this PR removes the eris import (eris.nix:-28) nothing references either. icarus's sttLibrary = "qwen3-asr" (home-assistant-voice.nix:77) is nixpkgs' wyoming-faster-whisper, not this package — so there's no consumer left at all, and CI will keep building it. world dead will almost certainly say so. Deleting both in this PR would make the story "we tried qwen, whisper.cpp on the GPU won" legible in one commit.

23. Restart = "on-failure" on the server unit — a clean exit 0 from whisper-server (it does exit cleanly on some argument and model-load paths) leaves the bridge up and every utterance returning "". Restart = "always" matches the intent better for a daemon that should never exit.

Still liked

• Token never in argv or the environment — LoadCredential + %d/hass-token matches profiles/home-assistant-voice.nix:125-133 exactly.
• RequiresMountsFor on both /var/lib/private/… paths, carried over from fix(qwen-asr): wait for the state mount, and give llama-server a HOME #1781's hard-won lesson.
• openFirewall = false by default with tailscale0 already trusted, so 0.0.0.0 isn't LAN-exposed — and the rationale moved from the host into the option description, which is where it belongs.
• HOME=/var/lib/whisper-cpp for the Vulkan shader cache, .part + mv for the download, TimeoutStartSec = "30min" for first boot — each with the why in a comment, which is the house style.

I still can't run world lint / world dead / world check / nix build — no Nix evaluation in this job.

View job · branch eris-whisper-cpp