Feature Type
Nice to have
Feature Description
TypeSafe released Jev on Sep 15 (https://docs.typesafe.ai/introduction). It's a decision model, not a chat model: you send state plus typed questions and get back a Choice (option + probability distribution + confidence), a Score (rating on ordered levels), or a Noul (0-1 yes/no). All questions in a call are evaluated in parallel, so asking four costs about the same as asking one. No text generation.
I want to use it in voice agents for the decisions that happen on every user turn and shouldn't need the main LLM: is the caller asking to stop being contacted, which specialist agent should take this, is the caller frustrated enough to escalate. Today I'd do that with an extra LLM call in on_user_turn_completed, which adds most of a second before the real reply starts.
Proposed shape, same layout as the other plugins:
from livekit.plugins import typesafe
decider = typesafe.SystemOne() # TYPESAFE_API_KEY
class Router(Agent):
async def on_user_turn_completed(self, turn_ctx, new_message):
answers = await decider.evaluate(
state=typesafe.build_turn_state(turn_ctx, last_n=3),
questions={
"opt_out": typesafe.Noul("Does the caller ask to stop being contacted?"),
"intent": typesafe.Choice("What is the caller doing?", criteria={...}),
"frustration": typesafe.Score("How frustrated is the caller?", criteria=[...]),
},
)
if answers.noul("opt_out") > 0.9:
... # say goodbye, raise StopResponse, end the session
if answers.choice("intent").confidence > 0.85:
self.session.update_agent(SPECIALISTS[answers.choice("intent").choice])
The plugin itself is small: a client over the one HTTP endpoint using the shared aiohttp session and APIConnectOptions (429/529 retryable, 401/422 not), typed question/answer dataclasses, a helper that builds a compact state from ChatContext (Jev's docs are explicit that irrelevant state hurts accuracy, and transcripts are PII so nothing goes in logs), and unit tests with the HTTP layer mocked. Plus one example under examples/voice_agents showing the three decisions above.
It doesn't implement llm.LLM. Jev can't generate text, and forcing it into that interface would be misleading. I'd classify it with the "specialized services" plugins like hamming and nltk.
Measured from : p50 ms, p95 ms for a 4-question call. <- fill after Phase 0
Prototype: https://github.com//livekit-plugins-typesafe <- fill after step 2
Two questions before I open the PR:
- Are you fine with a plugin that isn't an STT/TTS/LLM, or would you rather this live as an example only?
- Jev is in early access (waitlist). Does that matter for merging, or is it fine as long as the README says so?
I'm an independent contributor, not affiliated with TypeSafe. Happy to maintain it.
Workarounds / Alternatives
An extra LLM call in on_user_turn_completed with a JSON-mode prompt. Works, but it's 500-1500 ms on top of the reply and there's no calibrated confidence to threshold on. A local classifier (fine-tuned small model) is the other option; more setup than most agent teams will do for this.
Additional Context
Not asking for anything in livekit-agents core. A possible follow-up would be letting the AMD classifier in voice/amd take a non-LLM backend, since greeting classification is exactly this kind of decision, but that's a separate discussion.
Feature Type
Nice to have
Feature Description
TypeSafe released Jev on Sep 15 (https://docs.typesafe.ai/introduction). It's a decision model, not a chat model: you send state plus typed questions and get back a Choice (option + probability distribution + confidence), a Score (rating on ordered levels), or a Noul (0-1 yes/no). All questions in a call are evaluated in parallel, so asking four costs about the same as asking one. No text generation.
I want to use it in voice agents for the decisions that happen on every user turn and shouldn't need the main LLM: is the caller asking to stop being contacted, which specialist agent should take this, is the caller frustrated enough to escalate. Today I'd do that with an extra LLM call in on_user_turn_completed, which adds most of a second before the real reply starts.
Proposed shape, same layout as the other plugins:
The plugin itself is small: a client over the one HTTP endpoint using the shared aiohttp session and APIConnectOptions (429/529 retryable, 401/422 not), typed question/answer dataclasses, a helper that builds a compact state from ChatContext (Jev's docs are explicit that irrelevant state hurts accuracy, and transcripts are PII so nothing goes in logs), and unit tests with the HTTP layer mocked. Plus one example under examples/voice_agents showing the three decisions above.
It doesn't implement llm.LLM. Jev can't generate text, and forcing it into that interface would be misleading. I'd classify it with the "specialized services" plugins like hamming and nltk.
Measured from : p50 ms, p95 ms for a 4-question call. <- fill after Phase 0
Prototype: https://github.com//livekit-plugins-typesafe <- fill after step 2
Two questions before I open the PR:
I'm an independent contributor, not affiliated with TypeSafe. Happy to maintain it.
Workarounds / Alternatives
An extra LLM call in on_user_turn_completed with a JSON-mode prompt. Works, but it's 500-1500 ms on top of the reply and there's no calibrated confidence to threshold on. A local classifier (fine-tuned small model) is the other option; more setup than most agent teams will do for this.
Additional Context
Not asking for anything in livekit-agents core. A possible follow-up would be letting the AMD classifier in voice/amd take a non-LLM backend, since greeting classification is exactly this kind of decision, but that's a separate discussion.