How Currai uses Jev to detect user intent and policy violations
A look inside Currai's classification pipeline: how Jev matches user intent, checks assistant responses against policy, and leaves reports and improvement proposals to a generative LLM.
A customer asks for a refund. The assistant promises the money will arrive tomorrow, even though it has no authority to guarantee that.
There are two things to capture in that exchange: what the customer wanted and what the assistant did wrong. Keeping them separate helps a team understand both demand and agent behavior.
Currai uses Jev for these semantic checks. It sends captured conversation text and the workspace's intent definitions and violation rules to TypeSafe through @typesafe-ai/sdk. Jev returns probabilities. Currai applies decision thresholds, stores the results, and connects matches to the original conversation.
Generative LLMs have a separate job: writing improvement proposals and commentary for reports.
Start with a definition the team can agree on
An intent describes a user goal or a conversational signal the team wants to track. A useful definition explains where the category starts and stops.
For example:
Refund request: The user asks to receive money back for a purchase or subscription. Questions about the refund policy alone do not count.
That distinction matters. "Please refund my subscription" and "What is your refund policy?" both contain the word refund, but they describe different requests.
Currai loads approved intents for the workspace and gives Jev each intent's name and description. The question asks whether the user-authored messages express that intent. The instructions explicitly prevent assistant wording from being used to infer the user's intent.
Teams can also track signals such as frustration. A definition might cover repeated complaints, dissatisfaction with an answer, or loss of trust after an unresolved request. Clear descriptions make these judgments easier to evaluate when someone reviews a match later.
Check assistant behavior separately
Violation rules describe behavior the assistant should avoid. They include a name, description, policy instructions, and any configured include hints or exceptions.
Consider this rule:
Never guarantee a refund: Flag unconditional promises that a refund will happen. Allow explaining eligibility or offering to submit a request for review.
The following examples illustrate the intended labels. They are examples of the rule, not recorded Jev results.
| User message | Assistant response | Intended result |
|---|---|---|
| "Please refund my subscription." | "You will definitely get your money back." | Refund intent and refund-guarantee violation |
| "Please refund my subscription." | "I can submit a request. Approval depends on eligibility." | Refund intent only |
| "What is your refund policy?" | "Requests are reviewed for eligibility." | Neither of these signals |
For violations, Jev evaluates the assistant's output. User messages can provide context, but the behavior being checked belongs to the assistant. Include phrases are supplied as hints; the semantic question also tells Jev to honor exceptions.
Currai retains separate explicit-metadata and phrase-matching paths. Jev handles the semantic classification described here. An intent match also does not perform the requested action: detecting a refund request never issues a refund.
Each definition gets its own question
A conversation can contain several signals. Someone may ask for a refund while also expressing frustration. An assistant response may break more than one rule.
Currai therefore creates an independent yes/no question for each intent and each violation rule. It batches questions that share the same conversation state, splitting requests when needed to stay within the application's size budget.
These questions use Jev's Noul type, which returns the probability that the answer is yes. Currai does not force all intents into a single category selection. TypeSafe documents the question types and their independent evaluation in its primitives guide.
The integration calls TypeSafe directly through its SDK. Classification does not pass through OpenRouter, and there is no generative LLM fallback that silently takes over when Jev is unavailable.
What a percentage means
If an intent card shows 98%, that is Jev's returned probability of the intent applying to the supplied text. It is not a claim that Currai has measured 98% accuracy across conversations.
Noul probabilities also differ from the separate confidence statistic returned for Jev's Choice and Score answers. Noul does not have that extra field. TypeSafe's confidence guide explains the distinction.
Currai's default positive threshold is 0.75. With sufficient context, it interprets a result as follows:
| Returned probability | Stored outcome |
|---|---|
| At least 0.75 | Match |
| At most 0.25 | No match |
| Between those values | Uncertain |
Intent and violation thresholds are configurable separately in the backend. These defaults are starting points. A team should evaluate them against human-labeled examples, especially where a missed violation or an incorrect accusation would be costly.
Only positive semantic matches are published into the match tables. The backend evaluation record also preserves negative, uncertain, and insufficient-evidence outcomes.
Missing context should stay visible
For every definition, Currai asks Jev a second question: is the supplied state sufficient to make the judgment? If that probability falls below 0.75, Currai records insufficient evidence, even if the match probability is high.
This matters for rules such as "Never claim a refund succeeded unless the payment tool confirmed it." The assistant's words alone cannot establish whether the payment tool succeeded. The relevant facts need to be available to the evaluator.
The current classifier works with the user messages and assistant output supplied in a captured event. It does not automatically retrieve every earlier turn or external tool result. Missing context remains a limitation, and the sufficiency check is itself a model prediction.
Oversized input is also marked insufficient rather than silently clipped and treated as complete. If a provider request fails, Currai records a failure or missing-provider status. A failed evaluation must not be interpreted as a clean conversation.
The conversation is available for review
The Signals tab shows matched intents and violations, their descriptions, probabilities, and a link to the trace. The inline Evidence section is currently hidden.
That is deliberate. Jev returns decisions; it does not produce a written rationale or choose an exact supporting quotation in this implementation. Currai stores a bounded source excerpt, but that excerpt is copied from the start of the relevant text. It may not contain the sentence that best supports the match.
Reviewers can open the trace to read the captured conversation. They should judge the classification against that source material rather than treat a percentage as proof.
Where the LLM still helps
Once signals have been collected, a generative model can help explain patterns and suggest what to change.
Currai's Auto Improve feature uses selected production events and stored matches to draft diagnoses, recommended changes, and evaluation cases. Those proposals are suggestions for review; they do not automatically change the agent.
Slack reports can also include a short LLM-written commentary. The report's counts are calculated by application code. If commentary generation fails, the factual report can still be delivered. Reviewed false-positive violations are excluded from report summaries and improvement inputs.
This gives each part of the application a specific responsibility: Jev makes bounded semantic judgments, Currai manages the records and counts, and the generative model writes analysis that a team can act on.
Putting it to work with voice conversations
For the Vapi webhook connector, Currai waits for end-of-call-report. In-call updates are acknowledged and ignored. After the completed call is accepted, Currai imports its conversation turns and schedules classification for eligible captured events.
Repeated deliveries are deduplicated. Classification runs also track their input and definition versions so repeated jobs do not publish duplicate results. The backend supports a shadow mode that stores Jev decisions without publishing Jev matches, which is useful when evaluating a configuration before enabling it.
Start with a small set of definitions your team can label consistently. Capture examples that clearly match, examples that clearly do not, and cases that depend on missing context. Then inspect the predictions and the conversations together.
A useful first pair is a refund-request intent and a rule against unconditional refund promises. They expose the distinction between user demand and assistant behavior without requiring a large policy catalog. Connect your application, then define intents and violation rules around the conversations your team needs to understand.