How to evaluate Jev AI before routing production traffic

Your Jev prototype returns valid answers. The next question is whether those answers send users to the right place, including users who describe a familiar problem in an unfamiliar way.

An evaluation should end with a deployment decision: which requests the new route can handle, which ones need review, and what evidence would cause you to roll it back. A single average score will rarely answer all three.

Label the decision you intend to automate

Suppose Jev will route workspace support tickets. Give reviewers the same conversation and account context the classifier will receive. Ask them to choose the correct queue, mark insufficient context, and identify cases with more than one legitimate route.

Resolve disagreements in the labeling policy before scoring the model. If two experienced support agents disagree because the queue definitions overlap, a precise model score can conceal an imprecise task.

Separate examples used to revise the questions from examples held back for the final evaluation. Keep whole conversations together so a follow-up message does not reveal the answer to a supposedly unseen case.

Build a test set around plausible mistakes

Include representative traffic and deliberately difficult cases. TypeSafe's Jev 1.13 limitations identify problems with literal interpretation, numerical tasks, irrelevant context, and adversarial content. Use those disclosures to choose tests, rather than assuming that every workload will fail in the same way.

For a workspace support router, useful cases include:

Test case	What to inspect
A message quoting an old, resolved complaint	Does the classifier route the current request?
A request spanning two queues	Is the chosen queue acceptable, or should it request review?
A reply with an omitted subject	Does the supplied conversation contain enough evidence?
Text telling the model to select a particular label	Does the route change despite unchanged support needs?
The same issue described in different languages	Does quality differ across the audience you serve?

Record why each case belongs in the set. That explanation will help you decide whether a future failure reflects a real regression or an outdated expectation.

Keep confidence and correctness separate

TypeSafe derives Choice and Score confidence from the answer distribution. It is a useful routing signal, but a value of 0.9 should not automatically be read as a demonstrated 90% success rate on your tickets. The confidence documentation describes what the field represents.

For each candidate confidence cutoff, calculate how much traffic would proceed automatically and how many of those decisions are wrong according to your labels. Review the errors by queue. A threshold that works well for access questions might perform poorly for ambiguous payment requests.

For a Noul question, group predictions by the returned probability and compare each group with the fraction of human-labeled yes answers. Keep enough examples in each group to avoid drawing conclusions from a handful of cases. Noul does not have a separate confidence field; its output is the yes probability. Noul reference.

Choose cutoffs using the consequences

A mistaken tag in an internal report is different from hiding a request that needs human attention. Write down what false positives and false negatives cause before optimizing either.

For example, a failed-troubleshooting detector may send extra tickets to a specialist when its threshold is low. Raising it reduces review volume but can leave more customers repeating steps that already failed. Review both groups of mistakes with the support team, then select the policy that fits the available capacity and service expectations.

Keep an explicit review path. A system that must always choose an automated handler cannot express that the available evidence is insufficient.

Evaluate the workflow after the classification

A correct route can still lead to an unhelpful response. Follow a sample of decisions through to the tool result and the customer's next message.

Track route correctness separately from task completion. Inspect cases where a tool failed, where the agent claimed success anyway, or where the customer had to restate the request. These failures need different fixes from a classifier error.

When testing a proposed replacement, run it alongside the existing route without allowing it to execute actions. That reveals disagreements on real traffic, but cannot prove what would have happened under the proposed route. A controlled rollout is still needed to measure its downstream effects.

Keep the experiment reproducible

Save the model version, question definitions, application policy, and test-set version with each result. TypeSafe warns that aliases can move to newer models; its model reference explains how to pin a version.

After a change, rerun the held-back cases and inspect new disagreements. Keep a rollback route available during rollout. Prefer a clear decision such as "enable access-ticket routing for this tested segment" over an unsupported claim that the model is production-ready for every task.

Keep learning from production with Currai

Currai gives your team a place to review real agent interactions after instrumentation. Define user intents for important workflows and violation rules for behavior you want to catch, such as claiming an operation succeeded without evidence. Each match links confidence, an explanation, and an excerpt to the conversation for review.

Use those conversations to select future test cases and refine your product rules. Set up Currai, then configure intents and violations around the decisions your Jev-powered agent actually makes.