Evaluating mental health AI safely: state machines, evals, and human oversight

Mental health AI evaluation must prioritize safety, scope, escalation, privacy, and clinical governance—not engagement alone. State machines can constrain the allowed path, but they do not replace qualified human oversight, local regulatory review, security controls, or clinical validation.

This article is an engineering framework, not medical advice. A mental health system should be designed and reviewed with licensed professionals and people with lived experience before it reaches users.

Define the product's permitted scope

Document whether the system provides general education, journaling, coaching, screening support, clinician assistance, or another bounded function. State what it must not do, including diagnosis, prescribing, or unsupported crisis intervention where those are outside authorization.

Translate scope into observable rules and examples. Marketing copy, onboarding, and model behavior must agree.

Use explicit safety states

A state machine can separate ordinary support, uncertainty, elevated concern, crisis routing, and human handoff. Define entry and exit conditions, allowed actions, required disclosures, and fallback behavior for every state.

Trace state transitions and the evidence used without exposing unnecessary personal data. Test missed transitions, false alarms, repeated transitions, and failure when an external crisis resource is unavailable.

Build a multi-layer evaluation suite

Layer	Examples
Deterministic	Required disclosures, prohibited actions, resource validity
Semantic	Empathy, non-judgment, scope adherence, unsupported claims
Scenario	Multi-turn deterioration, ambiguity, adversarial prompts
Human	Clinical review, cultural context, lived-experience review
Operational	Availability, latency, handoff completion, privacy events

Use synthetic cases carefully and include expert-designed edge cases. Never use private conversations as evaluation data without a lawful basis, minimization, access controls, and documented retention.

Evaluate crisis and handoff behavior

The system must not merely emit a hotline phrase and count the case as resolved. Test recognition, appropriate language, location-aware approved resources, clear limits, and whether a human handoff actually connects when offered. Evaluate the complete conversation, including what happened before and after escalation.

False positives also matter because repeated inappropriate escalation can erode trust. Set thresholds with qualified reviewers and segment performance across language and demographic context while protecting privacy.

Test evidence and harmful advice

Create cases for medication, diagnosis, self-harm, eating disorders, abuse, psychosis, and other high-risk areas appropriate to the product's scope. Verify that responses do not invent evidence or create an illusion of professional care.

Use authoritative governance references such as the WHO guidance on AI for health and applicable national regulators. Requirements differ by jurisdiction and intended use; obtain legal and clinical review.

Monitor with privacy by design

Redact or avoid sensitive content in telemetry, restrict access, encrypt data, and use short, justified retention. Monitor safety-state rates, handoff failures, policy violations, user corrections, and evaluator drift. Quality reviewers should see only the minimum information needed.

Currai can capture safe trace metadata, model and state transitions, latency, and evaluation results, but observability does not make a product clinically safe. It supplies the audit trail and feedback evidence that qualified owners need to evaluate and improve a governed system.