OpenAI Launches MentalHealthBench: 80 Experts, 22 Countries, Three Tiers of Acuity
OpenAI released MentalHealthBench on September 23, an open evaluation framework for measuring how AI systems respond across a structured range of mental health scenarios. The benchmark extends the clinical methodology developed for HealthBench into mental health, replacing generalist health criteria with rubrics built by more than 80 licensed psychologists and psychiatrists drawn from 22 countries, 19 languages, and approximately 20 mental health subspecialties.
Three tiers, not just emergencies
Prior mental health evaluations for AI have concentrated heavily on crisis and emergency scenarios, leaving a gap in how models perform across the more common, lower-acuity conversations. MentalHealthBench is structured around three tiers:
- Non-acute: Everyday conversations with emotional components — relationship stress, workplace difficulty, sleep anxiety
- High-acuity: Significant distress that warrants careful response but is not an immediate emergency
- Emergencies: Immediate safety concerns requiring urgent real-world support
Scenarios cover four user types: adults, teens, caregivers, and clinicians. Some scenarios include synthetic background context — a recent bereavement, for example — to test whether models use prior conversational signals to calibrate their responses appropriately.
Expert-weighted rubrics
Each conversation was reviewed by at least three experts. Rubric criteria were retained only when agreed upon by at least two experts and not contradicted by a third. Criteria carry weights ranging from -10 to +10: positive scores reward beneficial behaviors such as seeking context, preserving user agency, and providing actionable referrals; negative scores penalise harmful responses. Larger absolute weights indicate greater clinical importance within a given scenario.
Automated grading uses GPT-5.6 Sol to assess model responses against the expert-written criteria. The evaluation covers the newest available model from each major provider.
Benchmark mechanics
Synthetic conversations were generated using privacy-preserving techniques to reflect real-world usage patterns of AI for mental health. The benchmark is released openly — third parties can run their own evaluations and build on the methodology.
OpenAI notes that ChatGPT is not a substitute for therapy or professional care. The benchmark is framed as a measurement instrument for the gap between current model performance and clinically appropriate response — including the ability to surface localised crisis hotlines and trusted-contact referrals.
Results show steady improvement in model performance across successive model generations, consistent with the trend seen in HealthBench runs. Scores do not uniformly increase across acuity tiers — the paper notes a calibration trade-off between emergent and non-acute scenarios, where optimising for crisis response can affect performance in everyday conversations. Scores for 17 models were published alongside the launch:
| Model | MentalHealthBench |
|---|---|
| GPT-6 Astra | 57.3% |
| GPT-6 Sol | 53.9% |
| Claude Opus 5.5 | 52.4% |
| GPT-4o | 32.1% |
| Gemini 2.5 Pro | 29.5% |
The benchmark paper describes the grading process, evaluation settings, and full model rankings.
The release comes as frontier labs face increasing pressure to quantify model safety not just in adversarial settings but in the everyday high-sensitivity conversations that now account for a substantial share of consumer AI usage.