← Back to feed

OpenAI’s alignment team published the monitorability evaluations dataset and reference code on April 23, releasing the infrastructure behind a safety methodology that has been quietly part of its frontier model system cards since 2025.

The release covers datasets and code for 12 individual evaluations — sorted into three archetypes — plus the metric formula (g-mean2) used to score them. Back-filled results for GPT-5.4 thinking, GPT-5.2 thinking, GPT-5 thinking, and o3 are now published in updated system cards.

What Monitorability Measures

The premise: if an AI model’s reasoning is visible in a chain-of-thought trace, can a monitor reliably catch when the model is behaving deceptively or failing to stay aligned? Monitorability evaluates whether that monitoring is actually possible — not whether the model is safe, but whether a human or automated monitor can detect misalignment from CoT output alone.

The three eval archetypes in the open-source release:

• Intervention evals (7 benchmarks): GPQA, GPQA encoded, AIME, DAFT, Memory, Scruples, and Sandbagging. These test whether a monitor can detect when a model is deliberately underperforming or hiding capability.
• Process evals (2 benchmarks): AIME-Math and GPQA-Physics. Assess whether the reasoning process is legible and non-deceptive step-by-step.
• Outcome-property evals (3 benchmarks): Flaky-Tools (CharXiv), ShadeArena, and Impossible Coding Tasks (LiveBench). Evaluate whether final outputs give any signal about misalignment even without full CoT inspection.

The g-mean2 Metric

OpenAI’s g-mean2 is the geometric mean of two quantities: the rate at which a monitor correctly detects true misalignment events, and the rate at which it avoids false positives on aligned behavior. Standard accuracy metrics collapse precision and recall into a single number in ways that can mask monitor failures at low base rates. g-mean2 addresses this — a monitor that calls everything misaligned scores poorly, as does one that never triggers.

The released code includes a cross-fit filtering strategy that makes intervention-style estimates more robust to noise-dominated problem instances — an update developed after the paper was published. This filtering update is retroactively applied to the back-filled system card results.

Back-Filled Results for Four Models

The updated system cards now report g-mean2 over CoT token length at different reasoning effort levels for:

• GPT-5.4 thinking
• GPT-5.2 thinking
• GPT-5 thinking
• o3 (including o1 results on the non-tool subset)

These results make it possible to track whether monitorability is improving, degrading, or staying flat as models become more capable. The pattern across the four model generations — which OpenAI has not summarized in prose — will be visible to anyone running the released evaluation suite on the published data.

The Strategic Implication

Releasing this infrastructure outside OpenAI serves a specific purpose stated explicitly in the paper: “other frontier model developers may also begin reporting the monitorability of their models.” OpenAI is positioning monitorability as a standard safety reporting metric, comparable to MMLU or SWE-Bench, but purpose-built for alignment evaluation.

If Anthropic, Google, and xAI adopt g-mean2 reporting — even voluntarily — it creates an external accountability layer for whether the reasoning traces of frontier models are legible enough to supervise. That’s a harder requirement than capability benchmarks, and one that becomes more consequential as models approach agentic autonomy.

The GitHub repo is available at github.com/openai/monitorability-evals. The scaffold is described as illustrative, not production-ready. OpenAI has explicitly stated it will not be supporting it.