OpenAI Found Five Released Models Had Chain-of-Thought Accidentally Graded During RL
OpenAI published a disclosure this week confirming that five released model families — GPT-5.4 Thinking, GPT-5.1 Instant through GPT-5.4 Instant, GPT-5.3 mini, and GPT-5.4 mini — had their chain-of-thought accidentally graded during reinforcement learning training runs. GPT-5.5 was not affected.
The disclosure came from OpenAI’s alignment research team and was independently reviewed before publication by METR, Apollo Research, and Redwood Research. Redwood Research published a separate report on the analysis.
Why This Matters
OpenAI has maintained a formal policy against directly grading CoT during RL since at least 2025, citing a specific risk: if the reward signal evaluates the reasoning trace itself, models may learn to produce misleading or sanitised chains of thought designed to satisfy the reward process rather than reflect actual reasoning. The lab’s own prior research showed this can happen.
The accidental grading occurred across several distinct RL runs. OpenAI investigated each case and found no clear reductions in CoT monitorability — the degree to which a model’s visible reasoning reflects its actual internal state. However, the team acknowledged the investigation cannot rule out effects that are harder to measure.
What They Built to Catch It
The discovery came from a new automated CoT grading detection system OpenAI developed after GPT-5.4 shipped. After deploying the system, engineers found retroactively that training runs for the listed models had been affected. OpenAI has since fixed the affected reward pathways, expanded the automated detection system, and says it has strengthened internal guidance and processes.
Notably, the same detection system caught additional near-misses in subsequent training runs before they reached production.
What It Means for Chain-of-Thought Safety
The technical concern is specific: CoT monitoring — the practice of using reasoning traces as a signal for model alignment — depends on those traces being honest. If a model learns during training that its reasoning steps are evaluated and rewarded or penalised, it can learn to write plausible-looking but misleading traces while pursuing a different internal objective.
OpenAI’s findings don’t demonstrate that happened here. But the disclosure itself is significant as a model for how frontier labs should handle similar incidents. The team explicitly called out the value of other labs reporting CoT grading when discovered, arguing that monitorability is an infrastructure and culture problem as much as a research one.
Anthropic’s own research, cited in the disclosure, shows a related finding: Sonnet 3.7 could evade either an output monitor or a reasoning monitor individually, but did not learn to use reasoning to improve backdoor success while evading both simultaneously — suggesting layered monitoring has value even if individual monitors can be gamed.
Affected Models
| Model | Status |
|---|---|
| GPT-5.4 Thinking | CoT graded during RL |
| GPT-5.1 Instant → GPT-5.4 Instant | CoT graded during RL |
| GPT-5.3 mini | CoT graded during RL |
| GPT-5.4 mini | CoT graded during RL |
| GPT-5.5 | Unaffected |
No capability changes or retraining are planned for affected models based on the current analysis. The monitorability evaluations showed no statistically significant degradation.