← Back to feed

The standard way to fix a broken production AI system is slow and manual: observe the failure, inspect edge cases, adjust the prompt, deploy, repeat. At Thrive Holdings, OpenAI’s forward-deployed engineering team built a system that does the debugging itself.

The result is Tax AI, deployed across Crete, a network of 30+ accounting firms that processes tens of thousands of tax returns each season. Published Tuesday in an OpenAI engineering post, the case study is one of the first documented production deployments of an agentic self-improvement loop applied to a high-stakes professional workflow.

The Numbers

Tax AI handles 1040 and 1041 returns. For medium-to-large filings, manual data entry alone typically takes eight hours per return — gathering and reconciling prior-year documents, messy source files, client notes.

From the first full deployment season:

• 7,000 returns processed across the participating Crete firms
• Up to 97% accuracy in correct field completion
• ~33% reduction in prep time per return
• ~50% increase in throughput, creating more capacity for client-facing work

The more revealing figure is where the system started. At launch, only 25% of returns reached 75% correct field completion. Within six weeks, the system was measurably better — not because engineers ran a sprint, but because the improvement loop ran itself.

The Architecture

The self-improvement mechanism has three layers:

Production traces. Every return processed by Tax AI generates a structured record of the full input-to-output path: source files, client notes, intermediate reasoning, and final field completion. Failures are captured in context, not as summary logs.

Practitioner feedback. Accountants at Crete firms submit corrections and flag specific errors. That feedback is typed and attached to the relevant production trace — not free-form commentary, but structured improvement signals tied to specific return failures.

Codex-driven iteration. Given the traces and practitioner feedback, a Codex agent identifies failure patterns, writes new evals targeting those patterns, proposes fixes to the system’s logic or prompts, and routes changes through test infrastructure before production deployment. Codex doesn’t just help write the agent — it runs the improvement loop that makes the agent better.

The result is a system where the engineer’s role shifts from writing fixes to designing the infrastructure that surfaces them. Codex handles the rest.

What This Demonstrates

This is distinct from fine-tuning on user corrections or retrieval-augmented generation. Tax AI doesn’t learn by updating weights. It learns by generating better evaluation logic, identifying the precise failure modes that evals missed, and deploying targeted fixes through an automated pipeline.

The architecture — production traces, structured practitioner feedback, Codex iteration — is domain-agnostic. Tax preparation was chosen because the feedback loop is fast (returns are either right or wrong), practitioners are available, and the cost of errors is concrete. The same pattern applies anywhere an AI system operates in a feedback-rich environment: legal document review, medical coding, financial reconciliation.

Crete’s accountants gained roughly a third of their time back on data entry this season. The system that delivered that gain is also materially better today than the one that was first deployed three months ago — and it got there without anyone having to find each bug.