Microsoft Research: Frontier LLMs Corrupt 25% of Document Content in Delegated Workflows
Microsoft Research published DELEGATE-52, a benchmark testing AI system readiness for delegated knowledge work across 52 professional domains. The study ran 19 LLMs through long-horizon document editing workflows and found consistent, compounding document corruption — even from the best available frontier models.
The paper is by Philippe Laban, Tobias Schnabel, and Jennifer Neville (arXiv:2604.15597, April 2026). It hit Hacker News front page this week.
The Setup
DELEGATE-52 simulates the “vibe coding” paradigm more broadly: a knowledge worker delegates document editing tasks to an AI, which executes them over an extended workflow. The benchmark covers 310 work environments across 52 domains — from Python and Docker configs to crystallography, music notation, EDIFACT, and earning statements — each seeded from real-world documents.
The evaluation methodology uses backtranslation round-trips: for each editing task, there is a forward instruction and an inverse instruction. A perfect model returns the document to its original state after applying both. After 10 round-trips (20 interactions), the benchmark measures how much content survives — the Reconstruction Score.
This approach is reference-free and compositional, enabling evaluation without annotated ground truth.
The Numbers
After 20 delegated interactions:
| Model | RS@20 |
|---|---|
| Gemini 3.1 Pro | 80.9% |
| Claude 4.6 Opus | 73.1% |
| GPT 5.4 | 71.5% |
| Grok 4 | 59.3% |
| Kimi K2.5 | 64.1% |
| Average (all 19 models) | ~50% |
The best result is Gemini 3.1 Pro at 80.9% — meaning it corrupts or loses nearly 1 in 5 units of semantic content across a 20-interaction workflow.
Domain performance varies sharply. Python is the only domain where a majority of models achieve RS@20 ≥ 98% (the study’s “ready” threshold). The best model (Gemini 3.1 Pro) is ready in only 11 of 52 domains. Catastrophic degradation (>20% loss) occurs in 80% of all model-domain combinations.
Critical Failures, Not Death by a Thousand Cuts
The failure mode matters. Document corruption is not gradual — it is dominated by sparse, severe critical events. A critical error is defined as a single round-trip where the reconstruction score drops by 10+ points. These account for 80–98% of total degradation across all models.
Stronger models don’t avoid small errors better. They delay critical failures and experience them less frequently. Gemini 3.1 Pro averages 22 points of critical drop when a failure occurs; GPT 5.4 averages 26 points. The difference is that the better model has fewer such events.
For weaker models, corruption is primarily deletion — content simply disappears. For frontier models, the dominant failure mode is corruption: content is present but hallucinated, structurally distorted, or numerically wrong. Both are invisible without careful review.
Agentic Tool Use Makes It Worse
Adding tools (read/write files, code execution) to the agentic loop made all four tested models perform worse, not better. Average additional degradation: 6% by end of simulation. Best case: GPT 5.4 goes from 71.5% to 68.3%.
The explanation: tool use generates 2–5x more input tokens, and maintaining precision over long contexts is a known LLM weakness. Tools also don’t help much with the actual failure mode — tasks require textual understanding and domain reasoning, not just file I/O.
The only sign of upside: GPT 5.4 uses code execution (vs. file writing) in 45% of agentic edits, compared to 10% for GPT 4.1. Better models begin to leverage code for deterministic operations, which is the direction this needs to go.
Compounding Factors
Three factors multiply each other:
Document size: Each additional 1,000 tokens in the seed document increases degradation by 0.7% after two interactions — but 3.6% after 20 interactions. The effect is not linear; it compounds with time.
Interaction length: Performance continues declining through 100 interactions with no plateau in sight. The most capable model tested (GPT 5.4) drops below 60% after 50 round-trips.
Distractor context: Including irrelevant files in the workspace (simulating realistic retrieval conditions) compounds errors over time. Removing distractors improves scores by 2–8% after 20 interactions.
Short-horizon evaluations systematically underestimate all three effects.
What It Means
The study’s practical framing: delegation currently works reliably only in code, and specifically in Python. For most professional document types — legal, financial, scientific, creative — current models are not ready for autonomous long-horizon editing.
The implication for vibe coding tools: short demos show good results; long production workflows do not. A coding agent that tests well on individual commits may compound errors across a 20-step refactor.
The DELEGATE-52 benchmark is publicly released as a monitoring tool for AI readiness in knowledge work.