I Analyzed the Hidden Flaw Breaking Long-Running Agents

Description

👉 Access our AI Architects course & join hundreds of serious AI builders in our community https://www.theaiautomators.com/?utm_source=youtube&utm_medium=video&utm_campaign=tutorial&utm_content=delegate52 🔗 Microsoft Research: https://www.microsoft.com/en-us/research/publication/llms-corrupt-your-documents-when-you-delegate/ 🔗 Paper (arXiv): https://arxiv.org/abs/2604.15597 🔗 Benchmark repo (MIT-licensed): https://github.com/microsoft/DELEGATE52 🔗 Anthropic Edit tool docs: https://docs.anthropic.com/en/docs/build-with-claude/tool-use 🔗 Anthropic Word document skill (the "use Edit, do not write Python" steering): https://github.com/anthropics/skills Microsoft Research just published a paper that empirically measures something every builder of long-running document agents has felt. Frontier LLMs corrupt about 25% of document content by the end of a 20-step delegated workflow, and across all 19 tested models the average degradation is 50%. The benchmark they released — DELEGATE-52 — is MIT-licensed and reproducible. The mechanism the paper names is silent compounding corruption with cliff-edge drops. Models look competent for many turns, then take a single-turn 10–30 point loss. Frontier models preserve structure while corrupting content, so the failures are harder to detect than weaker models that delete rows outright. About 80% of total degradation comes from these rare critical failures. And the part most directly relevant to builders: adding a basic agentic harness with file read, file write, and code execution made performance an average of 6% worse, not better. In this video I walk through the benchmark, run it locally myself against a ledger in the accounting domain, and then push back on what the paper actually proves. The main experiment removes the harness. The agentic experiment uses a generic harness without a surgical edit tool. Neither tests an opinionated harness. I lay out three production principles for builders editing documents at scale: separate Edit from Write, enforce read-before-edit, and require exact string matching. The reason Claude Code works is that the harness encodes the assumptions the model can't enforce on its own — failure becomes the safety mechanism rather than the failure mode. ⏱️ Timestamps: 00:00 Delegate52 05:19 My Tests 09:12 The Agent Harness 18:00 Extra Experiments 21:52 Eight Builder Takeaways #AI #HarnessEngineering #AIAgents #DocumentEditing #Microsoft #DELEGATE52 #LLM #ContextEngineering #AIBuilder #ClaudeCode #MultiEditTool #AgentReliability #LongHorizonAgents #AIArchitects