Claude, GPT-5, Gemini 3: Only 1 of 52 Jobs They Can Actually Do

Description

Microsoft new benchmark shows how frontier LLMs corrupt 25% of document content on long-horizon delegated work. For links to the research, check out the full video here - https://www.youtube.com/watch?v=T8EW3mD626E Their headline takeaway is that LLMs are unreliable when it comes to delegating document editing tasks and that adding tools doesn't help. However from my deep dive into it, parts of this just don't add up and there are some important lessons for those building custom AI systems. The paper introduces Delegate52, a serious new benchmark simulating real document editing across 52 professional domains … Ledgers, schemas, slides, contracts. Forward edits and backward edits across 20 turns where the final document should match the seed exactly. None of the frontier models managed it. Even Gemini 3.1 Pro hits a catastrophic single-round failure in 86% of runs. The part that doesn't add up is the agentic experiment. They gave four models a basic harness (read, write, delete, run Python) and the models performed 6% worse than without tools. Their conclusion was that tools do not help. They admitted that adding tools should improve things. If giving the model tools made it worse, that is a signal about the harness, not about tool use as a concept. What they have actually proved, perhaps inadvertently, is that a generic basic agent harness is no better than an agent with no tools at all. Which is the real lesson here. Production AI systems on documents need three things: 1. An opinionated system prompt that steers the model toward surgical edits, not full rewrites 2. An opinionated harness that enforces the workflow (read before edit, exact string match, unique match) so the model cannot bypass it 3. An edit tool with checkpointing, so changes are surgical and you can cleanly revert when something does go wrong This is exactly how Claude Code and the various Claude products are designed. The edit tool requires the file to have been read in the current conversation, requires character-for-character string matching including whitespace, and requires the match to be unique. If any of those fail, the edit does not apply. The file cannot be silently corrupted because the patch has nowhere to land. Combined with checkpointing, you have a system you can actually trust. I break the whole paper down in the new video, run the benchmark myself, show where the catastrophic failures kick in, and close with 8 takeaways for anyone building production document workflows.