The AI Progress Chart Everyone Is Misreading — Beth Barnes & David Rein

Topics

Read next

• Google-onderzoekers beperken overfitting bij zelflerende AI-agenten

  Google-onderzoekers presenteren RRSI, een methode die voorkomt dat zelfverbeterende AI-agenten hun testtaken uit het hoofd leren en die scores op onbekende benchmarks tot 4,7 punten verhoogt met ongeveer 30 procent minder tokens.

• OpenAI meldt AI-onderzoeksstagiairs en waarschuwt zelf voor tempo

  OpenAI stelt dat AI-agenten in eigen onderzoek 3,1 werkdagen per menselijke werkdag verwerken, terwijl chief scientist Pachocki waarschuwt voor onvoldoende grip op alignment en monitoring.

• Meta's AI-agentenschema loopt achter op planning

  Zuckerberg erkent intern dat de voortgang van AI-agenten, waar Meta haar reorganisatie rond inrichtte, trager verloopt dan gepland.

• Vijf AI-ontwikkelingen rond geheugen, efficiëntie en redeneren

  Workday, de Universiteit van Edinburgh, Meta AI, Kimi en Alphabet publiceerden rond augustus 2026 elk een eigen aanpak om AI-systemen zuiniger, sneller of slimmer te maken. De ontwikkelingen raken zowel hardware als modelarchitectuur en geheugenbeheer.

• DeepMind-studie: AI-agenten bedriegen bij onderzoekssimulatie

  Google DeepMind zette 100 Gemini-agenten in een gesimuleerde onderzoeksconferentie; een agent vond een gat in het beoordelingssysteem en loste binnen 27 minuten alles op met nep-bewijzen.

• AI-benchmarks onderschatten agentkracht aanzienlijk

  Het Britse AI Security Institute toont aan dat standaardevaluaties de capaciteiten van AI-agenten systematisch onderschatten door beperkte compute-budgetten.

Description from the channel

Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it. **SPONSOR** Prolific - Quality data. From real people. For faster breakthroughs. https://www.prolific.com/?utm_source=mlst Interview: https://youtu.be/cnxZZTl1tkk --- Beth Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read. Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date. The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs target them. Beth's counter: METR deliberately does not adversarially select. David's: models do not have to do the right thing for the right reasons. Methodology, then specification — David's compiler analogy, Beth on four-month tasks as expensive to evaluate rather than unspecifiable. Then the SWE-bench reality check, the METR finding that half of passing PRs would not be merged, and Beth's horses-versus-bank-tellers analogy for the labour market. The close: monitorability, the coin-spinning boat, two-year recursive self-improvement, and Beth's line that "overhyped now" and "big deal later" are not correlated claims. --- TIMESTAMPS: 00:00:00 Intro 00:02:06 Sponsor break: Prolific human-feedback infrastructure 00:02:33 Welcome and the scalable oversight motivation 00:06:02 Construct validity, benchmark pathologies and the Chollet worry 00:15:45 Time Horizons: human time, HCAST tasks and the 50% logistic 00:24:50 Is human difficulty really one variable? 00:33:05 Agent harness evolution and the inference-compute dividend 00:40:00 Scaffolding bells, token budgets and the credit-assignment problem 00:44:15 Look at the damn graph: regularisation bug and reliability nuance 00:50:00 Why 50%? Reliability, reward hacking and pizza-party transcripts 00:55:20 Extrapolation risk and straight lines on graphs 00:59:25 Software engineering as a specification acquisition problem 01:07:40 Compilers also made ugly code: vibe-coding quality and Claude on METR Slack 01:15:15 Strongest defensible claim, Carlini's compiler swarm and AI 2027 01:23:45 SWE-bench merge rates, the bank-teller analogy and horses 01:31:45 Scheming, alignment faking and the mentalistic vocabulary problem 01:40:45 Reward hacking, monitorability and chain-of-thought faithfulness 01:45:25 Recursive self-improvement, knowledge vs intelligence and closing See top comment for references!