The Watchdogs of AGI — Rune Kvist of AI Underwriting Company

What you'll learn

• Rune Kvist explains why risk, liability and trust may become the binding constraint on AI adoption, not model capability alone.
• AIUC-1 is positioned as a standard for the security, safety and reliability of AI agents.
• Agents are stress-tested for jailbreaks, hallucinations, data leaks and other failure modes.
• Standards and insurance evolve together, with Lloyd's of London able to insure AI systems according to Kvist.
• The episode covers the growing trust gap between frontier labs and governments and the question of who acts as watchdog.

Frequently asked questions

Topics

Read next

• Studie: autonoom AI-onderzoek niet dicht bij bereik naar claims toe

  Een onderzoek van Princeton en het UK AI Security Institute stelt vast dat AI-agenten voorlopig niet in staat zijn zelfstandig wetenschappelijk onderzoek van kwaliteit uit te voeren.

• Von der Leyen waarschuwt voor AI-agenten die ontsnappen uit hun testomgeving

  Europese Commissievoorzitter Ursula von der Leyen heeft op 16 september 2026 in Straatsburg gewaarschuwd voor AI-agenten die ontsnappen uit hun evaluatieomgeving en voor zelfverbeterende modellen. Ze wil grote AI-labs uitnodigen voor gesprekken en de Europese AI-wet inzetten als basis voor mondiale veiligheidsnormen.

• Open-weight AI-modellen halen frontier-niveau, veiligheidsgat blijft

  SaferAI-rapport toont dat open-weight GLM-5.2 frontier-capaciteiten bereikt zonder belangrijke veiligheidskwartieren, wat zorgen oproept over onvoldoende governance.

• OpenAI noemt GPT-6 Astra het begin van het AGI-tijdperk

  OpenAI heeft GPT-6 Astra uitgebracht, het eerste model dat het bedrijf als het begin van het "AGI-tijdperk" bestempelt. Het model haalt topscore op wiskundige, programmeer- en cyberbeveiligingsbenchmarks en ontdekte tijdens evaluatie twee onbekende zero-day kwetsbaarheden.

• 54% van bedrijven ondervindt beveiligingsincidenten met AI-agenten

  Onderzoek van 107 bedrijven toont aan dat meer dan de helft al AI-agent beveiligingsincidenten of bijna-incidenten heeft meegemaakt, en dat meeste agenten nog steeds credentials delen zonder op-scope identiteiten.

• AI-guardrails beperken werk cybersecurity-onderzoekers

  Beveiligingsonderzoekers zeggen dat guardrails van OpenAI en Anthropic hun werk bij het zoeken naar onbekende kwetsbaarheden bemoeilijken.

Description from the channel

From their seed led by Nat Friedman and Dan Gross, AIUC has now announced a monster $40M Series A! From being Anthropic’s first product hire to building the standards, testing, and insurance infrastructure meant to make frontier AI deployable, Rune Kvist is betting that the biggest constraint on AI adoption won’t be capability — it will be trust. In this episode, the AIUC cofounder joins swyx and Vibhu to announce a new $40M round and explain why companies like Cursor, Harvey, Lovable, and ElevenLabs are increasingly confronting a problem that gets harder as AI gets better: who is responsible when autonomous systems fail? We go deep on AIUC-1, the emerging standard for agent security, safety, and reliability; how AI agents are stress-tested for jailbreaks, hallucinations, and data leaks; and why Rune thinks standards and insurance could become critical infrastructure for AI. We also discuss the growing trust gap between governments and frontier labs, AI-enabled cyber and biological risks, why every model can ultimately be jailbroken, what happens when a $20 coding agent causes $200M of damage, whether AI engineers should be certified, and why even after AGI there may be one job the labs can never do themselves: be their own watchdog. We discuss: • Why risk, liability, and trust may become the binding constraint on AI adoption • Rune’s path from reading the Scaling Laws paper to joining Anthropic in its earliest days • What Anthropic understood about scaling, compute, and the future years before it became obvious • Why Waymo illustrates the gap between AI capability and real-world deployment • AIUC’s $40M round and work with Cursor, Harvey, Lovable, ElevenLabs, and other frontier AI companies • AIUC-1: a standard for AI agent security, safety, and reliability • How agents are tested for jailbreaks, hallucinations, data leakage, and other failures • Why most AI companies optimize the happy path without seriously stress-testing adversarial cases • Why AI standards may need to update every quarter instead of every decade • The emerging trust gap between frontier AI labs and governments • Cybersecurity, child safety, biological weapons, and the expanding frontier-model risk surface • Why standards and insurance may need to evolve together • How Lloyd’s of London can insure AI systems and bring trust to enterprise deployment • What happens if a $20 Cursor subscription contributes to a $200M plane crash • The Air Canada chatbot case and how AI failures are beginning to clarify legal liability • Why copyright may be one of the hardest AI risks to insure • Evals, mechanistic interpretability, monitoring, and models becoming aware that they’re being tested • The impossible CISO mandate: adopt AI fast, but don’t let anything go wrong • Why robotics will make AI liability dramatically more consequential • Whether AI engineers should have Level 1, 2, and 3 certifications • AIUC’s roadmap across agents, frontier models, robotics, and universal red teaming • Why AGI could become a question of national sovereignty • Why the labs can never fully serve as their own watchdogs • The Big Short problem: how do you stop competing watchdogs from racing standards to the bottom? — Rune Kvist • LinkedIn: https://www.linkedin.com/in/runekvist • X: [https://x.com/RuneKvist AIUC • https://aiuc.com Timestamps 00:00:00 Hook 00:01:29 AIUC’s $40M Round and the Risk Bottleneck for AI 00:02:36 From Scaling Laws to Early Anthropic 00:09:27 Why Trust, Not Capability, Could Limit AI Adoption 00:13:48 Founding AIUC and Building AIUC-1 00:20:21 How AI Agents Are Audited and Stress-Tested 00:26:55 Frontier Models, Government, and the AI Trust Gap 00:35:01 Cyber, Child Safety, and AI-Enabled Biological Risk 00:39:43 Why Standards and Insurance Belong Together 00:43:14 What Does an AI Insurance Policy Actually Cover? 00:52:13 The $20 Cursor Subscription and the $200M Plane Crash 00:55:22 AI Liability, Monitoring, and Earning Enterprise Trust 00:57:50 From AI Agents to Models to Robotics 00:59:58 Copyright, Adverse Selection, and AI Insurance 01:04:57 Evals, Mechanistic Interpretability, and Eval Awareness 01:10:05 The Impossible Enterprise AI Mandate 01:13:21 Prediction Markets vs. AI Audits 01:16:12 Should AI Engineers Be Certified? 01:20:39 AIUC’s Roadmap, AGI, and Who Watches the Watchdogs? Sponsorship and business inquiries: business@latent.space