Does Your Agent Know It's Lost? Uncertainty & Progress Signals for Reliable LLM Agents, w/ Sharon Li

Name: Does Your Agent Know It's Lost? Uncertainty & Progress Signals for Reliable LLM Agents, w/ Sharon Li
Uploaded: 2026-07-24T18:42:34.000Z
Duration: 1 h
Channel: Cohere
Description: Sharon Li presents methods for measuring uncertainty and progress signals in LLM agents, enabling systems to detect when they go off track. She introduces AgentUQ and demonstrates how RL training implicitly yields useful signals for reliable agent operation.

Cohere24 July 2026Watch on YouTube

What you'll learn

LLM agents need reliable methods to detect when they go off-track during multi-step tasks with irreversible actions and unpredictable feedback
AgentUQ provides a formalization of uncertainty specific to agent interactions, distinct from traditional uncertainty quantification for QA tasks
Progress advantage, derived from RL training, yields implicit step-level signals that improve reliability without additional annotations or reward model training
Test-time scaling and RL post-training offer practical ways to enhance agent confidence and failure detection
Standard leaderboards and existing uncertainty methods fall short for complex agent scenarios involving tool use and long-horizon objectives

Frequently asked questions

What is AgentUQ and how does it differ from traditional uncertainty quantification?

AgentUQ is a framework for measuring uncertainty in interactive LLM agents with multi-step tasks. It differs from standard UQ because it must handle tool use, irreversible actions, and uncertainty evolving over longer interactions, rather than single-turn questions.

How can an agent know it is lost or off-track?

Sharon Li presents progress advantage as an implicit signal from RL training that indicates whether an agent is making progress. This signal helps agents detect mid-episode when they have gone off-track without additional annotations.

What is progress advantage and how does it emerge from RL training?

Progress advantage is a step-level signal that emerges implicitly from RL post-training. It recovers the optimal advantage function and outperforms simple confidence baselines and even dedicated trained reward models.

Why are traditional leaderboards insufficient for evaluating agent reliability?

Leaderboards measure final outcomes but tell nothing about whether an agent knows it is lost mid-episode. Agents performing long-horizon tasks with irreversible actions need granular uncertainty measures during execution, not just final scores.

Topics

llm agents uncertainty quantification reinforcement learning reliability

Description from the channel

Does Your Agent Know It's Lost? Uncertainty and Progress Signals for Reliable LLM Agents Talk by Sharon Li, Associate Professor, University of Wisconsin-Madison Hosted by Beyza Ermis, Senior Research Scientist, Cohere Labs 00:00 Welcome and Speaker Intro 02:06 Why Agent Reliability Matters 04:02 Limits of Leaderboards 05:27 Uncertainty Methods Today 06:52 Process Rewards Primer 08:28 Why Agents Break Assumptions 10:35 Two-Part Talk Roadmap 11:20 AgentUQ Setup and Example 14:59 Formalizing Agent Uncertainty 19:17 Estimator Challenges and Gaps 21:37 Open Problems in AgentUQ 23:16 Spec for a Good Signal 24:37 Progress Advantage Motivation 26:23 RL Objective and Implicit Rewards 29:44 Key Idea Target Advantage 31:05 PERES Advantage Defined 32:16 News Tool Call Example 33:57 Practical Recipes Setup 35:12 Aggregation Strategy Choices 36:05 Test Time Scaling Results 38:35 Uncertainty Quantification AUROC 40:08 Failure Attribution Benchmark 41:57 Ablations and Best Aggregation 43:01 Key Takeaways and Wrap Up 43:51 Future Directions and Credits 46:23 Q and A Confidence Meaning 50:02 Calibration Discussion 54:25 Generalization to Multi Agent 56:42 Checkpoint Choice Sweet Spot 59:18 Closing Remarks LLM agents are increasingly deployed on long-horizon tasks with tool use, irreversible actions, and unpredictable feedback. Yet we have few principled ways to tell, mid-episode, whether an agent is on track or quietly failing. Most uncertainty quantification (UQ) research still centers on single-turn QA, a poor match for interactive agents. In this talk, I'll present a general formulation of agent UQ and the challenges unique to agentic settings, from choosing uncertainty estimators to modeling how uncertainty evolves over an interaction. I'll then show that a powerful answer has been hiding in plain sight: RL post-training already yields an implicit step-level signal, the progress advantage, which recovers the optimal advantage function with no annotation or reward-model training. Across test-time scaling, UQ, and failure attribution, this free byproduct beats confidence baselines and even dedicated trained reward models. This talk is part of Cohere Labs in Conversation, a limited series of talks, in which Cohere Labs scientists and engineers host external researchers for techincal talks and Q&A discussions on subjects related to our current explorations at Cohere Labs. We look forward to sharing these talks with you, giving you a glimpse into the problems we're exploring, and learning together from some of the greatest minds in the field.