How Listen Labs stopped reviewing traces manually with LangSmith Engine

Description

Ollie Elmgren, engineer at Listen Labs, walks through how LangSmith Engine changed the way his team evaluates their AI agents. Listen Labs runs AI-moderated customer research, fleets of AI interviewers that synthesize findings into reports. Before LangSmith Engine, Ollie was manually piecing through traces or dumping them into Claude Code to spot problems. Now, an agent runs in the background and surfaces patterns automatically, catching systemic issues instead of making him wonder whether a single bad trace is a fluke. He shares how the tool fits both early development and ongoing production monitoring, and why the shift to automated evaluation was a big time saver for his team. How Listen Labs stopped reviewing traces manually with LangSmith Engine 0:00 The "aha" moment: workflow transformation 0:14 Who is Ollie and what does Listen Labs do? 0:34 The analysis agent: how it works and what success looks like 1:07 Before LangSmith Engine: manual trace review and Claude Code workarounds 1:23 After LangSmith Engine: automated pattern detection across traces 1:41 Early development and production monitoring use cases 2:08 The time savings, and working with the LangSmith team Extra resources: LangSmith Engine: https://www.langchain.com/langsmith/engine LangSmith: https://www.langchain.com/langsmith/ LangChain: https://langchain.com/