All videos
0:00 / 0:00
research

Harbor x LangChain: A Unified Stack for Evaluating Agents

LangChain1 July 2026Watch on YouTube

Part of series

Ep. 6 · Deep Agents met LangChain

LangChain legt uit hoe je met het Deep Agents framework complexe AI-agenten bouwt, beheert en naar productie deployt.

View the series

What you'll learn

  • You learn why output-string grading no longer suffices for agents that act in a full environment.
  • You see the three building blocks Harbor needs: an agent, a sandbox and a dataset.
  • You understand the structure of a Harbor task with task.toml, instruction.md, environment and tests.
  • You learn how to integrate a Deep Agent with Harbor and how results show up in LangSmith.

Frequently asked questions

What does Harbor do differently from existing agent evaluations?
Harbor is an open-source framework that runs agents in their own clean, isolated environment per run. A deterministic pytest script decides whether a task passes, instead of judging the agent's output text.
What three things do you need to use Harbor?
You need an agent with a function that compiles all tools, prompts and logic, a sandbox either locally with Docker or in the cloud with LangSmith, and a dataset of tasks.
What does a task in a Harbor dataset look like?
A task is a subfolder with a task.toml for configuration such as timeout and memory, an instruction.md that drives the task, an environment folder with the Docker image and a test folder with the pytest script.
Where can you view the results of a Harbor evaluation?
When you use the LangSmith plugin, runs appear in LangSmith under the datasets and experiments tab. There you see per task the reward as one or zero, latency and the token count.

Topics

agent evaluationlangchainai infrastructuredeep agentstesting & observability

Read next

Sources

What is known about this topic outside the broadcast, and where it says so.

Description from the channel

As agents increase in capabilities, evaluations have gotten more difficult. Agent harnesses like Claude Code, Pi, and Deep Agents now give agents access to entire computers to read files, execute scripts, run code, and more. Every agent now needs to run in its own clean, reproducible environment for a given task. Evaluating long-running, stateful agents requires a new eval runner. Harbor has emerged as the industry leader in this space. In this blog, we first explain why everyone running agent evals should know what Harbor is and then show how to integrate Deep Agents, LangSmith Sandboxes, and LangSmith Experiments into Harbor. We ultimately need to run agents in a real, reproducible, isolated environment, many times in parallel, with a deterministic check at the end. Harbor solves this problem and is now wired directly into Deep Agents, LangSmith Sandboxes, and LangSmith Observability. Join LangChain software engineer Nick for a look at: - Why traditional output-based evals fall short for complex, long-running AI agents. - Harbor, an open source evaluation framework that runs agents in isolated, reproducible sandboxes - The full workflow: building a Deep Agent, structuring an eval dataset, and tracking results in LangSmith 0:00 Why agent evals needed to change 0:32 From output strings to real environments 1:07 What makes a Deep Agent different 1:40 Building a deep research agent 2:47 Running the agent: noise pollution demo 3:55 Introducing Harbor 4:33 Datasets and tasks in Harbor 6:15 Turning a demo into a verifiable eval 7:57 Integrating Deep Agent with Harbor 8:32 Running Harbor via CLI 9:21 Viewing results in LangSmith 10:05 Recap and getting started https://www.harborframework.com/docs https://docs.langchain.com/oss/python/deepagents/overview https://docs.langchain.com/langsmith/harbor-integrations