Trace Any AI Agent with OTel, MLflow, and Unity Catalog

Description

AI agents generate massive volumes of trace data, but traditional observability tools make this data expensive to retain and difficult to govern. This demo explores how to use OpenTelemetry (OTel), MLflow, and Unity Catalog to unify your AI observability stack. See how streaming agent traces directly into the Databricks Platform allows you to securely govern your data, build custom token cost dashboards, and run continuous LLM evaluations without the risk of PII deadlocks. Learn more about Agent Tracing and AI Observability with managed MLflow here: https://www.databricks.com/product/managed-mlflow Read the launch blog to learn more about Governing AI agents at scale with Unity Catalog: https://www.databricks.com/blog/governing-ai-agents-scale-unity-catalog Read the blog Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks: https://www.databricks.com/blog/observability-any-agent-anywhere-production-ready-tracing-opentelemetry-unity-catalog-databricks TIMESTAMPS: 00:00 – Challenges in AI Agent Observability 02:28 – The Continuous Improvement Flywheel 04:20 – Demo: Building a Support Manager Assistant 05:39 – Setup: Trace Integration with MLflow and Unity Catalog 08:27 – Analyzing Traces and Native Dashboards 10:13 – Offline Evaluation and LLM Judges 13:04 – Closing the Loop for Continuous Improvement