Key Takeaways
- Proactively detect agent issues. LangSmith Engine v2 introduces red teaming, expanded issue detection, and automated fix validation. - Run agents with identity-scoped auth and memory. Managed Deep Agents makes it simple build and ship agents, and now includes user-level memory and built-in web search. - Turn agent sessions into fine-tuning data. Trajectories is a new conversational view of traces. LangSmith Fine-Tuning turns uses trajectories to tune models for specialized tasks. - Build custom UIs for your agent data. With Custom Apps, teams can build and publish custom LangSmith interfaces for workflows like annotation queues and experiments. This week, hundreds of AI engineers joined us at Interrupt NYC to hear how teams are taking domain-specific agents to production and to discover the latest LangSmith announcements. Our latest releases focus on two pillars of the agent development lifecycle that help teams build with control and flexibility, compound the intelligence of their agents, and ensure strong governance: runtime and observability + evaluations. Runtime gives agents what they need to operate in production (such as memory, channels, tools and secure execution environments). Observability and evals help teams understand behavior, measure quality, and improve agents they more they are used. Sitting above is the intelligence layer, which works over the business logic, traces, evals and production data to accelerate this agent development loop. Here’s what we shipped.
LangSmith Engine v2: Proactively detect agent issues and ship fixes faster
Improving agents is a multi-step process; a typical workflow requires engineers to spot issues, determine a root cause, propose a fix, test it, deploy it, and monitor for regressions. We built LangSmith Engine as an in-platform agent that automates work at each step of the agent development lifecycle. Since launching in May, Engine has already helped engineers analyze more than 60M traces to diagnose tens of thousands of issues. Engine v2 introduces major advancements for issue detection, testing and validation.
Proactively troubleshoot with Red Teaming
New Red Teaming capabilities help detect issues before they surface in production. Using production traces and repos, Engine can generate hypotheses about issues that may not have appeared in production yet, test those hypotheses, and surface confirmed failures for review.
Detect harder-to-spot issues
Engine now detects more issue types. In addition to errors and unmet user requests, it can identify performance trends across error rate, latency, and cost, as well as inefficient agent work such as repetitive tool calls or unnecessarily long trajectories.
Automatically test proposed fixes
For agents running on LangSmith Deployment, Engine can now validate proposed fixes prior to human review and deployment. Engine v2 runs the offending inputs against your agent to confirm the issue. Then it tests candidate fixes against a broader eval set until it finds one that resolves the issue. Once Engine has a satisfactory fix, it’s presented to the end user, where they can open a PR with a click for quick deployment. The next release of self-hosted LangSmith will support BYOK for Engine.
Managed Deep Agents v0.8: New auth, memory, and channels
Managed Deep Agents combines the Deep Agents harness with managed infrastructure, creating the easiest way to go from business logic to an agent in production. Our latest release addresses four challenges teams face when running agents in production: agent memory, authentication, channels and tool management.
Identity-scoped auth and memory
Managed Deep Agents already supported durable agent memory, and we’re excited to introduce new user-level memory so that agents can store context scoped to each authenticated user. This new layer provides a place to store caller-specific context that keeps it separate from shared agent-level memory. The runtime does not copy content between layers, and access policies can be defined at each level to ensure that user-level information isn’t leaked into conversations. Managed Deep Agents also now supports both agent and user-owned credentials for external connections (such as GitHub and Notion).
New and expanded channels
Agents work best when they operate where users already are. We expanded our Slack support to include file transfer, so users can send logs, spreadsheets, contracts, screenshots, and other files directly to an agent. New HTTP channel support lets teams connect agents to any service that can send a JSON webhook. This is especially useful for customer-facing agents that live on multiple channels.
Built-in web search