Automated LLM Tracing Infrastructure for AI Agent Development
Public- 09 Nov, 2026

Introducing AI agents often leads to a critical bottleneck: the lack of a structured foundation for collecting execution histories needed for evaluation and improvement. Forward Deployed Engineers (FDEs) at AI Shift, a subsidiary of CyberAgent, face this challenge daily while building diverse AI applications for clients. Their mission is to maximize value by deeply integrating into client workflows, but the priority for adding observability instrumentation is frequently low. Since applications function correctly without tracing, and deciding how to instrument each component is complex, FDEs often defer this task. This delay hampers the ability to assess output quality and demonstrate cost-effectiveness to clients, creating a gap between rapid development and necessary operational oversight.

To solve this, the team developed a dual-layer tracing architecture that balances automation with business context. The first layer utilizes an LLM Gateway, specifically Agent Router (formerly Envoy AI Gateway), to capture infrastructure-level telemetry. By routing all LLM calls through this gateway, the system automatically records inputs, outputs, model names, and token counts. This data is sent via OpenTelemetry to a collector and then filtered so that only LLM-related traces are sent to Langfuse, an LLM engineering platform. This approach removes the decision-making burden from FDEs: they do not need to manually decide whether to instrument every LLM call, as the gateway handles it automatically at the infrastructure level.

However, the LLM Gateway alone cannot capture business-specific semantics or maintain context across complex workflows. Without application-level context propagation, the gateway generates independent, shallow traces for each LLM call, making analysis difficult. To address this, the team developed a custom Tracing SDK. This SDK is not used for instrumenting the LLM calls themselves, which remain the gateway's responsibility, but for attaching business metadata. It allows developers to define traces with specific inputs, user IDs, and session IDs, providing a clear hierarchy of operations. For example, a trace can encompass a "review" process, linking the user context and the specific purpose of the LLM invocation, such as a guardrail check or evaluation, to the resulting output.

To further reduce the development load, the team created an AI Skill that allows coding assistants to automatically insert the necessary Tracing SDK instrumentation. This automation ensures that even as development speeds up with AI assistance, the required observability code is consistently applied. The resulting data in Langfuse is highly structured, showing clear traces with attached metadata like user sessions and call purposes. This setup ensures that even if manual SDK instrumentation is missed, the LLM Gateway still records the basic call data, providing a fallback for minimal analysis. This infrastructure enables FDEs to focus on solving client problems while maintaining the data quality required for continuous agent evaluation and improvement.