A customer asks a support agent to cancel an order and confirm the refund route. The assistant retrieves the order, checks its fulfilment status and produces a polished reply. Yet the cancellation never happens. The log shows successful requests at every step. This is where agent observability becomes an engineering discipline rather than a dashboard: the team must reconstruct the execution path, identify the missing state transition and prove that a fix changes the outcome.
This walkthrough uses an explicitly hypothetical commerce workflow. It is designed to illustrate how to investigate multi-step agent behaviour without pretending that a transcript or trace proves more than it does. A production system will have its own tool permissions, confirmation requirements, order states and retention rules.
Write the expected path before opening the trace
Before touching the instrumentation, define the authorised sequence. The user requests cancellation; the agent identifies the order; the system checks eligibility; the agent obtains any required consent; the cancellation endpoint performs the state change; and the agent confirms the actual recorded result. Refund information may be a separate policy explanation or a later transaction. A response that says “cancelled” before a successful state change is a failure even if its language sounds helpful.
Document the input, each required action and the end-state evidence. For the hypothetical example, the order database changing from eligible to cancelled is stronger proof than an assistant sentence. If a cancellation is prohibited because dispatch has begun, a correct refusal with an appropriate alternative is also a valid outcome. The reference path must encode those branches or the debugger risks “fixing” behaviour that was compliant.
This expected sequence is the diagnostic ruler. Without it, engineers often stare at a long trace and inspect whichever slow or expensive step looks unusual. Cost matters, but it can distract from the missing action that caused the customer-facing failure.
Agent Observability begins with a correlated run
A distributed trace links the parent workflow to child spans for model calls, retrieval, tool execution and relevant application operations. OpenTelemetry's GenAI semantic conventions describe operations including agent invocation, workflow execution and tool calls. They are useful naming guidance, but several conventions remain in development, so teams should document the particular schema and instrumentation version they deploy.
For the cancellation journey, the outer span should identify the workflow version, correlation ID and non-sensitive task category. Child spans may record an order lookup, eligibility check, model decision and cancellation tool call. Capture timing, status and a sanitised error classification. Where sensitive messages are needed for debugging, use purposeful collection, access controls and retention limits; OpenTelemetry warns that raw model inputs and outputs may contain personal data.
A useful minimal record is often smaller than the full transcript. It might say: order lookup succeeded; eligibility returned true; model selected a policy explanation; cancellation tool was never invoked; no cancellation event was emitted. Those five observations narrow the cause more effectively than hundreds of unfiltered debug messages.
Find the earliest missing or incorrect decision
Start at the final state and walk backwards. Did the application record show a cancellation event? If yes, the problem may be confirmation messaging or event propagation. If no, was the cancellation endpoint called? If called, did it accept the correct order identifier and authorisation context? If never called, inspect the routing decision that was supposed to select it.
In our hypothetical failure, the trace contains an order lookup and a response-generation span, but no cancellation span. This makes a tool timeout unlikely as the primary cause. Possible explanations include an overly restrictive instruction, an outdated tool definition, a missing confirmation state or a route that handles “refund policy” before “cancel order.” The trace supports those hypotheses; it does not distinguish them without inspecting configuration and the expected workflow.
If the missing span is surprising, check instrumentation itself. An auto-instrumentation package may trace model calls but omit a custom Python function, queue consumer or external workflow engine. Absence of a span is evidence of absence only when the path is known to be instrumented. Cross-check application event logs before declaring the tool was never attempted.
Distinguish bad retrieval from bad execution
A retrieval-augmented agent can fail before it chooses a tool. The search layer might return an old returns policy, the model might ignore a relevant passage, or the agent might misuse a correct policy. These are different failure points. Inspect retrieval query, document identifier, document version and permitted context, then compare the response against an approved source. Do not infer retrieval quality solely from a model-generated relevance score.
Tool failures need their own inspection. A transport-level success does not prove an action occurred. For example, an API can return a successful response containing a business-rule rejection, an idempotent no-op or a queued request that has not completed. Record business status separately from HTTP status. In asynchronous systems, correlate the tool response with the eventual event; otherwise the agent might announce success while the queue later rejects the operation.
For multi-agent systems, add the handoff boundary. A planner may assign work to a fulfilment agent, which returns a policy summary instead of a cancellation outcome. The parent agent then mistakes that message for task completion. Trace the task contract across the handoff: expected action, delegated scope, returned evidence and final validation. More agents create more boundaries where an apparently fluent answer can hide an incomplete job.
Reproduce without copying live customer data everywhere
Build a minimal test case with an invented order, authorised state and expected cancellation result. Preserve the structural feature that caused the failure, such as the sentence “cancel this order and explain when I get a refund.” Replace personal identifiers and remove unnecessary transcript content. A faithful synthetic reproduction is easier to share across engineering teams and safer for development environments.
Run the current workflow with a fixed tool-response fixture. Verify that the cancellation step is omitted. Then change one component at a time: routing rules, tool description, state storage or confirmation logic. LLM outputs can vary between runs, so repeat meaningful cases and evaluate the decision path, not only exact words. If you cannot reproduce the defect, broaden the investigation to the production context rather than declaring it resolved.
We recommend recording both deterministic assertions and semantic expectations. Deterministic checks can confirm the correct tool was called with the expected order ID, or that no action occurred without consent. Semantic checks can assess whether the final explanation is accurate and understandable. The two complement each other; an elegant answer with a missing action still fails.
Fix the boundary, not the entire prompt
The temptation is to append “always cancel orders when requested” to a system prompt. That change is unsafe if some orders require explicit confirmation or are ineligible. A stronger fix may be a typed workflow state requiring a cancellation result before the assistant can claim completion. Another may be a router that separates action requests from information requests. The right remedy depends on which boundary the trace exposed.
Record the probable root cause in the issue tracker with a minimal example, affected version, observed evidence and proposed regression test. Keep alternative hypotheses visible until disproved. If the data indicates missing instrumentation rather than missing behaviour, fix the telemetry first. An unobservable workflow cannot be evaluated reliably by simply increasing the volume of LLM judgements.
Before release, replay the failed case and neighbouring branches: cancelled, already dispatched, permission denied, duplicate request and user withdraws consent. Test for harmful regressions, especially unwanted state changes. The production rollout should monitor the targeted defect as well as adjacent cancellation and refund intents.
Verify the customer outcome after deployment
A successful local test is only the first check. After release, compare eligible cancellation requests under the old and new workflow versions, accounting for intent mix and channel. Track cancellation events, incorrect success confirmations, repeated requests and escalation quality. Investigate cases where the trace claims completion but the order record disagrees. Sample enough journeys to detect unintended behaviour; a handful of positive examples is insufficient evidence of a systemic fix.
For an agent that only advises users rather than performing actions, the verification target changes. You may need approved-answer checks, explicit feedback or downstream support contacts. There is no universal completion signal. A team should state that limitation rather than promote a speculative “resolution rate” as ground truth.
Phoenix, LangSmith and Langfuse document trace inspection, evaluations or session-level scoring workflows that can support parts of this process. Tool selection matters less than ensuring the traces can be joined to the application events needed for a reliable verdict. A good debugger can explain why an agent failed and show how the proposed correction was tested.
What to give product teams, not just engineers
Technical root-cause notes are necessary but insufficient for prioritisation. Product teams need to know which customer journey failed, how often it failed among eligible attempts, how severe the outcome was and whether customers could recover. Keep that product context attached to the engineering trace instead of presenting a stack of spans without a meaningful customer outcome.
VoxMith's published positioning focuses on production conversation analysis and turning observed agent failures into suggested engineering actions that teams review before release. That aligns with a useful handoff: conversation evidence identifies the broken journey; a trace narrows the component; tests check the fix; production measurement establishes whether outcomes improved. Its exact capabilities should still be assessed against the team's instrumentation and governance requirements.
Agent observability pays off when a debugging session ends with a verified change rather than a more colourful trace. Define the task, reconstruct the path, identify the earliest wrong decision, protect user data, test the smallest defensible correction and confirm the real action succeeded. That is how a “green” failure becomes an issue the team can actually close.
Sign in to leave a comment.