Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance monitoring you already run: uptime, latency, error rate) says 200 OK, latency fine, everything green.
That is the part nobody shows you. Nested traces and per-agent cost are becoming common; the primitives are easy to find now. What stays rare is a data model that lets you act on them: catch the run that looks completely successful while it burns money in the middle. The paper "Why Do Multi-Agent LLM Systems Fail?" (MAST, arXiv:2503.13657) hand-annotated 150 traces across 7 state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa=0.88, and measured failure rates from 41% to 86.7%. The uncomfortable finding: many of those failures do not crash. They complete. They look fine.
In this article I build a small read-only "AWS Account Investigator" crew, wire real cost into every trace span, and then reproduce three silent-waste patterns with real Amazon Nova Pro dollars. You can run the whole thing for $0 locally. Nothing gets created, modified, or deleted in your AWS account.
If you only have two minutes, jump straight to the unique part: catching silent waste. The build up to it matters, but that section is the payoff.
I spent about a week on this against a real AWS account: a few days probing the SDK's behavior before I trusted it, then several more building the crew, watching the trace design break twice, and reading the SDK source when the docs ran out. What follows is written from that, not from a quickstart. The scars are in here on purpose, because they are the part that saves you the week.
This is for people already building AI agents who have never put a real observability layer under them. You know agents, tools, and crews. Where the tracing vocabulary (spans, traces, OpenTelemetry) is new, I define it the first time it shows up.
Contents Why agent observability is a different problem Picking the instrumentation Prerequisites Adding Traccia to your code The stack: AWS native and read only The cost bridge and one gotcha Modeling a multi-agent crew in traces The unique part: catching silent waste Watching it happen: the live control panel Build your own, at zero cost and read only An honest take on Traccia Honest caveats FAQ
Traditional application monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that is enough, because the work is deterministic and the failure modes are loud. An AI agent breaks all three assumptions. It decides its own control flow at runtime, it calls tools in an order you did not hardcode, and it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it is a trend, not an event.
So agent observability has to record things classic APM never needed: how many reasoning cycles an agent took, which tools it called versus which it was allowed to call, the token count and dollar cost of each step, and which agent in a multi-agent crew did what. Those attributes are what make an invisible regression visible.
This is not a fringe opinion. AWS's own Well-Architected Agentic AI Lens frames the baseline state (its "Level 1") as exactly this problem: agent costs are visible only at the account level, Cost Explorer cannot separate agents or workflows, and "teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing." The whole point of what follows is to move off Level 1: to make spending "attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level," which is AWS's own words for the target.
A quick vocabulary anchor, since the rest of the article leans on it. A span is one timed step with attributes attached (one LLM call, one tool call, one AWS read). A trace is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes (status, latency). Agent observability is the same trace structure carrying agent-specific attributes: cycle count, tokens, cost, and which agent owned the step. That is the whole idea; everything below is just putting the right attributes on the right spans.
At one run, a 1.4x overspend is a rounding error. At enterprise scale it is a budget line and a governance problem, and it shows up in four concrete ways: Cost control. A 1.03x-to-1.4x silent overspend per run (the real range I measured across three waste patterns), multiplied across thousands of daily runs and dozens of agents, is real money leaking with no alarm attached. Per-agent, per-tool cost on the trace is the only way to attribute and cap it. Accountability. When a crew misbehaves, "which agent, owned by which team, cost what" needs to be answerable. Trace-level ownership metadata turns a vague incident into a routed ticket. Regression detection. Agents change when prompts, models, or tools change. A known-good baseline plus per-run deltas catches the day a prompt tweak silently doubled token usage, before finance does. Auditability. In regulated environments you need a record of what the agent read, what it decided, and what it cost. A trace is that record.
The theme throughout: a correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose. Here is the before and after in one line. Before, a typical demo gives you one lump token count for the whole run, and an inefficient run looks identical to an efficient one. After, every reasoning step and every AWS read is a span carrying real cost, tokens, cycle count, and the owning agent's identity, so two runs that both return the correct answer and both show 200 OK are no longer indistinguishable when one of them costs 43% more.
Once you know you need per-agent cost, cycle counts, and tool-call attributes on every span, the next question is what to write them with. You could do a lot of this with raw OpenTelemetry, and I nearly did. The reason I did not is that agents need a vocabulary plain OTel does not ship: token counts turned into dollars, a span-level agent identity so one process can render as a real fleet, ownership metadata, and a way to view per-agent cost grouped by session. You end up building all of that yourself, or you find an SDK that already speaks it.
There are options here: LangSmith, Langfuse, and Arize Phoenix all do LLM tracing, and each is worth a look depending on your stack. I went looking for one I would trust in a codebase, which for me means two hard requirements: I can read the source, and I am not locked in. Traccia cleared both cleanly. The SDK is open source, Apache-2.0 licensed, and built on OpenTelemetry (OTel, the vendor-neutral open standard for traces and metrics, the reason you are not locked into any one backend). The spans it produces are standard OTel, the file exporter works with no account and no network, and I could read exactly what it does to my data before committing to it (I did, and the source-grounded critique later in this article is the result). It runs at $0 locally; the hosted dashboard at app.traccia.ai is optional and only comes in when you want the visualization. An open, inspectable SDK with an optional commercial backend is a split I am comfortable adopting, because the instrumentation does not trap me.
That is the real reason it is in this build: agent-native plumbing I did not want to hand-roll, source I could audit, and a real $0 offline path. It also has sharp edges, and I hit several of them; those are documented in full near the end rather than glossed over.
Why not Amazon Bedrock AgentCore Observability or Langfuse, the two obvious AWS-native alternatives? Both are good, and for many teams either is the right call. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on the AgentCore runtime, but AWS's own Well-Architected lens is blunt about the cost gap: "cost reporting stops at the AWS account level, so teams can't separate supervisor overhead from worker execution." Per-agent dollars are something you still assemble. Langfuse is the strong open-source incumbent and I would happily use it; it just was not the tool I was asked to put through its paces here. The point of this build is not "Traccia beats them." It is that whichever tracer you pick, the per-agent cost attribute and the baseline-delta detection are things you wire on purpose, and this article shows exactly how.
Nothing exotic. Three things to run this yourself: Python 3.10+ and the SDKs (strands-agents, strands-agents-tools, traccia, boto3). The repo pins the exact tested versions in requirements.txt. AWS credentials with read-only permissions for the services the crew reads (Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, GuardDuty), plus bedrock:InvokeModel so the agent can actually call the model. The repo ships a ready-to-use policy at iam/read-only-policy.json; AWS's managed SecurityAudit + ViewOnlyAccess cover the reads, but you still add bedrock:InvokeModel on top of them. Amazon Nova Pro, which is two separate steps: (a) enable model access once in the Bedrock console (us-east-1, amazon.nova-pro-v1:0) under Model access, and (b) allow bedrock:InvokeModel in your IAM policy. The console grant is not an IAM permission, so you need both.