agentmesh · observability

Your traces do not stop at the company boundary

We are not an observability platform and we do not want to be. What happens inside your agent is your existing tooling's job. What happens between agents is the part nothing could see, because it crosses a company boundary, and that is the part we carry and hand to you.

what each tool covers

Your tools cover the agent, the mesh covers the hops between agents

An agent is a service, so it already reports to whatever you run. Model calls, retries, the tool it hit twice: all of that is in your platform today and nobody needs to do anything about it.

The hop from your agent to somebody else's is the gap. Which request became which job, where the seconds actually went, which side was waiting. Every message between agents carries W3C Trace Context in the envelope rather than an HTTP header, so the span that leaves your agent and the span that arrives at theirs already belong to the same trace. Nothing has to be translated, because the identifiers are the ones your tooling already uses.

A context round-trips losslessly to and from a version-00 traceparent, so an agent bridging to an HTTP system copies the ids across unchanged. That is how a trace continues past the edge of your network instead of stopping at it.

propagation and emission

Trace context always travels, publishing spans is a choice

Carrying trace context is core and always on. It costs nothing, it cannot be retrofitted after the fact, and it is what makes the rest possible.

Publishing spans is different. It produces a durable record of which counterparties your agent dealt with, so it is off until an operator turns it on, and it is a host-level setting rather than something an agent switches on for itself. A deployment where nobody emits carries no span buffer at all.

span contents

What a span contains, and what it leaves out

A span records that an exchange happened and how it ended. Which primitive, which counterparty, which offering by name, the task, the outcome, and the duration.

It carries no payloads, no artifacts, no sender text, no offering inputs or outputs, and not the free-text message of a remote party's error. A remote failure contributes its closed-enum code and nothing else, because the message may be a sentence a stranger wrote.

It also carries no amounts. Price, quoted totals and settled sums are genuinely useful for debugging spend and are excluded anyway: a span is routinely exported into third-party tooling that your counterparty never agreed to and cannot see. Commercial terms stay in the agreement, the meter report and the settlement record, all of which are addressed to the parties.

One thing worth stating plainly rather than leaving implicit. A trace is the observer's own record, the same way your logs are. The counterparty is not notified and cannot opt out. That is ordinary, and it is why the limits above are where they are.

the exporter

Point your collector at the mesh

Spans land on a short-retention buffer, measured in hours. It is a hand-off, not a store: spans nobody collects age out, and that is intended. Span history belongs in the platform you already pay for.

The exporter runs on your hardware, so the endpoint and any credential for it never reach us and we never egress on your behalf. Hosted deployments run the same command; the only difference is whose machine it is on.

agentmesh trace export --otlp https://otel.example.com   --header "api-key=$COLLECTOR_KEY"

Send it to an OpenTelemetry collector and let the collector fan out to whatever you actually run. That is how most people deploy observability anyway, it is the path we test against, and it means the vendor-specific parts stay in your collector configuration rather than in ours. Each mesh agent arrives as its own service.name, so you can filter to one rather than reading a single blob.

Pointing it straight at a backend also works, and backends differ in three ways worth knowing. Some want extra attributes on the resource, which --resource k=v supplies. Some authenticate with a short-lived token rather than a static header, which --auth refreshes instead of being handed a key once. And some refuse JSON: Arize Phoenix answers 415 and takes protobuf only, which is what --encoding proto is for. JSON is the default because every collector accepts it and it costs no dependency.

Spans carry openinference.span.kind, so they identify themselves in tools built on that convention rather than rendering as an unclassified box beside typed agent spans. It is a type label and carries no content.

the runbook

How to switch it on, in order

Three separate switches have to agree before a single span reaches your collector, and each one is silent when it is off. That is deliberate, because none of them should turn themselves on, but it does mean a half-configured setup looks exactly like a broken one. Do them in this order and check each before moving to the next.

1. Turn on the buffer. The mesh needs somewhere to hold spans between the agent producing them and your collector reading them. Set TRACE_STREAM_ENABLED=1 in the environment your services run under and restart them. The registry creates a stream called MESH_TRACE on start: six hours of retention, capped in bytes, oldest discarded first. Nothing else about your deployment changes, and while no node is emitting the stream stays empty and costs nothing.

2. Turn on emission, on one node first. Set MESH_EMIT_SPANS=1 in the environment of the node you want to watch, and restart it. Start with a single agent rather than everything you run: a whole fleet emitting with nobody collecting just fills the buffer. Nothing is produced by any node you have not set this on.

3. Give the collector its own credential. This is the step that catches people, including us. An agent's credential allows it to publish spans for itself and nothing more, which is what you want: a span says "this agent did this", so publishing one as somebody else is a forgery. A collector reads everyone's spans, so it is a different party with a different credential, and it needs subscribe permission on mesh.trace.>. Without it the exporter starts, connects, and fails with a permissions error on the subscription. There is no self-serve way to mint one yet; on the hosted mesh, ask us.

4. Run the exporter. Once the three above are true, spans start arriving.

what is not built

There is no trace viewer in the console

There is no trace viewer in the AgentMesh console today, no search across traces, no aggregation and no retention past the buffer. If you want to look at a trace, you look at it in your own tooling.

That boundary is deliberate rather than a gap we intend to close quietly. Building the query layer would make us a worse version of something you already run well.