> ## Documentation Index
> Fetch the complete documentation index at: https://docs.augmentcode.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Use your data

> Find sessions, count tokens and cost, build dashboards and set up alerts with Cosmos Trace Export data.

> [!NOTE]
>
> This feature is in early access and behind a feature flag. Contact your Augment Solutions Architect to enable it for your tenant

Everything Cosmos sends has `service.name` = `augment-cosmos`. Start every query with that filter.

## Find a session

| To find | Filter or group by |
| - | - |
| One session | `gen_ai.conversation.id` = the session ID, or its trace ID. Every span and log event carries it |
| Sessions of one expert | `gen_ai.agent.id` = the expert ID, on run and setup spans. Prefer the ID to the run's name, which changes when the expert is renamed |
| Sessions of one user | `user.id` or `user.email` (with **Include user identity** on) |
| Failed runs | `invoke_agent` spans with status `ERROR`, grouped by `error.type` |
| Slow setup | `wait_for_agent` spans, or the time from a run's start to its `agent_ready` event |
| Spans from one agent version | The resource attribute `service.version` |

Sort a session's spans by start time to read it in order.

### A run started by another session

When an agent delegates to a worker in another session, the worker's run span has a link to the tool call that started it. The link's `gen_ai.conversation.id` attribute holds the calling session's ID, so you can jump between the two traces.

### The session span can arrive twice

While a session is open, Cosmos sends a provisional `session` span when its first run ends, and again when a later run ends at least 5 minutes after the last copy, so the session span covers its runs before the session is over. When the session ends, Cosmos sends the final `session` span with the same trace ID and span ID and the real end time. If work resumes later, it's sent once more.

Most trace views show only the latest copy. Span search and span counts may count every copy: to count sessions, use `cosmos.session.count` or `cosmos.session.start` events instead of `session` spans, and for session duration use the copy with the latest end time.

## Count tokens and cost

**Use the metrics if you can.** They're already aggregated and count each token once.

> [!WARNING]
>
> Subagents' model requests aren't traced yet, so every Trace Export usage and cost total, from metrics, spans or log events, leaves out subagent inference. For billing, use your Augment invoice and usage reports.

| Question | Metric | Group by |
| - | - | - |
| Tokens by model | `cosmos.token.usage` | `gen_ai.response.model` (the model that served it), `gen_ai.token.type` |
| Credits by model or user | `cosmos.credit.usage` | `gen_ai.response.model`, `user.email` |
| Cost in US dollars | `cosmos.cost.usage` | `gen_ai.response.model`, `user.email` |
| Sessions started, by expert | `cosmos.session.count` | `gen_ai.agent.id` |

`cosmos.cost.usage` is sent only for organizations billed in US dollars. Credits and cost are what Augment records for each request; your invoice remains the source of truth.

**To count tokens from spans**, sum `gen_ai.usage.*` over `chat` spans. Only the spans of individual attempts carry usage, so a retried request's parent span is never double-counted.

* `gen_ai.usage.input_tokens` already includes tokens read from and written to the prompt cache. Don't add the cache attributes to it.
* Spans can arrive more than once after a retry. If your platform doesn't remove duplicates, count each span once by trace ID and span ID.

**To cost a session or run from traces**, sum `cosmos.credits` or `cosmos.cost_usd` over its `chat` spans. Like usage, they're on attempt spans only. The `cosmos.api.request` and `cosmos.api.error` log events carry the same values per request.

Group by `gen_ai.response.model` rather than `gen_ai.request.model`: a request for the default model has no requested model, and a request can be served by a different model than the one requested.

## Build dashboards

Useful panels, mostly from metrics:

| Panel | Query |
| - | - |
| Model latency, P50 / P90 / P99 | `gen_ai.client.inference.duration` by `gen_ai.response.model` |
| Time to first token | `gen_ai.client.inference.time_to_first_chunk` by model |
| Slowest tools | `gen_ai.execute_tool.duration` by `gen_ai.tool.name` |
| MCP server health | `mcp.client.operation.duration` by `server.address` and `error.type`; `cosmos.mcp.connection.failures` by `cosmos.mcp.server.name` |
| Agent invocations by name | `gen_ai.invoke_agent.duration` by `gen_ai.agent.name`. This includes subagents as well as runs: runs carry the expert's name, subagents their own name such as `explore`. For runs only, use `invoke_agent` run spans or `cosmos.agent.run` events, which you can group by `gen_ai.agent.id` |
| Run duration and error rate | `cosmos.agent.run` log events (`cosmos.duration_ms`, `cosmos.success`), or run spans. `gen_ai.invoke_agent.duration` mixes in subagents |
| System prompt changes | From traces only: distinct `cosmos.system_prompt.hash` values on `chat` spans over time. The hash isn't in metrics or log events, and `chat` spans don't carry the expert, so to break it down by expert, join each `chat` span to its run span in the same trace, which carries `gen_ai.agent.id` |
| Token and cost trend | `cosmos.token.usage`, `cosmos.cost.usage` |
| Export health | `otel.sdk.exporter.span.exported` by `error.type` |

Metrics are cumulative and sent every 60 seconds from each Cosmos server instance (`service.instance.id`). Use rate or increase functions and sum across instances. See [Aggregating metrics](https://docs.augmentcode.com/cosmos/trace-export/reference.md#aggregating-metrics).

## Set up alerts

Every failed span has status `ERROR` and an [`error.type`](https://docs.augmentcode.com/cosmos/trace-export/reference#error-types).

| Alert when | Query |
| - | - |
| An MCP server's credentials are rejected | `cosmos.mcp.connection.failures` with `error.type` = `mcp_auth`, grouped by `cosmos.mcp.server.name` and `gen_ai.agent.id`. Or `initialize <server>` spans or `cosmos.agent.setup` events with that `error.type` |
| Any MCP server fails to connect | An increase in `cosmos.mcp.connection.failures`, or `error.type` starting with `mcp_` on `initialize` or `mcp_server_unavailable` spans |
| A scheduled run fails | `cosmos.agent.run` log events, or `invoke_agent` run spans, with `cosmos.trigger` = `schedule` and a failure (`cosmos.success` = `false`, or status `ERROR`) |
| The startup script fails | `startup_script` spans with `error.type` = `startup_script_failed` |
| An organization hits a usage limit | `error.type` = `usage_limit_reached` or `credits_depleted` on `invoke_agent` spans |
| The model is rate limited | `chat` spans with `error.type` = `rate_limited` |
| Model latency regresses | P99 of `gen_ai.client.inference.duration` above your threshold |
| Tool or MCP calls start failing | `gen_ai.execute_tool.duration` or `mcp.client.operation.duration` points with `error.type` set. For MCP auth failures during tool calls, match both `auth_error` and `401`/`403` |
| Cosmos can't deliver to your platform | `otel.sdk.exporter.span.exported` with `error.type` set, or the [destination status](https://docs.augmentcode.com/cosmos/trace-export/delivery.md#destination-status) |

### Example: an expert's MCP credentials expired

An expert runs every night on a schedule and uses a Linear MCP server. Its Linear token expires.

1. In the next session, the `initialize linear` span fails with `error.type` = `mcp_auth` and the status message "MCP server rejected the credentials; reconnect or re-authorize it."
   The span carries the expert's `gen_ai.agent.id` and `cosmos.trigger` = `schedule`.
2. With **Send metrics** on, `cosmos.mcp.connection.failures` goes up by 1 for `error.type` = `mcp_auth`, `cosmos.mcp.server.name` = `linear`, the expert's `gen_ai.agent.id` and `cosmos.trigger` = `schedule`.
3. With **Send logs** on, a `cosmos.agent.setup` event with severity `ERROR` carries the same values and `cosmos.setup.step` = `initialize linear`.
4. An alert on the metric or the event, grouped by server and expert, says which expert's scheduled run lost which server. No join is needed.

## Send data to more than one place

Each destination receives its own copy, so you can, for example, send metrics to Datadog and full content to Langfuse. To send to many backends, filter or transform the data, or reach a private backend, run your own [OpenTelemetry Collector](https://docs.augmentcode.com/cosmos/trace-export/providers.md#your-own-collector) as a destination and route from it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.