> ## Documentation Index
> Fetch the complete documentation index at: https://docs.augmentcode.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Delivery and troubleshooting

> How Trace Export delivers spans, what each destination status means, and how to fix delivery errors.

> [!NOTE]
>
> This feature is in early access and behind a feature flag. Contact your Augment Solutions Architect to enable it for your tenant

Start with **Troubleshooting** if data isn't arriving. The rest of the page explains each destination status and error, how Cosmos delivers data, and its [capacity limits](#capacity-and-buffering).

## Troubleshooting

| Problem | What to check |
| - | - |
| **No spans arrive** | The destination is enabled and **Healthy**. The session is in **Sessions to export**. Your platform is filtered on `service.name` = `augment-cosmos` and the right time range. Spans can take up to a minute to appear after the settings change |
| **The test fails on metrics or logs** | Your platform may accept only traces at this endpoint, or the key may allow only traces. Turn off **Send metrics** or **Send logs**, or use a key that allows them |
| **Spans have no prompts or responses** | Content levels are off by default. Turn them on in **Content**, and check that the destination doesn't [receive less content](https://docs.augmentcode.com/cosmos/trace-export/content-and-privacy.md#send-less-content-to-one-destination) than the organization (for example after **Turn off tool outputs**) |
| **Content is cut short** | The span exceeded **Max content size** or the 128 KB limit. See [Content limits](https://docs.augmentcode.com/cosmos/trace-export/content-and-privacy.md#content-limits) |
| **A run is missing** | Runs are sent when the agent finishes responding. A run whose agent stopped first ends as `error.type` = `runtime_lost` |
| **Model names don't match my model IDs** | `gen_ai.request.model` and span names use the Cosmos model picker name, such as `Claude Opus 5.5`. `gen_ai.response.model` is the public model ID, such as `claude-opus-5-5`. Group usage and cost by `gen_ai.response.model` |
| **A session span appears twice** | See [The session span can arrive twice](https://docs.augmentcode.com/cosmos/trace-export/use-your-data.md#the-session-span-can-arrive-twice) |
| **Metrics totals jump or reset** | Metrics are cumulative per Cosmos server instance (`service.instance.id`). Aggregate across instances, and use rate or increase functions. Totals restart when the destination's sessions-to-export setting or user identity changes, or after two hours without spans |
| **Metric series labeled `otel.metric.overflow`** | A metric exceeded 2,000 attribute combinations per destination (10,000 with user identity). Further combinations are counted in one overflow series |

## Destination status

The Trace Export page shows each destination's status for the last hour. It can be up to about a minute late.

| Status | Meaning |
| - | - |
| **Healthy** | Spans were delivered in the last hour. Shows when the last span arrived |
| **Waiting** | Enabled, but nothing has been delivered yet. Start a session or send a test span |
| **Idle** | Enabled, and nothing was sent in the last hour |
| **Dropping some spans** | Some spans were dropped in the last hour. The page shows the main reason and what to do |
| **Failing** | The latest delivery attempts failed. Shows how long it has been failing and the cause |
| **Paused** | An admin disabled the destination. Nothing is queued while it's paused |
| **Turned off** | Cosmos disabled it automatically after 24 hours of failed deliveries. See [Automatic disabling](#automatic-disabling) |

The page also shows how many spans Cosmos sent and redacted in the last hour, and each destination's delivered and dropped counts.

### Dropped spans

| Reason | Meaning | What to do |
| - | - | - |
| **Queue full** | Your endpoint was slower than incoming spans, so the oldest queued spans were dropped | Short bursts are normal. If it continues, check your platform's ingest limits |
| **Retries exhausted** | The endpoint kept failing or timing out for the whole retry window | Check **Recent requests** for the errors |
| **Rejected** | The endpoint refused the request, or rejected spans in its response | Check credentials and ingest limits. **Recent requests** shows the HTTP status, and for partial rejections one of four categories (attribute limits, timestamps out of range, rate or quota limit, invalid span data). For the vendor's own message, check its or your collector's logs |
| **Too large** | The endpoint rejected a span with HTTP 413 even when sent alone | Lower **Max content size**, or stop sending tool inputs and outputs to this destination |
| **Rejected at ingest** | The span failed validation before export | Contact support if this continues |
| **Dropped by agent**, **Dropped by Cosmos** | Spans dropped under load before export | No action needed |
| **Shutdown** | Spans dropped during a Cosmos restart | No action needed |

### Recent requests

**View requests** shows the destination's last 20 delivery attempts from the last 24 hours, newest first. Each shows the time, outcome (OK, partial or failed), HTTP status or error, span count, compressed request size, latency and attempt number. It never shows span content.

## Delivery errors

| Error | Likely cause |
| - | - |
| `http_401`, `http_403` | A wrong or expired API key, a key from another region or environment, or a key that doesn't allow this signal |
| `http_404`, `http_405` | A wrong path. Most OTLP endpoints end in `/v1/traces` |
| `http_408` | The endpoint timed out |
| `http_413` | The request was too large. Lower **Max request size**, or **Max content size** for single large spans |
| `http_415` | The URL isn't an OTLP/HTTP protobuf endpoint |
| `http_429` | The endpoint is rate limiting. Try later, or raise the limit with the vendor |
| `http_530` | The vendor's edge couldn't route the request, usually a wrong region in the host name |
| `http_5xx` | The endpoint returned a server error |
| `http_3xx` | The endpoint redirected the request. Use the URL it redirects to |
| `spans_rejected` | The endpoint accepted the request but rejected every span. Check the vendor's ingestion settings |
| `invalid_url` | The URL isn't a valid public HTTPS URL |
| `blocked_address` | The host resolves to a private or reserved address |
| `dns` | The host name doesn't resolve. Check for typos, including the region |
| `connect` | Cosmos couldn't connect. Check the port, and that the endpoint accepts connections from the internet |
| `proxy` | Augment's outbound proxy couldn't reach the endpoint |
| `tls` | The TLS handshake failed. The certificate must come from a public certificate authority and match the host name |
| `timeout` | The endpoint didn't respond within 10 seconds |
| `network` | A network error interrupted the request |

## Automatic disabling

Cosmos turns a destination off when all its deliveries have failed for 24 hours, with at least 20 failed attempts. Failures from before the destination was last changed don't count. While a destination is failing, the page shows the time left before it turns off.

While a destination is turned off, its spans are dropped and aren't sent after it's enabled again. To resume:

1. Fix the cause, which the destination shows.
2. Click **Test and re-enable**. The destination is enabled only if the test passes. If it fails, the form opens on the field to fix.

## How spans are delivered

| Behavior | Details |
| - | - |
| **Protocol** | OTLP/HTTP, `POST`, `Content-Type: application/x-protobuf` |
| **Compression** | gzip, with `Content-Encoding: gzip`. Databricks AI Gateway endpoints get uncompressed requests |
| **User agent** | `User-Agent: augment-cosmos-trace-export/1` |
| **Headers** | Your destination's headers on every request. Cosmos sets `Content-Type`, `Content-Encoding`, `Content-Length`, `Host`, `User-Agent`, `Connection`, `Transfer-Encoding`, `TE`, `Upgrade`, `Proxy-Authorization` and `Proxy-Connection` itself, so you can't use those names |
| **When** | Spans are sent when they end. Cosmos batches up to 512 spans per request and sends a batch within about a second |
| **Request size** | A batch whose compressed body is larger than **Max request size** (4 MB by default), or that the endpoint rejects with HTTP 413, is split in half and sent again. A single span is never split |
| **Timeouts** | 5 seconds to connect and 10 seconds for each request |
| **Redirects** | Not followed. A `3xx` response is a failed delivery; use the URL it redirects to |
| **Best effort** | A span can be lost, for example when your endpoint fails for longer than the retry window or a queue is full. Lost spans aren't sent again |
| **Duplicates possible** | A span can arrive more than once, for example when your endpoint received a request but Cosmos didn't get the response and retried. Remove duplicates by trace ID and span ID |
| **Partial success** | If your endpoint returns OTLP `partial_success` with rejected spans, those spans are counted as rejected and the rest as delivered |

### Retries

Cosmos retries a delivery after a timeout, connection or DNS failure, a network error, or an HTTP `408`, `429`, `502`, `503` or `504` response. Retries use exponential backoff with jitter, from 0.5 seconds up to 10 seconds, and follow a `Retry-After` header of up to 30 seconds. Retries for one batch stop after 5 minutes, and retries for a destination are limited to about one for every ten requests, so retries can't overload an endpoint that's already failing.

Cosmos doesn't retry other responses, such as `400`, `401`, `403`, `404` or `500`, or a TLS failure or blocked address. Those spans are dropped as **Rejected**.

When an endpoint keeps failing, Cosmos pauses sending to it for 30 seconds at a time, then tries one request. It also sends fewer requests in parallel when an endpoint rate-limits, times out or returns server errors, and more when it keeps up.

### Metrics and logs delivery

* **Metrics** are cumulative and sent every 60 seconds. A failed metrics request isn't retried, because the next one carries the running totals.
* **Logs** are sent with the spans they describe and retried like traces. When an endpoint is too slow, log events are dropped rather than delaying traces.

## Capacity and buffering

Trace Export is built so that a slow or failing destination can't slow down your agents or other destinations. That means data is dropped, not delayed, when a limit is reached. Drops caused by delivery problems are counted on the Trace Export page and in the metrics.

Changing settings can also drop data that isn't shown on the page. When you turn off a content level or user identity, lower **Max content size**, change resource attributes or **Sessions to export**, or remove or disable a destination, batches already queued or being retried under the old settings are dropped rather than re-encoded. These drops appear only in `otel.sdk.exporter.span.exported`, as `policy_changed` or `destination_removed`.

### No durable queue

Spans are buffered only in memory, on the agent and in Cosmos. Nothing is written to disk, and there is no dead-letter queue. Spans are lost, and not sent again, when:

* a destination is down or failing for longer than its buffer or the 5-minute retry window;
* a Cosmos server restarts while spans are still queued (counted as **Shutdown**);
* a destination is disabled, turned off automatically, or deleted.

If you need every span, send to your own [OpenTelemetry Collector](https://docs.augmentcode.com/cosmos/trace-export/providers.md#your-own-collector) with a persistent queue, and forward from there.

### How much outage is buffered

Each destination has its own queue on each Cosmos server, of up to 65,536 spans or 32 MB, within 128 MB shared by all destinations on that server. When a queue is full, its oldest spans are dropped (**Queue full**). When the shared memory is full, the destination whose oldest span has waited longest gives up memory first, so a stalled destination doesn't take space from healthy ones.

How long that lasts depends on your volume and content settings:

* **Metadata only:** spans are small, so a short outage is usually absorbed.
* **With content levels on:** spans are much larger. In Augment's load test at 10,000 concurrent sessions with content on, the buffer covered about 15 seconds of destination outage. A 60-second outage dropped about 15% of spans.

### Metrics and logs don't multiply the load

Turning on metrics and logs doesn't record anything three times. Each span is received and queued once, and the three signals are split off only when sending:

| Signal | When it's sent | Grows with | If the destination is slow |
| - | - | - | - |
| **Traces** | Batches of up to 512 spans or 4 MB, within about a second | Span volume | Up to 16 requests in flight per destination, then the queue fills |
| **Logs** | With each trace batch, about one extra request | Span volume | Up to 4 requests in flight per destination. Log events that don't fit are dropped and never delay traces |
| **Metrics** | One request every 60 seconds | Fixed, whatever the volume | Not retried; the next request carries the running totals |

### Metric series

Metrics are cumulative per Cosmos server instance, so your platform receives one copy of each series per instance (`service.instance.id`), and sees more series when Cosmos scales out. Each metric holds at most 2,000 attribute combinations per destination and instance (10,000 for combinations with a user). Further combinations go to one series with `otel.metric.overflow` = `true`. Sum across instances in your queries.

### Overload on the Cosmos side

* **Agents:** each agent buffers at most 1,000 spans or 8 MB and never waits on tracing. If uploads fail, it backs off and drops its oldest spans first (**Dropped by agent**). A single span larger than 1 MB is dropped.
* **Cosmos:** when Cosmos is overloaded, it refuses uploads with a retryable error rather than slowing down. Agents send them again later. Spans that still can't be accepted are counted as **Dropped by Cosmos**.
* **Your sessions aren't affected.** Trace export runs separately from chat. In the 10,000-session load test, response times and time to first token matched runs without export.

### Single large spans

A batch is split in half until it fits **Max request size**, but one span can't be split: it's sent on its own even if it's larger. It's dropped as **Too large** only if the endpoint rejects it with HTTP 413. With content on, keep **Max content size** well below **Max request size**.

## Limits

| Limit | Value |
| - | - |
| Destinations per organization, including disabled ones | 5 |
| Request headers per destination | 10 |
| Header name | A valid HTTP header name (letters, digits and common symbols such as `-` and `_`), at most 256 characters |
| Header value | 8 KB, no control characters |
| Traces URL | 2,048 characters, `https`, public host, no user info or fragment |
| Destination name | 100 characters |
| Resource attributes per destination | 8 (key 128 characters, value 256 bytes) |
| Max request size | 64 KB to 4 MB (default 4 MB) |
| Content per span | About 120 KB captured. Max content size: 1 KB to 128 KB (default: everything captured) |
| Experts in **Only selected experts** | 200 |
| Connection tests per organization | 10 per minute |
| Spans per request | 512 |
| Queue per destination, per Cosmos server | 65,536 spans or 32 MB (128 MB shared by all destinations) |
| In-flight requests per destination | Traces up to 16; logs 4 |
| Metric series per metric, per destination and server | 2,000 (10,000 with user identity) |
| Agent buffer | 1,000 spans or 8 MB; spans over 1 MB are dropped |
| Attributes per span | 64 |
| Metadata text attribute | 256 characters |
| Retry window | 5 minutes |
| Automatic disabling | After 24 hours and 20 failed attempts |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.