This feature is in early access and behind a feature flag. Contact your Augment Solutions Architect to enable it for your tenant
Troubleshooting
Destination status
The Trace Export page shows each destination’s status for the last hour. It can be up to about a minute late.
The page also shows how many spans Cosmos sent and redacted in the last hour, and each destination’s delivered and dropped counts.
Dropped spans
Recent requests
View requests shows the destination’s last 20 delivery attempts from the last 24 hours, newest first. Each shows the time, outcome (OK, partial or failed), HTTP status or error, span count, compressed request size, latency and attempt number. It never shows span content.Delivery errors
Automatic disabling
Cosmos turns a destination off when all its deliveries have failed for 24 hours, with at least 20 failed attempts. Failures from before the destination was last changed don’t count. While a destination is failing, the page shows the time left before it turns off. While a destination is turned off, its spans are dropped and aren’t sent after it’s enabled again. To resume:- Fix the cause, which the destination shows.
- Click Test and re-enable. The destination is enabled only if the test passes. If it fails, the form opens on the field to fix.
How spans are delivered
Retries
Cosmos retries a delivery after a timeout, connection or DNS failure, a network error, or an HTTP408, 429, 502, 503 or 504 response. Retries use exponential backoff with jitter, from 0.5 seconds up to 10 seconds, and follow a Retry-After header of up to 30 seconds. Retries for one batch stop after 5 minutes, and retries for a destination are limited to about one for every ten requests, so retries can’t overload an endpoint that’s already failing.
Cosmos doesn’t retry other responses, such as 400, 401, 403, 404 or 500, or a TLS failure or blocked address. Those spans are dropped as Rejected.
When an endpoint keeps failing, Cosmos pauses sending to it for 30 seconds at a time, then tries one request. It also sends fewer requests in parallel when an endpoint rate-limits, times out or returns server errors, and more when it keeps up.
Metrics and logs delivery
- Metrics are cumulative and sent every 60 seconds. A failed metrics request isn’t retried, because the next one carries the running totals.
- Logs are sent with the spans they describe and retried like traces. When an endpoint is too slow, log events are dropped rather than delaying traces.
Capacity and buffering
Trace Export is built so that a slow or failing destination can’t slow down your agents or other destinations. That means data is dropped, not delayed, when a limit is reached. Drops caused by delivery problems are counted on the Trace Export page and in the metrics. Changing settings can also drop data that isn’t shown on the page. When you turn off a content level or user identity, lower Max content size, change resource attributes or Sessions to export, or remove or disable a destination, batches already queued or being retried under the old settings are dropped rather than re-encoded. These drops appear only inotel.sdk.exporter.span.exported, as policy_changed or destination_removed.
No durable queue
Spans are buffered only in memory, on the agent and in Cosmos. Nothing is written to disk, and there is no dead-letter queue. Spans are lost, and not sent again, when:- a destination is down or failing for longer than its buffer or the 5-minute retry window;
- a Cosmos server restarts while spans are still queued (counted as Shutdown);
- a destination is disabled, turned off automatically, or deleted.
How much outage is buffered
Each destination has its own queue on each Cosmos server, of up to 65,536 spans or 32 MB, within 128 MB shared by all destinations on that server. When a queue is full, its oldest spans are dropped (Queue full). When the shared memory is full, the destination whose oldest span has waited longest gives up memory first, so a stalled destination doesn’t take space from healthy ones. How long that lasts depends on your volume and content settings:- Metadata only: spans are small, so a short outage is usually absorbed.
- With content levels on: spans are much larger. In Augment’s load test at 10,000 concurrent sessions with content on, the buffer covered about 15 seconds of destination outage. A 60-second outage dropped about 15% of spans.
Metrics and logs don’t multiply the load
Turning on metrics and logs doesn’t record anything three times. Each span is received and queued once, and the three signals are split off only when sending:Metric series
Metrics are cumulative per Cosmos server instance, so your platform receives one copy of each series per instance (service.instance.id), and sees more series when Cosmos scales out. Each metric holds at most 2,000 attribute combinations per destination and instance (10,000 for combinations with a user). Further combinations go to one series with otel.metric.overflow = true. Sum across instances in your queries.
Overload on the Cosmos side
- Agents: each agent buffers at most 1,000 spans or 8 MB and never waits on tracing. If uploads fail, it backs off and drops its oldest spans first (Dropped by agent). A single span larger than 1 MB is dropped.
- Cosmos: when Cosmos is overloaded, it refuses uploads with a retryable error rather than slowing down. Agents send them again later. Spans that still can’t be accepted are counted as Dropped by Cosmos.
- Your sessions aren’t affected. Trace export runs separately from chat. In the 10,000-session load test, response times and time to first token matched runs without export.