> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Explore Your Traces
> Search, filter, and organize your traces, then analyze failures across many traces to find and fix error modes
Your app is instrumented and traces are flowing in. Every request your app handles is now captured, but you might have thousands of them. This page walks you through how to find the ones that matter, understand what happened inside them, organize your workflow, and analyze failures across many traces to find and fix recurring error modes.
Open your project and you'll see several tabs, each giving you a different lens on your data:
### The Traces tab
This is where most investigations start. The **Traces** tab shows every request your app has handled. Click any trace to expand its span tree and see exactly what happened: which components were called, what inputs and outputs flowed through, how long each step took, and where things went wrong.
### The Spans tab
Sometimes you need to look across all traces at a specific type of operation -- all LLM calls, all tool invocations, or all retrievals. The **Spans** tab lets you do that, showing individual units of work across your entire project.
### The Sessions tab
Your users don't send one-off requests -- they have conversations. The **Sessions** tab groups traces into conversation threads by `session.id`, so you can follow an entire interaction from start to finish and spot where things go off track.
* Sort and filter sessions by duration, total traces, user ID, or token count
* Click into a session to see the full conversation with inputs and outputs for each turn
* Export session data to CSV or via the Python SDK
* Run session-level evals to detect patterns like user frustration or unresolved issues
`session.id` must be a non-empty string. Spans with an empty session ID won't appear in the Sessions tab.
## Token counts
Token usage directly impacts cost. Arize AX automatically computes token counts for LLM spans using the `llm.token_count.prompt`, `llm.token_count.completion`, and model name attributes, and aggregates them up to the trace level.
In the span tree, each span shows its total token count. Hover over the number to see a breakdown of prompt tokens and completion tokens, along with how they were computed (user-provided, tiktoken, or character estimate).
From here you can sort traces or sessions by token count to find expensive requests, or define [custom metrics](/docs/ax/observe/projects/custom-metrics-api) and [monitors](/docs/ax/observe/production-monitoring) on token usage for cost control.
***
## Querying and filters
Once you know what you're looking for, use filters to narrow down. Click the filter bar to start querying -- you can type, select from suggestions, or use [AI Search](/docs/ax/alyx/using/search-bar-agent) to generate filters from natural language.
**Let Alyx build the filter for you.** Press **Cmd+L** (macOS) or **Ctrl+L** (Windows/Linux) to open [Alyx](/docs/ax/alyx) and describe what you want in plain language. Try: *"Find tool spans from user X in the last day"*
* Format: `"dimension name" = 'value'`
* Operators: `=`, `!=`, `<`, `>`, `IN`, `contains`
* Null checks: `"attributes.input.value" = null` or `is null`
* Combine with `AND` / `OR`
* Example: `"attributes.openinference.span.kind" = 'LLM' AND "status_code" = 'ERROR'`
### Multi-span filters
Need to find traces based on complex patterns -- like traces where an LLM call follows a retrieval, or traces with errors but no tool calls? Create multiple span queries and define relationships between them.
1. Define your first **Span Query**
2. Click **Add Span Query** (up to 5)
3. Use **Find Traces Matching** to combine them with operators: `AND`, `OR`, `NOT`, `->` (temporal), `=>` (parent-child)
* **AND** -- traces with spans matching both A and B
* **OR** -- traces with spans matching A or B
* **NOT** -- traces without spans matching the query
* **->** (Indirectly Calls) -- A starts before B
* **=>** (Directly Calls) -- A is the direct parent of B
* Combine with parentheses: `(A AND B) OR C`, `A AND NOT(B -> C)`
### Time filters
Use the **Time Selector** to focus on the period that matters. Choose presets, use the calendar, or type a custom range directly.
* **Presets:** Last 15 Min, Last Hour, Last 12 Hours, Last 7 Days, Last Month, Last 3 Months, Last 6 Months
* **Custom range:** Select "Custom" to pick a specific start and end date/time
* **Time zone:** Set your preferred timezone; it persists across sessions
***
## Saved Views
Found a useful filter + column setup? Save it so you don't have to rebuild it every time. Saved Views preserve your exact Tracing table configuration -- filters, columns, sort order, and time range -- so you and your team always come back to the same starting point.
Arize AX provides two built-in views: **Arize Default** (no filters) and **Recent Errors (15m)** (pre-filtered to `status_code = 'ERROR'`). You can't overwrite them, but you can fork them with **Save as new view**.
To create a view: set up the table how you want it (filters, columns, sort, time range). A **Save** button (disk icon) appears in the toolbar when you have unsaved changes. Click it and choose **Save as new view**. Views are shared per project. Each user can set their own **personal default** via the view's actions menu.
| Setting | Stored |
| -------------- | ------------------------------------ |
| **Filter** | Full query including subqueries |
| **Columns** | Visible columns and order |
| **Sort** | Column and direction |
| **Time range** | Preset or custom window and timezone |
***
## Columns
Not every column is relevant to every investigation. Customize which columns are visible using the **Columns** button in the table toolbar. Columns are organized into groups:
* **Span columns** -- TraceID, SpanID, Input, Output, Start/End Timestamp, Total Tokens, Status
* **Span Evaluations** -- scores and labels from your span-level evals
* **Session Evaluations** -- scores and labels from your session-level evals
* **Annotations** -- human labels and scores from your team
Toggle individual columns or select/deselect entire groups. Your column configuration is saved when you create a [Saved View](#saved-views).
## Source mapping
Every framework stores user input and model output in different span attributes. **Source mapping** tells Arize AX which fields to treat as **Input** and **Output** for a project, so the Tracing table, filters, exports, and evaluators all read from the same place.
Open your project → **Config** → **Source Mapping**. For each field, enter a dot-notation path into the span's attribute tree, or leave it blank to use Arize's defaults.
| Field | Example path | When to use it |
| :--------- | :------------------------------- | :---------------------------------------- |
| **Input** | `my_app.user_query` | Your question lives on a custom attribute |
| **Output** | `output.value.final_answer.text` | The answer is nested inside JSON |
| **Input** | `input.value.actions.-1` | You want the last item in an array |
Paths are relative to OpenInference attributes; omit the `attributes.` prefix. Use `.` between segments; use numeric indices for arrays (`0`, `-1` for last element).
**What changes after you save**
* New spans get human-readable **Input** and **Output** columns derived from your paths
* You can filter with `input` / `output` instead of long attribute names (`input contains 'refund'`)
* Evaluator tasks can map `{input}` and `{output}` directly
Source mapping applies to **new spans only**. Existing data is not backfilled. Plan mapping before heavy traffic, or accept that history and new data may differ.
For attribute naming in code, see [Customize your traces](/docs/ax/instrument/customize-your-traces). For mapping variables in eval tasks, see [Run online evals on traces (Filters)](/docs/ax/evaluate/run-evals-on-traces#filters).
***
## Export traces
Need the data outside Arize? Export trace data for evaluations, fine-tuning, or further analysis.
* **Export to Notebook** -- generates a prefilled Python snippet using the Arize SDK to export your data to a DataFrame. Copy it into a Jupyter notebook to start analyzing.
* **Export as CSV** -- downloads the top 10,000 rows currently displayed. Customize columns before exporting.
* **Configure Export via Agent (Skills)** -- opens the Arize Skills setup for exporting via your coding agent.
* **Configure Export via Code** -- shows CLI, Python SDK, and REST API snippets for programmatic export.
### Fetch a trace by trace ID
A trace is the set of all spans that share the same `context.trace_id`. To fetch a single trace programmatically, query spans with a trace ID filter using the [CLI](/docs/api-clients/cli/spans), the [Python SDK](/docs/api-clients/python/version-8/client-resources/spans), or the [REST API](/docs/ax/rest-reference/overview).
```bash CLI theme={null}
ax spans export \
--trace-id \
--start-time "2024-01-01T00:00:00Z" \
--end-time "2024-01-02T00:00:00Z" \
--stdout
```
```python Python SDK theme={null}
resp = client.spans.list(
project="",
filter="context.trace_id = ''",
)
for span in resp.spans:
print(span.span_id, span.name)
```
```bash REST API theme={null}
curl -X POST "https://api.arize.com/v2/spans?limit=500" \
-H "Authorization: Bearer $ARIZE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project_id": "",
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-02T00:00:00Z",
"filter": "context.trace_id = '\'''\''"
}'
```
The REST API is span-shaped, so a trace-level fetch means filtering spans by `context.trace_id`. All spans in the response share that trace ID.
***
## Analyze traces for error modes
You can explore individual traces and see what happened in a single request. But any non-trivial LLM app (agent, RAG, or chatbot) can fail in many different ways on the same input. When something goes wrong, the problem could be anywhere in the chain.
A single failing trace is a data point. It can surface a pattern you actually want to fix. Three approaches get you from failures to labeled error modes: **Arize Skills** in your coding agent, **Alyx** in the product, or **manual** annotation. Skills and Alyx use AI to produce categories directly; the manual approach walks through annotation and clustering step by step.
Run the `arize-trace` skill in your AI coding agent (Claude Code, Cursor, Codex, Windsurf, and 40+ others) to export traces and analyze them locally.
**Install skill**
```bash theme={null}
npx skills add Arize-ai/arize-skills --skill "arize-trace" --yes
```
**Set up authentication**
```bash theme={null}
export ARIZE_API_KEY="YOUR_API_KEY"
export ARIZE_SPACE_ID="YOUR_SPACE_ID"
```
**Ask your agent**
```
# Ask your AI coding agent:
"Export the last day of traces with errors from my project
and tell me what's going wrong"
```
Other prompts to try: *"find me traces that have long latency, and summarize what they have in common"*, *"why are these traces failing"*, *"group last night's errors by root cause"*. The skill runs `ax traces export` under the hood; your agent reads the output and reports patterns.
**What you get back:** a written summary of common patterns across the exported traces. Fast to run. Best for quickly spotting *what* is going wrong before you commit to deeper analysis. Works with Cursor, Claude Code, Codex, and more.
**Alyx** is the AI assistant built into the product. Open it from the Traces page or a trace detail view (or press **Cmd+L** / **Ctrl+L**) and ask for filters or categories directly:
* "What are the most common error types in the last 24 hours?"
* "Find me the common types of questions users are asking"
* "Filter for LLM spans with latency over 2 seconds"
* "Group failing traces by root cause"
* "Why did this trace fail?"
* "Create a dataset from the filtered spans"
You can also type natural language directly in the filter bar, Alyx converts it to query syntax automatically.
**What you get back:** a **Categorized Spans** widget with category names, counts, short descriptions, and example spans you can click into. You can also apply the categories back to your spans as annotations. Great for a fast, structured first pass across large volumes of traces.
This is the deepest approach. A small pipeline: **traces in → human annotations → named error modes → an evaluator out**. Skills and Alyx give you fast coverage; manual annotation gives you categories you can stake a fix on.
Filter to failures in the **Traces** tab above. Plan to read 20–50 failing traces.
**Output:** a filtered list of failing traces to work through.
For each trace, write a short free-form note about what went wrong. Don't force it into buckets yet; just capture what's weird. Examples: *"not specific enough"*, *"made up info"*, *"too verbose"*, *"called wrong tool"*.
Use [Human Annotations](/docs/ax/evaluate/human-annotations) to capture these notes directly on the traces. Once patterns emerge, formalize them as labels in an annotation config and apply them to matching traces.
**Output:** a set of **human annotations** attached to real traces, the raw material for clustering.
Group your labeled annotations into error modes. Several notes about bad hotel IDs might cluster into a named pattern such as `invalid_hotel_id`. Expect 3–5 such patterns in total. Others could be `tool_call_hallucination`, `retrieval_miss`, or `agent_loop`.
Track the rate of each error mode as a [custom metric](/docs/ax/observe/projects/custom-metrics-api) so you can see whether a fix actually lowers it.
**Output:** 3–5 named error modes, each tracked as a **custom metric** so you can watch the rate over time.
For each labeled error mode, write an eval that catches it automatically (LLM-as-judge or a code check). The labeled traces become your regression suite. See [Evaluators](/docs/ax/evaluate/evaluators).
**Output:** one **evaluator per error mode**, with the labeled traces as its regression suite.
Skills and Alyx are the fastest way to do a first pass across large volumes of traces. Manual annotation takes longer but typically produces sharper, more reliable categories. Use it on the error modes that matter most.
For scheduled, always-on issue detection from production traces, see [Surface issues with Signal](/docs/ax/observe/signal).
## Fix the error mode
If you used Skills or Alyx, pick the most impactful pattern from the output and give it a name (e.g., `invalid_hotel_id`). Then validate a fix before shipping:
* **Save failing traces to a dataset**: those traces become your regression suite for future prompt tweaks, retrieval updates, tool changes, or any other fix.
* **[Run an experiment](/docs/ax/develop/datasets-and-experiments)**: compare the old vs new version against the dataset; ship the fix only if the experiment shows improvement without regressions on other examples.
After shipping, track the rate of this error mode as a [custom metric](/docs/ax/observe/projects/custom-metrics-api) and put a [monitor](/docs/ax/observe/production-monitoring) on it so you're alerted if it comes back.
***
## Next step
You've found your error modes. Now turn their rates into metrics so you can track whether fixes actually move them: