Alyx

​

Alyx meets you where you are

​

Getting Started

​

How to open Alyx

• macOS: Cmd+L
• Windows / Linux: Ctrl+L

​

Adding context to Alyx

• Highlight - Select any text on the page (a span attribute, an error message, a prompt snippet) then press Cmd+L on macOS or Ctrl+L on Windows/Linux. Alyx opens and adds the selected text to your message. If nothing is selected, the shortcut just opens or closes Alyx.
• Mention - Type @ in the Alyx input to open a menu. Mention a dataset, experiment, project, or span and Alyx receives the IDs so it can scope the conversation to that data.
• Type it - Include context directly in your message. Reference a trace, dataset, or experiment by ID, or start with “Additional context:” and add what Alyx needs to know.

​

Alyx Settings

​

Alyx across your workflow

• Observe
• Improve
• Evaluate

A single trace tells you something failed. It doesn’t tell you why or how widespread it is. Alyx is grounded in live trace and span context including inputs, outputs, tool calls, errors, and your selected time range, so you can go from noticing a problem to understanding its scope in one conversation.

​

Where to find Alyx

On the Traces page, in the trace slideover, and as AI Search in the traces table filter bar (view and manage traces).

​

Skills

Theme	Skill	Description
Diagnose & understand	Trace preview	Get an overview of the trace structure and span hierarchy
	Find in trace	Search for specific content across the trace
	Display query params	Show the current filter and time range
	Span data	Inspect detailed input/output, latency, and attributes for any span
Assess quality	Annotations	Create configs and annotate single or multiple spans with labels
	Build eval	Write a custom LLM-as-a-judge eval for your use case
	Choose evals	Select or attach evals; list dataset evals and online tasks
Curate training data	Create dataset from spans	Create a dataset from selected trace spans
	Append spans to dataset	Add spans from this trace to an existing dataset
	Synthetic datasets	Create or append synthetic data for testing
Version & reuse prompts	Prompt Hub	List, load, and save prompts; save new versions
Find anything fast	Search	Build filters or find spans via natural language

​

Example prompts

• “What’s causing the latency in this trace?”
• “Annotate this span as a hallucination”
• “Build an eval to check if the response answers the question”
• “Find all spans that call the search tool”
• “Create a dataset from the spans with errors”

Prompt iteration involves more than writing prompts. It means attaching datasets, mapping columns, wiring evals, running experiments, and keeping track of what changed between runs. Alyx holds the full context of your prompt, dataset, evals, and experiment history in one thread so each iteration builds on the last.

​

Where to find Alyx

Alyx is available in the Prompt Playground, and on Datasets and Experiments for comparing runs, inspecting eval breakdowns, and scaling what you learned from traces into repeatable tests.

​

Skills

Theme	Skill	Description
Set up your data	List datasets	List available datasets
	Dataset preview	Inspect dataset structure and sample rows
	Attach dataset / select columns	Attach a dataset and choose which columns to use
Curate training data	Create synthetic dataset	Create a new dataset with synthetic examples
	Append synthetic data	Add synthetic examples to an existing dataset
	Create dataset from spans	Build a dataset from trace spans
	Append spans to dataset	Add spans from traces to an existing dataset
Iterate on prompts	Create or edit prompt	Modify the prompt in the playground
	Optimize or align prompt	Improve prompt based on eval performance or goals
	Load / save prompts	Load from hub, save to hub, save new versions
Assess quality	Build eval	Write a custom eval for your application
	Choose evals	Select or attach evals; list evals attached to datasets
Run & compare	Run experiment	Execute an experiment; get results and compare runs
	Get experiments	Fetch experiment data for analysis and comparison

​

Example prompts

• “Attach the regression-test dataset and use only the question and answer columns”
• “Create a synthetic dataset with 100 examples for regression testing”
• “Create a synthetic dataset with 50 examples that test edge cases”
• “Optimize this prompt to reduce hallucinations”
• “Save this prompt to the hub as a new version”
• “Build an eval to check if the response is helpful”
• “Which experiment has the best hallucination score?”
• “Summarize the eval results across these experiments”
• “Run an experiment and compare the results to the last run”
• “Analyze and compare the outputs of my last two experiments”
• “What’s the difference in token usage between experiment A and B?”

Setting up an eval means picking a template, mapping variables, and wiring a data source before you’ve even started thinking about what good looks like. Alyx understands your project or experiment and builds the eval for you, and even maps variables in a way that makes sense for your data.

​

Where to find Alyx

In the eval building, and from the trace slideover or trace page when you want to turn a bad output into a guardrail without starting from a blank template.

​

Skills

Theme	Skill	Description
Target the right data	List datasets / experiments	List datasets and projects for task targeting
	Dataset preview	Inspect structure and sample rows to choose columns
Build eval	Build eval	Write an eval based on your goals and data structure
	Create / update eval form	Configure eval parameters and columns
Use eval	Configure dataset task	Set dataset, filters, and sampling for a dataset-based task
	Configure project task	Set project, filters, and sampling for a trace-based task
	Choose evals	Select or attach evals
	Propose task name	Suggest a descriptive name for the task

​

Example prompts

• “Build an eval that checks if the response answers the question”
• “Change this task to use the customer-support dataset”
• “Point this task at the production traces project”
• “What columns should I use for input and output?”
• “Update the sampling rate to 10%“

​

Use your own integrations with Alyx

​

Data privacy

​

Third-party integrations