Skip to main content
Your automated evals say a response is “grounded” — but is it really? Sometimes you need a human to weigh in. Annotations let your team add ground-truth labels and scores directly on spans, building a feedback loop between humans and your AI.

​
How to do it

  1. Open a trace and click into any span
  2. Click the Annotate toggle in the span toolbar
  3. Select an annotation config (e.g., “Correctness”, “Helpfulness”) or create a new one
  4. Add your label or score — saves automatically

​
Annotation levels

Annotations apply to a single span, a whole trace, or a whole session. Use a span-level annotation when you are judging one operation, such as whether a retrieved document was relevant. Use a trace-level annotation when the judgment concerns one complete request, such as end-to-end response quality. Use a session-level annotation when the judgment covers a multi-turn conversation, such as whether the user reached their goal across every turn. To annotate a session, open it from the Sessions view in your project, click the Annotate Session toggle in the session toolbar, then pick an annotation config in the annotation pane. Session labels and scores use the session_annotation prefix, so you can filter on them from the query filter dropdown and read them as columns in the sessions table, the traces table, and the dataset examples table.

​
Annotation configs

Configs define the schema for your labels. Shared across the project so everyone uses the same schema.
  • Categorical — fixed labels (e.g., “correct”, “incorrect”, “partially correct”)
  • Continuous — numeric scores on a range (e.g., 1–5)
Create new configs on the fly from the annotation panel: click + New Config, choose type, add options, save.

​
Annotation notes

In addition to labels and scores, you can attach free-text notes to any annotation. Notes are useful for explaining edge cases, providing context for disagreements, or flagging spans for follow-up discussion.

​
Measure eval quality with annotations

Use annotations as ground truth to measure how well your automated evals perform:

​
Annotations vs. evals

Use both: evals for scale, annotations for accuracy.