LLM Evaluators
LLM evaluators use a judge model to assess the quality of outputs. These are useful for subjective or nuanced evaluations where simple rules don’t suffice.Faithfulness
Measures whether a response is faithful to (grounded in) the provided context. Detects hallucinations and unsupported claims.
Hallucination
Detects claims in a response that are unsupported by, or contradict, the conversation. The conversation-level counterpart to Faithfulness.
Conciseness
Evaluates whether a response is concise and free of unnecessary content like filler, hedging, and meta-commentary.
Correctness
Evaluates the general correctness of an LLM response.
Retrieval Relevance
Assesses whether externally retrieved information is relevant to the request, from any source: RAG, tools, MCP, or web search.
Tool Selection
Determines whether the correct tool was selected for a given context from the available options.
Tool Invocation
Checks if a tool was invoked correctly with proper arguments, formatting, and safe content.
Tool Response Handling
Evaluates whether an agent correctly processed a tool’s result, including error handling, data extraction, and safe information disclosure.
Refusal
Detects when an LLM refuses, declines, or avoids answering a user query.
User Friction
Detects corrections, retries, frustration, and challenges expressed in a user’s follow-up message.
Toxicity
Detect hateful, demeaning, abusive, or threatening text in model outputs or user inputs.
Code Evaluators
Code evaluators use deterministic logic for evaluation. These are faster, cheaper, and provide consistent results for objective criteria.Exact Match
Checks if the output exactly matches an expected value. Supports optional normalization.
Matches Regex
Validates that output matches a specified regular expression pattern.
Precision / Recall / F-Score
Computes precision, recall, and F1 scores for comparing predicted vs actual values.
Looking to create custom evaluators? See the Building Custom Evaluators guide.