Classification

Use the classification helpers when you want an LLM to choose from a fixed set of labels and return a structured explanation.

Create A Classifier Function

import { openai } from "@ai-sdk/openai";
import { createClassifierFn } from "@arizeai/phoenix-evals";

const classify = createClassifierFn({
  model: openai("gpt-4o-mini"),
  choices: { relevant: 1, irrelevant: 0 },
  promptTemplate:
    "Question: {{input}}\nContext: {{context}}\nAnswer: {{output}}\nLabel as relevant or irrelevant.",
});

const result = await classify({
  input: "What is Phoenix?",
  context: "Phoenix is an AI observability platform.",
  output: "Phoenix helps teams inspect traces and experiments.",
});

Decision Models

Any classification evaluator, including every built-in, can run on an AI SDK decision model such as OpenAI’s gpt-6-luna (Decisions API) or TypeSafe’s Jev. Decision models pick one of the labels directly and return a probability for each, instead of generating text. They are much faster and cheaper than a chat model judge. With OpenAI, use @ai-sdk/openai 4.0.89 or later.

import { openai } from "@ai-sdk/openai";
import { createCorrectnessEvaluator } from "@arizeai/phoenix-evals";

const correctness = createCorrectnessEvaluator({
  model: openai.decisionModel("gpt-6-luna"),
});

const result = await correctness.evaluate({
  input: "What is the capital of Australia?",
  output: "Sydney.",
});
// {
//   label: "incorrect",
//   score: 0,
//   metadata: {
//     probabilities: { correct: 0.02, incorrect: 0.98 },
//     confidence: 0.96,
//     modelId: "gpt-6-luna",
//   },
// }

How it differs from a language model:

No explanation. Decision models don’t generate text, so explanation is always undefined.
Probabilities in metadata. metadata.probabilities maps each label to its probability. metadata.confidence is included when the provider reports one (OpenAI and TypeSafe do).
Prompt shape. The template is rendered the same way as for a language model, then sent as one plain-text state. Message roles are dropped and only text content is sent.
Choices. Each provider sets its own limits on how many choices a question can have. A classification outside them fails with the provider’s error.
Refusals. A decision model may refuse to classify, for example when asked to infer sensitive attributes rejected by the chosen model. On a refusal, the evaluator throws the AI SDK’s Experimental_DecisionRefusalError. In runExperiment, it’s recorded as that evaluation’s error.
Plain model IDs. A string model ID is always treated as a language model. Use the provider’s decisionModel() to get a decision model.

Decision models can’t work through a problem step by step. They do well on judgments you can read off the text, such as relevance, faithfulness, tool selection, and toxicity. They’re unreliable for checking answers that need a calculation, such as unit conversions or arithmetic, and their probabilities can be confidently wrong on those. Use a language model for those evaluations.

Lower-Level API

Use generateClassification directly when you already have a rendered prompt and only need structured label generation. It also accepts a decision model, in which case the rendered prompt is sent as one plain-text state.

Source Map

src/llm/createClassifierFn.ts
src/llm/createClassificationEvaluator.ts
src/llm/generateClassification.ts
src/decision/decideClassification.ts
src/decision/decisionPrompt.ts
src/types/evals.ts

Python

TypeScript

Rest API

OpenInference SDK

Create A Classifier Function

Decision Models

Lower-Level API

Source Map

​Create A Classifier Function

​Decision Models

​Lower-Level API

​Source Map

Create A Classifier Function

Decision Models

Lower-Level API

Source Map