> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Decision Judge

> Score project spans or dataset examples with a decision model from TypeSafe AI, OpenAI, Perplexity, or Cloudflare Clef. Typed boolean, choice, and score questions answered in one call, with confidence instead of free text.

**Experimental**

Decision Judge is an experimental feature. The in-product UI and workflow may change as we continue to improve the experience. It was previously called **Jev-as-a-Judge**.

A **Decision Judge** evaluates spans or dataset examples with a **decision model**: a model built for classification, scoring, and routing rather than text generation. You send one shared **state** plus typed **questions**; the model returns a structured label and a confidence (or rubric) score for every question in a single call, with no free-text explanation.

Decision Judge supports four decision model providers: TypeSafe AI, OpenAI, Perplexity, and Cloudflare Clef. You choose one in the evaluator editor's AI Provider dropdown. The same evaluator definition works with any of them.

For when to choose a decision model over an [LLM-as-a-judge](/docs/ax/evaluate/evaluators/llm-as-a-judge.md), see [Can decision models replace LLM judges?](https://arize.com/blog/typesafe-jev-llm-judge/).

![New Evaluator form with Evaluator Type set to Decision Judge, AI Provider set to jev-latest, a JSON state template with input and output variables, a boolean question named resolves\_request with instructions and criteria, and a configuration panel that picks project spans, maps input and output to span columns, and offers Test Evaluator On Spans](https://storage.googleapis.com/arize-assets/doc-images/evaluate/decision-judge-editor.png)

Creating a Decision Judge evaluator: evaluator type, AI provider, shared state template, a typed boolean question, and the span mapping step

## Supported providers

| Provider | Model | How to connect |
| :- | :- | :- |
| [TypeSafe AI](/docs/ax/security-and-settings/integrations-playground/typesafe.md) | `jev-latest` | Add a TypeSafe AI key under **Settings → AI Providers**. Eval-only. |
| [OpenAI](/docs/ax/security-and-settings/integrations-playground/openai.md) | `gpt-6-luna` | Reuses your existing OpenAI integration. The integration must use the default OpenAI base URL (or the US or EU regional endpoint) and have `gpt-6-luna` enabled. |
| [Perplexity](/docs/ax/security-and-settings/integrations-playground/perplexity.md) | `pplx-decider-v1.1-27b` | Add a Perplexity API key under **Settings → AI Providers**. Eval-only. |
| [Cloudflare Clef](/docs/ax/security-and-settings/integrations-playground/cloudflare-clef.md) | `clef`, `clef-flash` | Add a Cloudflare API token and your 32-character Account ID under **Settings → AI Providers**. Eval-only. |

Each provider serves the decision models listed in the table. Cloudflare Clef serves two. The **AI Provider** dropdown in the evaluator editor lists only the integrations that can run Decision Judge evaluations in your space.

TypeSafe AI, Perplexity, and Cloudflare Clef integrations are **restricted-use**: they power Decision Judge evaluators only and do not appear in the prompt playground or Alyx model pickers. They appear in the `/v2/ai-integrations` REST API only when a request filters by provider. OpenAI keys keep all their other uses.

## Question types

Each question needs a **name** (used for the published column only) and **instructions** (what the model actually judges). An evaluator can hold up to 64 questions. Results land on:

`eval.<evaluator name> <question name>.{label,score,...}`

| Type | Returns | Score |
| :- | :- | :- |
| **Boolean** | `true` / `false` | Confidence in the winning answer |
| **Choice** | Selected option (2–255 options) | Confidence in the selected option |
| **Score** | Rubric position (2–10 levels) | Position on your levels, with a maximize or minimize direction |

## Create a Decision Judge evaluator

**1. Connect a decision model provider**

Go to **Settings → [AI Providers](https://app.arize.com/account/ai-providers)** and add a [TypeSafe AI](/docs/ax/security-and-settings/integrations-playground/typesafe.md) key, a Perplexity key, or a Cloudflare Clef key with your Account ID. Or confirm you already have an [OpenAI](/docs/ax/security-and-settings/integrations-playground/openai.md) integration with `gpt-6-luna` enabled. You can also add any of the four providers inline from the evaluator form.

**2. Create the evaluator**

Go to **Evaluators → Create → Decision Judge**. In the **AI Provider** dropdown, select the integration and decision model to use.

**3. Write the state and questions**

Define the shared **state** (plain text or JSON with `{variables}`), map variables to span columns, then add one or more boolean, choice, or score questions. Put the full judgment in **instructions**. The question name is sent with the question as a short label, so keep the judgment itself in the instructions.

When you refer to a state field in a question, wrap it in backticks (for example `` `output` ``) so the model treats it as data, not as part of the instruction text.

**4. Test and save**

Optionally run **Test Evaluator On Spans**, then **Create Evaluator** or **Create and Run Evaluator**. Attach it to an [online eval task](/docs/ax/evaluate/run-evals-on-traces.md) like any other evaluator.

## Start from a template

Most [Arize eval templates](/docs/ax/evaluate/evaluators/llm-as-a-judge.md#arize-eval-templates) have a Decision Judge version, including Hallucination, Q\&A Correctness, User Frustration, Toxicity, Profanity, Prompt Injection, Jailbreak, RAG Relevancy, Summarization, and Code Readability. Open a template card and pick a TypeSafe AI model in its model dropdown to load the Decision Judge version; pick a chat model to return to the LLM-as-a-judge version. Template versions currently run on TypeSafe AI only.

> [!NOTE]
>
> Decision Judge evaluators run on project spans and dataset examples. They cannot run on experiments or at trace or session granularity. A task holds a single Decision Judge evaluator, and you cannot mix it with LLM, code, or [remote evaluators](/docs/ax/evaluate/remote-evaluators.md) on the same task. Decision Judge calls are not traced, so runs do not produce eval traces.
