Create evaluator
Creates a new evaluator with an initial version.
Payload Requirements
- The evaluator
namemust be unique within the given space. type(top-level) selects the evaluator kind:TEMPLATE,CODE,REMOTE, orDECISION_JUDGE. WithTEMPLATE, provideversion.template_config. WithCODE, provideversion.code_config— wherecode_config.typeisMANAGEDorCUSTOM(a separate discriminator withincode_config, independent of the top-leveltype: CODE). WithREMOTE, provideversion.remote_config.integration_idreferencing an accessibleEVALUATORintegration. Remote evaluator creation requires the remote evaluators feature to be enabled; otherwise the request returns403. WithDECISION_JUDGE, provideversion.decision_judge_config. Decision Judge must be enabled for the account; otherwise the request returns403. Anai_integration_idthat does not exist or is not available in the space returns404(LLM integration not found or not accessible to this space). Amodel_namethat is not a Decision Judge model, or questions that break the question rules, return422. The evaluator runs on spans only. Each question writes its own result, namedeval.<evaluator name> <question name>.labeland.score.- For template evaluators:
version.template_config.nameis the eval column name; must match^[a-zA-Z0-9_\s\-&()]+$. - For template evaluators:
version.template_config.templateis the prompt template; use{variable}for placeholders (f-string format, e.g.{input},{output}). - For template evaluators:
version.template_config.classification_choicesis required and maps choice labels to numeric scores (e.g.{"relevant": 1, "irrelevant": 0}). - For code evaluators: see
CodeConfig— managed evaluators (code_config.type: MANAGED) usemanaged_evaluatorandvariables; custom evaluators (code_config.type: CUSTOM) usecode, optionalimports, andvariables. - System-managed fields (
id,created_at,updated_at,created_by_user_id) are rejected on input.
Valid example (template evaluator)
{
"name": "Hallucination Detector",
"space_id": "U3BhY2U6MTpWNEth",
"type": "TEMPLATE",
"version": {
"commit_message": "Initial version",
"template_config": {
"name": "hallucination",
"template": "Given the input: {input}\nand the output: {output}\nIs the output a hallucination?",
"include_explanations": true,
"use_function_calling": true,
"classification_choices": {"hallucinated": 0, "factual": 1},
"llm_config": {
"ai_integration_id": "TGxtSW50ZWdyYXRpb246MTI6YUJjRA==",
"model_name": "gpt-4o",
"invocation_parameters": {"temperature": 0},
"provider_parameters": {}
}
}
}
}
Invalid example (type/config mismatch — TEMPLATE type with code_config)
{
"name": "Bad Evaluator",
"space_id": "U3BhY2U6MTpWNEth",
"type": "TEMPLATE",
"version": {
"commit_message": "Wrong config",
"code_config": {
"type": "CUSTOM",
"name": "my_eval",
"code": "class Evaluator: ...",
"variables": ["input"]
}
}
}
Authorizations
Most Arize AI endpoints require authentication. For those endpoints that require authentication, include your API key in the request header using the format
Service accounts may instead present an access token issued by an external identity provider (Microsoft Entra ID) when the account admin has configured that provider. Send it in the same header: Authorization: Bearer <access-token>.
Body
Body containing evaluator creation parameters with an initial version.
type: TEMPLATE, type: CODE, type: REMOTE, and type: DECISION_JUDGE are
accepted on creation.
Body containing evaluator creation parameters with an initial version.
The evaluator type:
TEMPLATE— LLM-based evaluator.CODE— managed built-in evaluators or custom Python code (both are subtypes ofCODE, discriminated by the nestedCodeConfig.type=MANAGED|CUSTOM).HARNESS— test harness evaluator.REMOTE— remote evaluator.DECISION_JUDGE— Decision Judge evaluator. It asks a set of typed questions about each span and writes one result per question. Requires Decision Judge to be enabled for the account.GET /v2/evaluatorsreturns these evaluators only whentype=DECISION_JUDGEis passed.
Applies to both the parent Evaluator.type field and every version's type
discriminator — a version's type must always match its parent evaluator's type.
TEMPLATE, CODE, HARNESS, REMOTE, DECISION_JUDGE Response
Returns an evaluator with a resolved version
An evaluator defines reusable evaluation logic that can be attached to evaluation tasks. The type field determines the kind of evaluation: TEMPLATE (LLM-based template evaluation), CODE (custom code evaluation), REMOTE (externally hosted evaluation), or DECISION_JUDGE (typed questions answered by a Decision Judge model).
The evaluator type:
TEMPLATE— LLM-based evaluator.CODE— managed built-in evaluators or custom Python code (both are subtypes ofCODE, discriminated by the nestedCodeConfig.type=MANAGED|CUSTOM).HARNESS— test harness evaluator.REMOTE— remote evaluator.DECISION_JUDGE— Decision Judge evaluator. It asks a set of typed questions about each span and writes one result per question. Requires Decision Judge to be enabled for the account.GET /v2/evaluatorsreturns these evaluators only whentype=DECISION_JUDGEis passed.
Applies to both the parent Evaluator.type field and every version's type
discriminator — a version's type must always match its parent evaluator's type.
TEMPLATE, CODE, HARNESS, REMOTE, DECISION_JUDGE Was this page helpful?