Skip to main content
Webhooks notify your systems when a prompt or evaluator changes or when an evaluation task run fails. Arize AX sends the event to your endpoint as it happens, so you can react right away, for example by redeploying when a prompt version is labeled or alerting when a task starts failing, and your systems do not need to poll for changes.

Events

A webhook defines the destination URL and authentication. A subscription connects one webhook to one event on one source. ONLINE_TASK_RUN_FAILED fires when a task run fails, including when a run writes some evaluations and then stops on an error. It is sent once when a task starts failing and again each time the cause changes.

Create a webhook

  1. Go to Settings → Org Settings → Webhooks and click Add Webhook.
  2. Enter a Name and the URL that receives events. The name must be unique within the organization, and a description is optional.
  3. Enter an Authorization Token if your endpoint expects a bearer token or secret, and add custom headers if it needs them.
  4. Click Add Webhook.
Add Webhook panel with Name, Description, URL, and Authorization Token fields and an Add Custom Headers button

The Add Webhook panel, with the name, URL, and optional authorization token.

The REST API can also create a webhook:
Webhooks created in the UI use BEARER. HMAC_SHA256 webhooks are created with the REST API, the Python SDK, the TypeScript SDK, the Go SDK, or the ax webhooks CLI. Arize AX does not send events to private, loopback, or link-local addresses, and it does not follow redirects, so the URL must be the exact public address that receives events.

Authenticate deliveries

The auth type is fixed at creation. To switch, create a new webhook.

Bearer token

BEARER sends the saved token verbatim in the Authorization header, so the token must include the scheme your endpoint expects, such as Bearer my-secret-token. The token is write-only and never returned after saving. Updating the webhook replaces or clears it.

HMAC signature

HMAC_SHA256 signs every delivery with a signing secret Arize AX generates. The secret is returned once, in the create response, and cannot be rotated or retrieved again; afterwards the webhook exposes only a signing_secret_hint such as whsec_…abcd. If the secret is lost, create a new webhook. Each signed delivery includes these headers. The signature is HMAC-SHA256 over <timestamp>.<raw request body>. The key is the full signing secret, including its whsec_ prefix.
  1. Read the raw request body as bytes, before JSON parsing.
  2. Compute HMAC-SHA256 over <X-Arize-Webhook-Timestamp>.<raw body> with the full signing secret as the key.
  3. Hex-encode the result, prefix v1=, and compare to X-Arize-Webhook-Signature in constant time.
  4. Reject timestamps older than your tolerance, for example five minutes. Each retry carries a fresh timestamp and signature.

Custom headers

Both auth types support up to 20 write-only custom headers, for example a routing key. Connection headers such as Host and Content-Length are rejected. Content-Type: application/json, the bearer token, and the signature headers override custom headers with the same names.

Subscribe to events

Subscriptions are created with the REST API:
The webhook and the source must belong to the same organization, and each event works only with its matching source type. Prompt events use PROMPT sources, evaluator events use EVALUATOR sources, and task run events use ONLINE_TASK and SPACE sources. An ONLINE_TASK source must be an eval task; other task types return 422. A SPACE subscription covers every eval task in the space, including ones created later. A webhook subscribed to both a task and its space receives each alert once.

Task failure alerts

online_task.run.failed covers tasks that run LLM-as-a-judge, code, agent-as-a-judge, and remote evaluators. Playground experiments and prompt optimization tasks never send it.

When it fires

Only the webhook payload uses partial_completed; the UI, REST API, and SDKs show that run as CANCELLED. An invalid config adds one failed run with the config error to the task’s run logs each time the task starts failing on config. This applies to scheduled runs only; Playground and date-range runs show the error directly and record no run.

One alert per cause

Arize AX sends one alert when a task starts failing, and another each time the cause changes.
  • Runs that keep failing with the same error_type send nothing more, even when they alternate between failed and partial_completed.
  • A run with a new error_type sends a new alert.
  • A completed run ends the streak. The next failure alerts again.
  • A run that finds no data neither alerts nor ends the streak.
Each webhook tracks its own streak. Subscribing to a task that is already failing alerts on the task’s next failed run.

When it does not fire

  • A user stops the run. It ends cancelled with the counts it had scored.
  • The run had nothing to evaluate, or found no data.
  • Some parts could not be scored, but the scored parts saved.
  • A single eval fails, such as one LLM call returning an unparsable label. The run still succeeds.
  • The run is a sub-run. Only the top-level run alerts, and it completes silently when its sub-runs split between partly finished and completed with none failed.
  • The run never reaches a final status.

Payloads

Each delivery is an HTTP POST with Content-Type: application/json. Event deliveries share these top-level fields. All IDs are opaque strings. data fields by event. changed_by is the ID of the user who made the change.
label is the label that was added or removed.
The payload carries project_id or dataset_id, whichever the task runs on. status is failed or partial_completed; treat any other value as a failure. failure_reason is the run log text, up to 1,000 characters. error_type is missing when the failure could not be classified, and a run with several evaluators reports the first failing evaluator’s type in name order.
New fields may be added; ignore fields you do not use. A test.test_event delivery carries only event, event_uuid, and timestamp.

Retries

Arize AX counts any 2xx response as delivered. It attempts a delivery up to 6 times in total, with retries about 1, 4, 16, 64, and 256 minutes after the previous attempt. Connection failures, timeouts, and HTTP 408, 429, and 5xx responses are retried. Other 4xx responses and 3xx redirects are not. Retries resend the same payload and event_uuid, so deduplicate on event_uuid. Return 2xx quickly and do slow work afterwards.

Delivery history

Every delivery attempt is recorded. Fetch them with GET /v2/webhooks/{webhook_id}/delivery-attempts or from the webhook’s Event Delivery tab.

Test a webhook

Send a test event from the webhook’s page or with POST /v2/webhooks/{webhook_id}/test. The test sends one test.test_event request and reports your endpoint’s HTTP status, or 502 when no response came back. Tests do not retry, do not appear in delivery history, and do not work for HMAC_SHA256 webhooks; subscribe to an event and trigger it instead.

Limits

  • Webhook names must be unique within an organization.
  • The delivery timeout is between 1 and 60 seconds. The default is 30 seconds.
  • A webhook can have at most 20 custom headers.
  • At most 200 webhooks can subscribe to the same event on one source.
  • A webhook can subscribe to a given event on a given source only once.

Permissions

Account Admins can do everything an Organization Admin can. See Role-based access control.