Skip to main content

Sample code: Arize AX and Dynatrace dual export

Collector config, Python and TypeScript examples, and a script that checks both backends.
Many teams run Dynatrace for application performance monitoring and Arize AX for their AI apps. With OpenTelemetry you can send one stream of telemetry to both, and keep the same trace id in each:
  • Arize AX gets the AI spans (agents, LLM calls, tools, retrievers) with full prompts and responses, for evaluation and debugging. Every trace has a root, so it appears in the Traces view.
  • Dynatrace gets every span (HTTP, database, internal, and the AI spans without their prompt content), the logs, and the metrics.
When an answer looks wrong in Arize, search Dynatrace for its trace id to see the database queries, outbound calls, and logs of that request.
An application exports one trace over OTLP to an OpenTelemetry Collector. The collector sends all spans with AI content removed, plus logs and delta metrics, to Dynatrace, and sends the AI spans plus the trace root as a CHAIN span to Arize AX. Both backends show the same trace id.

Choose an approach

You can split the telemetry in two places:
  • Through an OpenTelemetry Collector. The app exports to one OTLP endpoint, and the collector filters, redacts, and fans out to both backends.
  • In application code. One TracerProvider with two span processors, one for each backend. No extra infrastructure, which suits serverless and edge runtimes.

Before you start

You need:
  • An Arize AX account, with your API key and Space ID
  • A Dynatrace environment and an ingest token. A platform token (dt0s16.*) needs openpipeline:traces:ingest, openpipeline:logs:ingest, and openpipeline:metrics:ingest. An access token (dt0c01.*) needs openTelemetryTrace.ingest, logs.ingest, and metrics.ingest.
  • Python 3.10 or later, Node 20 or later for the TypeScript example, and an OpenAI API key for the example agent
  • Optional, to check both backends from the command line: the AX CLI and a Dynatrace platform token with storage:logs:read, storage:metrics:read, storage:spans:read, and storage:buckets:read

Set up the sample

The sample app is a small FastAPI support agent. Each request runs a sqlite query, an OpenAI tool-calling loop, a tool that queries the database, and a tool that calls a weather API over HTTP. It emits logs and three metrics.
1

Clone the sample

2

Create the .env file

Fill in ARIZE_SPACE_ID, ARIZE_API_KEY, DYNATRACE_ENV_URL (for example https://{your-environment-id}.live.dynatrace.com, without a path), DYNATRACE_API_TOKEN, and OPENAI_API_KEY. Set DYNATRACE_AUTH_SCHEME to Bearer for platform tokens or Api-Token for access tokens.
3

Install the Python dependencies

One request produces this trace:

Send through an OpenTelemetry Collector

The app needs no Arize or Dynatrace credentials and no filtering code. It exports traces, logs, and metrics with the standard OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318. The collector config is collector/config.yaml. It works with the Dynatrace distribution of the collector (0.60.0) and with otelcol-contrib (0.165.0).

How the config fans out

Shared processing, such as sampling, runs once in traces/in, before the fan-out, so both backends keep the same traces. The forward connector copies every span into each branch. It ships in both distributions, and unlike the routing connector it sends each span to every branch, so AI spans reach both backends. The Arize branch keeps spans that follow the OpenInference or the OpenTelemetry GenAI semantic conventions. Arize AX ingests gen_ai.* spans natively and derives the span kind from gen_ai.operation.name.
The Dynatrace branch keeps every span but deletes prompt and response attributes, so AI content lives only in Arize. Model names, token counts, tool names, and timings stay, so the Dynatrace trace is one tree with no gaps. Remove the processor to send content to both.
The exporters. Dynatrace accepts OTLP over HTTP/protobuf only. For Arize, use gRPC; if you must use the otlp_http exporter, set compression: none, because Arize’s HTTP endpoint rejects gzip bodies.
Dynatrace ingests delta metrics, so the metrics pipeline runs cumulative_to_delta. Logs pass through unchanged: the SDK puts the trace and span id on each record, and Dynatrace exposes them as trace_id and span_id.

Run it

1

Start the collector

The script downloads the Dynatrace distribution into bin/, validates the config, and runs it. Use DISTRO=contrib scripts/run-collector.sh for otelcol-contrib, or docker compose up -d to run the collector in a container.
2

Send a request

In another shell:
The app prints the trace id:
The collector guide in the sample, docs/collector.md, covers tail sampling, scaling with the load_balancing exporter, Kubernetes, and the debug overlay.

Send from application code

Use one TracerProvider with two span processors. Each processor sees every span that ends and decides what to forward:
  • The Arize processor keeps only AI spans and makes the top AI span the root.
  • The Dynatrace processor exports every span through an exporter that removes AI content.
  • A LoggerProvider and a MeterProvider (with delta temporality) export to Dynatrace.
The Python OpenInference packages don’t include a span filter or root processor, so the sample has a small one in example/openinference_processors.py. OpenInferenceSpanProcessor forwards AI spans to the Arize exporter, and ContentRedactingExporter removes prompt and response attributes before the Dynatrace export.
Then add your HTTP and database instrumentation (FastAPIInstrumentor, SQLite3Instrumentor, HTTPXClientInstrumentor, and so on) on the same provider.If you use arize.otel.register(), keep it as the only provider. Its own Arize exporter receives every span, including APM spans, so to keep APM spans out of Arize, build the provider as shown above.Run the sample:
Two rules keep the trace ids in sync:
  • Use one provider per process. Two TracerProviders, for example one from arize.otel.register() and one for Dynatrace, produce two separate traces for the same request.
  • Don’t set trace_id as a log attribute yourself. The SDK already sends it as a top-level field. If you add it again, Dynatrace stores trace_id as an array and filter trace_id == "..." stops matching.
Across services, propagate the W3C traceparent header. Each service exports with its own two processors, and both backends join the spans into one trace by id. The in-code guide in the sample, docs/in-code-instrumentation.md, covers sampling, flushing in serverless functions, and the tradeoffs of sending AI spans to Dynatrace.

How traces stay rooted in Arize

Arize lists traces by their root span, the span with no parent. The top AI span of a request usually has an APM parent, such as the HTTP server span, and the Arize filter drops that parent. Without a fix, Arize stores the AI spans but has no root for the trace, so the trace is missing from the Traces view. Both setups fix this by default, and neither changes what Dynatrace receives.
The traces/arize-root branch sends the trace’s root span to Arize as a CHAIN:
  1. tail_sampling/traces-with-ai-spans waits decision_wait (30 seconds by default) and passes a trace on only if it contains an AI span. Requests without AI spans send nothing to Arize.
  2. filter/non-ai-trace-root keeps only the span with no parent, and only if it is not already an AI span.
  3. transform/root-as-chain sets openinference.span.kind to CHAIN on that span.
In Arize, the request span is the root and the AI spans sit under it at their real positions. No parent ids change, so nested agents stay nested. The AI spans themselves never pass through tail_sampling.This works when the top AI span’s parent is the trace root and all spans of a trace reach the same collector instance. With several collector replicas, route spans by trace id with the load_balancing exporter. If you can change the app, tagging the request span as a CHAIN there is still better, because it also puts the request input and output on the root.
If you build your own collector, Arize’s filter-remap-processor is an alternative. It buffers each trace, drops the non-AI spans, and points every kept span at its nearest kept ancestor. It is not part of the Dynatrace distribution or otelcol-contrib, so it needs a custom build with the OpenTelemetry Collector Builder.

Verify both backends

Look up the trace id the app printed in each backend.
1

Find the trace in Arize AX

Open the project named by ARIZE_PROJECT_NAME and find the trace id in the Traces view. The trace has one root: POST /ask as a CHAIN through the collector, or support_agent in code. With the AX CLI:
2

Find the trace and logs in Dynatrace

Open the trace id in the Distributed Tracing app, or run these queries in a notebook:
Dynatrace has the full tree: the HTTP, database, and AI spans, with one root (POST /ask) and no input.value or output.value on the AI spans.
3

Run the automated check

scripts/verify.py sends one request per scenario and checks what each backend stored: all AI spans and only AI spans in Arize, one root with no dangling parents, the trace listed in Traces, the full redacted trace in Dynatrace, correlated logs, and metrics. It needs the AX CLI and DYNATRACE_QUERY_TOKEN in .env.
TESTING.md lists every scenario and check.

Troubleshooting

Learn more

OpenTelemetry Collector

Collector concepts, deployment models, and common pitfalls.

Advanced patterns

Collector authentication and dynamic project routing for Arize AX.

Semantic conventions

How Arize AX maps OpenInference and OTel GenAI attributes.

Dynatrace OpenTelemetry Collector

Install and configure the Dynatrace distribution of the collector.