> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Send Telemetry to Arize AX and Dynatrace

> Send AI spans to Arize AX and every span, log, and metric to Dynatrace with the same trace id, from application code or through an OpenTelemetry Collector, with every trace rooted in Arize.

> [card] [Sample code: Arize AX and Dynatrace dual export](https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/dynatrace_dual_export)
>
> Collector config, Python and TypeScript examples, and a script that checks both backends.

Many teams run Dynatrace for application performance monitoring and Arize AX for their AI apps. With OpenTelemetry you can send one stream of telemetry to both, and keep the same trace id in each:

* Arize AX gets the AI spans (agents, LLM calls, tools, retrievers) with full prompts and responses, for evaluation and debugging. Every trace has a root, so it appears in the Traces view.
* Dynatrace gets every span (HTTP, database, internal, and the AI spans without their prompt content), the logs, and the metrics.

When an answer looks wrong in Arize, search Dynatrace for its trace id to see the database queries, outbound calls, and logs of that request.

![An application exports one trace over OTLP to an OpenTelemetry Collector. The collector sends all spans with AI content removed, plus logs and delta metrics, to Dynatrace, and sends the AI spans plus the trace root as a CHAIN span to Arize AX. Both backends show the same trace id.](https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/cookbooks/dynatrace-dual-export-architecture.svg)

## Choose an approach

You can split the telemetry in two places:

* Through an OpenTelemetry Collector. The app exports to one OTLP endpoint, and the collector filters, redacts, and fans out to both backends.
* In application code. One `TracerProvider` with two span processors, one for each backend. No extra infrastructure, which suits serverless and edge runtimes.

| | Collector | In code |
| - | - | - |
| App needs Arize and Dynatrace credentials | No | Yes |
| Filtering and redaction | Collector config (OTTL) | Span processors in code |
| Tail sampling | Yes | No (head sampling only) |
| Delta metrics for Dynatrace | `cumulative_to_delta` processor | SDK exporter setting |
| Root span in Arize | The trace's root span is sent as a CHAIN | The top AI span becomes the root |
| Extra infrastructure | A collector | None |

## Before you start

You need:

* An [Arize AX account](https://app.arize.com/auth/join), with your [API key and Space ID](/docs/ax/security-and-settings/api-keys.md)
* A Dynatrace environment and an ingest token. A platform token (`dt0s16.*`) needs `openpipeline:traces:ingest`, `openpipeline:logs:ingest`, and `openpipeline:metrics:ingest`. An access token (`dt0c01.*`) needs `openTelemetryTrace.ingest`, `logs.ingest`, and `metrics.ingest`.
* Python 3.10 or later, Node 20 or later for the TypeScript example, and an OpenAI API key for the example agent
* Optional, to check both backends from the command line: the [AX CLI](/docs/api-clients/cli/overview.md) and a Dynatrace platform token with `storage:logs:read`, `storage:metrics:read`, `storage:spans:read`, and `storage:buckets:read`

## Set up the sample

The sample app is a small FastAPI support agent. Each request runs a sqlite query, an OpenAI tool-calling loop, a tool that queries the database, and a tool that calls a weather API over HTTP. It emits logs and three metrics.

**1. Clone the sample**

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
git clone https://github.com/Arize-ai/tutorials.git
cd tutorials/python/cookbooks/dynatrace_dual_export
```

**2. Create the .env file**

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
cp .env.example .env
```

Fill in `ARIZE_SPACE_ID`, `ARIZE_API_KEY`, `DYNATRACE_ENV_URL` (for example `https://{your-environment-id}.live.dynatrace.com`, without a path), `DYNATRACE_API_TOKEN`, and `OPENAI_API_KEY`. Set `DYNATRACE_AUTH_SCHEME` to `Bearer` for platform tokens or `Api-Token` for access tokens.

**3. Install the Python dependencies**

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
python3 -m venv .venv && .venv/bin/pip install -r example/requirements.txt
```

One request produces this trace:

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
POST /ask                    SERVER   FastAPI          Dynatrace, and Arize as the CHAIN root (collector)
├── SELECT                   CLIENT   sqlite3          Dynatrace
└── support_agent            AGENT    OpenInference    both (the Arize root in code)
    ├── ChatCompletion       LLM      OpenInference    both
    ├── get_order_status     TOOL     OpenInference    both
    │   └── SELECT           CLIENT   sqlite3          Dynatrace
    ├── execute_tool get_weather  TOOL  OTel GenAI     both
    │   └── GET              CLIENT   httpx            Dynatrace
    └── ChatCompletion       LLM      OpenInference    both
```

## Send through an OpenTelemetry Collector

The app needs no Arize or Dynatrace credentials and no filtering code. It exports traces, logs, and metrics with the standard `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318`. The collector config is [`collector/config.yaml`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/collector/config.yaml). It works with the [Dynatrace distribution](https://github.com/Dynatrace/dynatrace-otel-collector) of the collector (0.60.0) and with `otelcol-contrib` (0.165.0).

### How the config fans out

| Pipeline | Processors | Exports to |
| - | - | - |
| `traces/in` | `memory_limiter`, `batch` (add `tail_sampling` here if you need it) | `forward/traces` |
| `traces/dynatrace` | `transform/redact-ai-content` | Dynatrace |
| `traces/arize` | `filter/ai-spans-only` | `forward/arize` |
| `traces/arize-root` | `tail_sampling/traces-with-ai-spans`, `filter/non-ai-trace-root`, `transform/root-as-chain` | `forward/arize` |
| `traces/arize-out` | `resource/arize-project` | Arize AX |
| `metrics` | `memory_limiter`, `cumulative_to_delta`, `batch` | Dynatrace |
| `logs` | `memory_limiter`, `batch` | Dynatrace |

Shared processing, such as sampling, runs once in `traces/in`, before the fan-out, so both backends keep the same traces. The `forward` connector copies every span into each branch. It ships in both distributions, and unlike the `routing` connector it sends each span to every branch, so AI spans reach both backends.

The Arize branch keeps spans that follow the OpenInference or the OpenTelemetry GenAI semantic conventions. Arize AX [ingests `gen_ai.*` spans natively](/docs/ax/concepts/otel-openinference/semantic-conventions) and derives the span kind from `gen_ai.operation.name`.

```yaml theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
filter/ai-spans-only:
  error_mode: ignore
  trace_conditions:          # a span is dropped when the condition is true
    - >-
      span.attributes["openinference.span.kind"] == nil and
      span.attributes["gen_ai.operation.name"] == nil and
      span.attributes["gen_ai.request.model"] == nil and
      span.attributes["gen_ai.tool.name"] == nil
```

The Dynatrace branch keeps every span but deletes prompt and response attributes, so AI content lives only in Arize. Model names, token counts, tool names, and timings stay, so the Dynatrace trace is one tree with no gaps. Remove the processor to send content to both.

```yaml theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
transform/redact-ai-content:
  error_mode: ignore
  trace_statements:
    - delete_key(span.attributes, "input.value")
    - delete_key(span.attributes, "output.value")
    - delete_key(span.attributes, "llm.prompt_template.variables")
    - delete_matching_keys(span.attributes, "^llm\\.(input|output)_messages\\.")
    - delete_matching_keys(span.attributes, "^llm\\.prompts?\\.")
    - delete_matching_keys(span.attributes, "^gen_ai\\.(input|output)\\.messages$")
    - delete_key(span.attributes, "gen_ai.system_instructions")
    - delete_matching_keys(span.attributes, "^gen_ai\\.tool\\.call\\.(arguments|result)$")
```

The exporters. Dynatrace accepts OTLP over HTTP/protobuf only. For Arize, use gRPC; if you must use the `otlp_http` exporter, set `compression: none`, because Arize's HTTP endpoint rejects gzip bodies.

```yaml theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
exporters:
  otlp_http/dynatrace:
    endpoint: ${env:DYNATRACE_ENV_URL}/api/v2/otlp   # appends /v1/traces, /v1/logs, /v1/metrics
    headers:
      Authorization: "${env:DYNATRACE_AUTH_SCHEME:-Bearer} ${env:DYNATRACE_API_TOKEN}"
  otlp_grpc/arize:
    endpoint: otlp.arize.com:443
    headers:
      space_id: ${env:ARIZE_SPACE_ID}
      api_key: ${env:ARIZE_API_KEY}
```

Dynatrace ingests delta metrics, so the metrics pipeline runs `cumulative_to_delta`. Logs pass through unchanged: the SDK puts the trace and span id on each record, and Dynatrace exposes them as `trace_id` and `span_id`.

### Run it

**1. Start the collector**

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
scripts/run-collector.sh
```

The script downloads the Dynatrace distribution into `bin/`, validates the config, and runs it. Use `DISTRO=contrib scripts/run-collector.sh` for `otelcol-contrib`, or `docker compose up -d` to run the collector in a container.

**2. Send a request**

In another shell:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
.venv/bin/python example/app.py --mode collector
```

The app prints the trace id:

```json theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{"trace_id": "<32 hex characters>", "answer": "...", "mode": "collector"}
```

The collector guide in the sample, [`docs/collector.md`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/docs/collector.md), covers tail sampling, scaling with the `load_balancing` exporter, Kubernetes, and the debug overlay.

## Send from application code

Use one `TracerProvider` with two span processors. Each processor sees every span that ends and decides what to forward:

* The Arize processor keeps only AI spans and makes the top AI span the root.
* The Dynatrace processor exports every span through an exporter that removes AI content.
* A `LoggerProvider` and a `MeterProvider` (with delta temporality) export to Dynatrace.

> [tab] Python
>
> The Python OpenInference packages don't include a span filter or root processor, so the sample has a small one in [`example/openinference_processors.py`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/example/openinference_processors.py). `OpenInferenceSpanProcessor` forwards AI spans to the Arize exporter, and `ContentRedactingExporter` removes prompt and response attributes before the Dynatrace export.
>
> ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
> import logging, os
> from opentelemetry import metrics, trace
> from opentelemetry._logs import set_logger_provider
> from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
> from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
> from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
> from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
> from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
> from opentelemetry.sdk.metrics import Counter, Histogram, MeterProvider, ObservableCounter
> from opentelemetry.sdk.metrics.export import AggregationTemporality, PeriodicExportingMetricReader
> from opentelemetry.sdk.resources import Resource
> from opentelemetry.sdk.trace import TracerProvider
> from opentelemetry.sdk.trace.export import BatchSpanProcessor
> from openinference.instrumentation.openai import OpenAIInstrumentor
>
> from openinference_processors import ContentRedactingExporter, OpenInferenceSpanProcessor
>
> resource = Resource.create({
>     "service.name": "support-api",
>     "openinference.project.name": "support-agent",   # Arize project
> })
>
> dt_base = os.environ["DYNATRACE_ENV_URL"].rstrip("/") + "/api/v2/otlp"
> token = os.environ["DYNATRACE_API_TOKEN"]
> dt_headers = {"Authorization": ("Api-Token " if token.startswith("dt0c") else "Bearer ") + token}
>
> provider = TracerProvider(resource=resource)
> provider.add_span_processor(OpenInferenceSpanProcessor(BatchSpanProcessor(OTLPSpanExporter(
>     endpoint="https://otlp.arize.com/v1/traces",
>     headers={"space_id": os.environ["ARIZE_SPACE_ID"], "api_key": os.environ["ARIZE_API_KEY"]},
> ))))
> provider.add_span_processor(BatchSpanProcessor(ContentRedactingExporter(
>     OTLPSpanExporter(endpoint=f"{dt_base}/v1/traces", headers=dt_headers)
> )))
> trace.set_tracer_provider(provider)
>
> logger_provider = LoggerProvider(resource=resource)
> logger_provider.add_log_record_processor(
>     BatchLogRecordProcessor(OTLPLogExporter(endpoint=f"{dt_base}/v1/logs", headers=dt_headers)))
> set_logger_provider(logger_provider)
> logging.getLogger("support").addHandler(LoggingHandler(logger_provider=logger_provider))
>
> # Dynatrace ingests delta metrics.
> delta = {Counter: AggregationTemporality.DELTA, ObservableCounter: AggregationTemporality.DELTA,
>          Histogram: AggregationTemporality.DELTA}
> metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[PeriodicExportingMetricReader(
>     OTLPMetricExporter(endpoint=f"{dt_base}/v1/metrics", headers=dt_headers, preferred_temporality=delta))]))
>
> OpenAIInstrumentor().instrument(tracer_provider=provider)
> ```
>
> Then add your HTTP and database instrumentation (`FastAPIInstrumentor`, `SQLite3Instrumentor`, `HTTPXClientInstrumentor`, and so on) on the same provider.
>
> If you use `arize.otel.register()`, keep it as the only provider. Its own Arize exporter receives every span, including APM spans, so to keep APM spans out of Arize, build the provider as shown above.
>
> Run the sample:
>
> ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
> .venv/bin/python example/app.py --mode incode
> ```

> [tab] TypeScript
>
> The Node example uses the Vercel AI SDK. `ArizeRootProcessor` in [`example-ts/arize-root-processor.ts`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/example-ts/arize-root-processor.ts) wraps the OpenInference processor so that AI spans whose local parent is a non-AI span reach Arize without a parent. `redactAiContent` in [`example-ts/index.ts`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/example-ts/index.ts) removes AI content from the Dynatrace copy.
>
> ```ts theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
> import { registerTelemetry } from "ai";
> import { OpenTelemetry } from "@ai-sdk/otel";
> import { enrichSpanWithOpenInference, isOpenInferenceSpan, OpenInferenceBatchSpanProcessor } from "@arizeai/openinference-vercel";
> import { SEMRESATTRS_PROJECT_NAME } from "@arizeai/openinference-semantic-conventions";
> import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
> import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-proto";
> import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";
> import { AggregationTemporalityPreference } from "@opentelemetry/exporter-metrics-otlp-http";
> import { resourceFromAttributes } from "@opentelemetry/resources";
> import { BatchLogRecordProcessor, LoggerProvider } from "@opentelemetry/sdk-logs";
> import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
> import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
> import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
> import { ArizeRootProcessor } from "./arize-root-processor";
>
> registerTelemetry(new OpenTelemetry({ enrichSpan: enrichSpanWithOpenInference })); // AI SDK v7
>
> const dtBase = `${process.env.DYNATRACE_ENV_URL}/api/v2/otlp`;
> const token = process.env.DYNATRACE_API_TOKEN!;
> const dtHeaders = { Authorization: `${token.startsWith("dt0c") ? "Api-Token" : "Bearer"} ${token}` };
> const resource = resourceFromAttributes({ "service.name": "support-api", [SEMRESATTRS_PROJECT_NAME]: "support-agent" });
>
> const tracerProvider = new NodeTracerProvider({
>   resource,
>   spanProcessors: [
>     new ArizeRootProcessor(
>       new OpenInferenceBatchSpanProcessor({
>         exporter: new OTLPTraceExporter({
>           url: "https://otlp.arize.com/v1/traces",
>           headers: { space_id: process.env.ARIZE_SPACE_ID!, api_key: process.env.ARIZE_API_KEY! },
>         }),
>         spanFilter: isOpenInferenceSpan,
>       }),
>     ),
>     new BatchSpanProcessor(redactAiContent(new OTLPTraceExporter({ url: `${dtBase}/v1/traces`, headers: dtHeaders }))),
>   ],
> });
> tracerProvider.register();
>
> const loggerProvider = new LoggerProvider({
>   resource,
>   processors: [new BatchLogRecordProcessor({ exporter: new OTLPLogExporter({ url: `${dtBase}/v1/logs`, headers: dtHeaders }) })],
> });
> const meterProvider = new MeterProvider({
>   resource,
>   readers: [new PeriodicExportingMetricReader({
>     // Dynatrace ingests delta metrics.
>     exporter: new OTLPMetricExporter({ url: `${dtBase}/v1/metrics`, headers: dtHeaders,
>       temporalityPreference: AggregationTemporalityPreference.DELTA }),
>   })],
> });
> ```
>
> Each AI SDK call needs `experimental_telemetry: { isEnabled: true }`. On Vercel, `registerOTel({ spanProcessors, logRecordProcessors, metricReaders })` from `@vercel/otel` takes the same processors and readers.
>
> Run the sample:
>
> ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
> cd example-ts && npm install && npm start
> ```

Two rules keep the trace ids in sync:

* Use one provider per process. Two `TracerProvider`s, for example one from `arize.otel.register()` and one for Dynatrace, produce two separate traces for the same request.
* Don't set `trace_id` as a log attribute yourself. The SDK already sends it as a top-level field. If you add it again, Dynatrace stores `trace_id` as an array and `filter trace_id == "..."` stops matching.

Across services, propagate the W3C `traceparent` header. Each service exports with its own two processors, and both backends join the spans into one trace by id.

The in-code guide in the sample, [`docs/in-code-instrumentation.md`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/docs/in-code-instrumentation.md), covers sampling, flushing in serverless functions, and the tradeoffs of sending AI spans to Dynatrace.

## How traces stay rooted in Arize

Arize lists traces by their root span, the span with no parent. The top AI span of a request usually has an APM parent, such as the HTTP server span, and the Arize filter drops that parent. Without a fix, Arize stores the AI spans but has no root for the trace, so the trace is missing from the Traces view. Both setups fix this by default, and neither changes what Dynatrace receives.

> [tab] Collector
>
> The `traces/arize-root` branch sends the trace's root span to Arize as a CHAIN:
>
> 1. `tail_sampling/traces-with-ai-spans` waits `decision_wait` (30 seconds by default) and passes a trace on only if it contains an AI span. Requests without AI spans send nothing to Arize.
> 2. `filter/non-ai-trace-root` keeps only the span with no parent, and only if it is not already an AI span.
> 3. `transform/root-as-chain` sets `openinference.span.kind` to `CHAIN` on that span.
>
> ```yaml theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
> tail_sampling/traces-with-ai-spans:
>   decision_wait: ${env:ARIZE_ROOT_DECISION_WAIT:-30s}
>   num_traces: 50000
>   decision_cache:
>     sampled_cache_size: 100000
>   policies:
>     - name: has-ai-span
>       type: ottl_condition
>       ottl_condition:
>         error_mode: ignore
>         span:
>           - attributes["openinference.span.kind"] != nil
>           - attributes["gen_ai.operation.name"] != nil
>           - attributes["gen_ai.request.model"] != nil
>           - attributes["gen_ai.tool.name"] != nil
>
> filter/non-ai-trace-root:
>   error_mode: ignore
>   trace_conditions:
>     - span.parent_span_id != SpanID(0x0000000000000000)
>     - span.attributes["openinference.span.kind"] != nil
>     - span.attributes["gen_ai.operation.name"] != nil
>     - span.attributes["gen_ai.request.model"] != nil
>     - span.attributes["gen_ai.tool.name"] != nil
>
> transform/root-as-chain:
>   error_mode: ignore
>   trace_statements:
>     - set(span.attributes["openinference.span.kind"], "CHAIN")
> ```
>
> In Arize, the request span is the root and the AI spans sit under it at their real positions. No parent ids change, so nested agents stay nested. The AI spans themselves never pass through `tail_sampling`.
>
> This works when the top AI span's parent is the trace root and all spans of a trace reach the same collector instance. With several collector replicas, route spans by trace id with the `load_balancing` exporter. If you can change the app, tagging the request span as a CHAIN there is still better, because it also puts the request input and output on the root.
>
> > [!NOTE]
> >
> > If you build your own collector, Arize's [filter-remap-processor](https://github.com/Arize-ai/filter-remap-processor) is an alternative. It buffers each trace, drops the non-AI spans, and points every kept span at its nearest kept ancestor. It is not part of the Dynatrace distribution or `otelcol-contrib`, so it needs a custom build with the OpenTelemetry Collector Builder.

> [tab] In code
>
> When an AI span's local parent is a non-AI span, the Arize processor exports that AI span without a parent, so it becomes the root in Arize. The change applies only to the copy the Arize processor sees, so Dynatrace keeps the real parent and one complete tree. AI spans under other AI spans keep their parents, so nested agents stay nested.
>
> A parent that comes from another service (a remote `traceparent`) is kept, because the upstream service may export its own AI spans to Arize under the same trace.
>
> > [!NOTE]
> >
> > In `@arizeai/openinference-vercel`, don't use `reparentOrphanedSpans: true` when Dynatrace also receives the AI spans. It clears the parent on the shared span, so Dynatrace shows the AI subtree as a second root. `ArizeRootProcessor` changes only Arize's copy.

## Verify both backends

Look up the trace id the app printed in each backend.

**1. Find the trace in Arize AX**

Open the project named by `ARIZE_PROJECT_NAME` and find the trace id in the Traces view. The trace has one root: `POST /ask` as a CHAIN through the collector, or `support_agent` in code. With the AX CLI:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
ax spans export <project> --space <space-id> --trace-id <trace-id> --days 1 --stdout
ax traces list <project> --space <space-id> --limit 50 --output json
```

**2. Find the trace and logs in Dynatrace**

Open the trace id in the Distributed Tracing app, or run these queries in a notebook:

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
fetch spans | filter trace.id == toUid("<trace-id>")
fetch logs | filter trace_id == "<trace-id>" | sort timestamp asc
```

Dynatrace has the full tree: the HTTP, database, and AI spans, with one root (`POST /ask`) and no `input.value` or `output.value` on the AI spans.

**3. Run the automated check**

`scripts/verify.py` sends one request per scenario and checks what each backend stored: all AI spans and only AI spans in Arize, one root with no dangling parents, the trace listed in Traces, the full redacted trace in Dynatrace, correlated logs, and metrics. It needs the AX CLI and `DYNATRACE_QUERY_TOKEN` in `.env`.

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
.venv/bin/python scripts/verify.py                    # every scenario
.venv/bin/python scripts/verify.py collector incode   # some scenarios
```

[`TESTING.md`](https://github.com/Arize-ai/tutorials/blob/main/python/cookbooks/dynatrace_dual_export/TESTING.md) lists every scenario and check.

## Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| AI trace missing from the Arize Traces view, but its spans exist | The top AI span's parent was dropped and nothing made it a root | Use the `traces/arize-root` branch, `OpenInferenceSpanProcessor` (Python), or `ArizeRootProcessor` (TypeScript) |
| Different trace ids in Arize and Dynatrace | Two `TracerProvider`s, or work started outside the request context without passing context | Use one provider, pass context into threads and tasks, and propagate `traceparent` between services |
| `filter trace_id == "..."` misses logs | The app set a `trace_id` log attribute in addition to the SDK's field | Remove the manual attribute |
| Arize returns `400` from the HTTP endpoint | gzip compression | Set `compression: none`, or use gRPC |
| Dynatrace returns `401` or `403` | Wrong scheme for the token type, or a missing ingest scope | Use `Bearer` for platform tokens and `Api-Token` for access tokens, and add the ingest scopes |
| Dynatrace shows two roots for one request | `reparentOrphanedSpans: true` changed the shared span | Use `ArizeRootProcessor` instead |
| Last spans of a run never arrive | The process exited before the batch was exported | Call `force_flush()` or `shutdown()` on the providers before exit |

## Learn more

> [card] [OpenTelemetry Collector](/docs/ax/concepts/otel-openinference/otel-collector.md)
>
> Collector concepts, deployment models, and common pitfalls.

> [card] [Advanced patterns](/docs/ax/instrument/advanced-patterns.md)
>
> Collector authentication and dynamic project routing for Arize AX.

> [card] [Semantic conventions](/docs/ax/concepts/otel-openinference/semantic-conventions.md)
>
> How Arize AX maps OpenInference and OTel GenAI attributes.

> [card] [Dynatrace OpenTelemetry Collector](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector)
>
> Install and configure the Dynatrace distribution of the collector.
