> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Sync traces with Data Fabric
> Data Fabric continuously syncs your Arize AX traces, evaluations, and annotations to your own cloud data warehouse.
**Data Fabric** automatically syncs your production trace data, evaluations, and annotations from Arize AX to your own cloud data warehouse every 60 minutes, in open Iceberg format. You get a single, always-current source of truth you can query directly in BigQuery, Snowflake, Databricks, and other warehouses, with no export jobs to manage and no lock-in.
Data Fabric is currently on waitlist - reach out to Arize Support to get started.
Data Fabric is powered by **adb**, Arize's proprietary, AI-native online analytical processing (OLAP) database and the engine behind every managed Arize AX instance. adb stores your traces, evaluations, and annotations in open, standard formats, so Data Fabric can hand that data off to the warehouse of your choice without copying it into a proprietary format. For a deeper look at adb, see the [documentation](/docs/ax/concepts/adb/overview).
## Why is this better than a standard data export?
* **No lock-in:** Your data is always available to you. You can move it to data warehouse of your choice so you can use it in other tools you already use.
* **Single source of truth, always:** No need to maintain separate copies or manage export jobs
* **Automatic updates:** Your data, up-to-date. Any updates via evaluations or annotations, even on months-old data, are captured in the next sync regardless of timestamp. Data syncs every 60 minutes.
* **Query-ready format:** Data is stored in Iceberg format for direct querying in BigQuery, Snowflake, and other data warehouses
* **Time-partitioned:** Leverages Hive standard storage for efficient time-based queries
## Query and analyze synced traces
Once your traces land in the warehouse, they behave like any other table, with typed columns for tokens, cost, tool calls, latency, and evaluation scores. You can filter, aggregate, and join them with standard SQL instead of parsing unstructured logs. The value comes from joining traces against the business data already in your warehouse:
1. **Cost.** Join against your cloud billing export to see which prompt templates and sessions drive spend.
2. **Infrastructure.** Join against your warehouse's `INFORMATION_SCHEMA` to tell model latency apart from infrastructure contention, so you tune the right layer.
3. **Customer impact.** Join against CRM and revenue data to measure AI spend relative to account value and correlate evaluation scores with outcomes like CSAT.
Data Fabric syncs the data, but useful joins depend on your own data modeling: consistent session and user identifiers across systems, clear attribution between interactions and outcomes, and well-structured CRM and revenue tables.
## How does Data Fabric work?
* Data Fabric is only enabled for Enterprise accounts. Reach out to Arize Support ([support@arize.com](mailto:support@arize.com)) if you'd like to trial access.
* To set up Data Fabric, you must have write permissions to your target cloud storage bucket and at least one tracing project in your space
* Connectors are a connection to a filepath within your bucket. When you create a connector, you'll be able to specify a bucket and namespace, as well as projects to sync. You can add any number of projects or create one connector per project. Each connector must have a unique filepath.
* Once you've created your connectors and added your projects, your data will sync automatically every 60 minutes. This includes updates to historical data that may have changed.
* Project syncs can be paused, resumed or deleted as needed.
* **Supported Blobstore Providers:** Google Cloud Storage (GCS) and Amazon S3
* Coming Soon: Azure Storage
* **Supported Big Table Providers**: BigQuery (GCS), Databricks Unity Catalog
* Coming Soon: Snowflake
## Setting up Data Fabric
### Step 1: Create a Data Connector
1. Navigate to **Settings > Data Fabric** in your space
2. Click **New Connector**
3. Fill out the basic connector information:
1. **Connector Name:** A descriptive name for your connector
2. **Select Projects:** Choose which tracing projects to sync. You can modify these projects later
### Step 2: Configure Cloud Storage
1. **Select Data Storage:** Choose your cloud storage provider.
2. **File Path:** Enter your bucket path in the format shown below.
Enter your GCS path: `my-data-bucket/arize-sync/production`
Enter your S3 path: `my-bucket/optional-prefix`
### Step 3: Set Up Permissions
1. **Label Your Bucket:** In the GCS bucket, set a bucket **label** with a key of `arize-ingestion-key` and the corresponding value copied from the setup dialog. This proves ownership of the bucket.
* Key: `arize-ingestion-key`
* Value: See setup dialog
2. **Create IAM Role:** Run the provided command to create a custom IAM role.
```bash theme={null}
gcloud iam roles create arizeDataFabric \
--project=YOUR_PROJECT_ID \
--title="Arize Data Fabric Role" \
--description="Custom IAM role for Arize Data Fabric" \
--permissions=storage.buckets.get,storage.objects.get,storage.objects.list,storage.objects.create,storage.objects.update,storage.objects.delete \
--stage=ALPHA
```
3. **Apply IAM Permissions:** Grant the IAM role permission to your bucket.
```bash theme={null}
gsutil iam ch serviceAccount:arize-data-fabric@production-269901.iam.gserviceaccount.com:projects//roles/arizeDataFabric
```
1. **Tag Your Bucket:** In the S3 bucket, add a bucket **tag** with a key of `arize-ingestion-key` and the corresponding value copied from the setup dialog. This proves ownership of the bucket.
* Key: `arize-ingestion-key`
* Value: See setup dialog
You can add the tag via the AWS Console or CLI:
```bash theme={null}
aws s3api put-bucket-tagging \
--bucket YOUR_BUCKET_NAME \
--tagging 'TagSet=[{Key=arize-ingestion-key,Value=YOUR_VALUE_FROM_SETUP_DIALOG}]'
```
2. **Create IAM Policy:** Create an IAM policy that grants the Arize Data Fabric service role access to your bucket.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetBucketTagging"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME",
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
]
}
]
}
```
3. **Apply IAM Permissions:** Attach the policy to the Arize Data Fabric IAM role. The role ARN is provided in the setup dialog.
```bash theme={null}
aws iam put-role-policy \
--role-name arize-data-fabric \
--policy-name arize-data-fabric-s3-access \
--policy-document file://policy.json
```
The exact Arize IAM role ARN is provided in the Data Fabric setup dialog. Use this ARN when configuring trust policies or cross-account access.
### Step 4: Validate and Start Sync
1. **Validate:** Click Validate to verify your configuration
2. **Start Syncing:** Once validated, click Start Job to begin syncing. Your first sync will begin immediately and then continue every 60 minutes
### Step 5: Set Up Query Tables
1. **Allow the initial sync to complete:** Allow the first sync to complete. Sync time depends on data size and shape, and may vary.
2. **Create Table:** Once your data is syncing, create external tables to query the data directly. For each project being synced, create an external table using Iceberg or Delta format:
```sql theme={null}
CREATE EXTERNAL TABLE `your-project.your-dataset.your-table`
OPTIONS (
format = 'ICEBERG',
uris = ['gs://your-bucket/path/namespace/project-name/metadata/latest.metadata.json']
);
```
Once the first sync finishes, follow the standard Unity Catalog pattern for governing external S3 data: create a storage credential, create an external location that uses it, then register the Iceberg or Delta table at that location.
1. **Create a storage credential** that wraps the IAM role with access to the Arize sync bucket. See the Databricks [storage credentials docs](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-storage-credentials) for the full reference.
```sql theme={null}
CREATE STORAGE CREDENTIAL arize_data_fabric_cred
WITH AWS_IAM_ROLE 'arn:aws:iam::123456789012:role/arize-data-fabric'
COMMENT 'Credential for the Arize Data Fabric S3 sync bucket';
```
2. **Create an external location** that binds the credential to the Arize sync prefix in your bucket.
```sql theme={null}
CREATE EXTERNAL LOCATION arize_traces_loc
URL 's3://my-bucket/arize-sync/production'
WITH (CREDENTIAL arize_data_fabric_cred)
COMMENT 'Arize Data Fabric trace sync output';
```
3. **Register the external table** at the Iceberg metadata file under that location.
```sql theme={null}
CREATE TABLE my_catalog.arize_traces.agent_project
USING DELTA
LOCATION 's3://my-bucket/arize-sync/production/namespace/agent_project_a/metadata/latest.metadata.json';
```
Arize provides dual compatibility for both Delta and Iceberg formats without disrupting your existing Databricks pipelines. Unity Catalog seamlessly takes care of the rest, enforcing permissions, tracking end-to-end lineage, and maintaining audit logs across both formats.\\
Support for Snowflake and Athena is coming soon. Reach out to Arize Support for early access.
## Frequently Asked Questions
* **How often does data sync?** Data syncs every 60 minutes automatically.
* **Can I sync multiple spaces to the same bucket?** Yes, you can configure multiple connectors to write to the same bucket using different namespaces or prefixes. For stronger access isolation, we recommend using separate buckets or prefixes per space. See [Access Control Best Practices](/docs/ax/security-and-settings/data-fabric/access-control) for details.
* **What happens if I delete a project that's being synced?** The sync will stop for that project, but existing data in your storage will remain.
* **Can I change the sync frequency?**\
Currently, the sync frequency is fixed at 60 minutes and cannot be customized. Customization is coming soon.\\
* **Is historical data included in the sync?** Yes, all historical data is included in the initial sync, and any changes to historical data between syncs will be included in the next sync.
* **What's the difference between Data Fabric and manual exports?** Data Fabric provides automatic, continuous syncing with evaluations and annotations, while exports are manual snapshots at a point in time.
To learn more about adb, see the concept pages: