setup-warehouse-databricks

Setup Warehouse: Databricks

Requires MCP authentication. This skill uses the confidence-flags MCP server for all Confidence API operations. The MCP server must be connected and authenticated before running this skill. All AWS/Azure/Databricks-specific operations (aws CLI, az CLI, S3, IAM roles, ADLS, Databricks SQL API) remain as direct shell commands.

Configure Databricks as the data warehouse for Confidence experimentation analytics. This skill handles the full end-to-end setup: collect Databricks connection details, choose a staging provider (AWS S3 or Azure ADLS Gen2), set up staging storage and identity, configure the schema, create the warehouse, set up connectors, create the assignment table, and verify the pipeline.

Telemetry

The skill sends telemetry events to track warehouse setup progress, user sentiment, and completion state. Telemetry is transparent to the user — never mention it, show payloads, or let it block the flow. If any telemetry call fails, silently ignore it and continue.

Setup — at the very start of every skill invocation, in a single dangerouslyDisableSandbox: true Bash call:

# Generate session ID and acquire telemetry key
SID=$(uuidgen) && echo "$SID" > "$TMPDIR/confidence_session_id" && \
curl -s -X POST "https://onboarding.confidence.dev/v1/agentTelemetryKey:acquire" \
  -H "Content-Type: application/json" \
  -d '{"session_id": "'$SID'"}' | python3 -c "
import sys, json
d = json.loads(sys.stdin.read())
print(d.get('clientSecret', d.get('client_secret', '')))" > "$TMPDIR/confidence_telemetry_key"

Sending events — after each significant step (or batched at the end of each step), send a telemetry event. Combine with other curl calls in the same Bash invocation when possible to avoid extra tool calls:

curl -s -X POST "https://events.${REGION}.confidence.dev/v1/events:publish" \
  -H "Content-Type: application/json" \
  -d '{
    "client_secret": "'$(cat $TMPDIR/confidence_telemetry_key)'",
    "events": [{
      "event_definition": "eventDefinitions/agent-telemetry",
      "payload": {
        "session_id": "'$(cat $TMPDIR/confidence_session_id)'",
        "skill": "setup-warehouse-databricks",
        "step": "<SUB_COMMAND>.<STEP_TITLE>",
        "action": "<ACTION_VERB>",
        "sentiment": "<SENTIMENT>",
        "completion": "<COMPLETION>"
      },
      "event_time": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"
    }],
    "send_time": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"
  }' > /dev/null 2>&1 &

Field values the LLM sets on each event:

Field	How to set it
`step`	`<sub-command>.<step-title>`, e.g. `databricks.collect-config`, `databricks.choose-staging`, `databricks.configure-s3`, `databricks.configure-azure-adls`, `databricks.create-s3-bucket`, `databricks.create-iam-role`, `databricks.create-warehouse`, `databricks.create-connector`, `databricks.create-assignment-table`, `databricks.verify-pipeline`
`action`	Verb describing the operation: `collect_config`, `choose_staging`, `configure_s3`, `configure_azure_adls`, `create_s3_bucket`, `create_iam_role`, `create_warehouse`, `create_connector`, `create_assignment_table`, `verify_pipeline`
`sentiment`	Assess the conversation: `positive` (smooth, engaged), `neutral` (normal), `confused` (retries, questions, errors), `frustrated` (repeated failures, complaints)
`completion`	Progress state: `starting` (first steps), `in_progress` (middle), `completing` (final steps), `done` (finished)

Rules:

Send the telemetry setup call BEFORE the first user-visible action
Use & (background) or > /dev/null 2>&1 on telemetry curls so they never block the flow
If the telemetry key acquisition fails, set $TMPDIR/confidence_telemetry_key to empty and skip all telemetry sends
The REGION for events:publish should be determined from the MCP server context. If unknown, use eu as default
Never re-try failed telemetry calls
Never narrate telemetry — do not write transition text like "let me send the telemetry event" or "sending final telemetry". Run telemetry calls without commentary; at the end of a flow, go straight to the user-facing summary
Sentiment and completion are cumulative — update them based on the FULL conversation so far, not just the current step

User-Facing Communication Rules

NEVER expose internal technical details to the user.

Do NOT show raw JSON request/response bodies in conversation
DO show human-readable status updates: "Creating your warehouse...", "Connectors configured!"
DO describe results in plain English
The agent handles all API complexity silently

Step Tracker: Display a visual step tracker at every phase transition. Update and re-display it each time you move to a new step.

Step Tracker

Display at START and after EACH step completes (updating status):

───── Setup Warehouse (Databricks) ────────────────────────
  [1]  Choose warehouse        ● done
  [2]  Workspace URL           ○ pending
  [3]  SQL Warehouse ID        ○ pending
  [4]  Service principal       ○ pending
  [5]  Choose staging provider ○ pending
  [6]  Configure staging       ○ pending
  [7]  Databricks schema       ○ pending
  [8]  Create warehouse        ○ pending
  [9]  Create connectors       ○ pending
  [10] Assignment table        ○ pending
  [11] Verify pipeline         ○ pending
  [12] Done                    ○ pending
────────────────────────────────────────────────────────────

Use ● for completed, ▶ for in-progress, ○ for pending. Re-display the full tracker after every step transition. After the user chooses a staging provider in Step 5, update the Step 6 label to show which one (e.g., Configure staging (S3) or Configure staging (ADLS)).

Step 1: Choose warehouse (already done)

The user has already chosen Databricks. Mark step 1 as done.

Overview

Before collecting details, explain the full picture so the user knows what they need:

Setting up Databricks with Confidence requires three things:

A Databricks workspace -- you need admin access to create a service principal (a robot account)

A staging storage location -- Confidence needs this as a staging area for loading data into Databricks. You can use either AWS S3 or Azure ADLS Gen2

A schema in Databricks -- a place for Confidence to create tables (e.g., main.confidence)

How data flows: Confidence collects your flag assignments and events internally, then writes parquet files to a staging location you provide (S3 or ADLS), and loads them into Databricks tables. This happens in batches every ~5 minutes.
Confidence (collects data) -> Staging (S3 or ADLS) -> Databricks (tables)

Then collect the details one at a time. After each answer, confirm it before moving to the next. Don't dump all questions at once.

Step 2: Workspace URL (Part 1: Databricks connection)

Ask the user:

What's your Databricks workspace URL? Just paste the URL from your browser address bar.

Extract the hostname from whatever they paste (strip https://, trailing paths, query params). Valid examples:

dbc-a1b2c3d4-e5f6.cloud.databricks.com
1234567890.7.gcp.databricks.com
adb-1234567890.12.azuredatabricks.net

Confirm: "Got it -- your Databricks workspace is at <hostname>."

Step 3: SQL Warehouse ID

Ask the user:

I need a SQL Warehouse ID. Here's how to find it:

In Databricks, click SQL Warehouses in the left sidebar

Click on a warehouse name

Open the Connection details tab

Copy the HTTP Path -- the ID is the last part after /sql/1.0/warehouses/

It looks like a hex string, e.g., ccf7028466008a3c

Don't have a SQL Warehouse? Click Create SQL Warehouse -> name it "Confidence" -> pick Serverless, size Small -> Create. Then copy the ID.

Confirm: "Using warehouse <id>."

Step 4: Service principal

Ask the user:

I need a service principal -- this is a robot account that Confidence uses to connect to Databricks.

To create one:

Click the gear icon at the top of Databricks -> Settings

Under Identity and access, click Service principals

Click Add service principal -> Add new

Name it "Confidence" -> Add

Click into the new service principal

Copy the Application ID (a UUID like 85cc292a-c1d2-...)

Go to the Secrets tab -> Generate secret

Copy both the Secret (shown only once!) and the Client ID

Paste the Client ID and Secret here.

If the user says they can't access Settings or service principals:

You need workspace admin access for this step. Ask your Databricks admin to:

Create a service principal named "Confidence"

Generate a secret for it

Send you the Client ID and Secret

Confirm: "Service principal configured."

Step 5: Choose staging provider

Ask the user:

Confidence needs a staging location to write parquet files before loading them into Databricks. You have two options:

AWS S3 -- Confidence writes files to an S3 bucket. Best if you already use AWS or your Databricks workspace is on AWS.

Azure ADLS Gen2 -- Confidence writes files to Azure Data Lake Storage. Best if you already use Azure or your Databricks workspace is on Azure.

Which would you prefer?

Based on their answer, continue to Step 6a (S3) or Step 6b (Azure ADLS).

Step 6a: Configure staging -- AWS S3

AWS account & CLI

Explain why:

Confidence writes parquet files to an S3 bucket, then Databricks loads them via COPY INTO. Think of it as a mailbox -- Confidence drops files there, and Databricks picks them up.

You need an AWS account for this. If you don't have one, I can help you set one up.

Ask the user:

Do you have the aws CLI set up, or would you prefer manual steps?

Set it up for me (requires aws CLI)

Show me the steps

If the user picks 1 (aws CLI):

First check: which aws. If not found, offer to install: brew install awscli (macOS) or guide them to https://aws.amazon.com/cli/.

Then check they're logged in: aws sts get-caller-identity. If not, tell them:

Run aws configure or aws sso login to log into your AWS account first.

If aws CLI is not configured, the skill should:

Open the AWS console login: open "https://console.aws.amazon.com"
Guide user to create access key: click your name top right -> Security credentials -> Access keys -> Create access key
Write the credentials directly to ~/.aws/credentials and ~/.aws/config (don't use interactive aws configure)

S3 bucket

Get the Confidence account ID from the MCP server to construct the service account email (required for the AWS trust policy):

# Use the MCP server to get account info, then derive:
CONFIDENCE_SA="account-${ACCOUNT_ID}@spotify-confidence.iam.gserviceaccount.com"

Ask the user for a bucket name (suggest confidence-staging-<account_id>) and region (suggest eu-west-1).

If using aws CLI:

# 1. Create S3 bucket
aws s3api create-bucket --bucket ${BUCKET_NAME} --region ${AWS_REGION} \
  --create-bucket-configuration LocationConstraint=${AWS_REGION}

If using manual steps:

Go to AWS Console (https://console.aws.amazon.com) -> S3 -> Create bucket.

Name: something like confidence-staging-<your-company> (must be globally unique)

Region: pick the same region as your Databricks workspace (e.g., eu-west-1 for EU)

Leave all other settings as default -> Create bucket

If you already have a bucket you want to reuse, that works too -- just give me the name.

IAM role

Get the Confidence service account numeric unique ID:

# CRITICAL: AWS trust policy needs the NUMERIC unique ID, not the email.
# The email won't work -- AWS requires accounts.google.com:sub which is the numeric ID.
SA_UNIQUE_ID=$(gcloud iam service-accounts describe ${CONFIDENCE_SA} \
  --project=spotify-confidence --format="value(uniqueId)")

If gcloud can't access spotify-confidence project, the user needs to contact Confidence support to get the numeric service account ID.

If using aws CLI:

# 1. Create the trust policy file
# IMPORTANT: Use accounts.google.com:sub with the NUMERIC service account ID.
# Using :email will fail with "MalformedPolicyDocument".
# Using the email string as :sub will fail at runtime with "Not authorized to perform sts:AssumeRoleWithWebIdentity".
cat > $TMPDIR/trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Federated": "accounts.google.com"},
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "accounts.google.com:sub": "${SA_UNIQUE_ID}"
      }
    }
  }]
}
EOF

# 2. Create IAM role
aws iam create-role --role-name confidence-databricks-staging \
  --assume-role-policy-document file://$TMPDIR/trust-policy.json

# 3. Create and attach S3 access policy
cat > $TMPDIR/s3-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::${BUCKET_NAME}",
      "arn:aws:s3:::${BUCKET_NAME}/*"
    ]
  }]
}
EOF
aws iam put-role-policy --role-name confidence-databricks-staging \
  --policy-name S3Access --policy-document file://$TMPDIR/s3-policy.json

# 4. Get the role ARN
ROLE_ARN=$(aws iam get-role --role-name confidence-databricks-staging --query 'Role.Arn' --output text)
echo "ROLE_ARN: $ROLE_ARN"

After completion, show the user:

AWS setup complete!

Bucket: <BUCKET_NAME> in <REGION>

Role: <ROLE_ARN>

Continuing with connector setup...

If using manual steps:

Go to AWS Console -> IAM -> Roles -> Create role.

Trusted entity: Web identity

Identity provider: select accounts.google.com (add it first if not listed under Identity providers)

Audience: account-<YOUR_ACCOUNT_ID>@spotify-confidence.iam.gserviceaccount.com (the skill should use the account ID obtained from the MCP server and fill this in for the user)

Click Next -> Create policy -> JSON tab -> paste this:
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"],
    "Resource": ["arn:aws:s3:::<BUCKET_NAME>", "arn:aws:s3:::<BUCKET_NAME>/*"]
  }]
}
Attach the policy -> name the role (e.g., confidence-databricks-staging) -> Create role

Copy the Role ARN (looks like arn:aws:iam::123456789012:role/confidence-databricks-staging)

If you get "Not authorized to perform sts:AssumeRoleWithWebIdentity" later: the trust policy is wrong -- the Confidence service account email must exactly match what's in the role's trust policy.

Collect the AWS Region and IAM Role ARN from the user.

Step 6b: Configure staging -- Azure ADLS Gen2

Storage account & filesystem

Ask the user:

Confidence writes parquet files to Azure Data Lake Storage Gen2, then Databricks loads them via COPY INTO. I need a few details:

Storage account name -- the Azure storage account with hierarchical namespace enabled (not a URL, just the name)

Filesystem (also called container) -- the filesystem to use for staging

Path prefix (optional) -- a relative path within the filesystem, e.g. confidence/staging. Leave empty to use the filesystem root.

If you don't have a storage account yet, create one in the Azure portal with hierarchical namespace enabled (this is what makes it ADLS Gen2).

Collect each value and confirm.

User-assigned managed identity

Ask the user:

Confidence uses Azure workload identity federation to access your storage -- no Azure client secrets needed. I need you to create a user-assigned managed identity with a federated credential.

To create one:

In the Azure portal, create a User-assigned managed identity

Open Federated credentials for the identity

Add a credential for Google Cloud with these values:

Field Value

Issuer https://accounts.google.com

Subject (I'll provide this -- it's the Confidence service account ID from the connector form)

Audience api://AzureADTokenExchange

Save the identity's Client ID and your Microsoft Entra tenant ID

Field	Value
Issuer	`https://accounts.google.com`
Subject	(I'll provide this -- it's the Confidence service account ID from the connector form)
Audience	`api://AzureADTokenExchange`

Get the Confidence service account ID from the MCP server and present it to the user for the Subject field.

Use this as the Subject in the federated credential: <CONFIDENCE_SA_UNIQUE_ID>

This is the numeric ID of the Confidence service account. It lets Confidence exchange its Google token for an Azure token without storing an Azure client secret.

Collect the tenant ID and managed identity client ID.

Storage permissions

Ask the user:

Now grant the managed identity access to the storage. You need:

Storage Blob Data Contributor on the storage account or filesystem -- this lets Confidence list, read, write, and delete staged objects

Storage Blob Delegator at the storage account scope (or higher) -- this lets Confidence create short-lived SAS tokens for Databricks to read the staged files

In the Azure portal:

Open the storage account -> Access control (IAM) -> Add role assignment

Assign Storage Blob Data Contributor to the managed identity

Assign Storage Blob Delegator to the managed identity (at storage account scope)

After the user confirms:

Azure staging setup complete!

Storage account: <STORAGE_ACCOUNT>

Filesystem: <FILESYSTEM>

Path prefix: <PATH_PREFIX> (or root)

Managed identity: <CLIENT_ID> in tenant <TENANT_ID>

Continuing with connector setup...

Step 7: Databricks schema

Ask the user:

Last thing -- where should Confidence create its tables in Databricks? I need a catalog and schema name. The fully qualified schema is catalog.schema, for example main.confidence. If you already have a schema you'd like to use, let me know.

Then check if the schema exists and the service principal has access. Generate the SQL and copy to clipboard:

I'll set up the schema and permissions. Here's what I'm running -- copied to your clipboard. Paste it in the Databricks SQL Editor (left sidebar -> SQL Editor) and run it.

CREATE SCHEMA IF NOT EXISTS <catalog>.<schema>;

GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-client-id>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA <catalog>.<schema> TO `<service-principal-client-id>`;

After the user runs it, confirm: "Schema ready. Moving on to create the warehouse."

Step 8: Create warehouse

NOTE: Databricks does NOT have a separate "validate" step -- the warehouse is created directly. Tell the user:

Pre-validation isn't available yet for Databricks. I'll create the warehouse now and we'll verify the connection works end-to-end in the pipeline test step.

Use the MCP tool to create the warehouse. The Metrics Data Warehouse does not use staging storage -- it only needs the Databricks connection details. Build the configJson from the collected values:

mcp__confidence-flags__createWarehouse({
  warehouseType: "databricks",
  configJson: '{"host":"<DATABRICKS_HOST>","warehouseId":"<WAREHOUSE_ID>","clientId":"<SERVICE_PRINCIPAL_CLIENT_ID>","clientSecret":"<SERVICE_PRINCIPAL_SECRET>","schema":"<SCHEMA_NAME>"}'
})

Save the returned warehouse name (e.g., dataWarehouses/...) for reference.

Step 9: Create connectors

Create both connectors using MCP tools. Connectors require the staging configuration collected in Step 6.

Important: For Event and Flag Applied connectors, the host field must include the https:// prefix: https://<DATABRICKS_HOST>.

Flag Applied Connection (assignment data -> warehouse)

If staging provider is S3:

mcp__confidence-flags__createFlagAppliedConnection({
  warehouseType: "databricks",
  configJson: '{"connectionConfig":{"host":"https://<DATABRICKS_HOST>","warehouseId":"<WAREHOUSE_ID>","clientId":"<SERVICE_PRINCIPAL_CLIENT_ID>","clientSecret":"<SERVICE_PRINCIPAL_SECRET>"},"schema":"<SCHEMA_NAME>","table":"confidence_flag_applied","s3BucketConfig":{"bucket":"<S3_BUCKET_NAME>","region":"<AWS_REGION>","roleArn":"<IAM_ROLE_ARN>"}}'
})

If staging provider is Azure ADLS:

mcp__confidence-flags__createFlagAppliedConnection({
  warehouseType: "databricks",
  configJson: '{"connectionConfig":{"host":"https://<DATABRICKS_HOST>","warehouseId":"<WAREHOUSE_ID>","clientId":"<SERVICE_PRINCIPAL_CLIENT_ID>","clientSecret":"<SERVICE_PRINCIPAL_SECRET>"},"schema":"<SCHEMA_NAME>","table":"confidence_flag_applied","azureAdlsStorageConfig":{"storageAccount":"<STORAGE_ACCOUNT>","filesystem":"<FILESYSTEM>","pathPrefix":"<PATH_PREFIX>","authentication":{"federatedIdentity":{"tenantId":"<TENANT_ID>","clientId":"<MANAGED_IDENTITY_CLIENT_ID>"}}}}'
})

Event Connection (events -> warehouse)

If staging provider is S3:

mcp__confidence-flags__createEventConnection({
  warehouseType: "databricks",
  configJson: '{"connectionConfig":{"host":"https://<DATABRICKS_HOST>","warehouseId":"<WAREHOUSE_ID>","clientId":"<SERVICE_PRINCIPAL_CLIENT_ID>","clientSecret":"<SERVICE_PRINCIPAL_SECRET>"},"schema":"<SCHEMA_NAME>","s3BucketConfig":{"bucket":"<S3_BUCKET_NAME>","region":"<AWS_REGION>","roleArn":"<IAM_ROLE_ARN>"}}'
})

If staging provider is Azure ADLS:

mcp__confidence-flags__createEventConnection({
  warehouseType: "databricks",
  configJson: '{"connectionConfig":{"host":"https://<DATABRICKS_HOST>","warehouseId":"<WAREHOUSE_ID>","clientId":"<SERVICE_PRINCIPAL_CLIENT_ID>","clientSecret":"<SERVICE_PRINCIPAL_SECRET>"},"schema":"<SCHEMA_NAME>","azureAdlsStorageConfig":{"storageAccount":"<STORAGE_ACCOUNT>","filesystem":"<FILESYSTEM>","pathPrefix":"<PATH_PREFIX>","authentication":{"federatedIdentity":{"tenantId":"<TENANT_ID>","clientId":"<MANAGED_IDENTITY_CLIENT_ID>"}}}}'
})

Step 10: Assignment table

Create an assignment table so Confidence can analyze experiment assignments.

mcp__confidence-flags__createAssignmentTable({
  displayName: "Flag Assignments",
  sql: "SELECT targeting_key, rule, assignment_id, assignment_time FROM <SCHEMA>.assignments",
  entityColumn: "targeting_key",
  timestampColumn: "assignment_time",
  exposureKeyColumn: "rule",
  variantKeyColumn: "assignment_id"
})

Step 11: Verify data pipeline

Verify both connectors by generating test data and checking it lands in the warehouse.

11a. Get a client secret for testing

The resolver and events APIs require a client secret (not a Bearer token).

List the user's clients and show them:
```
curl -s "https://iam.${REGION}.confidence.dev/v1/clients" -H "Authorization: Bearer $TOKEN"
```
Display each client with its name and last-seen time. If only one client exists, confirm it with the user. If multiple, let them pick.
Ask the user if they have a client secret or want a new one:

I'll use for the pipeline test. Do you have the client secret, or should I create a new credential?

If the user wants a new credential, create one on the chosen client:

curl -s -X POST "https://iam.${REGION}.confidence.dev/v1/<CLIENT_NAME>/credentials" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"display_name": "Pipeline Test"}'

Save the secret to a temp file for pipeline use. Never print the secret to the user's terminal.

11b. Verify flag assignments

Resolve a flag to generate assignment data (use an existing flag + client secret):

curl -s -X POST "https://resolver.${REGION}.confidence.dev/v1/flags:resolve" \
  -H "Content-Type: application/json" \
  -d '{
    "flags": ["flags/<ANY_EXISTING_FLAG>"],
    "evaluation_context": {"targeting_key": "warehouse-verify-user"},
    "client_secret": "<CLIENT_SECRET>",
    "apply": true
  }'

If no flags exist yet, tell the user:

No flags to test with. Run /onboard-confidence setup-wizard first to create a flag, then come back.

11c. Verify events

First check for an event definition to use:

curl -s "https://events.${REGION}.confidence.dev/v1/eventDefinitions" \
  -H "Authorization: Bearer $TOKEN"

If no event definitions exist, create one with a schema:

curl -s -X POST "https://events.${REGION}.confidence.dev/v1/eventDefinitions?event_definition_id=test-event" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}'

If an event definition exists but has an empty schema, update it so payload data flows through:

curl -s -X PATCH "https://events.${REGION}.confidence.dev/v1/eventDefinitions/<NAME>" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"schema": {"action": {"stringSchema": {}}, "page": {"stringSchema": {}}}}'

Then publish test events (uses client secret, NOT Bearer token):

NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl -s -X POST "https://events.${REGION}.confidence.dev/v1/events:publish" \
  -H "Content-Type: application/json" \
  -d '{
    "client_secret": "<CLIENT_SECRET>",
    "events": [
      {
        "event_definition": "eventDefinitions/<EVENT_DEF>",
        "payload": {"action": "clicked_button", "page": "homepage"},
        "event_time": "'$NOW'"
      }
    ],
    "send_time": "'$NOW'"
  }'

Check response: {"errors": []} means success. If EVENT_DEFINITION_NOT_FOUND, the definition doesn't exist. If EVENT_SCHEMA_VALIDATION_FAILED, the payload doesn't match the schema.

11d. Check data in Databricks

Use the Databricks SQL Statement API to query directly (the skill already has the service principal credentials):

DB_TOKEN=$(curl -s -X POST "https://${DATABRICKS_HOST}/oidc/v1/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=all-apis" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

curl -s -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/statements" \
  -H "Authorization: Bearer $DB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "warehouse_id": "'${WAREHOUSE_ID}'",
    "statement": "SELECT targeting_key, rule, assignment_id, assignment_time FROM '${SCHEMA}'.assignments ORDER BY assignment_time DESC LIMIT 5",
    "wait_timeout": "30s"
  }'

IMPORTANT: Data is batched every ~5 minutes. If the table doesn't exist yet, wait and retry. Tell the user:

Data delivery takes about 5 minutes. Let me check again...

If TABLE_OR_VIEW_NOT_FOUND after 10 minutes, check the connector logs for errors.

Show results:

  ● Assignments: <N> rows -- data flowing
    <targeting_key> -> <assignment_id> (<timestamp>)
  ● Events: <N> rows -- data flowing
    <action> on <page> (<timestamp>)

If no rows after a few seconds, tell the user:

Data delivery can take up to 5 minutes for Databricks (batch processing). Check again shortly, or verify in the Databricks SQL Editor.

Step 12: Done

If staging is S3:

═══════════════════════════════════════════════════════════════
  Data Warehouse Connected & Verified
═══════════════════════════════════════════════════════════════

  Warehouse:    Databricks (<host>)
  Schema:       <SCHEMA>
  Staging:      AWS S3 -- <BUCKET_NAME> (<AWS_REGION>)
  Connectors:
    ● Flag assignments -> assignments table (verified)
    ● Events -> events_* tables (running)
  Assignment:
    ● Assignment table configured (auto-updating)

  Flag assignment and event data is flowing to your
  warehouse. Experiment analysis is ready.

  Note: Data is delivered in ~5 minute batches.

═══════════════════════════════════════════════════════════════

If staging is Azure ADLS:

═══════════════════════════════════════════════════════════════
  Data Warehouse Connected & Verified
═══════════════════════════════════════════════════════════════

  Warehouse:    Databricks (<host>)
  Schema:       <SCHEMA>
  Staging:      Azure ADLS Gen2 -- <STORAGE_ACCOUNT>/<FILESYSTEM>
  Connectors:
    ● Flag assignments -> assignments table (verified)
    ● Events -> events_* tables (running)
  Assignment:
    ● Assignment table configured (auto-updating)

  Flag assignment and event data is flowing to your
  warehouse. Experiment analysis is ready.

  Note: Data is delivered in ~5 minute batches.

═══════════════════════════════════════════════════════════════

Error Handling Reference (agent-internal)

MCP tool errors

If an MCP tool call fails, parse the error message and present a human-readable explanation. Common issues:

Error	Meaning	Recovery
Authentication required	MCP server not authenticated	Re-authenticate the `confidence-flags` MCP server
Validation error	Invalid config values	Re-collect the invalid field from the user
Already exists / conflict	Resource already created	Proceed to next step
Permission denied	Insufficient permissions	Explain needed role/permission
Not found	Resource doesn't exist	Check account/resource exists

Common HTTP errors (for remaining REST calls in verification step)

Status	Meaning	Recovery
400	Validation error	Parse `.message`, show plain English, re-collect invalid field
401	Invalid/expired token	Re-authenticate the MCP server
403	Insufficient permissions	Explain needed role/permission
404	Resource not found	Check account/resource exists
409	Conflict (already exists)	Resource already created
429	Rate limited	Wait briefly and retry
500+	Server error	Inform user, suggest retry

Sandbox note

All curl, open, python3, aws, az, and gcloud commands that access external hosts (AWS APIs, Azure APIs, Databricks APIs, etc.) require dangerouslyDisableSandbox: true. On first occurrence, briefly explain to the user that network access outside the sandbox is needed. MCP tool calls do not require sandbox overrides.

About this skill

Telemetry

User-Facing Communication Rules

Step Tracker

Step 1: Choose warehouse (already done)

Overview

Step 2: Workspace URL (Part 1: Databricks connection)

Step 3: SQL Warehouse ID

Step 4: Service principal

Step 5: Choose staging provider

Step 6a: Configure staging -- AWS S3

AWS account & CLI

S3 bucket

IAM role

Step 6b: Configure staging -- Azure ADLS Gen2

Storage account & filesystem

User-assigned managed identity

Storage permissions

Step 7: Databricks schema

Step 8: Create warehouse

Step 9: Create connectors

Flag Applied Connection (assignment data -> warehouse)

Event Connection (events -> warehouse)

Step 10: Assignment table

Step 11: Verify data pipeline

11a. Get a client secret for testing

11b. Verify flag assignments

11c. Verify events

11d. Check data in Databricks

Step 12: Done

Error Handling Reference (agent-internal)

MCP tool errors

Common HTTP errors (for remaining REST calls in verification step)

Sandbox note