Overview

Monte Carlo integrates with GCP Dataform to provide data lineage, run history, and workflow failure alerts from your Dataform repositories. We scan your workflow configurations, compilation results, and workflow invocations to surface lineage between the tables your Dataform actions read from and write to, and track the execution status of every scheduled workflow run. Those lineage edges and run details appear alongside your warehouse-native observability in Monte Carlo.

When a Dataform workflow fails, Monte Carlo can surface it as an alert — giving you centralized incident triage, notification routing, and data quality reporting across all your data and system issues. You can optionally configure a webhook so failures are detected immediately rather than on the next hourly collection.

Feature Support

What We Collect

Monte Carlo ingests the following components from your Dataform environment:

Repositories

Each Dataform repository in your configured GCP project is discovered and used as an organizational grouping for the workflows and actions within it.

Workflow Configurations

Workflow configurations define which actions to execute and on what schedule. Monte Carlo treats each workflow configuration as a trackable job — you can view its run history, current status, and configure alerts for failures.

Actions

Monte Carlo extracts individual actions from each workflow configuration's compilation result. Actions include tables, views, incremental tables, materialized views, assertions, and operations. For each action, we capture:

• Type — table, view, incremental table, materialized view, assertion, or operation
• Outputs — which table or view the action creates or updates (for relation and output-producing operation types)
• Inputs — which tables the action depends on

Workflow Invocations (Runs)

Monte Carlo collects workflow invocations — the execution history of your workflow configurations. For each invocation we capture:

• status
• start and end times
• error message (if failure)

Prerequisites

Before adding the integration, you'll need:

• A GCP project with the Dataform API enabled
• At least one Dataform repository with workflow configurations in that project
• A GCP service account with the required permissions (see below)

Required GCP Permissions

The service account needs read-only access to Dataform resources. Grant the following IAM role on the GCP project:

Installation

Step 1: Create a Service Account in GCP

1. In the GCP Console, navigate to IAM & Admin → Service Accounts.
2. Click Create Service Account.
3. Configure the account:
   • Name: monte-carlo-dataform (or your preferred label)
   • Description: Read-only access for Monte Carlo Dataform integration
4. Click Create and Continue.
5. Grant the service account the Dataform Viewer role (roles/dataform.viewer) on your project.
6. Click Done.

Step 2: Create a Service Account Key

1. From the Service Accounts list, click on the service account you just created.
2. Go to the Keys tab → Add Key → Create new key.
3. Select JSON and click Create.
4. Save the downloaded JSON key file. You will provide this to Monte Carlo in the next step. Treat this file like a password — Monte Carlo stores it encrypted and never surfaces it back through the UI.

Step 3: Add the Dataform Integration in Monte Carlo

1. In Monte Carlo, navigate to Settings → Integrations.

2. Click Create and select GCP Dataform from the Orchestration section.

3. Fill in the form:

   Field	Value
   GCP Project ID	The GCP project ID hosting your Dataform repositories (e.g. my-project-123).
   Locations	The GCP region(s) where your Dataform repositories are located (e.g. us-central1). You can specify multiple.
   Service Account Key	The contents of the JSON key file downloaded in Step 2.

4. Click Test Connection. Monte Carlo authenticates against the Dataform API and verifies it can list repositories and workflow configurations.

5. On success, click Save.

Once the integration is saved, Monte Carlo begins collecting your Dataform metadata on an hourly schedule. Assets and lineage typically appear in the UI within an hour of the first successful collection.

Step 4: Confirm data is flowing

1. In Monte Carlo, navigate to Assets -> Jobs and search for one of your Dataform workflow configurations.
2. Open the asset and verify you can see:
   • Run history — recent workflow invocations with status and duration
   • Actions — the individual actions within the workflow
3. Navigate to the Table Lineage tab on a table that your Dataform actions write to. Look for the Dataform lineage edges connecting upstream and downstream tables.

Webhook Setup (Optional)

By default, Monte Carlo collects Dataform run data on an hourly schedule. You can optionally configure a webhook so that Monte Carlo is notified immediately when a workflow fails, triggering an on-demand collection to capture the failure details without waiting for the next scheduled cycle.

How It Works

1. You create a log-based alert in GCP Cloud Monitoring that fires when a Dataform workflow invocation reaches a FAILED terminal state.
2. The alert sends a webhook notification to a Monte Carlo URL (provided during onboarding).
3. Monte Carlo validates the webhook payload (project, repository) and triggers an immediate run collection.

If you don't set up the webhook, Monte Carlo will still detect the failure — it will just be picked up on the next hourly collection instead of immediately.

Step 1: Get Your Webhook URL

1. In Monte Carlo, navigate to Settings → Integrations → Orchestration
2. Click on your GCP Dataform integration and find the Webhook configuration section of the details page.
3. Copy the webhook URL provided. It will look like:

   https://integrations.getmontecarlo.com/webhooks/dataform/<your-unique-id>

Step 2: Create a Notification Channel in GCP

1. In the GCP Console, navigate to Monitoring → Alerting → Edit notification channels.
2. Under Webhooks, click Add New.
3. Configure the channel:
   • Display name: Monte Carlo Dataform Alerts
   • Endpoint URL: Paste the webhook URL from Step 1
4. Click Save.

Step 3: Create a Log-Based Alert

1. In the GCP Console, navigate to Monitoring → Alerting → Create Policy.

2. Click Add Condition and select Log match condition.

3. Paste the following log query filter:

   resource.type="dataform.googleapis.com/Repository"
   jsonPayload.@type="type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
   jsonPayload.terminalState="FAILED"

4. Click Next to configure notifications.

5. Select the Monte Carlo Dataform Alerts notification channel you created in Step 2.

6. Give the alert policy a name (e.g. Dataform Workflow Failure Alert) and click Create Policy.

Once configured, any Dataform workflow failure will immediately notify Monte Carlo, triggering a collection so the failure appears in your run history within minutes rather than up to an hour.

FAQs

Q: How often does Monte Carlo collect Dataform data?

Collections run hourly by default. If you configure the webhook, workflow failures trigger an immediate on-demand collection in addition to the scheduled hourly cycle.

Q: I added the integration but I don't see any lineage. What should I check?

In order of likelihood:

1. BigQuery isn't connected to Monte Carlo. Dataform lineage only resolves when the BigQuery warehouse that Dataform writes to is also connected as a Monte Carlo integration.
2. Your workflow configs don't have a release config. Monte Carlo extracts actions via the compilation result referenced by a release config. Workflow configurations without a linked release config won't produce lineage.
3. Allow up to 24 hours. Lineage is processed in batches and may take up to 24 hours to appear, even though assets and runs are available within an hour.

Q: Can I connect multiple GCP projects?

Yes. Create a separate integration for each GCP project at Settings → Integrations. Each integration requires its own service account key with roles/dataform.viewer on the target project.

Q: My Dataform repositories are in multiple regions. Do I need separate integrations?

No. When configuring the integration, you can specify multiple locations (e.g. us-central1, europe-west1). Monte Carlo scans all specified locations within the same integration.

Q: What happens if a workflow fails but I don't have the webhook configured?

Monte Carlo will still detect the failure during the next scheduled hourly collection. The webhook just ensures the failure is captured immediately rather than waiting up to an hour.