Overview

This guide walks through setting up a custom ETL connector for Matillion Data Productivity Cloud (DPC, also known as "Maia") using a pre-built example implementation.

Matillion DPC is a cloud-native data pipeline platform. This example connector talks to the DPC REST API directly (there is no official Matillion Python SDK) to collect pipeline metadata and run history, so Monte Carlo can track pipeline runs, alert on failures, and surface a performance dashboard. The agent framework handles scheduling, pagination, and delivery — you only supply credentials.

This implementation is available as a reference in the mcd-public-resources repository. You can use it as-is or as a starting point for your own customizations.

Feature support

Prerequisites

• A running custom-connector-setup repository — see Custom ETL Connectors for initial setup.
• A Matillion DPC project with API access enabled.
• DPC API credentials (OAuth2 client-credentials): a client ID and client secret.
• The project ID (UUID) of the DPC project you want to monitor.

Matillion-specific notes

Understanding how Matillion's concepts map to Monte Carlo's ETL data model is helpful if you plan to customize the connector:

• Jobs are discovered from schedules — fetch_metadata lists the project's schedules; each schedule carries its pipeline, environment, and cron in one call. A pipeline with several schedules is upserted idempotently by job_source_id; a pipeline with no schedule is not discovered.
• Tasks are best-effort — the DPC API exposes a pipeline's components only at runtime, so metadata enriches each job with the components from its most recent run. A pipeline that has never run reports no tasks.
• Runs of any trigger are reported — polling lists all executions in the collection window (scheduled and ad-hoc/manual alike), not just scheduled ones.
• Auth — OAuth2 client-credentials against id.core.matillion.com. The bearer token is short-lived (~30 min) and refreshed automatically on expiry.

Setup

1. Scaffold the connector

Use the create_connector script with the --etl flag to scaffold the connector directory. This generates a unique identifier for your connector in manifest.json — this ID is how Monte Carlo distinguishes connector types, so it must be generated per-installation.

python scripts/create_connector.py matillion_maia --etl

You will be asked what the Matillion name for "Group", "Job" and "Task" are. This will change how these assets are named in Monte Carlo. We recommend:

What does this tool call a group of jobs? (default: Group): Environment
What does this tool call a job? (default: Job): Pipeline
What does this tool call a task? (default: Task): Component
Icon URL (leave blank to skip): 

The script also prompts for an optional icon URL to display Matillion's logo alongside the connection in the Monte Carlo UI.

You will now have a manifest.json file created. Replace run_status_mapping and credentials_schema with the following:

  "run_status_mapping": {
    "SUCCESS": "success",
    "FAILED": "failed",
    "RUNNING": "in_progress",
    "CANCELLING": "cancelling",
    "CANCELLED": "cancelled",
    "SKIPPED": "skipped",
    "FORBIDDEN": "error",
    "UNKNOWN": "unknown"
  },
  "credentials_schema": {
    "connect_args": {
      "type": "dict",
      "required": true,
      "schema": {
        "client_id":    { "type": "string", "required": true,  "empty": false },
        "client_secret":{ "type": "string", "required": true,  "empty": false },
        "project_id":   { "type": "string", "required": true,  "empty": false },
        "region":       { "type": "string", "required": false, "allowed": ["us1", "eu1"] },
        "api_base_url": { "type": "string", "required": false }
      }
    }
  }

The full manifest file should look something like this with a unique connection_type:

{
  "connection_type": "custom-etl-connector-XXXXXXX",
  "connection_name": "matillion_maia",
  "asset_class": "etl",
  "terminology": {
    "group": "Environment",
    "job": "Pipeline",
    "task": "Component"
  },
  "run_status_mapping": {
    "SUCCESS": "success",
    "FAILED": "failed",
    "RUNNING": "in_progress",
    "CANCELLING": "cancelling",
    "CANCELLED": "cancelled",
    "SKIPPED": "skipped",
    "FORBIDDEN": "error",
    "UNKNOWN": "unknown"
  },
  "credentials_schema": {
    "connect_args": {
      "type": "dict",
      "required": true,
      "schema": {
        "client_id":    { "type": "string", "required": true,  "empty": false },
        "client_secret":{ "type": "string", "required": true,  "empty": false },
        "project_id":   { "type": "string", "required": true,  "empty": false },
        "region":       { "type": "string", "required": false, "allowed": ["us1", "eu1"] },
        "api_base_url": { "type": "string", "required": false }
      }
    }
  }
}

2. Replace with the example implementation

Clone or download the example from mcd-public-resources and replace the stub connector.py with the pre-built implementation:

cp <path-to-mcd-public-resources>/custom_connectors/matillion_maia/connector.py connectors/matillion_maia/

No vendor SDK is required — the connector calls the DPC REST API directly with requests, which is already available in the agent base image. Leave requirements.txt empty and no Dockerfile.extra is needed.

3. Configure credentials

Create connectors/matillion_maia/credentials.json using the example as a reference:

{
  "connect_args": {
    "client_id": "<dpc-api-client-id>",
    "client_secret": "<dpc-api-client-secret>",
    "region": "us1",
    "project_id": "<dpc-project-uuid>"
  }
}

4. Build and run the test suite

Build the test Docker image, then run each test section to validate the connector against the DPC API:

docker compose build

# Verify connection (fetches an OAuth token)
CONNECTOR=matillion_maia docker compose run --rm test -m etl_connection

# Metadata collection (pipelines discovered via schedules)
CONNECTOR=matillion_maia docker compose run --rm test -m etl_metadata

# Run details (polling + webhook modes)
CONNECTOR=matillion_maia docker compose run --rm test -m etl_run_details

The test suite validates that the connector returns correctly structured dicts matching the EtlAsset and EtlRunEvent schemas. Test lookback defaults to 7 days (configurable via the ETL_TEST_LOOKBACK_HOURS env var).

5. Export and build the agent image

Once all tests pass, export capabilities and build the deployable image:

# Export capabilities
CONNECTOR=matillion_maia docker compose run --rm test --export

# Build the agent image
python scripts/generate_agent_image.py --connector matillion_maia

6. Deploy, register, and connect

From here, follow the standard custom ETL connector workflow:

1. Push the image to your container registry and deploy the agent, providing your production DPC credentials via self-hosted credentials.
2. Register the connector by running agent validations in the Monte Carlo UI (Settings → Deployments → your agent → Validate).
3. Add the connection through the Monte Carlo UI (Settings → Integrations → Add → Orchestration → Custom ETL integration). Select Collect to have Monte Carlo pull on a schedule, and configure the webhook for near-real-time failure detection.

After setup, Monte Carlo collects pipeline metadata on the agent's schedule and polls run details every 60 minutes.

Near-real-time failure alerts (webhook)

Polling already catches failures with no extra setup — the agent collects run details every 60 minutes, so a failure surfaces within ~60 minutes. For many teams that's enough. For near-real-time alerting, Monte Carlo exposes a webhook you POST to when a pipeline fails.

Matillion can't call that webhook directly. Its Webhook Post component sends only a URL and a JSON body — it can't set custom request headers, and Monte Carlo's webhook requires two:

You get both values when you add the Custom ETL connection in Monte Carlo (step 6) — after choosing Collect, Monte Carlo lands you on a page with the webhook URL and a button to generate the integration key (its ID and secret).

To bridge the gap, put a small forwarding proxy between Matillion and Monte Carlo. The Matillion Webhook Post component calls your proxy; the proxy injects the two headers and forwards the request to the Monte Carlo webhook URL.

flowchart LR
    P["Matillion pipeline<br>Webhook Post (on failure)"] --> X["Your proxy<br>adds x-mcd-id / x-mcd-token"]
    X --> MC["Monte Carlo webhook"]
    MC --> A["Run collection<br>(failure alert)"]

Any lightweight, stateless HTTP endpoint works — AWS API Gateway + Lambda, a Google Cloud Function, a Cloudflare Worker, or a reverse proxy you already run. The proxy only needs to:

1. Add the x-mcd-id and x-mcd-token headers.
2. Forward the request to the Monte Carlo webhook URL.

You can forward to the bare webhook URL — Monte Carlo then collects all runs from the last hour and picks up the failed one, so the proxy doesn't need to read Matillion's payload at all. To scope collection to just the failed run, have the proxy parse the pipeline or run id out of the Matillion JSON payload and append ?job_source_id=<pipeline_name> (recent runs for that pipeline) or ?job_run_id=<run_id> (that specific run) when forwarding.

Configure the Webhook Post component on the pipeline's failure path so it fires only when a run fails.

Lineage via SQL query tagging

Table ↔ pipeline lineage is not available from the DPC API — it exposes no structured read/written-table information, so the connector intentionally omits inputs/outputs. To get warehouse lineage, tag each pipeline's SQL with a JSON comment carrying mcd_job_id set to the connector's job_source_id (the pipelineName):

-- {"mcd_job_id": "my_pipeline.orch.yaml"}
CREATE TABLE ... AS SELECT ...;

Optionally add mcd_task_id (a component's task_source_id) to attribute lineage to a specific component, or mcd_resource_id (the ETL connection's resource UUID) to disambiguate when multiple connections share pipeline names. Monte Carlo ingests these tags through its standard warehouse query-log collection and resolves them back to the jobs and tasks this connector reports — no extra connector code required. See Lineage Between Jobs and Tables for details.

Injecting the tag with a Matillion system variable

You don't have to hardcode the pipeline name in each component. Matillion exposes it as a system variable, ${sysvar.thisPipeline.fullName}, which resolves to the same pipelineName the connector uses as job_source_id. Reference it in a component's SQL definition so the tag stays correct as pipelines are renamed or copied:

-- {"mcd_job_id": "${sysvar.thisPipeline.fullName}"}
CREATE TABLE ... AS SELECT ...;

Ingest the warehouse's query logs

SQL query tagging only works if Monte Carlo is collecting query logs from the warehouse the pipelines run against (Snowflake, BigQuery, etc.). That warehouse must be connected to Monte Carlo through its own separate integration with query log collection enabled — this is what makes the tables appear in the catalog in the first place, and specifically what lets Monte Carlo read the tags and resolve the pipeline → table edges. Without it, the tags are never ingested and no lineage is produced.