--- updatedAt: 2026-09-17T20:21:02.000Z agentTools: projectIndex: https://docs.getmontecarlo.com/llms.txt --- # Example: Matillion Maia (Public Preview) # Overview This guide walks through setting up a [custom ETL connector](https://docs.getmontecarlo.com/docs/custom-connectors) for [Matillion](https://www.matillion.com/) Data Productivity Cloud (DPC, also known as "Maia") using a pre-built example implementation. Matillion DPC is a cloud-native data pipeline platform. This example connector talks to the DPC REST API directly (there is no official Matillion Python SDK) to collect pipeline metadata and run history, so Monte Carlo can track pipeline runs, alert on failures, and surface a performance dashboard. The agent framework handles scheduling, pagination, and delivery — you only supply credentials. This implementation is available as a reference in the [mcd-public-resources](https://github.com/monte-carlo-data/mcd-public-resources/tree/main/custom_connectors/matillion_maia) repository. You can use it as-is or as a starting point for your own customizations. **One project per connection** The connector is scoped to a single Matillion **project** (supplied as the `project_id` credential) — that project _is_ the Monte Carlo connection. To monitor multiple projects, add one integration per project. # Feature support | Category | Capability | Support | | --------- | ---------------------------- | -------------------- | | Pipelines | Pipeline Failure Alerts | ✅ | | Pipelines | Webhook triggered collection | ✅ | | Pipelines | Pipeline Run History | ✅ | | Lineage | Table-to-Table Lineage | ✅ | | Lineage | Task-to-Task lineage | :white\_check\_mark: | | Lineage | Column-level lineage | ❌ | Table-to-table lineage is produced via **SQL query tagging**, not declared `inputs`/`outputs` — the DPC API exposes no structured read/written-table information. See [Matillion-specific notes](#matillion-specific-notes) below. # Prerequisites * A running [custom-connector-setup](https://github.com/monte-carlo-data/custom-connector-setup) repository — see [Custom ETL Connectors](https://docs.getmontecarlo.com/docs/custom-connectors) for initial setup. * A Matillion DPC project with API access enabled. * DPC API credentials (OAuth2 client-credentials): a **client ID** and **client secret**. * The **project ID** (UUID) of the DPC project you want to monitor. # Matillion-specific notes Understanding how Matillion's concepts map to Monte Carlo's ETL data model is helpful if you plan to customize the connector: | Monte Carlo concept | Matillion concept | | :------------------ | :-------------------------------------------------------------------------------- | | **Connection** | A DPC **project** (from `project_id`) — one integration per project. | | **Group** | A DPC **environment** (a warehouse-connection config), carried on each run. | | **Job** | A published **pipeline**, keyed by its `pipelineName`. | | **Task** | A pipeline **component** / step, emitted at runtime as `task_runs` on run events. | * **Jobs are discovered from schedules** — `fetch_metadata` lists the project's schedules; each schedule carries its pipeline, environment, and cron in one call. A pipeline with several schedules is upserted idempotently by `job_source_id`; a pipeline with no schedule is not discovered. * **Tasks are best-effort** — the DPC API exposes a pipeline's components only at runtime, so metadata enriches each job with the components from its most recent run. A pipeline that has never run reports no tasks. * **Runs of any trigger are reported** — polling lists all executions in the collection window (scheduled and ad-hoc/manual alike), not just scheduled ones. * **Auth** — OAuth2 client-credentials against `id.core.matillion.com`. The bearer token is short-lived (\~30 min) and refreshed automatically on expiry. # Setup ## 1. Scaffold the connector Use the `create_connector` script with the `--etl` flag to scaffold the connector directory. This generates a unique identifier for your connector in `manifest.json` — this ID is how Monte Carlo distinguishes connector types, so it must be generated per-installation. ```bash python scripts/create_connector.py matillion_maia --etl ``` You will be asked what the Matillion name for "Group", "Job" and "Task" are. This will change how these assets are named in Monte Carlo. We recommend: ```text bash What does this tool call a group of jobs? (default: Group): Environment What does this tool call a job? (default: Job): Pipeline What does this tool call a task? (default: Task): Component Icon URL (leave blank to skip): ``` The script also prompts for an optional icon URL to display Matillion's logo alongside the connection in the Monte Carlo UI. You will now have a manifest.json file created. Replace `run_status_mapping` and `credentials_schema` with the following: ```text "run_status_mapping": { "SUCCESS": "success", "FAILED": "failed", "RUNNING": "in_progress", "CANCELLING": "cancelling", "CANCELLED": "cancelled", "SKIPPED": "skipped", "FORBIDDEN": "error", "UNKNOWN": "unknown" }, "credentials_schema": { "connect_args": { "type": "dict", "required": true, "schema": { "client_id": { "type": "string", "required": true, "empty": false }, "client_secret":{ "type": "string", "required": true, "empty": false }, "project_id": { "type": "string", "required": true, "empty": false }, "region": { "type": "string", "required": false, "allowed": ["us1", "eu1"] }, "api_base_url": { "type": "string", "required": false } } } } ``` The full manifest file should look something like this with a unique connection\_type: ```json { "connection_type": "custom-etl-connector-XXXXXXX", "connection_name": "matillion_maia", "asset_class": "etl", "terminology": { "group": "Environment", "job": "Pipeline", "task": "Component" }, "run_status_mapping": { "SUCCESS": "success", "FAILED": "failed", "RUNNING": "in_progress", "CANCELLING": "cancelling", "CANCELLED": "cancelled", "SKIPPED": "skipped", "FORBIDDEN": "error", "UNKNOWN": "unknown" }, "credentials_schema": { "connect_args": { "type": "dict", "required": true, "schema": { "client_id": { "type": "string", "required": true, "empty": false }, "client_secret":{ "type": "string", "required": true, "empty": false }, "project_id": { "type": "string", "required": true, "empty": false }, "region": { "type": "string", "required": false, "allowed": ["us1", "eu1"] }, "api_base_url": { "type": "string", "required": false } } } } } ``` ## 2. Replace with the example implementation Clone or download the example from [mcd-public-resources](https://github.com/monte-carlo-data/mcd-public-resources/tree/main/custom_connectors/matillion_maia) and replace the stub `connector.py` with the pre-built implementation: ```bash cp /custom_connectors/matillion_maia/connector.py connectors/matillion_maia/ ``` No vendor SDK is required — the connector calls the DPC REST API directly with `requests`, which is already available in the agent base image. Leave `requirements.txt` empty and no `Dockerfile.extra` is needed. ## 3. Configure credentials Create `connectors/matillion_maia/credentials.json` using the [example](https://github.com/monte-carlo-data/mcd-public-resources/blob/main/custom_connectors/matillion_maia/credentials_example.json) as a reference: ```json { "connect_args": { "client_id": "", "client_secret": "", "region": "us1", "project_id": "" } } ``` | Field | Description | Default | | :----------------- | :----------------------------------------------- | :------ | | **client\_id** | DPC API credential ID | | | **client\_secret** | DPC API credential secret | | | **region** | DPC region (`us1` or `eu1`) | `us1` | | **project\_id** | UUID of the DPC project this connection collects | | You can override the derived region base URL with an optional `api_base_url` key inside `connect_args` — useful for non-standard DPC deployments. ## 4. Build and run the test suite Build the test Docker image, then run each test section to validate the connector against the DPC API: ```bash docker compose build # Verify connection (fetches an OAuth token) CONNECTOR=matillion_maia docker compose run --rm test -m etl_connection # Metadata collection (pipelines discovered via schedules) CONNECTOR=matillion_maia docker compose run --rm test -m etl_metadata # Run details (polling + webhook modes) CONNECTOR=matillion_maia docker compose run --rm test -m etl_run_details ``` The test suite validates that the connector returns correctly structured dicts matching the [EtlAsset](https://docs.getmontecarlo.com/docs/push-ingest-api) and [EtlRunEvent](https://docs.getmontecarlo.com/docs/push-ingest-api) schemas. Test lookback defaults to 7 days (configurable via the `ETL_TEST_LOOKBACK_HOURS` env var). ## 5. Export and build the agent image Once all tests pass, export capabilities and build the deployable image: ```bash # Export capabilities CONNECTOR=matillion_maia docker compose run --rm test --export # Build the agent image python scripts/generate_agent_image.py --connector matillion_maia ``` ## 6. Deploy, register, and connect From here, follow the standard custom ETL connector workflow: 1. **Push the image** to your container registry and [deploy the agent](https://docs.getmontecarlo.com/docs/custom-connectors), providing your production DPC credentials via [self-hosted credentials](https://docs.getmontecarlo.com/docs/self-hosted-credentials). 2. **Register the connector** by [running agent validations](https://docs.getmontecarlo.com/docs/custom-connectors) in the Monte Carlo UI (Settings → Deployments → your agent → Validate). 3. **Add the connection** through the [Monte Carlo UI](https://docs.getmontecarlo.com/docs/custom-connectors) (Settings → Integrations → Add → Orchestration → Custom ETL integration). Select **Collect** to have Monte Carlo pull on a schedule, and configure the [webhook](https://docs.getmontecarlo.com/docs/custom-connectors#webhook-collection) for near-real-time failure detection. After setup, Monte Carlo collects pipeline metadata on the agent's schedule and polls run details every 60 minutes. # Near-real-time failure alerts (webhook) Polling already catches failures with no extra setup — the agent collects run details every 60 minutes, so a failure surfaces within \~60 minutes. For many teams that's enough. For near-real-time alerting, Monte Carlo exposes a [webhook](https://docs.getmontecarlo.com/docs/custom-connectors#webhook-collection) you `POST` to when a pipeline fails. Matillion can't call that webhook directly. Its [Webhook Post component](https://docs.maia.ai/docs/components/webhook-post) sends only a URL and a JSON body — it can't set custom request headers, and Monte Carlo's webhook requires two: | Header | Value | | ----------- | ---------------------- | | x-mcd-id | Integration key ID | | x-mcd-token | Integration key secret | You get both values when you add the Custom ETL connection in Monte Carlo (step 6) — after choosing **Collect**, Monte Carlo lands you on a page with the **webhook URL** and a button to **generate the integration key** (its ID and secret). To bridge the gap, put a small forwarding proxy between Matillion and Monte Carlo. The Matillion Webhook Post component calls your proxy; the proxy injects the two headers and forwards the request to the Monte Carlo webhook URL. ```mermaid flowchart LR P["Matillion pipeline
Webhook Post (on failure)"] --> X["Your proxy
adds x-mcd-id / x-mcd-token"] X --> MC["Monte Carlo webhook"] MC --> A["Run collection
(failure alert)"] ``` Any lightweight, stateless HTTP endpoint works — AWS API Gateway + Lambda, a Google Cloud Function, a Cloudflare Worker, or a reverse proxy you already run. The proxy only needs to: 1. Add the `x-mcd-id` and `x-mcd-token` headers. 2. Forward the request to the Monte Carlo webhook URL. You can forward to the **bare** webhook URL — Monte Carlo then collects all runs from the last hour and picks up the failed one, so the proxy doesn't need to read Matillion's payload at all. To scope collection to just the failed run, have the proxy parse the pipeline or run id out of the Matillion JSON payload and append `?job_source_id=` (recent runs for that pipeline) or `?job_run_id=` (that specific run) when forwarding. Configure the Webhook Post component on the pipeline's failure path so it fires only when a run fails. # Lineage via SQL query tagging Table ↔ pipeline lineage is **not** available from the DPC API — it exposes no structured read/written-table information, so the connector intentionally omits `inputs`/`outputs`. To get warehouse lineage, tag each pipeline's SQL with a JSON comment carrying `mcd_job_id` set to the connector's `job_source_id` (the `pipelineName`): ```sql -- {"mcd_job_id": "my_pipeline.orch.yaml"} CREATE TABLE ... AS SELECT ...; ``` Optionally add `mcd_task_id` (a component's `task_source_id`) to attribute lineage to a specific component, or `mcd_resource_id` (the ETL connection's resource UUID) to disambiguate when multiple connections share pipeline names. Monte Carlo ingests these tags through its standard warehouse query-log collection and resolves them back to the jobs and tasks this connector reports — no extra connector code required. See [Lineage Between Jobs and Tables](https://docs.getmontecarlo.com/docs/lineage-between-jobs-and-tables) for details. ### Injecting the tag with a Matillion system variable You don't have to hardcode the pipeline name in each component. Matillion exposes it as a [system variable](https://docs.maia.ai/docs/guides/system-variables#list-of-system-variables), `${sysvar.thisPipeline.fullName}`, which resolves to the same `pipelineName` the connector uses as `job_source_id`. Reference it in a component's SQL definition so the tag stays correct as pipelines are renamed or copied: ```sql -- {"mcd_job_id": "${sysvar.thisPipeline.fullName}"} CREATE TABLE ... AS SELECT ...; ``` ### Ingest the warehouse's query logs SQL query tagging only works if Monte Carlo is collecting query logs from the warehouse the pipelines run against (Snowflake, BigQuery, etc.). That warehouse must be connected to Monte Carlo through its **own separate integration** with query log collection enabled — this is what makes the tables appear in the catalog in the first place, and specifically what lets Monte Carlo read the tags and resolve the pipeline → table edges. Without it, the tags are never ingested and no lineage is produced.