---
updatedAt: 2026-09-01T17:02:06.000Z
agentTools:
  projectIndex: https://docs.getmontecarlo.com/llms.txt
---

# Custom ETL and Orchestration Integrations (Public Preview)

<Callout icon="ℹ️" theme="info">
  This feature requires an Enterprise or higher plan to use.

  This feature is in Public Preview. See [here](https://docs.getmontecarlo.com/docs/integration-feature-lifecycles) for more information on what this means.
</Callout>

<Callout icon="👍" theme="okay">
  **AI-first by design**

  Each building block ships with Claude skills that handle the implementation work: scaffolding, code generation, testing, and deployment.

  - Custom Connectors: skills scaffold, implement, and deploy a connector through guided conversation.
</Callout>

Monte Carlo natively supports many ETL and orchestration tools (Airflow, dbt, Fivetran, and [more](https://docs.getmontecarlo.com/docs/orchestration-and-transformation)). Custom ETL integrations let you bring pipeline tools that Monte Carlo doesn't natively support — or customize how Monte Carlo collects from a tool it does support.

<Callout icon="ℹ️" theme="info">
  In scope: pipeline orchestrators, transformation tools, and any system that runs jobs with trackable status (Airflow, Coalesce, Talend, Control-M, dbt Cloud, Azure Data Factory, etc.)

  Looking for SQL-based data source integrations instead? See [Custom SQL Integrations](https://docs.getmontecarlo.com/docs/custom-integrations).
</Callout>

Custom ETL integrations give you two building blocks. You can use them individually depending on your needs.

## Building blocks

|                        | [ETL Push Ingest API](https://docs.getmontecarlo.com/docs/push-ingest-api) | [Custom ETL Connector](https://docs.getmontecarlo.com/docs/custom-connectors) |
| :--------------------- | :----------------------------------------- | :-------------------------------------------- |
| **Job Metadata**       | Yes                                        | Yes                                           |
| Lineage                | Yes                                        | Yes                                           |
| **Run history**        | Yes                                        | Yes                                           |
| **Run Failure Alerts** | Yes                                        | Yes                                           |
| **Scheduling**         | No                                         | Yes                                           |

## When to use each

[ETL Push Ingest API](https://docs.getmontecarlo.com/docs/push-ingest-api) — You push pipeline metadata and run events directly to Monte Carlo using the pycarlo SDK or HTTP. Use it when:

* You already have orchestration that can call an API after each pipeline run
* You want full control over when and how data is sent
* You can't deploy the Monte Carlo agent in your infrastructure

[Custom ETL Connector](https://docs.getmontecarlo.com/docs/custom-connectors) — You implement two Python methods (`fetch_metadata` and `fetch_run_details`) and deploy a connector inside Monte Carlo's agent. The agent handles scheduling, pagination, and delivery. Use it when:

* You want Monte Carlo to automatically collect on a schedule
* You want webhook-triggered collection for near-real-time failure detection
* You prefer a managed framework over writing your own push pipeline

## Architecture

Both approaches deliver data to the same Monte Carlo backend and produce the same result — pipeline visibility, failure alerting, run duration tracking, and cross-domain lineage.

```mermaid
flowchart LR
    subgraph push ["Push Ingest API"]
        A["Your script, pycarlo SDK or HTTP"] --> B["Integration Gateway"]
    end
    subgraph connector ["Custom ETL Connector"]
        C["Vendor ETL API"] --> D["Your Connector, fetch_metadata | fetch_run_details"]
        D --> E["Agent Framework, scheduling + pagination"] --> B
    end 
    B --> F["Monte Carlo, pipeline catalog · lineage · alerting"]
```

## Data model

Both building blocks use the same data model:

* **EtlAsset** — structural metadata about a job (name, tasks, schedule, lineage)
* **EtlRunEvent** — execution data for a specific run (status, timing, errors, task-level breakdown)

The schemas are defined by [pycarlo](https://pypi.org/project/pycarlo/) dataclasses. See the [Push Ingest API](https://docs.getmontecarlo.com/docs/push-ingest-api) reference for the full field-by-field breakdown.

## Feature Support

For either deployment style (Push-based or Custom ETL Connector) the exact features supported will depend on how much of the event schema you are able to populate. Either style is able to support *up to&#x20;*&#x74;his feature set when fully-implemented:

| Category | Capability                   | Support |
| -------- | ---------------------------- | ------- |
| Jobs     | Job Failure Alerts           | ✅       |
| Jobs     | Webhook triggered collection | ✅       |
| Jobs     | Job Run history              | ✅       |
| Lineage  | Table-to-Table Lineage       | ✅       |
| Lineage  | Task-to-Task lineage         | ❌       |
| Lineage  | Column-level lineage         | ❌       |

## FAQs

### Is this the same as the existing orchestration integrations?

No. Monte Carlo's [native orchestration integrations](https://docs.getmontecarlo.com/docs/orchestration-and-transformation) (Airflow, dbt, Fivetran, etc.) are built-in and require no custom code. Custom ETL integrations are for tools Monte Carlo doesn't natively support.

### What's the difference between this and Custom SQL Integrations?

[Custom SQL Integrations](https://docs.getmontecarlo.com/docs/custom-integrations) are for SQL-based data **sources** (warehouses, lakehouses, transactional databases). They push table metadata, query logs, and lineage, and optionally execute SQL queries.

Custom ETL Integrations are for pipeline **orchestration tools**. They push job/pipeline metadata and run events — "did this pipeline run, and did it succeed?"

<br />