Custom ETL and Orchestration Integrations (Private Preview)

Monte Carlo natively supports many ETL and orchestration tools (Airflow, dbt, Fivetran, and more). Custom ETL integrations let you bring pipeline tools that Monte Carlo doesn't natively support — or customize how Monte Carlo collects from a tool it does support.

Custom ETL integrations give you two building blocks. You can use them individually depending on your needs.

Building blocks

When to use each

ETL Push Ingest API — You push pipeline metadata and run events directly to Monte Carlo using the pycarlo SDK or HTTP. Use it when:

• You already have orchestration that can call an API after each pipeline run
• You want full control over when and how data is sent
• You can't deploy the Monte Carlo agent in your infrastructure

Custom ETL Connector — You implement two Python methods (fetch_metadata and fetch_run_details) and deploy a connector inside Monte Carlo's agent. The agent handles scheduling, pagination, and delivery. Use it when:

• You want Monte Carlo to automatically collect on a schedule
• You want webhook-triggered collection for near-real-time failure detection
• You prefer a managed framework over writing your own push pipeline

Architecture

Both approaches deliver data to the same Monte Carlo backend and produce the same result — pipeline visibility, failure alerting, run duration tracking, and cross-domain lineage.

flowchart LR
    subgraph push ["Push Ingest API"]
        A["Your script, pycarlo SDK or HTTP"] --> B["Integration Gateway"]
    end
    subgraph connector ["Custom ETL Connector"]
        C["Vendor ETL API"] --> D["Your Connector, fetch_metadata | fetch_run_details"]
        D --> E["Agent Framework, scheduling + pagination"] --> B
    end 
    B --> F["Monte Carlo, pipeline catalog · lineage · alerting"]

Data model

Both building blocks use the same data model:

• EtlAsset — structural metadata about a job (name, tasks, schedule, lineage)
• EtlRunEvent — execution data for a specific run (status, timing, errors, task-level breakdown)

The schemas are defined by pycarlo dataclasses. See the Push Ingest API reference for the full field-by-field breakdown.

Feature Support

For either deployment style (Push-based or Custom ETL Connector) the exact features supported will depend on how much of the event schema you are able to populate. Either style is able to support up to this feature set when fully-implemented:

FAQs

Is this the same as the existing orchestration integrations?

No. Monte Carlo's native orchestration integrations (Airflow, dbt, Fivetran, etc.) are built-in and require no custom code. Custom ETL integrations are for tools Monte Carlo doesn't natively support.

What's the difference between this and Custom SQL Integrations?

Custom SQL Integrations are for SQL-based data sources (warehouses, lakehouses, transactional databases). They push table metadata, query logs, and lineage, and optionally execute SQL queries.

Custom ETL Integrations are for pipeline orchestration tools. They push job/pipeline metadata and run events — "did this pipeline run, and did it succeed?"