Lineage Between Jobs and Tables

ℹ️

This feature requires an Enterprise or higher plan to use.

This feature is in Public Preview. See here for more information on what this means.

Lineage connects your ETL jobs to the warehouse tables they read and write. Once it exists, Monte Carlo can show which pipelines feed which tables, trace downstream impact when a job fails, and link pipeline health to data quality monitors on the affected tables.

There are two ways to produce it. They are independent — use either, or both together.

Declared lineage

Report the tables a job touches as inputs and outputs when you push pipeline data:

• Static lineage — set inputs/outputs on the EtlAsset (job) or its tasks. Use for tables a job always reads and writes.
• Runtime lineage — set inputs/outputs on the EtlRunEvent. Use when the tables vary between runs.

Each entry is an asset reference — the asset type, its role, and its fully qualified name in the warehouse:

inputs=[AssetRef(asset_type="TABLE", role="INPUT", fully_qualified_name="raw.billing.charges")]
outputs=[AssetRef(asset_type="TABLE", role="OUTPUT", fully_qualified_name="staging.billing.charges_clean")]

The fully_qualified_name must match how the table is identified in the monitored warehouse, so the edge attaches to the asset Monte Carlo already tracks.

See Push Ingest API and Custom Connectors for where these fields live in the schema.

SQL query tagging

When a job runs SQL that the source API can't introspect, tag the query with the job's identifier instead. Monte Carlo ingests the tag through its normal query-log collection on the monitored warehouse and resolves it back to your pipeline — no inputs/outputs and no extra integration code required.

Add a JSON comment to the SQL (or use a native query tag where the warehouse strips comments, e.g. Snowflake's QUERY_TAG):

-- {"mcd_job_id": "my-pipeline", "mcd_task_id": "my-task"}
INSERT INTO analytics.public.daily_revenue
SELECT ... FROM raw.billing.charges;

The tag values are the same source_ids your integration already reports:

Monte Carlo creates the table ↔ pipeline edges from the tables the query reads and writes: input tables become inputs of the job, output tables become outputs.

Requirements:

• The warehouse running the SQL is monitored by Monte Carlo with query logs enabled.
• The ETL integration is registered.

ℹ️

Multiple connections. If the same mcd_job_id exists in more than one registered ETL connection, add mcd_resource_id with the connection's resource UUID so Monte Carlo resolves the right one. Without it, an ambiguous tag is skipped.

Choosing an approach

• Prefer declared lineage when the source API exposes input/output tables — it's exact and doesn't depend on query logs.
• Use SQL query tagging for jobs whose lineage only lives in the SQL they run.
• Combine them: declare what the API knows, tag the queries it can't see. Both resolve to the same job and merge into one lineage graph.