Push Ingest API

What is the BI Push Ingest API?

The BI Push Ingest API lets you push BI asset metadata — dashboards, reports, datasets, workbooks, and their lineage — directly to Monte Carlo using the pycarlo SDK or direct HTTP. Use it to integrate any BI tool that Monte Carlo doesn't natively support.

One endpoint covers everything:

EndpointWhat it ingestspycarlo method
POST /ingest/v1/bi/metadataBI asset structure — names, types, owners, timestamps, and lineagesvc.send_bi_metadata(...)
ℹ️

This is the push approach — you control when and how data is sent. If you prefer a turnkey setup where Monte Carlo handles scheduling and pagination, see Custom BI Connectors instead.


End-to-end workflow

  1. Set up your integration — Create a Custom BI integration in the Monte Carlo UI with Push selected for metadata.
  2. Create an Integration key — Generate a dedicated ingestion key for authenticating push requests.
  3. Push data — Send BI asset metadata using the pycarlo SDK or direct HTTP.
  4. View assets — Once data is in Monte Carlo, BI assets appear in your catalog with their lineage.

1. Set up your integration

Go to Settings → Integrations → Add → Business intelligence → Custom BI integration and create an integration for your BI tool. Select Push for metadata.

After the integration is created, the finish-setup step shows you:

  • The resource ID — the UUID that identifies this integration in your push payloads
  • The BI metadata URL — the ingest endpoint
  • The option to generate an integration key (see step 2)

Save the resource ID — you'll need it for every push request. You can also find it later on the integration's details page, or via GraphQL:

query {
  getUser {
    account {
      bi {
        uuid
        name
        type
      }
    }
  }
}

2. Create an Integration key

Push requests use a dedicated integration key with scope Ingestion. A standard Monte Carlo API key will not work.

Option A — UI (recommended)

After creating your push-based BI integration you can generate an ingestion key from the finish-setup step, or later from the integration's details page.

Option B — GraphQL mutation

Use the API Explorer with a GraphQL API key. Using the resource UUID from step 1, generate an integration key with:

mutation {
  createIntegrationKey(
    description: "BI push ingestion key"
    scope: Ingestion
    warehouseIds: ["<resource_uuid>"]
  ) {
    key { id secret }
  }
}
❗️

Store the key secret immediately

The key secret is shown only once at creation time. Save it to a secure secrets manager before closing the terminal.


3. Push data

Set up your environment variables for the examples below:

export MC_INGEST_KEY_ID="<ingestion-key-id>"
export MC_INGEST_KEY_TOKEN="<ingestion-key-secret>"
export MC_RESOURCE_UUID="<resource-uuid>"

Option A — pycarlo SDK (recommended)

Install the SDK (minimum version 0.15.240):

pip install "pycarlo>=0.15.240"

Initialize the client

import os
from pycarlo.core import Client, Session
from pycarlo.features.ingestion import IngestionService

service = IngestionService(mc_client=Client(session=Session(
    mcd_id=os.environ["MC_INGEST_KEY_ID"],
    mcd_token=os.environ["MC_INGEST_KEY_TOKEN"],
    scope="Ingestion",
)))

Push BI asset metadata

📘

Illustrative example

The asset names, IDs, and lineage references below are illustrative. Replace them with values from your BI tool.

from pycarlo.features.ingestion import (
    AssetRef, BiAsset, BiAssetRef, BiOwner,
)
from pycarlo.features.ingestion.models import Tag

dataset = BiAsset(
    # Required
    asset_source_id="dataset-42",       # stable vendor ID — survives renames
    name="Orders dataset",              # display name
    asset_type="dataset",               # free-form type label

    # Optional — metadata
    description="Curated orders for reporting",
    folder="Finance/Operations",
    owner=BiOwner(email="[email protected]", name="Data Engineering"),
    is_certified=True,
    properties=[Tag(key="team", value="data-eng")],

    # Optional — table lineage (tables this asset reads)
    inputs=[
        AssetRef(asset_type="TABLE", role="INPUT",
                 fully_qualified_name="analytics.public.orders"),
    ],
)

dashboard = BiAsset(
    asset_source_id="dashboard-7",
    name="Revenue dashboard",
    asset_type="dashboard",
    asset_url="https://bi.example.com/dashboards/revenue",
    owner=BiOwner(email="[email protected]"),
    view_count=128,

    # Optional — BI-to-BI lineage (this dashboard derives from the dataset)
    upstream_assets=[
        BiAssetRef(asset_source_id="dataset-42", relationship_type="DERIVES_FROM"),
    ],
)

result = service.send_bi_metadata(
    resource_uuid=os.environ["MC_RESOURCE_UUID"],
    resource_type="custom-bi-connector",
    events=[dataset, dashboard],        # 1–100 per batch
)
print("invocation_id:", service.extract_invocation_id(result))

A 202 response with an invocation_id means the push was accepted. Assets appear in Monte Carlo within a few minutes; lineage is processed asynchronously and follows shortly after.

Option B — Direct HTTP

If you prefer not to use the pycarlo SDK, you can call the endpoint directly.

Base URL (production): https://integrations.getmontecarlo.com

Authentication headers:

HeaderValue
x-mcd-idIntegration key ID
x-mcd-tokenIntegration key secret
Content-Typeapplication/json

Every request body wraps its events in a common envelope:

FieldValue
event_typeBI_METADATA
resourceObject with uuid (your resource ID) and resource_type ("custom-bi-connector")
eventsList of 1–100 BI asset objects. Events are flat — there is no per-event wrapper.
curl -X POST https://integrations.getmontecarlo.com/ingest/v1/bi/metadata \
  -H "x-mcd-id: $MC_INGEST_KEY_ID" \
  -H "x-mcd-token: $MC_INGEST_KEY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "BI_METADATA",
    "resource": {
      "uuid": "'"$MC_RESOURCE_UUID"'",
      "resource_type": "custom-bi-connector"
    },
    "events": [
      {
        "asset_source_id": "dataset-42",
        "name": "Orders dataset",
        "asset_type": "dataset",
        "inputs": [
          {"asset_type": "TABLE", "role": "INPUT", "fully_qualified_name": "analytics.public.orders"}
        ]
      },
      {
        "asset_source_id": "dashboard-7",
        "name": "Revenue dashboard",
        "asset_type": "dashboard",
        "upstream_assets": [
          {"asset_source_id": "dataset-42", "relationship_type": "DERIVES_FROM"}
        ]
      }
    ]
  }'

Response codes

The endpoint returns 202 Accepted on success:

{ "invocation_id": "<uuid>" }
StatusMeaning
400Schema validation failed. Check details in the response body for field-level errors.
401Authentication failed — wrong key ID or secret.
403The key is not authorized for the resource.uuid in the payload.
413Payload too large — compressed body exceeds 1 MB. Split into smaller batches.
5xxServer error — retry with exponential backoff.

4. View assets

Once data is pushed, BI assets appear in your Monte Carlo catalog. You can:

  • Search for assets and see their type, owner, folder, and certification status
  • See table → BI asset lineage on a table's lineage view (when inputs are populated or queries are tagged)
  • Trace BI → BI lineage between dashboards, datasets, and other assets (when upstream_assets / downstream_assets are populated)

Data schema

The pycarlo SDK provides typed dataclasses, but you can also pass plain dicts with the same keys when using direct HTTP.

BiAsset — BI asset metadata

Input to send_bi_metadata(). Describes one BI asset (dashboard, report, dataset, workbook, …).

FieldTypeRequiredDescription
asset_source_idstryesVendor-stable identifier, unique within the integration. Must survive renames and moves.
namestryesHuman-readable asset name
asset_typestryesFree-form type label (e.g. "dashboard", "report", "dataset") — displayed verbatim, never validated
descriptionstrnoHuman-readable description
asset_urlstrnoLink to the asset in the BI tool
folderstrnoFolder / namespace path
ownerBiOwnernoAsset owner
created_timestrnoISO 8601, timezone-aware
last_modified_timestrnoISO 8601, timezone-aware
last_viewed_timestrnoISO 8601, timezone-aware
view_countintnoView count (non-negative)
is_certifiedboolnoCertification flag
certification_notestrnoCertification note
is_archivedboolnoArchived flag
upstream_assetslist[BiAssetRef]noBI assets this asset derives from (enables BI → BI lineage)
downstream_assetslist[BiAssetRef]noBI assets that derive from this asset (enables BI → BI lineage)
inputslist[AssetRef]noWarehouse assets this asset reads (enables table → BI lineage)
propertieslist[Tag]noKey-value tags
attributesdictnoArbitrary vendor-specific metadata (passed through)
❗️

asset_source_id is the asset's identity

Monte Carlo keys each asset on asset_source_id. Use the vendor's stable ID — not a display name — and keep it stable across renames and moves. If the ID changes, Monte Carlo treats the asset as a new one.

Sub-schemas

BiOwner

FieldTypeRequiredDescription
emailstrnoOwner's email (preferred — resolves to a Monte Carlo user)
namestrnoOwner's display name
source_idstrnoOwner's identifier in the source system

BiAssetRef — BI → BI lineage reference

FieldTypeRequiredDescription
asset_source_idstryesThe related asset's source ID (within the same integration)
relationship_typestrnoCONTAINED_IN, DERIVES_FROM, or REFERENCES — a semantic label on the edge

Cross-integration references are not supported: a BI asset reference can only point to another asset in the same integration. Untyped references produce a generic lineage edge.

AssetRef — table lineage reference

FieldTypeRequiredDescription
asset_typestryesTABLE, VIEW, FILE, TOPIC, DATASET, or DASHBOARD
rolestryesAlways INPUT — BI assets only consume warehouse assets
fully_qualified_namestrone of theseAsset name in the warehouse (e.g. db.schema.table)
mconstrone of theseMonte Carlo Object Name (internal identifier)

At least one of fully_qualified_name or mcon must be provided. The fully_qualified_name must match how the table is identified in the monitored warehouse, so the edge attaches to the asset Monte Carlo already tracks.

Tag

FieldTypeRequiredDescription
keystryesTag key
valuestryesTag value

Lineage

Two kinds of lineage can be declared on each BiAsset:

  • Table → BI asset lineage — inputs lists the warehouse tables, views, and other data assets the BI asset reads. Monte Carlo draws edges from those tables to the asset.
  • BI → BI lineage — upstream_assets and downstream_assets describe how BI assets relate to one another: a dashboard derives from a dataset, a page contains cards, and so on. There is no depth limit — chains of BI assets resolve into a full graph.

Lineage is entirely optional. If the BI tool's API doesn't expose relationships or source tables, omit the fields.

Lineage via SQL query tagging

When a BI tool queries your warehouse directly and its API can't tell you which tables an asset reads, you can tag the SQL the BI tool generates instead. Monte Carlo ingests the tags through its standard query-log collection on the monitored warehouse and matches them back to the BI asset, creating table → BI asset lineage without declaring inputs.

Add a JSON comment to the query (or use a native query tag where the warehouse strips comments, e.g. Snowflake's QUERY_TAG):

-- {"mcd_bi_asset_id": "dashboard-7", "mcd_resource_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}
SELECT * FROM analytics.public.orders;
Tag keyValueRequired
mcd_bi_asset_idThe asset's asset_source_id — the raw vendor ID, exactly as pushed.Yes
mcd_resource_idThe integration's resource UUID (from step 1).Yes

Both tags are required — Monte Carlo uses them together to resolve the exact BI asset. The query's input tables become lineage edges into that asset.

Requirements:

  • The warehouse running the SQL is monitored by Monte Carlo with query logs enabled.
  • The BI asset already exists in Monte Carlo — push its metadata first (or have it collected by a connector), otherwise the tag can't be resolved.

Query tagging complements inputs: declare the table lineage the API knows, and tag queries for the lineage it can't see. Both resolve to the same asset and merge into one lineage graph.


Validation rules

The pycarlo SDK enforces these rules at push time. If you use direct HTTP, the server validates on receipt.

  • asset_source_id, name, and asset_type are required and non-empty
  • All datetime fields must be valid ISO 8601 with timezone
  • owner keys are limited to email, name, source_id
  • view_count must be a non-negative integer
  • inputs require a valid asset_type, role: "INPUT", and at least one identifier
  • Batch size: 1–100 events per request
  • Omit None values and empty lists — the API expects sparse payloads

Best practices

Batching

Keep batches to 1–100 events per request. The compressed body must not exceed 1 MB. Splitting large pushes into batches also makes debugging easier — each batch returns its own invocation_id.

BATCH_SIZE = 100
for i in range(0, len(all_assets), BATCH_SIZE):
    batch = all_assets[i : i + BATCH_SIZE]
    result = service.send_bi_metadata(
        resource_uuid=resource_uuid,
        resource_type="custom-bi-connector",
        events=batch,
    )
    print(f"Batch {i // BATCH_SIZE + 1} invocation_id: {service.extract_invocation_id(result)}")

Deleting pushed assets

Pushed BI assets are not automatically deleted — they are customer-managed, so you are responsible for cleaning them up when assets are removed from your BI tool. Use the deleteBiAssets mutation with a GraphQL API key:

mutation DeleteBiAssets($mcons: [String!]!) {
  deleteBiAssets(mcons: $mcons) {
    success
    deletedCount
  }
}

Maximum 1,000 MCONs per call. The mutation is all-or-nothing: if any MCON doesn't exist in your account or isn't a pushed BI asset, nothing is deleted. Deleting an asset also removes its lineage nodes and any asset links referencing it.


Troubleshooting

Push returns 401 — The key ID or secret is wrong, or the key does not exist. Check x-mcd-id and x-mcd-token.

Push returns 403 — The key is scoped to specific resource UUIDs and the resource.uuid in your payload is not in the allowed list. Use an unscoped key or add the resource UUID to the key's allowlist.

Push returns 400 — Schema validation failed. The response body contains field-level details — common causes are a missing asset_source_id / name / asset_type, a naive (timezone-less) timestamp, or an unknown owner key.

Push returns 413 — The compressed payload exceeds 1 MB. Split your events list into smaller batches.

Assets aren't appearing after a push — Assets typically appear within a few minutes; lineage follows asynchronously. If data still hasn't appeared, note the invocation_id and contact Monte Carlo support.

A renamed or moved asset appears twice — The asset_source_id changed. Monte Carlo keys assets on the source ID, so a changed ID creates a new asset. Push the asset with its stable vendor ID, and delete the stale copy with deleteBiAssets.

Tagged queries aren't producing lineage — Check that both mcd_bi_asset_id and mcd_resource_id are present, that mcd_bi_asset_id exactly matches the pushed asset_source_id, that the warehouse is monitored with query logs enabled, and that the asset exists in the catalog.


FAQs

What happens to my integration keys if I delete the integration?

Deleting an integration in Monte Carlo also deletes any integration keys (including Ingestion-scoped keys) associated with it. Any scripts or pipelines using those keys will start failing with 401 errors. Before deleting an integration, make sure no active push pipelines depend on its keys — or create new keys on a replacement integration first.

Can I push assets from more than one BI tool to the same integration?

No — each integration represents one BI tool. Create a separate integration per tool so each gets its own resource ID and its own asset_source_id namespace.

Do pushed assets expire?

No. Pushed assets and their lineage are customer-managed and remain in Monte Carlo until you delete them with deleteBiAssets.


Did this page help you?