Push Ingest API
What is the BI Push Ingest API?
The BI Push Ingest API lets you push BI asset metadata — dashboards, reports, datasets, workbooks, and their lineage — directly to Monte Carlo using the pycarlo SDK or direct HTTP. Use it to integrate any BI tool that Monte Carlo doesn't natively support.
One endpoint covers everything:
| Endpoint | What it ingests | pycarlo method |
|---|---|---|
POST /ingest/v1/bi/metadata | BI asset structure — names, types, owners, timestamps, and lineage | svc.send_bi_metadata(...) |
This is the push approach — you control when and how data is sent. If you prefer a turnkey setup where Monte Carlo handles scheduling and pagination, see Custom BI Connectors instead.
End-to-end workflow
- Set up your integration — Create a Custom BI integration in the Monte Carlo UI with Push selected for metadata.
- Create an Integration key — Generate a dedicated ingestion key for authenticating push requests.
- Push data — Send BI asset metadata using the pycarlo SDK or direct HTTP.
- View assets — Once data is in Monte Carlo, BI assets appear in your catalog with their lineage.
1. Set up your integration
Go to Settings → Integrations → Add → Business intelligence → Custom BI integration and create an integration for your BI tool. Select Push for metadata.
After the integration is created, the finish-setup step shows you:
- The resource ID — the UUID that identifies this integration in your push payloads
- The BI metadata URL — the ingest endpoint
- The option to generate an integration key (see step 2)
Save the resource ID — you'll need it for every push request. You can also find it later on the integration's details page, or via GraphQL:
query {
getUser {
account {
bi {
uuid
name
type
}
}
}
}2. Create an Integration key
Push requests use a dedicated integration key with scope Ingestion. A standard Monte Carlo API key will not work.
Option A — UI (recommended)
After creating your push-based BI integration you can generate an ingestion key from the finish-setup step, or later from the integration's details page.
Option B — GraphQL mutation
Use the API Explorer with a GraphQL API key. Using the resource UUID from step 1, generate an integration key with:
mutation {
createIntegrationKey(
description: "BI push ingestion key"
scope: Ingestion
warehouseIds: ["<resource_uuid>"]
) {
key { id secret }
}
}
Store the key secret immediatelyThe key secret is shown only once at creation time. Save it to a secure secrets manager before closing the terminal.
3. Push data
Set up your environment variables for the examples below:
export MC_INGEST_KEY_ID="<ingestion-key-id>"
export MC_INGEST_KEY_TOKEN="<ingestion-key-secret>"
export MC_RESOURCE_UUID="<resource-uuid>"Option A — pycarlo SDK (recommended)
Install the SDK (minimum version 0.15.240):
pip install "pycarlo>=0.15.240"Initialize the client
import os
from pycarlo.core import Client, Session
from pycarlo.features.ingestion import IngestionService
service = IngestionService(mc_client=Client(session=Session(
mcd_id=os.environ["MC_INGEST_KEY_ID"],
mcd_token=os.environ["MC_INGEST_KEY_TOKEN"],
scope="Ingestion",
)))Push BI asset metadata
Illustrative exampleThe asset names, IDs, and lineage references below are illustrative. Replace them with values from your BI tool.
from pycarlo.features.ingestion import (
AssetRef, BiAsset, BiAssetRef, BiOwner,
)
from pycarlo.features.ingestion.models import Tag
dataset = BiAsset(
# Required
asset_source_id="dataset-42", # stable vendor ID — survives renames
name="Orders dataset", # display name
asset_type="dataset", # free-form type label
# Optional — metadata
description="Curated orders for reporting",
folder="Finance/Operations",
owner=BiOwner(email="[email protected]", name="Data Engineering"),
is_certified=True,
properties=[Tag(key="team", value="data-eng")],
# Optional — table lineage (tables this asset reads)
inputs=[
AssetRef(asset_type="TABLE", role="INPUT",
fully_qualified_name="analytics.public.orders"),
],
)
dashboard = BiAsset(
asset_source_id="dashboard-7",
name="Revenue dashboard",
asset_type="dashboard",
asset_url="https://bi.example.com/dashboards/revenue",
owner=BiOwner(email="[email protected]"),
view_count=128,
# Optional — BI-to-BI lineage (this dashboard derives from the dataset)
upstream_assets=[
BiAssetRef(asset_source_id="dataset-42", relationship_type="DERIVES_FROM"),
],
)
result = service.send_bi_metadata(
resource_uuid=os.environ["MC_RESOURCE_UUID"],
resource_type="custom-bi-connector",
events=[dataset, dashboard], # 1–100 per batch
)
print("invocation_id:", service.extract_invocation_id(result))A 202 response with an invocation_id means the push was accepted. Assets appear in Monte Carlo within a few minutes; lineage is processed asynchronously and follows shortly after.
Option B — Direct HTTP
If you prefer not to use the pycarlo SDK, you can call the endpoint directly.
Base URL (production): https://integrations.getmontecarlo.com
Authentication headers:
| Header | Value |
|---|---|
x-mcd-id | Integration key ID |
x-mcd-token | Integration key secret |
Content-Type | application/json |
Every request body wraps its events in a common envelope:
| Field | Value |
|---|---|
event_type | BI_METADATA |
resource | Object with uuid (your resource ID) and resource_type ("custom-bi-connector") |
events | List of 1–100 BI asset objects. Events are flat — there is no per-event wrapper. |
curl -X POST https://integrations.getmontecarlo.com/ingest/v1/bi/metadata \
-H "x-mcd-id: $MC_INGEST_KEY_ID" \
-H "x-mcd-token: $MC_INGEST_KEY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"event_type": "BI_METADATA",
"resource": {
"uuid": "'"$MC_RESOURCE_UUID"'",
"resource_type": "custom-bi-connector"
},
"events": [
{
"asset_source_id": "dataset-42",
"name": "Orders dataset",
"asset_type": "dataset",
"inputs": [
{"asset_type": "TABLE", "role": "INPUT", "fully_qualified_name": "analytics.public.orders"}
]
},
{
"asset_source_id": "dashboard-7",
"name": "Revenue dashboard",
"asset_type": "dashboard",
"upstream_assets": [
{"asset_source_id": "dataset-42", "relationship_type": "DERIVES_FROM"}
]
}
]
}'Response codes
The endpoint returns 202 Accepted on success:
{ "invocation_id": "<uuid>" }| Status | Meaning |
|---|---|
400 | Schema validation failed. Check details in the response body for field-level errors. |
401 | Authentication failed — wrong key ID or secret. |
403 | The key is not authorized for the resource.uuid in the payload. |
413 | Payload too large — compressed body exceeds 1 MB. Split into smaller batches. |
5xx | Server error — retry with exponential backoff. |
4. View assets
Once data is pushed, BI assets appear in your Monte Carlo catalog. You can:
- Search for assets and see their type, owner, folder, and certification status
- See table → BI asset lineage on a table's lineage view (when
inputsare populated or queries are tagged) - Trace BI → BI lineage between dashboards, datasets, and other assets (when
upstream_assets/downstream_assetsare populated)
Data schema
The pycarlo SDK provides typed dataclasses, but you can also pass plain dicts with the same keys when using direct HTTP.
BiAsset — BI asset metadata
Input to send_bi_metadata(). Describes one BI asset (dashboard, report, dataset, workbook, …).
| Field | Type | Required | Description |
|---|---|---|---|
asset_source_id | str | yes | Vendor-stable identifier, unique within the integration. Must survive renames and moves. |
name | str | yes | Human-readable asset name |
asset_type | str | yes | Free-form type label (e.g. "dashboard", "report", "dataset") — displayed verbatim, never validated |
description | str | no | Human-readable description |
asset_url | str | no | Link to the asset in the BI tool |
folder | str | no | Folder / namespace path |
owner | BiOwner | no | Asset owner |
created_time | str | no | ISO 8601, timezone-aware |
last_modified_time | str | no | ISO 8601, timezone-aware |
last_viewed_time | str | no | ISO 8601, timezone-aware |
view_count | int | no | View count (non-negative) |
is_certified | bool | no | Certification flag |
certification_note | str | no | Certification note |
is_archived | bool | no | Archived flag |
upstream_assets | list[BiAssetRef] | no | BI assets this asset derives from (enables BI → BI lineage) |
downstream_assets | list[BiAssetRef] | no | BI assets that derive from this asset (enables BI → BI lineage) |
inputs | list[AssetRef] | no | Warehouse assets this asset reads (enables table → BI lineage) |
properties | list[Tag] | no | Key-value tags |
attributes | dict | no | Arbitrary vendor-specific metadata (passed through) |
asset_source_idis the asset's identityMonte Carlo keys each asset on
asset_source_id. Use the vendor's stable ID — not a display name — and keep it stable across renames and moves. If the ID changes, Monte Carlo treats the asset as a new one.
Sub-schemas
BiOwner
| Field | Type | Required | Description |
|---|---|---|---|
email | str | no | Owner's email (preferred — resolves to a Monte Carlo user) |
name | str | no | Owner's display name |
source_id | str | no | Owner's identifier in the source system |
BiAssetRef — BI → BI lineage reference
| Field | Type | Required | Description |
|---|---|---|---|
asset_source_id | str | yes | The related asset's source ID (within the same integration) |
relationship_type | str | no | CONTAINED_IN, DERIVES_FROM, or REFERENCES — a semantic label on the edge |
Cross-integration references are not supported: a BI asset reference can only point to another asset in the same integration. Untyped references produce a generic lineage edge.
AssetRef — table lineage reference
| Field | Type | Required | Description |
|---|---|---|---|
asset_type | str | yes | TABLE, VIEW, FILE, TOPIC, DATASET, or DASHBOARD |
role | str | yes | Always INPUT — BI assets only consume warehouse assets |
fully_qualified_name | str | one of these | Asset name in the warehouse (e.g. db.schema.table) |
mcon | str | one of these | Monte Carlo Object Name (internal identifier) |
At least one of fully_qualified_name or mcon must be provided. The fully_qualified_name must match how the table is identified in the monitored warehouse, so the edge attaches to the asset Monte Carlo already tracks.
Tag
| Field | Type | Required | Description |
|---|---|---|---|
key | str | yes | Tag key |
value | str | yes | Tag value |
Lineage
Two kinds of lineage can be declared on each BiAsset:
- Table → BI asset lineage —
inputslists the warehouse tables, views, and other data assets the BI asset reads. Monte Carlo draws edges from those tables to the asset. - BI → BI lineage —
upstream_assetsanddownstream_assetsdescribe how BI assets relate to one another: a dashboard derives from a dataset, a page contains cards, and so on. There is no depth limit — chains of BI assets resolve into a full graph.
Lineage is entirely optional. If the BI tool's API doesn't expose relationships or source tables, omit the fields.
Lineage via SQL query tagging
When a BI tool queries your warehouse directly and its API can't tell you which tables an asset reads, you can tag the SQL the BI tool generates instead. Monte Carlo ingests the tags through its standard query-log collection on the monitored warehouse and matches them back to the BI asset, creating table → BI asset lineage without declaring inputs.
Add a JSON comment to the query (or use a native query tag where the warehouse strips comments, e.g. Snowflake's QUERY_TAG):
-- {"mcd_bi_asset_id": "dashboard-7", "mcd_resource_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}
SELECT * FROM analytics.public.orders;| Tag key | Value | Required |
|---|---|---|
mcd_bi_asset_id | The asset's asset_source_id — the raw vendor ID, exactly as pushed. | Yes |
mcd_resource_id | The integration's resource UUID (from step 1). | Yes |
Both tags are required — Monte Carlo uses them together to resolve the exact BI asset. The query's input tables become lineage edges into that asset.
Requirements:
- The warehouse running the SQL is monitored by Monte Carlo with query logs enabled.
- The BI asset already exists in Monte Carlo — push its metadata first (or have it collected by a connector), otherwise the tag can't be resolved.
Query tagging complements inputs: declare the table lineage the API knows, and tag queries for the lineage it can't see. Both resolve to the same asset and merge into one lineage graph.
Validation rules
The pycarlo SDK enforces these rules at push time. If you use direct HTTP, the server validates on receipt.
asset_source_id,name, andasset_typeare required and non-empty- All datetime fields must be valid ISO 8601 with timezone
ownerkeys are limited toemail,name,source_idview_countmust be a non-negative integerinputsrequire a validasset_type,role: "INPUT", and at least one identifier- Batch size: 1–100 events per request
- Omit
Nonevalues and empty lists — the API expects sparse payloads
Best practices
Batching
Keep batches to 1–100 events per request. The compressed body must not exceed 1 MB. Splitting large pushes into batches also makes debugging easier — each batch returns its own invocation_id.
BATCH_SIZE = 100
for i in range(0, len(all_assets), BATCH_SIZE):
batch = all_assets[i : i + BATCH_SIZE]
result = service.send_bi_metadata(
resource_uuid=resource_uuid,
resource_type="custom-bi-connector",
events=batch,
)
print(f"Batch {i // BATCH_SIZE + 1} invocation_id: {service.extract_invocation_id(result)}")Deleting pushed assets
Pushed BI assets are not automatically deleted — they are customer-managed, so you are responsible for cleaning them up when assets are removed from your BI tool. Use the deleteBiAssets mutation with a GraphQL API key:
mutation DeleteBiAssets($mcons: [String!]!) {
deleteBiAssets(mcons: $mcons) {
success
deletedCount
}
}Maximum 1,000 MCONs per call. The mutation is all-or-nothing: if any MCON doesn't exist in your account or isn't a pushed BI asset, nothing is deleted. Deleting an asset also removes its lineage nodes and any asset links referencing it.
Troubleshooting
Push returns 401 — The key ID or secret is wrong, or the key does not exist. Check x-mcd-id and x-mcd-token.
Push returns 403 — The key is scoped to specific resource UUIDs and the resource.uuid in your payload is not in the allowed list. Use an unscoped key or add the resource UUID to the key's allowlist.
Push returns 400 — Schema validation failed. The response body contains field-level details — common causes are a missing asset_source_id / name / asset_type, a naive (timezone-less) timestamp, or an unknown owner key.
Push returns 413 — The compressed payload exceeds 1 MB. Split your events list into smaller batches.
Assets aren't appearing after a push — Assets typically appear within a few minutes; lineage follows asynchronously. If data still hasn't appeared, note the invocation_id and contact Monte Carlo support.
A renamed or moved asset appears twice — The asset_source_id changed. Monte Carlo keys assets on the source ID, so a changed ID creates a new asset. Push the asset with its stable vendor ID, and delete the stale copy with deleteBiAssets.
Tagged queries aren't producing lineage — Check that both mcd_bi_asset_id and mcd_resource_id are present, that mcd_bi_asset_id exactly matches the pushed asset_source_id, that the warehouse is monitored with query logs enabled, and that the asset exists in the catalog.
FAQs
What happens to my integration keys if I delete the integration?
Deleting an integration in Monte Carlo also deletes any integration keys (including Ingestion-scoped keys) associated with it. Any scripts or pipelines using those keys will start failing with 401 errors. Before deleting an integration, make sure no active push pipelines depend on its keys — or create new keys on a replacement integration first.
Can I push assets from more than one BI tool to the same integration?
No — each integration represents one BI tool. Create a separate integration per tool so each gets its own resource ID and its own asset_source_id namespace.
Do pushed assets expire?
No. Pushed assets and their lineage are customer-managed and remain in Monte Carlo until you delete them with deleteBiAssets.
Updated about 1 hour ago
