Custom Connectors

ℹ️

This feature requires an Enterprise or higher plan with the Advanced Networking add-on: custom BI connectors run on the generic agent, and deploying an agent requires Advanced Networking.

This feature is in Public Preview. See here for more information on feature lifecycles.

👍

AI-first by design

The connector repository ships with Claude skills that handle the implementation work: scaffolding, code generation, testing, and deployment — through guided conversation.

Custom BI connectors let you integrate any BI tool with Monte Carlo. You implement Python methods that talk to your vendor's API and returns plain dicts describing the BI assets, and the agent framework handles scheduling, pagination, and delivery to Monte Carlo.

Your connector runs inside a custom generic agent image deployed to your infrastructure. The agent is egress-only — it calls out to the vendor API and to Monte Carlo.

ℹ️

This is the connector approach — Monte Carlo handles scheduling and collection automatically. If you prefer to push data yourself from your own pipeline, see the Push Ingest API instead.


Prerequisites

  • Docker — the only host dependency. Everything else (vendor SDKs, test suite, build scripts) runs inside containers.
  • Vendor API credentials — read access to the BI tool's API for listing assets.
  • Claude Code (recommended) — an AI coding agent that can research the vendor API and implement the connector automatically. Not required — you can implement manually if you prefer.

Steps

  1. Set up the repo — Clone the custom-connector-setup repository.
  2. Implement your connector — Implement fetch_metadata and pass the test suite.
  3. Build the agent image — Build the deployable Docker image.
  4. Deploy the agent — Push the image to a container registry and deploy using any supported generic agent platform.
  5. Register the connector — Run agent validations so Monte Carlo discovers your custom BI connector.
  6. Add the connection — Connect through the Monte Carlo UI or GraphQL API.

1. Set up the repo

Clone the custom-connector-setup repository:

git clone https://github.com/monte-carlo-data/custom-connector-setup.git
cd custom-connector-setup

The repository contains:

Directory / filePurpose
bi_connectors/_base/connector.pyCanonical template with the stub methods. This is the API reference — don't edit it.
bi_connectors/_base/validators.pyCross-field validation rules your returned dicts must satisfy. Used by the test suite — don't edit it.
bi_connectors/<name>/One directory per connector you create, containing your implementation (connector.py), vendor SDK (requirements.txt), system dependencies (Dockerfile.extra), credentials (credentials.json), and a manifest (manifest.json).
tests/bi/A pytest test suite that validates your connector against the real vendor API.
scripts/Helper scripts for scaffolding (create_connector.py) and building the deployable image (generate_agent_image.py).

2. Implement your connector

Path 1: AI-assisted (recommended)

The repository includes Claude Code skills that automate the workflow:

StepCommandWhat happens
1/create-connector <name> --biScaffolds bi_connectors/<name>/ with stub files and an optional icon URL.
2/implement-bi-connector <name>Researches the vendor API, installs the client library, implements fetch_metadata, and runs the tests — pausing for you to fill in sandbox credentials.
3/build-agent-image <name>Builds the deployable Docker image.

The only manual step is filling in credentials.json when step 2 pauses.

Path 2: Manual

1. Scaffold your connector:

python scripts/create_connector.py <name> --bi

The script prompts for an optional icon URL (a publicly reachable SVG/PNG shown in the Monte Carlo UI).

2. Add your vendor SDK to bi_connectors/<name>/requirements.txt:

looker-sdk==25.0.0

If no SDK exists, requests is already available for raw HTTP calls.

3. Add system dependencies (if needed) to bi_connectors/<name>/Dockerfile.extra, then regenerate the test Dockerfile:

python scripts/generate_test_dockerfile.py

4. Fill in credentials in bi_connectors/<name>/credentials.json:

{
  "connect_args": {
    "base_url": "https://bi.example.com",
    "api_key": "secret"
  }
}

The keys inside connect_args are whatever your setup_connection() method reads from self.credentials. Use the same format when configuring self-hosted credentials in production.

5. Implement your connector — edit bi_connectors/<name>/connector.py:

📘

Illustrative example

The API calls and field mappings below are illustrative. Adapt the implementation to your vendor's API.

import requests

class Connector:
    credentials: dict

    def setup_connection(self):
        self.base_url = self.credentials["base_url"]
        self.session = requests.Session()
        self.session.headers["Authorization"] = f"Bearer {self.credentials['api_key']}"

    def close_connection(self):
        self.session.close()

    def fetch_metadata(self, limit, offset):
        resp = self.session.get(
            f"{self.base_url}/api/v1/dashboards",
            params={"limit": limit, "offset": offset},
        )
        resp.raise_for_status()
        return [
            {
                "asset_source_id": d["id"],           # stable vendor ID — survives renames
                "name": d["title"],
                "asset_type": "dashboard",
                "description": d.get("description"),
                "asset_url": f"{self.base_url}/dashboards/{d['id']}",
                "owner": {"email": d.get("owner_email")},
                # Optional table lineage — tables this dashboard reads:
                "inputs": [
                    {"asset_type": "TABLE", "role": "INPUT", "fully_qualified_name": t}
                    for t in d.get("source_tables", [])
                ],
                # Optional BI-to-BI lineage:
                "upstream_assets": [
                    {"asset_source_id": u["id"], "relationship_type": "DERIVES_FROM"}
                    for u in d.get("upstream_assets", [])
                ],
            }
            for d in resp.json().get("dashboards", [])
        ]

Omit None values and empty lists from returned dicts — the agent expects sparse dicts with only populated fields.

6. Build and test iteratively:

docker compose build

# Verify connection
CONNECTOR=<name> docker compose run --rm test -m bi_connection

# Metadata — validates fetch_metadata returns well-formed dicts
CONNECTOR=<name> docker compose run --rm test -m bi_metadata

The test suite validates that fetch_metadata returns a non-empty list of dicts, each with asset_source_id, name, and asset_type, and that all dicts pass the cross-field validators (see Validation rules).

Connector contract

class Connector:
    def setup_connection(self): ...                         # Initialize API client using self.credentials
    def close_connection(self): ...                         # Clean up sessions
    def fetch_metadata(self, limit, offset) -> list[dict]:  # Fetch metadata

The agent sets self.credentials (a dict from credentials.json's connect_args) as an attribute before calling any methods — no __init__ needed. fetch_metadata returns at most limit assets starting from offset; the agent keeps calling it with incrementing offset until a short page comes back.

Dict schema reference

Connectors return plain dicts matching the BiAsset model in pycarlo.features.ingestion.bi — the canonical schema reference. The test validators check returned dicts against this schema; connector code never imports the models directly.

SchemaRequired keysPurpose
BiAssetasset_source_id, name, asset_typeOne BI asset (dashboard, report, dataset, workbook, …)

Optional keys: description, asset_url, folder, view_count, is_certified, certification_note, is_archived, created_time / last_modified_time / last_viewed_time (ISO 8601, timezone-aware), plus:

StructureKeysUsed in
owneremail, name, source_id (all optional)Asset owner
bi_asset_refasset_source_id (required), relationship_type (CONTAINED_IN / DERIVES_FROM / REFERENCES)BI → BI lineage via upstream_assets / downstream_assets
asset_refasset_type, role (INPUT), fully_qualified_name and/or mconTable lineage via inputs
tagkey, valueproperties
attributesarbitrary dictVendor-specific metadata (passed through)

Two rules to keep in mind:

  • asset_source_id is the identity seed — it must be vendor-stable across renames and moves, or Monte Carlo treats the asset as new.
  • BI → BI references stay within the integration — cross-integration references are not supported and are dropped server-side.
  • asset_type is a free-form label (e.g. "dashboard", "look", "semantic model") — it becomes the type shown on the asset and is never validated.

Lineage

  • Table → BI asset — populate inputs with the warehouse tables the asset reads: {"asset_type": "TABLE", "role": "INPUT", "fully_qualified_name": "db.schema.table"}.
  • BI → BI — populate upstream_assets / downstream_assets with references to other assets by their asset_source_id.
  • SQL query tagging — if the BI tool queries the warehouse directly and its API can't tell you which tables an asset reads, tag the BI-generated SQL with the asset's asset_source_id instead. Monte Carlo resolves the tags from the warehouse's query logs. See Lineage via SQL query tagging.

All lineage is optional — omit the fields the vendor API can't provide.

Manifest format

The scaffold generates bi_connectors/<name>/manifest.json:

{
  "connection_type": "custom-bi-connector-de8d7c2",
  "connection_name": "looker",
  "asset_class": "bi",
  "credentials_schema": {},
  "icon_url": "https://example.com/vendor-icon.svg"
}
  • connection_type — the unique identifier Monte Carlo uses for your connector type. Generated by the scaffold.
  • icon_url — optional publicly reachable image URL (SVG/PNG) used as the integration's icon in the Monte Carlo UI. By providing an image URL, you confirm you have the right to use and display this image in connection with Monte Carlo's services.
  • credentials_schema — optional cerberus schema that validates self-hosted credentials at setup time, catching missing fields and typos before collection runs. The schema validates the entire credentials.json payload, so keys must be wrapped under a top-level connect_args dict:
{
  "connection_type": "custom-bi-connector-de8d7c2",
  "connection_name": "looker",
  "asset_class": "bi",
  "credentials_schema": {
    "connect_args": {
      "type": "dict",
      "required": true,
      "schema": {
        "base_url": { "type": "string", "required": true },
        "api_key": { "type": "string", "required": true }
      }
    }
  }
}

3. Build the agent image

After all tests pass, build the deployable image:

python scripts/generate_agent_image.py <name>

The script packages your connector into a Docker image layered on top of montecarlodata/agent:latest-generic:

What's included in the image:

  • connector.py — your connector code
  • manifest.json — connection type, name, icon
  • requirements.txt — vendor SDK dependencies
  • Dockerfile.extra — system dependencies

What's NOT included:

Useful flags:

FlagDefaultDescription
--versionlatestAgent base image version.
--docker-platformlinux/amd64Target platform.
--tagcustom-agent:{version}-genericOutput image tag.

You can package multiple connectors into a single image by passing multiple names:

python scripts/generate_agent_image.py looker domo

4. Deploy the agent

  1. Push to a container registry accessible from your deployment environment.
  2. Configure self-hosted credentials in Monte Carlo using the same credentials.json format — swap in production values.
  3. Deploy the agent using any of the supported generic agent platforms. The agent is egress-only.
ℹ️

Your agent deployment must use your custom agent image instead of the default montecarlodata/agent image. Update your deployment configuration (Docker Compose, Kubernetes manifest, ECS task definition, etc.) to pull from the registry where you pushed your custom image.

5. Register the connector

After deploying the agent, Monte Carlo needs to discover and register your custom BI connector type before you can add a connection.

Your custom connector type is registered automatically when Monte Carlo discovers it on your agent. This happens:

  • The first time you enable the generic agent
  • When you run agent validations from the Monte Carlo UI (Settings → Deployments → your agent → Validate)

In the agent validations, a Discover connectors step lists all custom and native connectors found on the agent. When custom connectors are discovered, Monte Carlo triggers an async job to register the connector details. This usually takes a couple of minutes to complete.

ℹ️

If you update your custom connector and redeploy the agent, re-run the agent validations to resync the connector details with Monte Carlo.

To confirm that your custom BI connector type has been registered, run this GraphQL query in the Monte Carlo API:

query {
  getCustomBiConnectors(agentUuid: "<agent_uuid>") {
    agentUuid
    customBiConnectors {
      id
      name
      jobTypes
      iconUrl
    }
  }
}

Replace <agent_uuid> with your agent's UUID. The response lists each registered custom BI connector the agent reported.

6. Add the connection

Once the connector type is registered, you can add the connection through the Monte Carlo UI or the GraphQL API.

Using the UI

Go to Settings → Integrations → Add → Business intelligence → Custom BI integration and select Collect for metadata. Then:

  1. Select the Deployment — the agent where your connector is deployed.
  2. Select the Connector — one of the custom BI connector types registered by that agent.
  3. Configure credentials — Monte Carlo reads them from a secret store in your environment (see Self-hosted credentials).
  4. Run the validations in the drawer and submit.

After setup, Monte Carlo begins collecting automatically — asset metadata collection runs every 60 minutes. Assets appear in your catalog shortly after the first collection completes.

Using the API

If you prefer to automate setup or need programmatic access, you can add the connection via GraphQL.

1. List supported validations for your custom connector type:

query {
  getSupportedValidationsV2(
    connectionType: "<connection_type>"
    dcId: "<dc_id>"
  ) {
    supportedValidations {
      name
      description
      isPrerequisite
    }
  }
}

Replace <connection_type> with the type from getCustomBiConnectors (e.g., custom-bi-connector-de8d7c2) and <dc_id> with your data collector ID.

2. Run validations to confirm everything is connected correctly. Run each validation, starting with prerequisites:

mutation {
  testCustomConnector(
    validationName: "<validation_name>"
    connectionDetails: {
      selfHostedCredentialsType: FILE
      filePath: "/etc/secrets/integrations/<credentials_file>.json"
      connectionType: "<connection_type>"
    }
  ) {
    key
    validationResult {
      success
      errors {
        stackTrace
      }
    }
  }
}

Work through the validations returned in step 1. The final validation to run is save_credentials, which returns a key you'll need in the next step.

3. Add the connection using the key from the save_credentials validation:

mutation {
  addCustomBiConnector(
    key: "<key_from_save_credentials>"
    connectionType: "<connection_type>"
    name: "<display_name>"
    connectionName: "<display_name>"
  ) {
    connection {
      name
      uuid
    }
  }
}

Once this mutation succeeds, your custom BI connection is live — Monte Carlo creates the BI container, schedules hourly asset collection, and the first collection job runs shortly after.


Updating your connector

When you change your connector — new fields, bug fixes, new vendor endpoints:

  1. Make your changes in bi_connectors/<name>/connector.py.
  2. Re-run the tests:
    CONNECTOR=<name> docker compose run --rm test -m bi_metadata
  3. Rebuild the agent image:
    python scripts/generate_agent_image.py <name>
  4. Push the new image to your container registry and redeploy the agent.
  5. Re-run agent validations in the Monte Carlo UI (Settings → Deployments → your agent → Validate) if you changed the manifest (name, icon, credentials schema) — the Discover connectors validation syncs Monte Carlo with the updated details.

Until you re-run validations, Monte Carlo will continue using the previously registered connector details.

FAQs

What BI tools can I build a connector for?

Any tool that exposes an API or SDK for listing assets (dashboards, reports, datasets, …). The framework is API-agnostic — you bring the SDK and implement the fetch method.

How often does Monte Carlo collect from my connector?

The agent calls fetch_metadata every 60 minutes, paginating with limit/offset until it has collected all assets (up to 100,000 per collection run). The interval is fixed and not customer-configurable.

What happens to assets that are deleted in my BI tool?

Collected assets are managed by Monte Carlo: when an asset stops being returned by your connector, it ages out of the catalog automatically. You don't need to delete collected assets explicitly.

Can I include multiple connectors in one agent?

Yes. Pass multiple connector names when building the image:

python scripts/generate_agent_image.py my_tool_1 my_tool_2

How do I debug a failing connector?

The test suite output shows exactly which check failed and the validation errors. Run individual test markers to isolate the problem:

CONNECTOR=<name> docker compose run --rm test -m bi_connection     # just connection
CONNECTOR=<name> docker compose run --rm test -m bi_metadata       # just metadata

How do I rotate credentials?

Credentials are managed through self-hosted credentials. Update the secret in your secret store (AWS Secrets Manager, GCP Secret Manager, etc.) and the agent picks up the new value on its next connection. Further details can be found here.

What happens to my integration keys if I delete the integration?

Deleting an integration in Monte Carlo also deletes any integration keys associated with it. Before deleting an integration, make sure no active pipelines depend on its keys.


Did this page help you?