Example: Qlik Cloud

Overview

This guide walks through setting up a custom BI connector for Qlik Cloud (Qlik Sense SaaS) using a pre-built example implementation.

The connector talks to the Qlik Cloud REST API directly (Qlik's Python SDK is unmaintained and officially discouraged for new integrations) to collect BI asset metadata — apps, data flows, QlikView documents, and datasets — so Monte Carlo can catalog your Qlik assets, show who owns them, and link them to the warehouse tables they read.

This implementation is available as a reference in the mcd-public-resources repository. You can use it as-is or as a starting point for your own customizations.

❗️

Read the lineage section before you deploy

Qlik's lineage API is rate-limited to roughly 20 requests per minute — 50x tighter than the rest of the API. In-agent lineage is practical only for small tenants. The lineage section below covers the trade-off and two alternatives (a standalone push script and SQL query tagging) — pick your approach before building.

Feature support

CategoryCapabilitySupport
BICollect a catalog of BI assets and their metadata✅
LineageBI → BI lineage✅
LineageTable → BI asset✅
ℹ️

Sheets and visualizations inside an app are not collected — no Qlik REST endpoint enumerates them (it requires Qlik's Engine WebSocket API). The app is the leaf asset. Owner email/name requires an additional admin scope (see Qlik-specific notes); the raw owner ID is always collected.

Prerequisites

  • A running custom-connector-setup repository — see Custom BI Connectors for initial setup.
  • A Qlik Cloud tenant with assets you want to catalog.
  • Tenant Admin access to Qlik (needed to create the OAuth client below).
  • Your Qlik assets in a shared space — see the personal-space note below.

Qlik-specific notes

Understanding how Qlik's concepts map to Monte Carlo's BI data model is helpful if you plan to customize the connector:

Monte Carlo conceptQlik concept
AssetAn item from the unified catalog (GET /api/v1/items)
Asset typeapp (Qlik Sense app), dataflow (a data-flow-prep app), qlikview, dataset
IdentityThe item's resourceId GUID — stable across renames and re-cataloguing
ContainerThe item's space (names require an additional scope — see below)

A few behaviors worth knowing:

  • view_count is a 28-day count, not all-time — Qlik's itemViews covers a trailing 28-day window.
  • folder (space name) and owner email/name are gated. An OAuth client that is merely a member of a space can read its items, but Qlik's spaces and users APIs return empty/404 for it. The connector always emits the raw owner ID as owner.source_id; names and emails appear automatically if the credential later gains admin_classic-level scope.
  • Auth is not least-privilege by default. A Qlik API key inherits all permissions of the user who created it, and an OAuth machine-to-machine client is granted Tenant-Admin-equivalent rights by default. Prefer OAuth (it's rotatable) and scope it down where your security posture requires.

Setup

1. Create the OAuth client in Qlik

The connector authenticates with an OAuth2 machine-to-machine (M2M) client:

  1. In the Qlik Administration activity center, go to OAuth and click Create new.
  2. Set Client type to Web, then enable Allow Machine-to-Machine (M2M) — not "M2M user impersonation".
  3. After creation, copy the Client ID and Client secret immediately — the secret is shown only once.
  4. Open the client's action menu → Change consent method → Trusted. Without this, every token request fails with OAUTH-5: oauth client is not approved with trusted consent method.
ℹ️

Personal spaces are private. A personal space is visible only to its owner — no other identity, not even a Tenant Admin, can be added to it. If your apps live in personal spaces, move them to a shared space, or add the admin.apps scope to the OAuth client (Qlik's capability for reading personal/private content).

Add the bot to your spaces. An M2M client only sees items in spaces it is a member of. Open each shared space → Members → add the client's bot user (named after the OAuth client) with Can view. Until you do, the connector sees nothing.

2. Scaffold the connector

Use the create_connector script with the --bi flag. This generates a unique identifier for your connector in manifest.json — this ID is how Monte Carlo distinguishes connector types, so it must be generated per-installation.

python scripts/create_connector.py qlik --bi

The script prompts for an optional icon URL (leave blank to skip). Then replace the empty credentials_schema in manifest.json with:

  "credentials_schema": {
    "connect_args": {
      "type": "dict",
      "required": true,
      "schema": {
        "tenant_url":      { "type": "string",  "required": true,  "empty": false },
        "api_key":         { "type": "string",  "required": false },
        "client_id":       { "type": "string",  "required": false },
        "client_secret":   { "type": "string",  "required": false },
        "space_id":        { "type": "string",  "required": false },
        "collect_lineage": { "type": "boolean", "required": false }
      }
    }
  }

3. Replace with the example implementation

Clone or download the example from mcd-public-resources and replace the stub connector.py:

cp <path-to-mcd-public-resources>/custom_connectors/qlik/connector.py bi_connectors/qlik/

No vendor SDK is required — the connector calls the Qlik REST API directly with requests, which is already available. Leave requirements.txt empty and no Dockerfile.extra is needed.

4. Configure credentials

Create bi_connectors/qlik/credentials.json using the example as a reference:

{
  "connect_args": {
    "tenant_url": "https://<tenant>.<region>.qlikcloud.com",
    "client_id": "<oauth-client-id>",
    "client_secret": "<oauth-client-secret>",
    "space_id": "",
    "collect_lineage": true
  }
}
FieldDescriptionDefault
tenant_urlYour tenant base URL, e.g. https://acme.us.qlikcloud.com
client_idOAuth client ID from step 1
client_secretOAuth client secret from step 1
api_keyAlternative to OAuth — a tenant API key. Supply either this or the OAuth pair
space_idRestrict collection to a single space; leave empty for the whole tenant""
collect_lineageResolve lineage in-agent — see belowtrue

5. Build, test, and preview

docker compose build

# Verify connection (exchanges the OAuth token)
CONNECTOR=qlik docker compose run --rm test -m bi_connection

# Metadata collection (validates fetch_metadata output)
CONNECTOR=qlik docker compose run --rm test -m bi_metadata

# Preview what will actually be collected — asset types, lineage
# edges by name, table inputs, and field coverage
CONNECTOR=qlik python scripts/preview.py

The preview command runs fetch_metadata and prints exactly what will be collected. Sanity-check that asset type labels read correctly (a data flow shows as dataflow, not app), lineage edges point where you expect, and table names are clean database.schema.table values.

ℹ️

For the first metadata test on a large tenant, set "collect_lineage": false — with lineage on, the test collects a full page of assets, each costing a rate-limited lineage call. Flip it back on once the basics are green.

6. Build the agent image

python scripts/generate_agent_image.py qlik

7. Deploy, register, and connect

From here, follow the standard custom BI connector workflow:

  1. Push the image to your container registry and deploy the agent, providing your production Qlik credentials via self-hosted credentials.
  2. Register the connector by running agent validations in the Monte Carlo UI.
  3. Add the connection through the Monte Carlo UI (Settings → Integrations → Add → BI → Custom BI integration).

Lineage: choose your approach

The connector's lineage — both BI → BI edges (an app reading from a data flow or dataset) and the warehouse tables each asset reads — comes from a single Qlik API: GET /api/v1/lineage-graphs/nodes/{qri}, one call per asset.

❗️

That endpoint is rate-limited to ~20 requests/minute. The catalog reads the connector also makes get 1000/minute — lineage is 50x tighter. In-agent lineage is resolved only for the assets on each collected page: a page of 100 assets already costs ~5 minutes of wall clock, and a tenant with thousands of apps cannot collect lineage in-agent at all.

Pick the approach that matches your tenant size before you build.

ApproachBest forHow
In-agent lineage (default)Small tenants (tens of assets)collect_lineage: true — nothing extra to run
Push scriptLarger tenants, or lineage on your scheduleCompanion script
SQL query taggingExact table lineage without the lineage APIQuery tags

Approach 1: in-agent lineage (default)

With collect_lineage: true, the agent resolves lineage for each page of assets it collects. Nothing to configure beyond the credential flag. Two caveats:

  • Cost scales with page size — each collected asset costs one rate-limited lineage call. Fine at tens of assets; impractical at thousands.
  • Table names are only as qualified as Qlik makes them. A table reached through a registered Qlik data connection arrives as a clean database.schema.table; a table loaded inline may arrive as a bare name (e.g. MY_TABLE) with no database or schema, because Qlik's lineage API exposes no separate database/schema attributes. Monte Carlo may only partially match those.

Approach 2: push script

For larger tenants, the example ships a standalone script — push_events.py — that decouples lineage from the agent's collection schedule:

flowchart LR
    A["Agent (pull)"] -->|"metadata only<br>collect_lineage: false"| MC["Monte Carlo"]
    Q["Qlik tenant"] -->|"paced lineage calls<br>~15/min"| S["push_events.py (you run)"]
    S -->|"assets + lineage edges<br>BI Push Ingest API"| MC

The script is self-contained — it doesn't need the custom-connector-setup repo or the agent at all, just pycarlo and your Qlik credentials. It collects the catalog (cheap), paces its lineage calls to stay under the rate limit (4 seconds apart by default), and pushes complete BI metadata events — assets with their lineage edges — to Monte Carlo's BI Push Ingest API via the pycarlo IngestionService. Pushed assets upsert on asset_source_id, so re-running the script is safe and is the intended way to keep lineage fresh (e.g. from cron).

To use it:

  1. In Monte Carlo, add a Custom BI integration and choose Push as the mode (see the push API guide for the integration and key setup). Note the resource UUID and generate an integration key (ID + secret, Ingestion scope).

  2. Install pycarlo>=0.15.240 using

    pip install -U pycarlo
  3. Put your Qlik credentials in a credentials.json next to the script (same shape as credentials_example.json).

  4. Run it — Monte Carlo credentials come from the environment, never CLI args:

export MCD_DEFAULT_API_ID="<integration-key-id>"
export MCD_DEFAULT_API_TOKEN="<integration-key-secret>"

python push_events.py --resource-uuid <resource-uuid>

# Qlik-side options:
python push_events.py --resource-uuid <resource-uuid> --interval 8       # slower pace, more headroom
python push_events.py --resource-uuid <resource-uuid> --space-id <id>    # one space only
python push_events.py --resource-uuid <resource-uuid> --no-lineage       # metadata only, no lineage calls

Pushes go to the public Integration Gateway, and the script prints the endpoint it's pushing to before it sends anything.

The runtime is honest about scale: 1,000 assets at the default pace is roughly one hour of wall clock — run it from cron or CI rather than expecting it to finish quickly.

ℹ️

If you keep the agent collecting metadata alongside the script, point its credential at "collect_lineage": false so the agent stays fast and the script owns lineage. Note that the agent connection and the push integration are separate connections in Monte Carlo — confirm with Monte Carlo support whether they can share one integration before relying on both for the same assets.

Approach 3: SQL query tagging

If exact table → asset lineage matters more than BI → BI edges, skip Qlik's lineage API entirely and tag the SQL Qlik sends to your warehouse. Monte Carlo resolves the tags from the warehouse's query logs and creates table → BI asset edges with no lineage calls at all. See Lineage via SQL query tagging for the general mechanism.

For Qlik, set your warehouse's native query tag — e.g. on Snowflake, execute an ALTER SESSION statement at the start of the app's load script, before the data loads:

LIB CONNECT TO 'Snowflake_connection';
SQL ALTER SESSION SET QUERY_TAG = '{"mcd_bi_asset_id": "<app-guid>", "mcd_resource_id": "<resource-uuid>"}';

Both tags are required: mcd_bi_asset_id is the asset's asset_source_id (the app's GUID, exactly as the connector reports it), and mcd_resource_id is your Custom BI integration's resource UUID.

Requirements:

  • The warehouse Qlik loads from is monitored by Monte Carlo with query logs enabled — this is what makes the tables appear in the catalog and lets Monte Carlo read the tags. Without it, tags are never ingested and no lineage is produced.
  • Each app needs its own tag value, so this works best for a small number of high-value apps rather than the whole tenant.

Did this page help you?