Example: Qlik Cloud
Overview
This guide walks through setting up a custom BI connector for Qlik Cloud (Qlik Sense SaaS) using a pre-built example implementation.
The connector talks to the Qlik Cloud REST API directly (Qlik's Python SDK is unmaintained and officially discouraged for new integrations) to collect BI asset metadata — apps, data flows, QlikView documents, and datasets — so Monte Carlo can catalog your Qlik assets, show who owns them, and link them to the warehouse tables they read.
This implementation is available as a reference in the mcd-public-resources repository. You can use it as-is or as a starting point for your own customizations.
Read the lineage section before you deploy
Qlik's lineage API is rate-limited to roughly 20 requests per minute — 50x tighter than the rest of the API. In-agent lineage is practical only for small tenants. The lineage section below covers the trade-off and two alternatives (a standalone push script and SQL query tagging) — pick your approach before building.
Feature support
| Category | Capability | Support |
|---|---|---|
| BI | Collect a catalog of BI assets and their metadata | ✅ |
| Lineage | BI → BI lineage | ✅ |
| Lineage | Table → BI asset | ✅ |
Sheets and visualizations inside an app are not collected — no Qlik REST endpoint enumerates them (it requires Qlik's Engine WebSocket API). The app is the leaf asset. Owner email/name requires an additional admin scope (see Qlik-specific notes); the raw owner ID is always collected.
Prerequisites
- A running custom-connector-setup repository — see Custom BI Connectors for initial setup.
- A Qlik Cloud tenant with assets you want to catalog.
- Tenant Admin access to Qlik (needed to create the OAuth client below).
- Your Qlik assets in a shared space — see the personal-space note below.
Qlik-specific notes
Understanding how Qlik's concepts map to Monte Carlo's BI data model is helpful if you plan to customize the connector:
| Monte Carlo concept | Qlik concept |
|---|---|
| Asset | An item from the unified catalog (GET /api/v1/items) |
| Asset type | app (Qlik Sense app), dataflow (a data-flow-prep app), qlikview, dataset |
| Identity | The item's resourceId GUID — stable across renames and re-cataloguing |
| Container | The item's space (names require an additional scope — see below) |
A few behaviors worth knowing:
view_countis a 28-day count, not all-time — Qlik'sitemViewscovers a trailing 28-day window.folder(space name) and owner email/name are gated. An OAuth client that is merely a member of a space can read its items, but Qlik's spaces and users APIs return empty/404 for it. The connector always emits the raw owner ID asowner.source_id; names and emails appear automatically if the credential later gainsadmin_classic-level scope.- Auth is not least-privilege by default. A Qlik API key inherits all permissions of the user who created it, and an OAuth machine-to-machine client is granted Tenant-Admin-equivalent rights by default. Prefer OAuth (it's rotatable) and scope it down where your security posture requires.
Setup
1. Create the OAuth client in Qlik
The connector authenticates with an OAuth2 machine-to-machine (M2M) client:
- In the Qlik Administration activity center, go to OAuth and click Create new.
- Set Client type to
Web, then enable Allow Machine-to-Machine (M2M) — not "M2M user impersonation". - After creation, copy the Client ID and Client secret immediately — the secret is shown only once.
- Open the client's action menu → Change consent method → Trusted. Without this, every token request fails with
OAUTH-5: oauth client is not approved with trusted consent method.
Personal spaces are private. A personal space is visible only to its owner — no other identity, not even a Tenant Admin, can be added to it. If your apps live in personal spaces, move them to a shared space, or add the
admin.appsscope to the OAuth client (Qlik's capability for reading personal/private content).Add the bot to your spaces. An M2M client only sees items in spaces it is a member of. Open each shared space → Members → add the client's bot user (named after the OAuth client) with Can view. Until you do, the connector sees nothing.
2. Scaffold the connector
Use the create_connector script with the --bi flag. This generates a unique identifier for your connector in manifest.json — this ID is how Monte Carlo distinguishes connector types, so it must be generated per-installation.
python scripts/create_connector.py qlik --biThe script prompts for an optional icon URL (leave blank to skip). Then replace the empty credentials_schema in manifest.json with:
"credentials_schema": {
"connect_args": {
"type": "dict",
"required": true,
"schema": {
"tenant_url": { "type": "string", "required": true, "empty": false },
"api_key": { "type": "string", "required": false },
"client_id": { "type": "string", "required": false },
"client_secret": { "type": "string", "required": false },
"space_id": { "type": "string", "required": false },
"collect_lineage": { "type": "boolean", "required": false }
}
}
}3. Replace with the example implementation
Clone or download the example from mcd-public-resources and replace the stub connector.py:
cp <path-to-mcd-public-resources>/custom_connectors/qlik/connector.py bi_connectors/qlik/No vendor SDK is required — the connector calls the Qlik REST API directly with requests, which is already available. Leave requirements.txt empty and no Dockerfile.extra is needed.
4. Configure credentials
Create bi_connectors/qlik/credentials.json using the example as a reference:
{
"connect_args": {
"tenant_url": "https://<tenant>.<region>.qlikcloud.com",
"client_id": "<oauth-client-id>",
"client_secret": "<oauth-client-secret>",
"space_id": "",
"collect_lineage": true
}
}| Field | Description | Default |
|---|---|---|
| tenant_url | Your tenant base URL, e.g. https://acme.us.qlikcloud.com | |
| client_id | OAuth client ID from step 1 | |
| client_secret | OAuth client secret from step 1 | |
| api_key | Alternative to OAuth — a tenant API key. Supply either this or the OAuth pair | |
| space_id | Restrict collection to a single space; leave empty for the whole tenant | "" |
| collect_lineage | Resolve lineage in-agent — see below | true |
5. Build, test, and preview
docker compose build
# Verify connection (exchanges the OAuth token)
CONNECTOR=qlik docker compose run --rm test -m bi_connection
# Metadata collection (validates fetch_metadata output)
CONNECTOR=qlik docker compose run --rm test -m bi_metadata
# Preview what will actually be collected — asset types, lineage
# edges by name, table inputs, and field coverage
CONNECTOR=qlik python scripts/preview.pyThe preview command runs fetch_metadata and prints exactly what will be collected. Sanity-check that asset type labels read correctly (a data flow shows as dataflow, not app), lineage edges point where you expect, and table names are clean database.schema.table values.
For the first metadata test on a large tenant, set
"collect_lineage": false— with lineage on, the test collects a full page of assets, each costing a rate-limited lineage call. Flip it back on once the basics are green.
6. Build the agent image
python scripts/generate_agent_image.py qlik7. Deploy, register, and connect
From here, follow the standard custom BI connector workflow:
- Push the image to your container registry and deploy the agent, providing your production Qlik credentials via self-hosted credentials.
- Register the connector by running agent validations in the Monte Carlo UI.
- Add the connection through the Monte Carlo UI (Settings → Integrations → Add → BI → Custom BI integration).
Lineage: choose your approach
The connector's lineage — both BI → BI edges (an app reading from a data flow or dataset) and the warehouse tables each asset reads — comes from a single Qlik API: GET /api/v1/lineage-graphs/nodes/{qri}, one call per asset.
That endpoint is rate-limited to ~20 requests/minute. The catalog reads the connector also makes get 1000/minute — lineage is 50x tighter. In-agent lineage is resolved only for the assets on each collected page: a page of 100 assets already costs ~5 minutes of wall clock, and a tenant with thousands of apps cannot collect lineage in-agent at all.
Pick the approach that matches your tenant size before you build.
| Approach | Best for | How |
|---|---|---|
| In-agent lineage (default) | Small tenants (tens of assets) | collect_lineage: true — nothing extra to run |
| Push script | Larger tenants, or lineage on your schedule | Companion script |
| SQL query tagging | Exact table lineage without the lineage API | Query tags |
Approach 1: in-agent lineage (default)
With collect_lineage: true, the agent resolves lineage for each page of assets it collects. Nothing to configure beyond the credential flag. Two caveats:
- Cost scales with page size — each collected asset costs one rate-limited lineage call. Fine at tens of assets; impractical at thousands.
- Table names are only as qualified as Qlik makes them. A table reached through a registered Qlik data connection arrives as a clean
database.schema.table; a table loaded inline may arrive as a bare name (e.g.MY_TABLE) with no database or schema, because Qlik's lineage API exposes no separate database/schema attributes. Monte Carlo may only partially match those.
Approach 2: push script
For larger tenants, the example ships a standalone script — push_events.py — that decouples lineage from the agent's collection schedule:
flowchart LR
A["Agent (pull)"] -->|"metadata only<br>collect_lineage: false"| MC["Monte Carlo"]
Q["Qlik tenant"] -->|"paced lineage calls<br>~15/min"| S["push_events.py (you run)"]
S -->|"assets + lineage edges<br>BI Push Ingest API"| MC
The script is self-contained — it doesn't need the custom-connector-setup repo or the agent at all, just pycarlo and your Qlik credentials. It collects the catalog (cheap), paces its lineage calls to stay under the rate limit (4 seconds apart by default), and pushes complete BI metadata events — assets with their lineage edges — to Monte Carlo's BI Push Ingest API via the pycarlo IngestionService. Pushed assets upsert on asset_source_id, so re-running the script is safe and is the intended way to keep lineage fresh (e.g. from cron).
To use it:
-
In Monte Carlo, add a Custom BI integration and choose Push as the mode (see the push API guide for the integration and key setup). Note the resource UUID and generate an integration key (ID + secret, Ingestion scope).
-
Install
pycarlo>=0.15.240usingpip install -U pycarlo -
Put your Qlik credentials in a
credentials.jsonnext to the script (same shape as credentials_example.json). -
Run it — Monte Carlo credentials come from the environment, never CLI args:
export MCD_DEFAULT_API_ID="<integration-key-id>"
export MCD_DEFAULT_API_TOKEN="<integration-key-secret>"
python push_events.py --resource-uuid <resource-uuid>
# Qlik-side options:
python push_events.py --resource-uuid <resource-uuid> --interval 8 # slower pace, more headroom
python push_events.py --resource-uuid <resource-uuid> --space-id <id> # one space only
python push_events.py --resource-uuid <resource-uuid> --no-lineage # metadata only, no lineage callsPushes go to the public Integration Gateway, and the script prints the endpoint it's pushing to before it sends anything.
The runtime is honest about scale: 1,000 assets at the default pace is roughly one hour of wall clock — run it from cron or CI rather than expecting it to finish quickly.
If you keep the agent collecting metadata alongside the script, point its credential at
"collect_lineage": falseso the agent stays fast and the script owns lineage. Note that the agent connection and the push integration are separate connections in Monte Carlo — confirm with Monte Carlo support whether they can share one integration before relying on both for the same assets.
Approach 3: SQL query tagging
If exact table → asset lineage matters more than BI → BI edges, skip Qlik's lineage API entirely and tag the SQL Qlik sends to your warehouse. Monte Carlo resolves the tags from the warehouse's query logs and creates table → BI asset edges with no lineage calls at all. See Lineage via SQL query tagging for the general mechanism.
For Qlik, set your warehouse's native query tag — e.g. on Snowflake, execute an ALTER SESSION statement at the start of the app's load script, before the data loads:
LIB CONNECT TO 'Snowflake_connection';
SQL ALTER SESSION SET QUERY_TAG = '{"mcd_bi_asset_id": "<app-guid>", "mcd_resource_id": "<resource-uuid>"}';Both tags are required: mcd_bi_asset_id is the asset's asset_source_id (the app's GUID, exactly as the connector reports it), and mcd_resource_id is your Custom BI integration's resource UUID.
Requirements:
- The warehouse Qlik loads from is monitored by Monte Carlo with query logs enabled — this is what makes the tables appear in the catalog and lets Monte Carlo read the tags. Without it, tags are never ingested and no lineage is produced.
- Each app needs its own tag value, so this works best for a small number of high-value apps rather than the whole tenant.
Updated about 1 hour ago
