Consumption of warehouse resources

Monitors in Monte Carlo will consume resources in your data warehouse. For customers that need to attribute the consumption of warehouse resources back to the appropriate lines of business, two options are available:

• Query tags: queries from monitors will include a query tag containing the monitor_id. This is currently available just for Snowflake and BigQuery. Join this with the Monitors Data Export to attribute consumption back to particular creators, domains, audiences, and more.
• Connections: when integrating a data source to Monte Carlo, administrators can create multiple connections. By associating these with distinct users or warehouses within your data source, you can track the consumption associated with that connection. Using authorization groups in Monte Carlo, administrators can then define with connections any user is permitted to use.

These capabilities are limited to customers on our Enterprise product tier.

Query tags

On monitor runs, Monte Carlo adds the identifier mc_monitor_id=<uuid> to queries and jobs on supported warehouses so you can map them back to monitors:

• Snowflake: each monitor-run query includes QUERY_TAG text mc_monitor_id=<uuid>
• BigQuery: each monitor-run job includes a label mc_monitor_id=<uuid>

Use it to:

• Map queries/jobs to monitor_id
• Join to the Monitors Data Export for attributes (i.e.created_by, domains, monitor_tags, audiences, and more)
• Group by those attributes to attribute warehouse usage/cost per team

Snowflake example

Example: attribute queries to audiences.

select
  m.audiences as audiences,
  count(*) as query_count,
  sum(q.execution_time) as execution_ms
from snowflake.account_usage.query_history q
join <your_share_schema>.monitors m
  on m.monitor_id = regexp_substr(q.query_tag, 'mc_monitor_id=([0-9a-f\-]{36})', 1, 1, 'i', 1)
where q.query_tag ilike '%mc_monitor_id=%'
  and q.start_time >= dateadd('day', -7, current_timestamp())
group by audiences
order by execution_ms desc

Notes:

• The monitors table used above can be loaded from the Monitors Data Export; it includes monitor_id, created_by, domains, monitor_tags, audiences, and more.
• For Snowflake credit accounting, combine query-level attribution with warehouse metering as needed. We recommend using connections per team for strict isolation and chargebacks; see adding a new query-engine connection.

BigQuery example

Example: attribute jobs to audiences.

select
  m.audiences as audiences,
  count(*) as job_count,
  sum(j.total_bytes_billed) as total_bytes_billed
from `region-<YOUR_REGION>`.INFORMATION_SCHEMA.JOBS_BY_PROJECT j
cross join unnest(j.labels) l
join `<your_dataset>`.monitors m
  on m.monitor_id = l.value
where l.key = 'mc_monitor_id'
  and j.creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
group by audiences
order by job_count desc

Notes:

• Replace region-<YOUR_REGION> with your BigQuery region (e.g., region-us).
• The monitors table used above can be loaded from the Monitors Data Export; it includes monitor_id, created_by, domains, monitor_tags, audiences, and more.
• We recommend using connections per team for strict isolation and chargebacks.

Connections

When integrating data sources like Snowflake, BigQuery, Redshift, and Databricks into Monte Carlo, administrators can create multiple connections. By associating these with distinct users or warehouses within your data source, administrators can track the consumption associated with that connection.

Going a step further, administrators can also limit what connections a Monte Carlo user is able to use via Authorization Groups. When a user is then creating a monitor or using Data Profiler, only the permissible connections are made available to them. This ensures that the consumption from a given team can be easily tracked.

Limitations

Metadata for Table Monitors is collected en masse through the connection that is specified for metadata collection. There is not an ability to select between connections for Table Monitors.