Bulk Monitor

Overview

Apply metric monitoring rules across many tables in a single definition. Instead of one metric monitor per table, define an asset_selection that matches tables by database and schema, then use field_pattern rules to automatically target matching columns across all of them.

📘
Reference scope
This page covers MaC YAML configuration. Bulk monitors are the code-first equivalent of multi-table metric monitors in the UI, with additional field_pattern matching available only through MaC.

MaC key: bulk_monitor. Two sub-types: bulk_metric for field-level metric monitoring, and bulk_pii for PII detection.

Quick Start

montecarlo:
  bulk_monitor:
    - name: id_null_rate_bulk
      description: Monitor null rates on all ID columns across analytics tables
      monitor_type: bulk_metric
      asset_selection:
        databases:
          - name: ANALYTICS
            schemas:
              - CORE
      alert_conditions:
        - metric: NULL_RATE
          operator: AUTO
          field_pattern:
            operator: ENDING_WITH
            value: _ID
            field_type: NUMERIC
      schedule:
        type: fixed
        interval_minutes: 1440
      domains:
        - my-domain

Configuration

description — what this monitor checks

string · required

Displayed in the Monte Carlo UI and in incident notifications. Max 512 characters.

description: Monitor null rates on ID columns across core analytics

monitor_type — bulk monitor sub-type

enum · required

Accepted values: bulk_metric · bulk_pii

bulk_metric for field-level metric monitoring, bulk_pii for PII detection.

monitor_type: bulk_metric

asset_selection — which tables to monitor

object · required

Select tables by database, schema, name patterns, or tags. Same structure as the table monitor asset selection.

asset_selection:
  databases:
    - name: ANALYTICS
      schemas:
        - CORE
  filters:
    - table_tags:
        - criticality:high
      table_tags_operator: HAS_ANY

Properties

databases — databases to monitor

array of objects · required

Each entry selects a database and optionally narrows to specific schemas.

Property	Type	Required	Description
`name`	string	yes	Database name
`schemas`	array of strings	no	Schemas to include. Omit to include all.
`tables`	object	no	Table-level selection within the database

databases:
  - name: ANALYTICS
    schemas:
      - CORE
      - STAGING
  - name: RAW

filters — narrow selection to matching tables

array of objects · optional

Each entry matches tables by name pattern, tag, or type. The type field is auto-inferred from the other fields present.

By name pattern:

Property	Type	Required	Description
`table_name`	string	yes	Pattern to match against table names
`table_name_operator`	string	yes	`STARTS_WITH` · `ENDS_WITH` · `CONTAINS` · `MATCH_PATTERN`
`negated`	boolean	no	Invert the filter. Default: `false`

By tag:

Property	Type	Required	Description
`table_tags`	array of strings	yes	Tag values to match
`table_tags_operator`	string	yes	`HAS_ALL` · `HAS_ANY`
`negated`	boolean	no	Invert the filter. Default: `false`

By type:

Property	Type	Required	Description
`table_type`	string	yes	`TABLE` · `VIEW` · `EXTERNAL`
`negated`	boolean	no	Invert the filter. Default: `false`

filters:
  - table_tags:
      - criticality:high
    table_tags_operator: HAS_ANY

exclusions — remove matching tables from selection

array of objects · optional

Same structure as filters. Tables matching any exclusion are removed even if they match a filter.

exclusions:
  - table_name: _deprecated
    table_name_operator: ENDS_WITH

alert_conditions — metric and field-matching rules

array of objects · required

Each entry defines a metric and field-matching pattern. Every alert condition in a bulk monitor must include a field_pattern. Supported alert condition types: threshold (default), noop.

Available operators: AUTO · AUTO_HIGH · AUTO_LOW · GT · GTE · LT · LTE · EQ · NEQ · INSIDE_RANGE · OUTSIDE_RANGE · NOOP

Threshold types:

Static — fixed numeric value via threshold_value with an explicit operator.
Range — lower_threshold and upper_threshold with INSIDE_RANGE or OUTSIDE_RANGE.
Anomaly detection (AUTO) — ML-based. Control aggressiveness with the monitor-level sensitivity field (if supported).

alert_conditions:
  - metric: NULL_RATE
    operator: AUTO
    field_pattern:
      operator: ENDING_WITH
      value: _ID
      field_type: NUMERIC

Properties

metric — built-in metric name

string · optional (one of metric or custom_metric required)

Built-in metric name (e.g., NULL_RATE, NUMERIC_MEAN). Mutually exclusive with custom_metric.

metric: NULL_RATE

custom_metric — custom SQL-based metric

object · optional (one of metric or custom_metric required)

Mutually exclusive with metric.

Property	Type	Required	Description
`uuid`	string	no	UUID of an existing custom metric to reuse
`display_name`	string	yes	Name for the metric
`sql_expression`	string	yes	SQL expression that evaluates to a single numeric value

custom_metric:
  display_name: Active Rate
  sql_expression: "COUNT(CASE WHEN status = 'ACTIVE' THEN 1 END)::FLOAT / COUNT(*)"

field_pattern — pattern-based field selection

object · required

Selects fields dynamically by name pattern across all tables matched by asset_selection.

Property	Type	Required	Description
`operator`	string	yes	`CONTAINING` · `ENDING_WITH` · `MATCHING` · `STARTING_WITH`
`value`	string	yes	Pattern string to match against field names
`case_sensitive`	boolean	no	Whether the match is case-sensitive. Default: `false`
`field_type`	enum	no	Restrict matching to a specific field type: `BOOLEAN` · `DATE` · `NUMERIC` · `TEXT` · `TIME` · `TIME_OF_DAY`

field_pattern:
  operator: ENDING_WITH
  value: _ID
  field_type: NUMERIC

fields — explicit column names

array of strings · n/a

Not usable for bulk monitors. field_pattern is required on every bulk alert condition, and the backend rejects an alert condition that sets both fields and field_pattern ("Bulk monitors match fields either by pattern or by an explicit list, not both"). Because field_pattern is mandatory, there is no valid configuration that uses fields here — match columns with field_pattern instead. To pin specific columns, use a metric monitor with an explicit fields list.

type — alert condition type

enum · optional · default: threshold

Accepted values: threshold · noop

Use noop to collect data without alerting.

type: threshold

operator — comparison operator

enum · optional

Accepted values: AUTO · AUTO_HIGH · AUTO_LOW · GT · GTE · LT · LTE · EQ · NEQ · INSIDE_RANGE · OUTSIDE_RANGE · NOOP

AUTO uses ML anomaly detection. GT, LT, etc. require threshold_value. INSIDE_RANGE / OUTSIDE_RANGE require lower_threshold and upper_threshold.

operator: AUTO

threshold_value — static threshold

number · yes, when using explicit operators (GT, LT, etc.)

Static threshold for comparison.

threshold_value: 0.05

lower_threshold — lower bound for range operators

number · yes, for INSIDE_RANGE / OUTSIDE_RANGE

lower_threshold: 0.01

upper_threshold — upper bound for range operators

number · yes, for INSIDE_RANGE / OUTSIDE_RANGE

upper_threshold: 0.10

baseline_trailing_days — ML baseline window

integer · optional

Number of trailing days for the ML baseline window. Minimum: 1.

baseline_trailing_days: 14

baseline_start — fixed baseline start date

string · optional

ISO 8601 format.

baseline_start: "2024-01-01"

baseline_end — fixed baseline end date

string · optional

ISO 8601 format.

baseline_end: "2024-03-31"

num_bins — histogram bins for drift metrics

integer · optional

Number of histogram bins. Range: 2--1000.

num_bins: 50

id — stable condition identifier

string · optional

Preserved across updates.

id: null_rate_id_check

schedule — execution schedule

object · required

Controls when the monitor runs. Bulk monitors do not support crontab-based scheduling and require a minimum interval of 60 minutes.

schedule:
  type: fixed
  interval_minutes: 1440

Properties

type — schedule type

enum · optional · default: fixed

Accepted values: fixed · dynamic

Crontab, loose, and manual are not supported.

type: fixed

interval_minutes — run interval

integer · optional

Run interval for fixed schedules. Minimum 60 minutes.

interval_minutes: 1440

start_time — schedule start time

string · optional

ISO 8601 format.

start_time: "2024-01-01T06:00:00Z"

timezone — schedule timezone

string · optional

timezone: America/New_York

dynamic_schedule_tables — tables that trigger the monitor

array of strings · yes, when type is dynamic (unless dynamic_schedule_jobs is set)

dynamic_schedule_tables:
  - raw:ingestion.events

dynamic_schedule_jobs — jobs that trigger the monitor

array of objects · optional

Property	Type	Required	Description
`job_type`	string	yes	`AdfJob` · `AirflowDag` · `DatabricksJob` · `DbtJob`
`job_name`	string	yes	Name of the job
`project_name`	string	yes	Project or workspace containing the job
`task_name`	string	no	Specific task within the job
`mcon`	string	no	MCON identifier for the job

dynamic_schedule_jobs:
  - job_type: DbtJob
    job_name: nightly_build
    project_name: analytics

min_interval_minutes — minimum interval between dynamic runs

integer · optional

min_interval_minutes: 120

domains — domain for this monitor

array of strings (exactly one entry) · required on all accounts created after January 2025

Set default_domain in montecarlo.yml to avoid repeating it on every monitor.

domains:
  - my-domain

warehouse — which warehouse to use

string · optional (yes if multiple warehouses)

Warehouse UUID or name. Overrides default_resource from montecarlo.yml.

warehouse: prod-snowflake

name — unique identifier within the namespace

string · required

Required for monitors created after Jan 29, 2024 (existing monitors keep working). Changing the name creates a new monitor and deletes the old one.

name: id_null_rate_monitoring

notes — internal notes

string · optional

Visible in the Monte Carlo UI. Not included in notifications.

notes: Owned by the analytics team. Reviewed quarterly.

audiences — notification channels

array of strings · optional

Audience names linking this monitor to channels defined in Notifications as Code.

audiences:
  - data-engineering-alerts
  - platform-alerts

failure_audiences — run-failure notification channels

array of strings · optional

Separate audiences for run-failure notifications. Falls back to audiences if not set.

failure_audiences:
  - oncall-alerts

priority — incident priority level

enum · optional

Accepted values: P1 · P2 · P3 · P4 · P5

priority: P3

tags — key-value pairs for organizing monitors

array of objects · optional

Property	Type	Required	Description
`name`	string	yes	Tag key
`value`	string	no	Tag value

tags:
  - name: team
    value: analytics
  - name: environment
    value: production

data_quality_dimension — data quality category

enum · optional

Accepted values: ACCURACY · COMPLETENESS · CONSISTENCY · TIMELINESS · UNIQUENESS · VALIDITY

data_quality_dimension: COMPLETENESS

collection_lag_hours — lag for late-arriving data

integer · optional

Number of hours to lag data collection, accounting for late-arriving data.

collection_lag_hours: 6

aggregate_time_field — timestamp column for time bucketing

string · optional

Timestamp column used to bucket data by time across all matched tables. The column must exist on every matched table.

aggregate_time_field: CREATED_AT

aggregate_by — time bucket granularity

enum · optional

Accepted values: hour · day · week · month

aggregate_by: day

sampling_config — row sampling configuration

object · optional

Only valid when monitor_type is bulk_pii.

Property	Type	Required	Description
`percentage`	number	no	Percentage of rows to sample (0--100)
`count`	integer	no	Fixed number of rows to sample

sampling_config:
  percentage: 10

auto_prune_enabled — automatically remove deleted tables

boolean · optional · default: false

Automatically remove tables from monitoring when they are deleted or become inaccessible. Only valid when monitor_type is bulk_pii.

auto_prune_enabled: true

lineage_narrowing_enabled — narrow by lineage importance

boolean · optional · default: false

Only monitor tables that are upstream of important downstream assets. Only valid when monitor_type is bulk_pii.

lineage_narrowing_enabled: true

is_draft — create as draft without activating

boolean · optional · default: false

Creates the monitor in a paused state. Omitting this on a later update resets to false (active) due to PUT semantics — always include it if you want the monitor to stay in draft.

is_draft: true

uuid — update an existing monitor

string · optional

Include the UUID of an existing monitor to update it instead of creating a new one.

uuid: 0dae7702-0950-45c7-909c-7e183bddca19

Deprecated fields

Field	Use instead
`resource`	`warehouse`
`domain`	`domains`
`domain_uuids`	`domains`
`labels`	`audiences`

📘
API-only fields
Some fields visible in the API or JSON Schema (domain_restrictions) are present in the schema but silently stripped during MaC YAML processing. They have no effect and should not be included.

Examples

Null rate monitoring on all ID columns

Monitors null rates across all columns ending in _ID in the analytics core schema.

montecarlo:
  bulk_monitor:
    - name: core_id_null_rate
      description: Monitor null rates on ID columns across core analytics
      monitor_type: bulk_metric
      warehouse: prod-snowflake
      asset_selection:
        databases:
          - name: ANALYTICS
            schemas:
              - CORE
      alert_conditions:
        - metric: NULL_RATE
          operator: AUTO
          field_pattern:
            operator: ENDING_WITH
            value: _ID
            field_type: NUMERIC
      schedule:
        type: fixed
        interval_minutes: 1440
      audiences:
        - data-engineering-alerts
      priority: P3
      data_quality_dimension: COMPLETENESS
      tags:
        - name: team
          value: analytics
      domains:
        - my-domain

PII detection across multiple schemas

Scans for PII patterns in text columns across raw data schemas.

montecarlo:
  bulk_monitor:
    - name: raw_pii_scan
      description: Scan for PII in raw ingestion tables
      monitor_type: bulk_pii
      warehouse: prod-snowflake
      asset_selection:
        databases:
          - name: RAW
            schemas:
              - INGESTION
              - API_IMPORTS
      alert_conditions:
        - metric: NULL_COUNT
          operator: GT
          threshold_value: 0
          field_pattern:
            operator: CONTAINING
            value: ""
            field_type: TEXT
      schedule:
        type: fixed
        interval_minutes: 1440
      audiences:
        - security-alerts
      priority: P1
      data_quality_dimension: VALIDITY
      domains:
        - my-domain

Dynamic scheduling with auto-pruning

Runs after table updates and automatically removes deleted tables. auto_prune_enabled and lineage_narrowing_enabled are only valid for bulk_pii monitors.

montecarlo:
  bulk_monitor:
    - name: timestamp_cols_bulk_pii
      description: Monitor freshness-sensitive columns after each load
      monitor_type: bulk_pii
      warehouse: prod-snowflake
      asset_selection:
        databases:
          - name: RAW
      auto_prune_enabled: true
      lineage_narrowing_enabled: true
      alert_conditions:
        - metric: NULL_RATE
          operator: AUTO
          field_pattern:
            operator: MATCHING
            value: ".*_TIMESTAMP"
            field_type: TIME
      schedule:
        type: dynamic
        dynamic_schedule_tables:
          - raw:ingestion.events
          - raw:ingestion.transactions
        min_interval_minutes: 120
      priority: P2
      domains:
        - my-domain

Troubleshooting

Monitor type and fields

Using monitor_type: metric instead of bulk_metric. The monitor_type enum values are bulk_metric and bulk_pii. Using metric (the MaC key for the single-table metric monitor) is not valid here.
Omitting field_pattern from alert conditions. Unlike the single-table metric monitor where you can list specific fields, bulk monitors require field_pattern on every alert condition.

Scheduling

Setting schedule.interval_minutes below 60. Bulk monitors enforce a minimum interval of 60 minutes. Lower values fail validation.
Using crontab scheduling. Bulk monitors do not support interval_crontab. Use interval_minutes for fixed schedules.
Using loose or manual schedule types. Bulk monitors only support fixed and dynamic schedule types.

Asset and field selection

Using aggregate_time_field with tables that lack the column. If any table matched by asset_selection does not have the specified timestamp column, the monitor fails on that table. Ensure the column exists across all matched tables, or split into separate monitors.
Wrong tags format. Tags must be objects with name and optional value keys. Writing tags: ["my-tag"] fails validation.

Updates and deprecated fields

Forgetting PUT semantics on updates. When updating a monitor by including uuid, every field you omit reverts to its default — it is not left unchanged. Always specify the complete desired configuration.
Using domain instead of domains. The singular domain field is deprecated. Use domains (plural) in all new configurations.