Job Monitor

Alert on job failures from Airflow, dbt, Databricks and other integrations.

Overview

A job monitor decides which job failures become alerts, and who is notified. It covers one integration, such as an Airflow environment or a dbt connection, and alerts on the failure types you choose.

📘
Reference scope
This page covers MaC YAML configuration. For how job monitors work, see Job Monitors.

MaC key: job. Job monitors have no schedule and no warehouse. They are evaluated each time a job failure arrives.

A job entry names its integration, jobs and dbt projects. It never uses uuids or mcons. A MaC file can be written in one account and applied to another, so apply resolves each name to exactly one integration, job or project, and fails when a name matches none or several. See How names resolve.

Quick Start

montecarlo:
  job:
    - name: nightly_etl_failures
      description: Nightly ETL failures
      integration:
        type: airflow
        name: prod-airflow
      asset_selection:
        filters:
          - type: JOB_TAG
            job_tags: ["tier:gold"]
            job_tags_operator: HAS_ANY
      alert_conditions:
        - condition: etl_job_failure
      audiences:
        - data-platform-oncall
      domains:
        - data-platform

Configuration

integration — the integration this monitor covers

object · required

A job monitor covers one integration. Name it by its type and its name in Monte Carlo. To alert on two integrations, create a monitor for each.

integration:
  type: dbt-cloud
  name: prod-dbt

Properties

type — integration type

string · required

Integration	`type`
Airflow	`airflow`
Azure Data Factory	`azure-data-factory`
Custom ETL connectors	`custom-etl-connector`
Databricks	`databricks`
GCP Dataform	`gcp-dataform`
Informatica (v2)	`informatica-v2`
Power BI Dataflows	`power_bi_dataflow`
Salesforce Data Cloud	`salesforce-data-cloud`
Snowflake tasks	`snowflake`
dbt	The dbt connection's type: `dbt-cloud`, `dbt-cloud-v2`, `dbt-cloud-webhook` or `dbt-core`

Fivetran, Informatica (v1) and MuleSoft have no job alert conditions, so they can't be used.

name — integration name

string · required

The integration's name on the Integrations page. For dbt, the connection's name, or its warehouse's name when the connection has none.

alert_conditions — the failure types that alert

array of objects · required

At least one condition is required. Each entry has one field, condition.

Integration	`condition`	UI name
All non-dbt integrations	`etl_job_failure`	Job run failed
dbt	`dbt_model_error`	Model error
dbt	`dbt_test_failure`	Test failure
dbt	`dbt_test_warning`	Test warning
dbt	`dbt_snapshot_error`	Snapshot error
dbt	`dbt_seed_error`	Seed error

alert_conditions:
  - condition: dbt_model_error
  - condition: dbt_test_failure

jobs — jobs picked one by one

array of objects · optional

Jobs the monitor covers, by name. A picked job is fixed: new jobs aren't covered unless a rule in asset_selection matches them. Prefer tag and project rules for jobs that change often.

jobs:
  - name: load_orders
    group: daily_dag

Field	Integration	Description
`name`	All	The job's name. Required.
`group`	Non-dbt only	The job's group, such as its Airflow DAG. Picks one of several jobs that share a name.
`project`	dbt only	The dbt project name. Required for dbt jobs.
`environment`	dbt only	The dbt environment. Picks one of several dbt jobs that share a project and name.

Set group or environment whenever the job has one. A job with neither resolves only while its name is unique in the integration.

asset_selection — rules for which jobs and tables are covered

object · optional

Rules that select jobs, plus table filters for dbt. Leave out asset_selection and jobs to cover every job in the integration.

asset_selection:
  filters:
    - type: JOB_TAG
      job_tags: ["tier:gold"]
      job_tags_operator: HAS_ANY
  exclusions:
    - type: JOB_MCON
      jobs:
        - name: legacy_load
          group: daily_dag

Properties

filters — rows that add jobs

array of objects · optional

Rules, so they cover jobs added later. Each row has a type:

`type`	Fields	Covers
`JOB_TAG`	`job_tags` (list of `key:value` strings), `job_tags_operator` (`HAS_ANY` or `HAS_ALL`)	Jobs with the tags. Each tag splits on its first colon.
`JOB_TAG_PAIR`	`job_tag_pairs` (list of `name` and optional `value`), `job_tags_operator`	Jobs with the tags, matched exactly. Use it for a tag name that contains a colon, or a tag with no value. A pair with no `value` matches only a tag with no value.
`JOB_MCON`	`jobs` (the same job references as `jobs`)	The named jobs.
`DBT_PROJECT`	`dbt_projects` (list of project names)	Every job in the projects, including jobs added later. dbt only.

exclusions — rows that remove jobs

array of objects · optional

The same row types as filters. An excluded job is never covered, even when it's also in jobs.

table_filters — limit to tables (dbt only)

array of objects · optional

A dbt failure alerts only when its table matches. Table filters don't change which jobs the monitor covers.

`type`	Fields
`FULL_TABLE_ID`	`full_table_id`, as `database:schema.table`
`DATABASE`	`databases` (list)
`SCHEMA`	`schemas` (list), and optional `database`

table_exclusions — exclude tables (dbt only)

array of objects · optional

The same rows as table_filters. A dbt failure on a table they match doesn't alert.

table_exclusions:
  - type: SCHEMA
    schemas: [staging]
    database: analytics

filters_joiner, exclusions_joiner, table_filters_joiner, table_exclusions_joiner — how rows combine

enum · optional · default: OR

Accepted values: OR · AND

With OR, a job or table matches when it matches any row. With AND, it must match every row.

alert_grouping — how failures are grouped into alerts

object · optional · default: group failures from the same job for 12 hours

Leave it out for the default. With the default, job monitors that cover the same job share one alert.

`mode`	Behavior
`group_into_open_alert`	New failures from a job join its open alert for up to `max_hours` (default `5`). `renotify_every_hours` (default `24`, `-1` to turn off) re-notifies while the alert stays open.
`no_grouping`	Every failed run opens its own alert. For dbt, all failures from one dbt run share an alert. Takes no other field: setting `max_hours` or `renotify_every_hours` fails.

alert_grouping:
  mode: group_into_open_alert
  max_hours: 6
  renotify_every_hours: 12

domains — domain for this monitor

array of strings (exactly one entry) · required when creating a monitor

Required on all accounts. Monitors that already exist without a domain can still be updated without adding one.

Every job the monitor picks must belong to its domain, and the monitor only covers jobs in that domain.

Set default_domain in montecarlo.yml to avoid repeating it on every monitor.

domains:
  - data-platform

name — unique identifier within the namespace

string · required

Identifies the monitor on re-apply. Changing the name creates a new monitor and deletes the old one — alert history does not transfer.

name: nightly_etl_failures

description — monitor description

string · required

Shown as the monitor's title. Max 512 characters.

description: Nightly ETL failures

audiences — notification channels

array of strings or mappings · optional

Audience names linking this monitor to channels defined in Notifications as Code. A monitor with no audience still creates alerts.

An entry can instead be a mapping that narrows an audience to specific triage priorities. Some audience must cover NOT_TRIAGED. See Routing alerts by triage priority.

audiences:
  - data-platform-oncall
  - audience: oncall
    triage_priority: [HIGH]

notes — internal notes

string · optional

Context for whoever responds, such as a runbook link.

notes: Runbook at https://wiki.example.com/etl-failures

priority — alert priority level

enum · optional

Accepted values: P1 · P2 · P3 · P4 · P5

priority: P2

tags — key-value pairs for organizing monitors

array of objects · optional

Property	Type	Required	Description
`name`	string	yes	Tag key
`value`	string	no	Tag value

tags:
  - name: team
    value: data-platform

data_quality_dimension — data quality category

enum · optional

Accepted values: ACCURACY · COMPLETENESS · CONSISTENCY · TIMELINESS · UNIQUENESS · VALIDITY

data_quality_dimension: TIMELINESS

is_draft — create as draft without activating

boolean · optional · default: false

A draft covers nothing until it is published. Omitting this on a later update resets to false (active) due to PUT semantics.

is_draft: true

auto_triage — override auto-triage for this monitor

boolean · optional

Force auto-triage on (true) or off (false) for this monitor, overriding the account or domain default. Omit to inherit. See the Triage Agent.

auto_triage: false

How names resolve

apply resolves each name when it runs, and stores the result with the template. A name that later resolves to a different job shows up as a change on the next apply.

Reference	Matches	Fails when
`integration`	A live integration of `type` with this name. dbt connection names can repeat in an account.	None or several match.
Non-dbt job	A job of the integration with this `name`, and with this `group` when set. Stale and deleted jobs don't match.	None or several match.
dbt job	A job of the connection with this `project` and `name`, and with this `environment` when set.	None or several match, or the job hasn't run yet.
`dbt_projects`	A project of the connection with this name.	None or several match.

Prefer tags to named jobs. A job renamed in its source tool can be recorded as a new job, and the old name stops resolving. The template then fails to apply, even though it didn't change, until you update or drop the reference. Tag and project rules keep working through renames.

Exports name the integration, jobs, projects and domains. An integration, job or project that no longer exists exports as <MISSING_INTEGRATION>, <MISSING_JOB> or <MISSING_DBT_PROJECT>. A placeholder fails to apply: replace it with a current name, or drop the entry.

Examples

All jobs in an integration

No jobs or asset_selection, so the monitor covers every Databricks job, including new ones.

montecarlo:
  job:
    - name: databricks_all_failures
      description: All Databricks job failures
      integration:
        type: databricks
        name: prod-databricks
      alert_conditions:
        - condition: etl_job_failure
      audiences:
        - data-platform-oncall
      priority: P2
      domains:
        - data-platform

Tagged DAGs, minus one

Covers every Airflow DAG tagged team:finance, except one task.

montecarlo:
  job:
    - name: finance_dag_failures
      description: Finance DAG failures
      integration:
        type: airflow
        name: prod-airflow
      asset_selection:
        filters:
          - type: JOB_TAG
            job_tags: ["team:finance"]
            job_tags_operator: HAS_ANY
        exclusions:
          - type: JOB_MCON
            jobs:
              - name: legacy_load
                group: finance_daily
      alert_conditions:
        - condition: etl_job_failure
      audiences:
        - finance-data
      domains:
        - finance

dbt project, model and test failures, outside staging

montecarlo:
  job:
    - name: analytics_dbt_failures
      description: Analytics dbt model and test failures
      integration:
        type: dbt-cloud
        name: prod-dbt
      asset_selection:
        filters:
          - type: DBT_PROJECT
            dbt_projects: [analytics]
        table_exclusions:
          - type: SCHEMA
            schemas: [staging]
            database: analytics
      alert_conditions:
        - condition: dbt_model_error
        - condition: dbt_test_failure
      audiences:
        - analytics-eng
      alert_grouping:
        mode: no_grouping
      domains:
        - analytics

Troubleshooting

Integration and jobs

Using ids instead of names. A job entry rejects integration_type, integration_uuid, job_mcons, and job_mcons or dbt_project_uuids on a filter row (JOB_MONITOR_MAC_FIELD_NOT_ACCEPTED). Use integration, jobs and dbt_projects.
Setting warehouse or resource. Job monitors have no warehouse (JOB_MONITOR_WAREHOUSE_NOT_ACCEPTED).
Leaving out integration, or an empty JOB_MCON or DBT_PROJECT row. Fails with JOB_MONITOR_MAC_FIELD_REQUIRED. So does a dbt job reference with no project.
Using the wrong job reference fields. Non-dbt jobs take name and group. dbt jobs take project, name and environment. Any other field fails with JOB_MONITOR_JOB_REFERENCE_FIELD_NOT_ALLOWED.
A name that matches several jobs (JOB_MONITOR_JOB_REFERENCE_AMBIGUOUS). Set group or environment.
A name that matches no job (JOB_MONITOR_JOB_REFERENCE_NOT_FOUND). The message says when only a stale or deleted job matches, which usually means the job was renamed.
A dbt job that hasn't run yet (JOB_MONITOR_DBT_JOB_HAS_NO_MCON). Wait for its first run, or select it by project or tag.
Two dbt connections with the same name (JOB_MONITOR_INTEGRATION_REFERENCE_AMBIGUOUS). Rename one connection.
Changing integration on an existing monitor. The integration can't change after the monitor is created. Give the entry a new name to replace the monitor.
Using DBT_PROJECT, table_filters or table_exclusions on a non-dbt integration. These are dbt only.

Alert conditions and grouping

No alert_conditions, or a condition from another integration. At least one is required, and non-dbt monitors only accept etl_job_failure.
Setting max_hours or renotify_every_hours with no_grouping. Fails with ALERT_GROUPING_FIELD_NOT_ALLOWED, whatever the value.

Update semantics

Forgetting PUT semantics on updates. Every field you omit reverts to its default. Always specify the complete desired configuration.

Updated 1 day ago

Did this page help you?