Job Monitors

A job monitor decides which job failures become alerts, and who is notified about them. You choose an integration, the jobs to cover, and the failure types to alert on. Monte Carlo then evaluates every job failure from that integration as it arrives.

Job monitors replace the earlier job alerting settings: the per-job alert toggles on integrations, the alert types on dbt projects, and the Job notifications tab on audiences. Job alerting now lives on the Monitors page with your other monitors, and uses the same audiences, priorities, tags, notes, domains and alert grouping.

How it works

flowchart LR
    E["Job run fails<br>(Airflow, dbt, Databricks, ...)"] --> C{"Covered by an<br>active job monitor?"}
    C -->|"no"| N["No alert"]
    C -->|"yes"| A{"Matches the monitor's<br>alert conditions?"}
    A -->|"no"| N
    A -->|"yes"| G["Alert grouping"]
    G --> R["Notify the monitor's<br>audiences"]

When a job failure arrives, Monte Carlo:

1. Finds the covering monitors. A monitor covers a failure when the job is in the monitor's job selection and, for dbt, the failing table matches the monitor's table filters. Coverage is worked out when the failure arrives, so rules such as "all jobs" or "jobs tagged tier:gold" include jobs added after the monitor was created.
2. Checks the alert conditions. Each covering monitor alerts only on the failure types it selects, for example dbt Test failure but not Test warning.
3. Groups the alert. The monitor's alert grouping decides whether the failure opens a new alert or joins an open one.
4. Notifies the audiences set on the monitor.

A job failure alerts only when an active job monitor covers it. Paused, draft and deleted monitors cover nothing. A job that no monitor covers does not alert.

Supported integrations

Fivetran, Informatica (v1) and MuleSoft have no job alert conditions, so they don't appear in the job monitor integration list.

Prerequisites

• A connected integration from the list above.
• A webhook, for webhook-based integrations. Airflow, Azure Data Factory, Databricks, custom ETL connectors, GCP Dataform, Snowflake, Power BI Dataflows, Salesforce Data Cloud and dbt Cloud send failures to Monte Carlo through a webhook. Without one, failures are only picked up when Monte Carlo next collects runs, so alerts may be delayed up to an hour. The job monitor form and detail page warn you when an integration has Webhook not set up or No recent webhook events (none in the last 7 days).
• Permission to manage job monitors. See Permissions.

Steps

1. Open the job monitor form — Choose Job from Add monitor.
2. Choose jobs — Pick the integration, then all of its jobs or a selection.
3. Choose alert conditions — Pick the failure types that should alert.
4. Send notifications — Choose audiences, notes and alert grouping.
5. Add details and save — Add a title, tags and priority, then save.

1. Open the job monitor form

Add monitor → Data observability → Job

The form opens as Add job monitor. To change an existing monitor, open it from the Monitors page and edit it. The form opens as Edit job monitor.

2. Choose jobs

Integration

Choose an integration from Integration. A job monitor covers jobs from one integration. To alert on jobs from two integrations, create a monitor for each.

If your account uses domains, choose the domain first. Every job you pick must belong to the monitor's domain, and the monitor only covers jobs in that domain.

Jobs to monitor

Choose jobs and rules opens a two-panel picker. Available jobs on the left lists the integration's active jobs. Selected jobs on the right shows what the monitor covers.

You can add three kinds of selection:

Rules are saved as rules, not as a list of today's jobs. Jobs matched by a rule carry a By rule badge in Selected jobs. Choose Count on a rule to see how many jobs it matches now.

Exclusions. To leave one job out of a rule, uncheck it in Selected jobs. Exclusions win over picked jobs: an excluded job is never covered.

The picker lists active jobs only. A dbt job appears once it has run in the last 7 days.

Limit to tables (dbt only)

For dbt integrations, Limit to tables narrows which failures alert to failures on certain databases, schemas or tables:

• Database or schema — search and select a database, or a schema within a database.
• Tables — enter fully qualified tables as database:schema.table.

Table filters apply to each failing model, test, snapshot or seed. A dbt run can produce failures that alert and failures that don't. Table filters don't change which jobs the monitor covers, and the job list on the monitor is not narrowed by them.

3. Choose alert conditions

Under Alert when, check the failure types that should alert. Every condition is checked on a new monitor, and at least one is required.

Conditions have no thresholds or parameters.

4. Send notifications

Send alerts to — choose the audiences to notify. You can limit each audience to certain triage priorities, the same as on other monitors. A monitor with no audience still creates alerts, and shows No audience in Monte Carlo.

Notes — add context for whoever responds, such as a runbook link or the owning team.

Alert grouping

Reduce noisy notifications controls how failures are grouped into alerts. See Reducing noise from monitors for how grouping works across monitors.

A grouping change applies to alerts opened after you save it.

5. Add details and save

Choose Add to save, or Save as draft. A draft covers nothing until it is published.

When more than one monitor covers a job

A job can be covered by several job monitors, for example one for every Airflow DAG that pages the platform team, and one for DAGs tagged team:finance that notifies the finance channel.

• Every covering monitor is evaluated. Each monitor applies its own conditions and notifies its own audiences.
• Monitors on the default grouping share one alert. The shared alert takes the priority of the highest-priority monitor, and notifies the audiences of all of them.
• Monitors with any other grouping get their own alerts.
• A failure is never alerted twice by the same monitor, even when the run is collected again.

Viewing job monitors

Monitors page

Filter Type by Job to list your job monitors. Last run and Next run are blank for job monitors, and Status shows only Enabled, Disabled or Draft.

Monitor detail

A job monitor has Details, Alerts and Activity tabs. Details shows:

• Integration — the integration type and name.
• Jobs — what the monitor Covers (for example All Airflow jobs, All Airflow jobs except 2, or 14 jobs), the rules and picked jobs that select them, and the full job list. For dbt, it also lists Projects and Tables. The list is worked out from the selection when you open it, so it follows jobs as they are added, removed or retagged.
• Alert conditions — the conditions under Alert when.
• Notifications — audiences, notes and alert grouping.
• Details — title, tags, priority and data quality dimension.

Jobs page

The Monitors column on the Jobs page lists the job monitors that cover each job, highest priority first. A job with No monitors won't alert when it fails. The column only lists monitors you have access to.

Monitors as code

Job monitors use the job key in monitors as code. Reference the integration and jobs by name. See the Job Monitor MaC reference for every field:

montecarlo:
  job:
    - name: nightly_etl_failures
      description: Nightly ETL failures
      integration:
        type: airflow
        name: prod-airflow
      jobs:
        - name: load_orders
          group: daily_dag
      asset_selection:
        filters:
          - type: JOB_TAG
            job_tags: ["tier:gold"]
            job_tags_operator: HAS_ANY
        exclusions:
          - type: JOB_MCON
            jobs:
              - name: legacy_load
                group: daily_dag
      alert_conditions:
        - condition: etl_job_failure
      audiences:
        - data-platform-oncall

For dbt, reference jobs by project, name and an optional environment, and select projects with a DBT_PROJECT filter:

montecarlo:
  job:
    - name: analytics_model_errors
      description: Analytics dbt model errors
      integration:
        type: dbt-cloud
        name: prod-dbt
      asset_selection:
        filters:
          - type: DBT_PROJECT
            dbt_projects: [analytics]
      alert_conditions:
        - condition: dbt_model_error
        - condition: dbt_test_failure

alert_grouping, priority, notes, tags, domains, failure_audiences, data_quality_dimension and table_filters (dbt only) work the same as on other monitor types.

Permissions

See IAM resources and permissions for how permissions map to roles.

FAQs

Why didn't a job failure alert?

Check, in order:

1. Is the job covered? On the Jobs page, the Monitors column shows the monitors that cover it. No monitors means no active job monitor covers the job.
2. Is the monitor active? Paused and draft monitors cover nothing.
3. Does the failure match the alert conditions? For example, a dbt test warning only alerts on monitors with Test warning checked.
4. For dbt, does the failing table match Limit to tables?
5. Did the failure join an open alert? With grouping on, the failure is added to the existing alert rather than opening a new one.

Why are my job alerts late?

Webhook-based integrations need a working webhook for near-real-time alerts. Without one, failures are picked up when Monte Carlo next collects runs, which can delay alerts up to an hour. Open the monitor to check for a Webhook not set up or No recent webhook events warning.

Do new jobs get covered automatically?

Yes, when they match All jobs in the integration, a tag rule or a dbt project rule. A picked job is fixed, so new jobs are not covered unless a rule matches them.

Will a job covered by two monitors alert twice?

Monitors on the default grouping share one alert and notify all their audiences once. Monitors with other grouping options open their own alerts, so each one notifies its audiences.

Can I run a job monitor manually?

No. A job monitor has nothing to run. It is evaluated each time a job failure arrives.

Where do job monitors appear in data exports?

The Monitors export includes job monitors with a monitor type of Job, and the integration name in connection_name. See Data exports.