Most production model failures start as data failures: an input feature's distribution shifts, a categorical column picks up a value the model was never trained on, or an upstream join starts dropping rows.

ML metrics detect those conditions on the tables your models already read from and write to. If your predictions land in the warehouse or lakehouse — a scoring table, a feature table, an inference log — you can monitor prediction accuracy, distribution drift, and input data quality without instrumenting the model or installing an SDK.

ML metrics are one family of available metrics supported by Metric Monitors, so everything that applies to metric monitors applies here: table, view, or custom SQL sources, WHERE filters, segmentation, backfill, automated or manual thresholds, and Track mode. They are available on any connection that supports metric monitors.

ML metric types

For metric descriptions, supported column types, and MaC names, see ML Metrics.

Performance metrics require actual outcomes, which arrive after the prediction — days for a delivery estimate, months for a credit decision. Drift and cardinality metrics require no labels and run as soon as data lands. Most setups use both: drift and cardinality on input features and the prediction column, and performance metrics once labels are joined.

What to monitor

ML metrics run against whichever of a model's tables you materialize:

A metric monitor runs against any table, view, or custom SQL query, so the names describe a common layout rather than a required one. Where features, predictions, and labels live in separate tables, join them in a view or in the monitor's custom SQL.

ML Regression Performance

Tracks prediction accuracy on continuous outputs, such as forecasted versus actual revenue.

Requirements

• The predicted value and the actual value must be two numeric columns in the same row. If they live in separate tables, join them in a view or in the monitor's custom SQL.
• Select the predicted column first and the actual column second. The order affects MAPE, R-Squared, and the sign of Mean Error.

Choosing a metric

Start with MAE — it is the most interpretable and the most robust to outliers. Use RMSE when one large error matters more than many small ones; it is noisier, so expect wider automated thresholds. MAPE is comparable across models but unstable when actual values approach zero. R-Squared responds to changes in the variance of your data as well as to model accuracy, which makes it weak for alerting.

Mean Error preserves the sign of the error, so it detects systematic bias that MAE and RMSE hide. Pair the two: MAE shows that error grew, Mean Error shows in which direction.

ML Classification Performance

Tracks how often the model predicts the correct category or label.

Requirements

• The predicted label and the actual label must be two columns in the same row.
• Both columns must be the same data type, and must be string, boolean, or integer. Floating-point columns are not supported — use regression metrics for continuous values.

⚠️

Accuracy is misleading on imbalanced classes

If 2% of transactions are fraudulent, a model that always predicts "not fraud" is 98% accurate. For binary labels, segment by the actual label to get recall within each true class, or by the predicted label to get precision. Adding a drift metric on the prediction column also catches a model that has collapsed to a single class, regardless of what accuracy reports.

On a table where labels arrive late, filter out unlabeled rows. Rows where both columns are null count as a match and inflate accuracy.

ML Drift Detection

Detects when a column's distribution shifts over time — upstream data changes in model inputs, or concept drift in model outputs.

Requirements

• The monitor must be grouped by a time field under Group data, since drift compares a recent period against a baseline.
• Supported column types are text, boolean, and numeric. Timestamp, date, array, and struct columns are not supported.

Choosing a metric

PSI works on any supported column type and is a reasonable default. Scores are conventionally read as below 0.1 no meaningful shift, 0.1 to 0.25 moderate, above 0.25 significant — an industry convention.

KS Test is numeric only, and is most sensitive to shifts in the body of a distribution rather than in the tails.

JS Divergence is symmetric and bounded, so scores stay comparable across columns and stable as categories appear or disappear. This suits high-cardinality categorical fields. Scores range from 0 for identical distributions to roughly 0.83 for completely disjoint ones — set manual thresholds against that range.

What to monitor

Point drift metrics at input features (where upstream breakage appears first), the prediction column, and any confidence or probability score column.

ML Cardinality

Tracks changes in the set of unique values in a column over time.

Requirements

• The monitor must be grouped by a time field under Group data, since cardinality compares a recent period against a baseline.
• Supported column types are text, boolean, and numeric. Timestamp, date, array, and struct columns are not supported.

Choosing a metric

NEW_VALUES above 0 on an encoded categorical feature means production contains a value the model has no encoding for; alerting on the first occurrence is usually appropriate. MISSING_VALUES means a category stopped appearing — sometimes a business change, more often a changed upstream filter or a join dropping rows.

📘

Cardinality vs. drift on the same column

Drift detects a change in the proportions of values. Cardinality detects a change in the set of values. A feature can drift substantially with no new values, or gain a rare new value with almost no measurable drift. On a critical categorical feature, monitor both.

Baselines

Drift and cardinality metrics compare recent data against either a rolling window (default: 14 days) or a fixed date range, configured in the alert condition builder.

A rolling window answers "did something change recently?" Its baseline moves with the data, so gradual drift will not trip it. A fixed date range pinned to the training window or the last deployment answers "how far has production data moved from what the model was trained on?" — the signal that indicates retraining.

Use hourly or daily buckets if you need a drift value per bucket. At weekly or monthly buckets, and with any fixed baseline, the monitor produces one value per run.

⚠️

Changing the baseline does not reset a trained threshold

If you change the baseline window or bin count on an existing condition, an automated threshold keeps learning across the change even though the metric now means something different. Add a new alert condition instead of editing one in place.

Segmentation

Aggregate metrics dilute cohort-level failure: a model that has stopped working for one region may barely move overall accuracy. Metric monitors support up to 5 segment fields, subject to the same segment limits as any other metric monitor. See Segmentation.

• Model version turns any ML metric monitor into a deployment guardrail.
• Customer tier, region, channel, device — where cohort-level failure is otherwise hidden.
• Actual label decomposes classification accuracy into per-class performance.
• Use Minimum segment size to keep small, noisy segments from alerting. They are still collected and visible in the monitor.

Example: churn model

Predictions land in ml.churn_scores daily; actual labels arrive 30 days later. Segment all of the below by model_version.

Available immediately, without labels:

• PSI on churn_probability, baseline fixed to the training window.

• PSI on the top input features, rolling 14-day window.

• NEW_VALUES on each encoded categorical feature, manual threshold of 0.
  Available once labels arrive:

• ACCURACY on predicted_label versus actual_churned, segmented by actual_churned.

As code

montecarlo:
  metric:
    - name: prediction_score_psi_drift
      description: PSI drift on prediction score vs a fixed baseline
      data_source:
        table: analytics:prod.ml.predictions
      aggregate_time_field: event_time
      aggregate_by: day
      alert_conditions:
        - metric: PSI
          fields: [score]
          operator: GT
          threshold_value: 0.25
          baseline_start: "2025-01-01T00:00:00+00:00"
          baseline_end: "2025-02-01T00:00:00+00:00"
          num_bins: 20
        - metric: ACCURACY
          fields: [predicted_label, actual_label]   # predicted first
          operator: AUTO_LOW

Use baseline_trailing_days for a rolling baseline, or baseline_start and baseline_end for a fixed one. The two are mutually exclusive. Metric names must be uppercase.

ML Metrics FAQs

📘

To configure metric monitors as code, see the Metric Monitor MaC reference.

Do I need to instrument my model or install an SDK?

No. ML metrics read the tables your pipeline already writes: point a metric monitor at a feature or inference table, or at a view or custom SQL query over them, in an existing warehouse connection.

What if I don't know what threshold to set?

Add the metric in Track mode, let it collect data, then edit the monitor to promote it to an alerting condition. Use manual absolute thresholds where an SLA or convention already exists, such as accuracy above 95% or no new values at all.

Which metrics should I start with?

PSI on the prediction column and NEW_VALUES with a manual threshold of 0 on each encoded categorical input. Neither requires ground truth. Add performance metrics once predictions and actuals are reliably joined.

Can I use ML metrics in a multi-table monitor?

Drift and cardinality metrics, yes. Performance metrics, no — multi-table monitors select fields by pattern and cannot express a predicted/actual pair.