❗️

Starting March 1, 2025, new monitors can no longer be created using the field_health property.

We strongly recommend using the more up to datemetric property. However, users can still deploy the legacy version of this monitor under the field_health property until March 1, 2025.

Monitors created with the field_health property do not receive any new improvements.

These are the parameters currently supported by field_health monitor definitions:

Parameters

• table: MC global table ID (format <database>:<schema>.<table name>
• alert_conditions: List of configurations for metrics, fields, and breach conditions. A field may only be included once in a monitor for a given metric. See section below for details.
• segmented_expressions: List of fields or SQL expressions used to segment the field (must have exactly one field in fields). Enables Monitoring by Dimension.
• segmentation_type: field or custom_sql_expression. Defaults to field.
  • field supports up to 2 fields. The fields must exist in the table.
  • custom_sql_expression supports 1 custom SQL expression.
• timestamp_field: Timestamp field
• timestamp_field_expression: Arbitrary SQL expression to be used as timestamp field, e.g. DATE(created). Must use either timestamp_field or timestamp_field_expression or neither.
• where_condition: SQL snippet of where condition monitor's query
• lookback_days: Lookback period in days. Default: 3. This option is optional and will be ignored unless timestamp_field is set.
• aggregation_time_interval: Aggregation bucket time interval, either hour (default) or day. This option is optional and will be ignored unless timestamp_field is set.
• min_segment_size: Minimum number of rows for a segment to be fetched. Defaults to 1. Can be used to avoid cardinality limits for datasets with a long tail of less-relevant segments.
• sensitivity_level: Set sensitivity level to One of high, medium or low.
• schedule
  • type: One of fixed or dynamic
  • interval_minutes: For fixed, how frequently to run the monitor
  • start_time: When to start the schedule. If unspecified, for fixed schedules, then start sometime within the next hour.
  • timezone: Optional - select regional timezone for daylight savings ex. America/Los_Angeles
• connection_name: Specify the connection (also known as query-engine) to use. Obtain the warehouse UUID via the getUser API as described here. Then obtain names of the connections in the warehouse via the getWarehouseConnections API as described here. Use ["sql_query"] as the jobType parameter in the getWarehouseConnections API call.

Each Metric Monitor supports up to 300 monitored fields (or custom metrics).

Metric monitor alert conditions

• operator: accepts AUTO, GT, GTE, LT, LTE, EQ, NEQ, OUTSIDE_RANGE, INSIDE_RANGE. AUTO uses ML to detect breaches. All other operators require an explicit threshold.
• fields: List of field names that should be monitored for this metric, operator, and threshold. This field is optional and defaults to all fields in the table. The field type must be compatible with the metric.
  • Each field may only be included once in the monitor for a given metric (including as part of an "all fields" metric).
  • When using a custom metric, fields should not be included; the fields used should be included directly in the SQL expression.
• threshold_value: Breach threshold for the operator. Not used for AUTO, OUTSIDE_RANGE, or INSIDE_RANGE. For rate metrics, ensure the threshold is between 0 and 1 (not 0%-100%).
• lower_threshold: Lower end of threshold range for operators OUTSIDE_RANGE and INSIDE_RANGE
• upper_threshold: Upper end of threshold range for operators OUTSIDE_RANGE and INSIDE_RANGE
• metric: See the list of available metrics for full details, including whether AUTO is supported for each metric and which field types are compatible. All operators besides AUTO are compatible with every metric. Use the value from the MaC Name column in this field.
• custom_metric: Allows defining a custom metric based on a SQL expression instead of a predefined metric indicated in metric. It can use any operator except AUTO.

Converting field_health property to metric property

Most of the parameters in the old format have the same representation in the new one, but under different names:

Converting using the CLI

You can use the export-as-latest command from the CLI to export the current field_metric monitors using the new metric definition. Provide the namespace of the monitors you want to export using metric, and the command will return all legacy monitors converted to use the new metric definitions.

Bare in mind that these will be new monitors that, once created, will need to be retrained. There's no way to preserve the history and training of current field_health monitors when converting. However, this training period is rapidly accelerated by backfills, which happen automatically when the metric monitor is created.

Legacy monitors and new monitors behave differently with respect to the monitored data time window when using a timestamp field or expression. Legacy monitors will look at all data since the time of execution and the 'lookback_days' parameter, while new monitors will look at all data since the last collection, ignoring up to the 'collection_lag' hours. These differences might require you to tweak the converted collection_lag parameter if you know data arrives with some significant delay to your warehouse with respect to the monitor's scheduled run time.

The legacy monitors also have some different restrictions compared to new style monitors:

• Legacy monitors cannot have more than 1,000 segments when monitoring more than one metric or field
• New monitors cannot monitor more than 10,000 time series, with each time series being defined by the combination of segment, metric and field.

These differences might cause some old style monitors to be rejected when specified in the new format. For example, an 'all fields, all metrics' legacy monitor might easily exceed the 10,000 time series limit for tables with more than 300 fields. You will need to break these apart if you want to preserve the same coverage, or narrow down your metric and field selection.

Converting using 'clone'

You can use the clone button on the UI to clone an existing legacy monitor. The cloned monitor will be automatically converted to a new style monitor. Using the 'Generate YAML' button at the bottom of the create monitor screen, you can export the definition using the new metric property.