HomeStatus PageAPI ExplorerAPI ReferenceCLI ReferenceBlogRequest a Demo

Field Quality Monitors

Suggest Edits

Field Quality monitors offer an easy way to define an alert condition on a field metric like % null, % unique, etc. Unlike Field Health monitors, which rely on Machine Learning models to surface anomalies based on historical patterns, Field Quality monitors ensure any alert condition breach will immediately trigger an Incident.

These monitors are ideal for important fields that have experienced data quality issues in the past or where metric changes (ex. % unique rate decreasing below 100%) would impact downstream processes and analytical use cases.

Configuring a Field Quality monitor

Steps to create a Field Quality monitor:

  1. Visit the Catalog page for the table with the field in question
  2. Click on Create Monitor and select Field Quality OR Find the field in Field Lineage and click the (+) icon
  3. Confirm the selected field
  4. Select the field metric you want to monitor (% null, % unique, etc)
  5. Define the alert condition for the metric
  6. Add field filters to refine the metric (most users apply datetime field lookback filters here!)
  7. Edit the monitor name, labels and/or notes
  8. Save the monitor

📘

Editing Field Quality Monitors during Alpha

Field Quality Monitors will create customized SQL Rules that monitor the specified metric. To differentiate, the Field Quality Monitors will include a [Field Quality] prefix in the SQL Rule name. Note that the SQL and alert condition will be fully editable if you wish to edit the monitor.

In subsequent releases, we plan to promote Field Quality Monitors into a first-class Monitor type where the metric, alert condition and field filters are editable via a standard input form.

Supported field metrics

The following field metrics are currently supported:

MetricSupported data typesSQL used to calculate metric
% nullAll (Numeric, Text, Boolean, Datetime, Time, Array)select 1 - COUNT({{field_name}}) / CAST(COUNT(*) AS NUMERIC) as NULL_RATE from {{table_name}}
% uniqueNumeric, Text, Boolean, Datetimeselect COUNT(DISTINCT {{field_name}}) / CAST(COUNT(*) AS NUMERIC) as APPROX_DISTINCTNESS from {{table_name}}
% negativeNumericselect COUNTIF({{field_name}} < 0) / CAST(COUNT(*) AS NUMERIC) as NEGATIVE_RATE from {{table_name}}
% zeroNumericselect COUNTIF({{field_name}} = 0) / CAST(COUNT(*) AS NUMERIC) as ZERO_RATE from {{table_name}}
MeanNumericselect AVG({{field_name}}) as NUMERIC_MEAN from {{table_name}}
MaxNumericselect MAX({{field_name}}) as NUMERIC_MAX from {{table_name}}
MinNumericselect MIN({{field_name}}) as NUMERIC_MIN from {{table_name}}
Mean LengthTextselect AVG(LENGTH({{field_name}})) as MEAN_LENGTH from {{table_name}}
Max LengthTextselect MAX(LENGTH({{field_name}})) as MAX_LENGTH from {{table_name}}
Min LengthTextselect MIN(LENGTH({{field_name}})) as MIN_LENGTH from {{table_name}}

Defining alert conditions

We currently support the following operators to define your alert condition.

  • Equal to (=)
  • Less than (<)
  • Less than or equal to (<=)
  • Greater than (>)
  • Greater than or equal to (>=)
  • Not equal to (!=)

Adding field filters

🚧

Datetime field filters highly encouraged

To reduce the risk of expensive and/or timed out queries, we highly encourage applying a filter on a Datetime column to limit how many records are returned in each monitor run. Follow the steps below to add a Datetime field filter.

Field filters apply basic WHERE clauses to the underlying query to reduce which records are included in the metric calculation query. The most common application is to reduce the lookback range of the query so only recent records are returned.

Steps to apply a field filter:

  1. Click the blue (+) icon
  2. Select a table field
  3. Use the operator to specify lookback range, string filter, etc.

Updated 21 days ago