Field Quality Rules

Field Quality rules offer an easy way to define an alert condition on a field metric like % null, % unique, etc. Unlike Field Health rules, which rely on Machine Learning models to surface anomalies based on historical patterns, Field Quality rules ensure any alert condition breach will immediately trigger an Incident.

These rules are ideal for important fields that have experienced data quality issues in the past or where metric changes (ex. % unique rate decreasing below 100%) would impact downstream processes and analytical use cases.

Configuring a Field Quality rule

Steps to create a Field Quality rule:

Visit the Catalog page for the table with the field in question
Click on Create Monitor and select Field Quality OR Find the field in Field Lineage and click the (+) icon
Confirm the selected field
Select the field metric you want to monitor (% null, % unique, etc)
Define the alert condition for the metric
Add field filters to refine the metric (most users apply datetime field lookback filters here!)
Edit the rule name, audiences, and/or notes
Save the rule

Supported field metrics

The following field metrics are currently supported:

Metric	Supported data types	SQL used to calculate metric
% null	All (Numeric, Text, Boolean, Datetime, Time, Array)	`select 1 - COUNT({{field_name}}) / CAST(COUNT(*) AS NUMERIC) as NULL_RATE from {{table_name}}`
% unique	Numeric, Text, Boolean, Datetime	`select COUNT(DISTINCT {{field_name}}) / CAST(COUNT(*) AS NUMERIC) as APPROX_DISTINCTNESS from {{table_name}}`
% negative	Numeric	`select COUNTIF({{field_name}} < 0) / CAST(COUNT(*) AS NUMERIC) as NEGATIVE_RATE from {{table_name}}`
% zero	Numeric	`select COUNTIF({{field_name}} = 0) / CAST(COUNT(*) AS NUMERIC) as ZERO_RATE from {{table_name}}`
Mean	Numeric	`select AVG({{field_name}}) as NUMERIC_MEAN from {{table_name}}`
Max	Numeric	`select MAX({{field_name}}) as NUMERIC_MAX from {{table_name}}`
Min	Numeric	`select MIN({{field_name}}) as NUMERIC_MIN from {{table_name}}`
Mean Length	Text	`select AVG(LENGTH({{field_name}})) as MEAN_LENGTH from {{table_name}}`
Max Length	Text	`select MAX(LENGTH({{field_name}})) as MAX_LENGTH from {{table_name}}`
Min Length	Text	`select MIN(LENGTH({{field_name}})) as MIN_LENGTH from {{table_name}}`

Defining alert conditions

We currently support the following operators to define your alert condition.

Equal to (=)
Less than (<)
Less than or equal to (<=)
Greater than (>)
Greater than or equal to (>=)
Not equal to (!=)

Adding field filters

🚧
Datetime field filters highly encouraged
To reduce the risk of expensive and/or timed out queries, we highly encourage applying a filter on a Datetime column to limit how many records are returned in each rule run. Follow the steps below to add a Datetime field filter.

Field filters apply basic WHERE clauses to the underlying query to reduce which records are included in the metric calculation query. The most common application is to reduce the lookback range of the query so only recent records are returned.

Steps to apply a field filter:

Click the blue (+) icon
Select a table field
Use the operator to specify lookback range, string filter, etc.