Field Health Monitors

When applied to a table or view, Field Health Monitors automatically begin tracking and reporting on anomalies across the metrics listed below. These monitors are ideal for important tables within your organization where null or non-distinct field values would result in a data quality issue. By tracking these statistics over time, we’re able to intelligently notify you when anomalies occur. Note that you need to configure Field Health monitoring on tables and views.

Configuring a field health monitor

Configuring a field health monitor requires a few simple steps including:

  • select a table
  • select 1 or more fields in selected table to monitor
  • specify a row creation time field to minimize the records analyzed on each run
  • specify the schedule for when the check should run

Once enabled for field(s) in a table, the Field Health monitor will run on the specified schedule regularly checking for anomalies across many metrics calculated on the record values for the field(s).

Advanced Field Health monitor options

Multiple advanced options are available to configure Field Health monitors to check for specific types of anomalies. Advanced options include

Monitoring by dimension
This option allows you to run field health monitors on the values of a field (i.e. contract_amount) segmented by another field value (i.e. account_type). This structure is ideal for situations where a segment of records may have unique patterns which the Field Health monitor can observe. In the example above, account_type=enterprise will likely have very different metrics in the contract_amount field which would be observed with this feature.

You can enable this option in the advanced settings of step 2 (select fields) in the Field Health Monitor creation flow.

Table Record filter
This option allows you to filter the records which the Field Health monitor looks at to train the model and monitor for changes. Similar to monitoring by dimension, this feature can be used to exclude or include specific records that you want the Field Health model to review.

You can enable this option in the advanced settings of step 2 (select fields) in the Field Health Monitor creation flow.

Aggregation window
This option allows you to define how records are aggregated when calculating the Field Health metrics. A daily aggregation window is ideal for tables that have sparse records being added daily.

You can configure the aggregation window in the advanced settings of step 3 (select schedule) in the Field Health Monitor creation flow.

Adjusting Field Health Metric Sensitivity
Once a Field Health monitor has been setup, you have the option to adjust the sensitivity for a given Field Health monitor. To do so, you need to navigate to a given monitors page and within the summary view on the Monitor Detail tab you will see Metric Sensitivity. The settings options are low, medium, and high with medium being the default.

Training the Field Health Model

Field Health monitors leverage ML to detect anomalies across many metrics, and thus require a training period to train the models and set a baseline. The length of the training period depends on the number of freshness events the model has to review. If a table does not receive many updates, it will take longer for the models to train. In most cases, this training period lasts less than 14 days.

Field health monitor metrics

The following list includes all metrics automatically tracked with Field Monitors. While trend reporting is available for all metrics, some metrics are not yet configured to notify you of anomalies. To enable alerting on an excluded metric please contact us using the Intercom bot or at [email protected].

MetricDefinitionSupported field typesAnalyticsAnomaly detection
% NullPercentage of rows that have a NULL valueAllYesYes
% UniqueNumber of distinct values divided by total number of valuesAllYesYes
% ZeroPercentage of rows that have the value 0NumericYesYes
% NegativePercentage of rows that have a negative valueNumericYesYes
MeanMean value across all rowsNumericYesYes
MinMinimum across all rowsNumericYesYes
p2020th percentileNumericYesYes
p4040th percentileNumericYesYes
p6060th percentileNumericYesYes
p8080th percentileNumericYesYes
MaxMaximum across all rowsNumericYesYes
StdStandard deviation of valuesNumericYesCurrently not supported
Min lengthMinimum count of characters over non-null valuesStringsYesCurrently not supported
Mean lengthMean count of characters over non-null valuesStringsYesCurrently not supported
Std lengthStandard deviation of the count over non-null valuesStringsYesCurrently not supported
Max lengthMaximum count of characters over non-null valuesStringsYesCurrently not supported
% Whitespace onlyPercentage of rows where the string contains only whitespace characters, e.g. \n, \t, spaceStringsYesYes
% IntegerPercentage of rows where the string represents a valid integer numberStringsYesYes
% "Null"/"None"Percentage of rows where the string is equal to 'null', 'None' or similar tokensStringsYesYes
% FloatPercentage of rows where the string represents a valid floating point numberStringsYesYes
% UUIDPercentage of rows where the string represents a valid UUID, e.g. '256bc0d3-dedd-4c3f-b6f8-aced33238bc6'StringsYesYes

Volume anomalies notifications from Field Health Monitors

Field Health Monitors track the row count in your table during each execution. If an unusual volume is detected, a 'Volume Anomaly' incident is generated. Note that these incidents are sent only to notification channels designated for Automated Monitors or specifically set to receive 'Volume Anomalies.' They are not sent to channels configured for the Field Health Monitor audience. We are actively working to resolve this limitation.