Freshness

Use cases

There are two types of Freshness monitors available, Time since last update and Time since last row count change. These can be used to address use cases like:

• Alert if a table has gone an unusually long without an update or row count change. Automatic thresholds are ideal for this.
• Alert if a table must be refreshed by a particular time. For example, if executives expect their dashboards to be updated by 8:00AM, and the jobs that update those dashboards run once/day at 7:40AM, then set a check at 7:55AM. This context is highly business-specific and so manual thresholds are recommended.

Interacting with freshness monitors

Freshness monitors are managed from Assets. From the Freshness sections on the Assets > Summary page, users can see the current threshold, adjust the sensitivity, switch to a manual threshold, or disable freshness monitoring from the Freshness widget.

Click View Details and then Edit to adjust the alert threshold

Users can select between automated or explicit (manual) thresholds, or choose to disable freshness monitoring. For automated thresholds, users can select between low, medium, and high sensitivity.

For explicit thresholds, users must indicate an alert condition and how often the check should run.

Configurations for Explicit thresholds

There are three configurations for explicit thresholds. Periodically, On a schedule, and CRON.

Periodically

The monitor automatically determines the check schedule to ensure updates occur at least as often as the threshold specifies (e.g., "at least every 6 hours"), running checks at a frequency that avoids missing breaches without requiring the user to set exact times.

Example 1:

A table that contains events from your web app updates irregularly. Sometimes it updates hourly, but there are also frequent gaps of 4-6 hours. You know it is a problem if that table ever goes more than 6 hours without an update, but the machine learning is a bit too desensitized.

Suggested configuration:

• Check that the table is updated Periodically. At least once every six hours,

This will check every 3 hours that the table has received an update in the trailing 6 hours. If the table ever goes more than 6 hours without an update, an Incident would be created and a notification would be sent.

On A Schedule

A fixed interval and start time (e.g., every day at 7:55AM), are specified so the monitor runs at those exact times.

Example 2:

Executives expect their dashboards to be updated by 8:00AM, and the jobs that update those dashboards run once/day at 7:40AM.

Suggested configuration:

• Notify when the table has not updated within the last: 1 hour
• Check every 1 day, starting at [today] 7:55AM

This will check every day at 7:55AM that the 7:40AM job has completed before your internal deadline of 8AM. If the 7:40AM job had not yet completed, an Incident would be created and a notification would be sent.

CRON

Allows you to specify when the monitor runs like the "On a schedule" setting, but allows for advanced, custom scheduling using CRON expressions (e.g., "0 10 13 * *" for 10:00 on the 13th of each month), giving precise control over when the monitor runs.

Notifications

For purposes of routing notifications, the incidents from Freshness monitors are freshness anomalies.

To configure notifications for freshness anomalies:

• Go to Settings > Notifications
• Create or find the audience you'd like to send notifications to, and click the (+) in the Other notifications column
• Under Alert types, select freshness anomalies
• Filter for the desired data using the Affected data and Table importance sections

Freshness rules

Up through February 2024, users had the ability to create Freshness Rules. These were standalone rules that set explicit Freshness expectations for a table, completely separated from the freshness monitor described above.

Freshness Rules will eventually be deprecated, in favor of managing Freshness in a single place in Assets. However, many customers still have full-functioning Freshness Rules configured on their Monitors page, and some customers can still create Freshness Rules through the UI and Monitors as Code.