Freshness

Use cases

Freshness monitors can easily address two use cases:

• Alert if a table has gone an unusually long without an update. Automatic thresholds are ideal for this.
• Alert if a table must be updated by a particular time. For example, if executives expect their dashboards to be updated by 8:00AM, and the jobs that update those dashboards run once/day at 7:40AM, then set a check at 7:55AM. This context is highly business-specific and so manual thresholds are recommended.

Interacting with freshness monitors

Freshness monitors are managed from Assets. From the Freshness section on the Assets > Summary page, users can see the current threshold, adjust the sensitivity, switch to a manual threshold, or disable freshness monitoring from the Freshness widget.

Click View Details and then Edit to adjust the alert threshold

Users can select between automated or explicit (manual) thresholds, or choose to disable freshness monitoring. For automated thresholds, users can select between low, medium, and high sensitivity. Learn more about how the machine learning for freshness works here.

For explicit thresholds, users must indicate an alert condition and how often the check should run.

Common configurations for explicit thresholds

Example 1:

Executives expect their dashboards to be updated by 8:00AM, and the jobs that update those dashboards run once/day at 7:40AM.

Suggested configuration:

• Notify when the table has not updated within the last: 1 hour
• Check every 1 day, starting at [today] 7:55AM

This will check every day at 7:55AM that the 7:40AM job has completed before your internal deadline of 8AM. If the 7:40AM job had not yet completed, an Incident would be created and a notification would be sent.

Example 2:

A table that contains events from your web app updates irregularly. Sometimes it updates hourly, but there are also frequent gaps of 4-6 hours. You know it is a problem if that table ever goes more than 6 hours without an update, but the machine learning is a bit too desensitized.

Suggested configuration:

• Notify when the table has not updated within the last: 6 hours
• Check every 1 hour, starting at [today] 00:00AM

This will check every 1 hour that the table has received an update in the trailing 6 hours. If the table ever goes more than 6 hours without an update, an Incident would be created and a notification would be sent.

Notifications

For purposes of routing notifications, the incidents from Freshness monitors are freshness anomalies.

Freshness rules

Up through February 2024, users had the ability to create Freshness Rules. These were standalone rules that set explicit Freshness expectations for a table, completely separated from the freshness monitor described above.

Freshness Rules will eventually be deprecated, in favor of managing Freshness in a single place in Assets. However, many customers still have full-functioning Freshness Rules configured on their Monitors page, and some customers can still create Freshness Rules through the UI and Monitors as Code.