Pipeline Observability

Monte Carlo makes it easy to instantly deploy broad monitoring of your data pipelines, with no manual configuration. It learns the normal patterns of updates, size changes, and size growth, and alerts when those patterns are violated. This is ideal for detecting breakages and stoppages in the flow of data.

For regularly updated tables, Monte Carlo can typically generate thresholds after 7 days of observing a table, and thresholds will be mature by 14 days. At most, monitors will incorporate a 6-week rolling window of training data to determine thresholds, including handling of weekly seasonality.

The sensitivity of thresholds can be adjusted between High, Medium, and Low. Users may also choose a manual threshold, instead of relying on thresholds generated by machine learning. Learn more about sensitivity settings.

For most integrations, these monitors start working immediately without user configuration and use metadata instead of directly querying the table. Types of Pipeline Observability monitors include:

• Freshness: alerts to unusually long periods of time between updates
• Volume: alerts to unusual changes in size, and unusually long periods of time between size changes
• Schema change: alerts to changes in the schema of the table (no machine learning involved here)