Monitors as Code
Overview
Monte Carlo developed a YAML-based monitors configuration to help teams deploy monitors as part of their CI/CD process. The following guide explains how to get started with monitors as code.
Prerequisites
- Install the CLI — https://docs.getmontecarlo.com/docs/using-the-cli
- When running
montecarlo configure
, provide your API key and you may leave the AWS settings blank
Integrating into your CI/CD pipeline
Deploying monitors within a continuous integration pipeline is straightforward. Once changes are merged into your main production branch, configure your CI pipeline to install the montecarlodata
CLI:
pip install montecarlodata
And run this command:
MCD_DEFAULT_API_ID=${MCD_DEFAULT_API_ID} \
MCD_DEFAULT_API_TOKEN=${MCD_DEFAULT_API_TOKEN} \
montecarlo monitors apply \
--namespace ${MC_MONITORS_NAMESPACE} \
--project-dir ${PROJECT_DIR}
These environment variables need to be populated:
MCD_DEFAULT_API_ID
: Your Monte Carlo API IDMCD_DEFAULT_API_TOKEN
: Your Monte Carlo API TokenMC_MONITORS_NAMESPACE
: Namespace to apply monitor configuration toPROJECT_DIR
: If themontecarlo
command is not run within your Monte Carlo project directory, you can specify it here. Optional.
Namespaces
You can think of namespaces like Cloudformation stack names. It’s a logical separation of a collection of resources that you can define. Monitors from different namespaces are isolated from each other.
Some examples of why this is useful -
- You have multiple people (or teams) working on managing monitors and don’t want to conflict or override configurations.
- You want to manage different groups monitors in different pipelines (e.g. dbt models in CI/CD x & non-dbt models in CI/CD y).
Using code to define monitors
First, you will need to create a Monte Carlo project. A Monte Carlo project is simply a directory which contains a montecarlo.yml
file, which contains project-level configuration options. If you are using DBT, we recommend placing montecarlo.yml
in the same directory as dbt_project.yml
.
The montecarlo.yml
format:
version: 1
default_resource: <string>
include_file_patterns:
- <string>
exclude_file_patterns:
- <string>
Description of options:
version
: The version of MC configuration. Set to1
default_resource
: The warehouse friendly name or UUID where YAML-defined monitors will be created.- If your account only has a single warehouse configured, MC will use this warehouse by default, and this option does not need to be defined.
- If you have multiple warehouses configured, you will need to either (1) define
default_resource
, or (2) specify the warehouse friendly name or UUID for each monitor explicitly in theresource
property (see YAML format for configuring monitors below).
include_file_patterns
: List of file patterns to include when searching for monitor configuration files. By default, this is set to**/*.yaml
and**/*.yml
. With these defaults, MC will search recursively for all directories nested within the project directory for any files with ayaml
oryml
extension.exclude_file_patterns
: List of file patterns to exclude when searching for monitor configuration files.
Example montecarlo.yml
configuration file, which should be sufficient for customers with a single warehouse:
version: 1
Example montecarlo.yml
configuration file, for customers with multiple warehouses configured.
version: 1
default_resource: bigquery
Defining individual monitors
Monitors are defined in YAML files within directories nested within the project. Monitors can be configured in standalone YAML files, or embedded within DBT schema.yml
files within the meta
property of a DBT model definition.
Standalone monitor YAML files can be contained in any .yml file in any directory nested within the project. Example:
montecarlo:
field_health:
- table: project:dataset.table_name
timestamp_field: created
dimension_tracking:
- table: project:dataset.table_name
timestamp_field: created
field: order_status
Example of monitor embedded within a DBT schema.yml
file:
version: 2
models:
- name: table_name
description: My table
meta:
montecarlo:
field_health:
- table: project:dataset.table_name
timestamp_field: created
dimension_tracking:
- table: project:dataset.table_name
timestamp_field: created
field: order_status
Tip: Using monitors as code with DBT
You may find that embedding monitor configurations within DBT
schema.yml
may make maintenance easier, as all configuration/metadata concerning a given table are maintained in the same location.
Monitor configuration reference
montecarlo:
field_health:
- table: <string> # required
fields:
- <string>
timestamp_field: <string>
timestamp_field_expression: <string>
where_condition: <string>
lookback_days: <int>
aggregation_time_interval: <one of 'day' or 'hour'>
schedule: # optional -- by default, loose schedule with interval_minutes=720 (12h)
type: <loose, fixed, or dynamic> # required
interval_minutes: <integer> # required if loose or fixed
start_time: <date as isoformatted string> # required if fixed
dimension_tracking:
- table: <string> # required
field: <string> # required
timestamp_field: <string>
timestamp_field_expression: <string>
where_condition: <string>
lookback_days: <int>
aggregation_time_interval: <one of 'day' or 'hour'>
schedule: # optional -- by default, loose schedule with interval_minutes=720 (12h)
type: <loose, fixed, or dynamic> # required
interval_minutes: <integer> # required if loose or fixed
start_time: <date as isoformatted string> # required if fixed
json_schema:
- table: <string> # required
field: <string> # required
timestamp_field: <string>
timestamp_field_expression: <string>
where_condition: <string>
schedule: # optional -- by default, loose schedule with interval_minutes=720 (12h)
type: <loose, fixed, or dynamic> # required
interval_minutes: <integer> # required if loose or fixed
start_time: <date as isoformatted string> # required if fixed
custom_sql:
- sql: <string> # required
comparisons: <comparison> # required
description: <string>
notes: <string>
schedule:
type: fixed # must be fixed
start_time: <date as isoformatted string>
interval_minutes: <integer>
freshness:
- table: <string> # required
freshness_threshold: <integer> # required
description: <string>
schedule:
type: fixed # must be fixed
start_time: <date as isoformatted string>
interval_minutes: <integer>
volume:
- table: <string> # required
comparisons: <comparison> # required
volume_metric: <row_count or byte_count> # row_count by default
description: <string>
schedule:
type: fixed # must be fixed
start_time: <date as isoformatted string>
interval_minutes: <integer>
Lookback Limits
Where we allow you to specify a longer lookback period on some monitors (in case the data in your table has historical timestamps), you cannot pick a number larger than 7. This is because for each day we "lookback", an additional query against your table is run. This is a safeguard to prevent specifying a very large period, like 90 days, and then having 90 queries run against your warehouse each time the monitor runs. If you need help with these windows, please feel free to reach out to [email protected] or the chat bot in the lower right hand corner.
field_health
Configures a field health monitor
table
: MC global table ID (format<database>:<schema>.<table name>
fields
: List of fields in table to monitor. Optional — by default all fields are monitoredtimestamp_field
: Timestamp fieldtimestamp_field_expression
: Arbitrary SQL expression to be used as timestamp field, e.g.DATE(created)
. Must use eithertimestamp_field
ortimestamp_field_expression
or neither.where_condition
: SQL snippet of where condition to add to field health querylookback_days
: Lookback period in days. Default: 3aggregation_time_interval
: Aggregation bucket time interval, eitherhour
(default) orday
schedule
type
: One ofloose
,fixed
, ordynamic
interval_minutes
: For loose or fixed, how frequently to run the monitorstart_time
: For fixed, when to start the schedule
dimension_tracking
Configures a dimension tracking monitor
table
: MC global table ID (format<database>:<schema>.<table name>
field
: Field in table to monitortimestamp_field
: Timestamp fieldtimestamp_field_expression
: Arbitrary SQL expression to be used as timestamp field, e.g.DATE(created)
. Must use eithertimestamp_field
ortimestamp_field_expression
or neither.where_condition
: SQL snippet of where condition to add to field health querylookback_days
: Lookback period in days. Default: 3aggregation_time_interval
: Aggregation bucket time interval, eitherhour
(default) orday
schedule
type
: One ofloose
,fixed
, ordynamic
interval_minutes
: For loose or fixed, how frequently to run the monitorstart_time
: For fixed, when to start the schedule
json_schema
Configures a JSON schema monitor
table
: MC global table ID (format<database>:<schema>.<table name>
field
: Field in table to monitortimestamp_field
: Timestamp fieldtimestamp_field_expression
: Arbitrary SQL expression to be used as timestamp field, e.g.DATE(created)
. Must use eithertimestamp_field
ortimestamp_field_expression
or neither.where_condition
: SQL snippet of where condition to add to field health queryschedule
type
: One ofloose
,fixed
, ordynamic
interval_minutes
: For loose or fixed, how frequently to run the monitorstart_time
: For fixed, when to start the schedule
custom_sql
sql
: SQL of rulecomparisons
: See comparisons belowdescription
: Friendly description of rulenotes
: Additional context for the ruleschedule
type
: Can befixed
ormanual
interval_minutes
: How frequently to run the monitor. Required forfixed
.start_time
: When to start the schedule. Required forfixed
.
freshness
table
: MC global table ID (format<database>:<schema>.<table name>
freshness_threshold
: Freshness breach threshold in minutesdescription
: Friendly description of ruleschedule
type
: Must befixed
interval_minutes
: How frequently to run the monitorstart_time
: When to start the schedule
volume
table
: MC global table ID (format<database>:<schema>.<table name>
volume_metric
: Must berow_count
orbyte_count
— defines which volume metric to monitorcomparisons
: See comparisons belowdescription
: Friendly description of ruleschedule
type
: Must be "fixed"interval_minutes
: How frequently to run the monitorstart_time
: When to start the schedule
comparisons
comparisons
are definitions of breaches, not expected return values. This section would be where you would define the logic for when to get alerted about anomalous behavior in your monitor. For example, if you make a custom SQL rule and pick:
type
:threshold
operator
:GT
threshold_value
: 100When Monte Carlo runs your monitor and the return results are greater than 100, we will fire an alert to any routes configured to be notified about breaches to this monitor.
type
:threshold
orchange
. Ifthreshold
,threshold_value
below is an absolute value. Ifchange
,threshold_value
as change from the historical baselineoperator
: One ofEQ
,NEQ
,GT
,GTE
,LT
,LTE
. Operator of comparison, =, ≠, >, ≥, <, ≤ respectively.threshold_value
: Threshold valuebaseline_agg_function
: If type =change
, the aggregation function used to aggregate data points to calculate historical baselinebaseline_interval_minutes
: If type =change
, the time interval in minutes (backwards from current time) to aggregate over to calculate historical baselineis_threshold_relative
: If type =change
, whether or notthreshold_value
is a relative vs absolute threshold.is_threshold_relative: true
would be a percentage measurement,is_threshold_relative: false
would be a numerical measurement. Relative means the threshold_value will be treated as a percentage value, Absolute means the threshold_value will be treated as an actual count of rows.
Example
montecarlo:
field_health:
- table: project:dataset.table_name
timestamp_field: created
schedule:
type: dynamic
dimension_tracking:
- table: project:dataset.table_name
timestamp_field: created
field: order_status
custom_sql:
- description: Test rule
sql: |
select foo from project.dataset.my_table
comparisons:
- type: threshold
operator: GT
threshold_value: 0
schedule:
type: fixed
interval_minutes: 60
start_time: "2021-07-27T19:00:00"
freshness:
- table: project:dataset.table_name
freshness_threshold: 30
schedule:
type: fixed
interval_minutes: 30
start_time: "2021-07-27T19:00:00"
Developing and testing locally
To apply monitor configuration to MC:
montecarlo monitors apply --namespace <namespace>
Monitors configured using the CLI are organized under namespaces. All apply
operations are scoped to a namespace. Namespaces make it easier to organize and manage monitors as code.
The apply
command behaves as follows:
- MC will search for all monitor configuration elements in the project, both in standalone and embedded in DBT schema files. All monitor configuration elements will be concatenated into a single configuration template.
- MC will apply the configuration template to your MC account:
- Any new monitors defined since last apply will be created
- Any previously defined monitors present in current configuration template will be updated if any attributes have changed
- Any previously defined monitors absent from current configuration template will be deleted
The apply
command also supports a --dry-run
argument which will dry run the configuration update and report each operation.
To delete (destroy) a namespace:
montecarlo monitors delete --namespace <namespace>
This will delete all monitors for a given namespace.
Updated 1 day ago