MotherDuck (beta)
What is MotherDuck?
MotherDuck is a cloud-based data analytics platform that leverages the power of the DuckDB engine for performing fast and efficient analytical queries on large datasets. DuckDB is an in-process SQL OLAP database management system that is designed to be lightweight and fast, making it suitable for data science, machine learning, and other analytical workloads.
Why Connect MotherDuck to Monte Carlo?
Integrating Monte Carlo with Motherduck allows you to monitor your Duckdb databases through custom SQL monitors, which can be created in either the UI wizard and/or programmatically via monitors as code (API/SDK too). These monitors can be used to generate notifications (incidents) to relevant stakeholders and circuit break pipelines. For a full list of supported monitor types see Monitor Support.
Monitor Support
Below are the supported monitors for the Motherduck integration today. Please reach out to your Monte Carlo representative if you have requirements for additional Monitors.
| Category | Monitor Type | Support |
|---|---|---|
| Automated Anomaly Detection | Freshness | |
| Automated Anomaly Detection | Volume | |
| Automated Monitoring | Schema Changes | ✅ |
| Opt-in Monitor | SQL Rule | ✅ |
| Opt-in Monitor | Comparison Rule | ✅ |
| Opt-in Monitor | Field Quality Rule | |
| Opt-in Monitor | Freshness Rule | |
| Opt-in Monitor | Volume Rule | |
| Opt-in Monitor | Volume Anomaly Detection | |
| Opt-in Monitor | Field Health Monitor | |
| Opt-in Monitor | Dimension Tracking | |
| Opt-in Monitor | JSON Schema Changes |
Connecting to MotherDuck
To connect Monte Carlo to your Motherduck database, follow these steps:
- Login to your Motherduck account and navigate to the settings page.
- Under the General section use the 'Copy token' button to copy the service account token. This will be needed in step 4.
- In your Monte Carlo workspace navigate to the Settings/Integrations page. Under the "Data lakes and warehouses" section use the "Create" drop down to select Motherduck.
- Fill in all required fields, using the token from step 2 for the Token field.
Memory Limit
If the validations fail with an internal server error it is most likely caused by a memory limit being exceeded in the connector to Motherduck. Please contact us at [email protected] for assistance.
FAQs
What is the default database?
This is the default database that will be used by the Monte Carlo connection. In your custom SQL monitors you can query any table in this database by just specifying the table name. You will still be able to set up custom SQL monitors on tables in other databases but will need to specify the database and table name in your query.
Do I need to provide a default database name that already exists?
If you provide a default database name that does not already exist in Motherduck, that database will be created when Monte Carlo connects to Motherduck. This is a feature of Motherduck's python connector called out in their documentation here .
Will Monte Carlo only monitor the tables in my default database?
No. Monte Carlo queries the Motherduck metadata tables to collect all database, schema and table names. If you would like Monte Carlo to only monitor certain databases or schemas, this can be controlled through the Usage controls .
What version of Motherduck do you support?
Currently we only support connecting to a Motherduck account with a DuckDB version of 0.9.2. Reach out to your Monte Carlo representative or [email protected] if you are interested in support for DuckDB version 0.10.2.
Updated 5 months ago