Monitoring Agent Technical Overview

Overview

Monte Carlo's Monitoring Agent is an internally developed application that can use licensed models from Anthropic (accessed via Amazon Bedrock) to generate intelligent recommendations for custom data quality monitors, also known as Advanced Recommendations. In addition, the Monitoring Agent will also provide heuristic recommendations without the use of AI.

We've created this section to provide a technical overview of the Monitoring Agent and answer frequently asked questions. For technical specs, such as Model Types and Versions click here.

Note: Data Sampling functionality must be enabled in order to use the Monitoring Agent.

SQL Query Assistance

There is an additional feature within the Monitoring Agent - SQL Query Assistance that is available to use and is enabled in conjunction with the main Monitoring Agent. Use of SQL Query Assistance feature does not interact with or affect the use of the main Monitoring Agent and can be disabled via the API if desired.

Primary & Intended Use

The Monitoring Agent was built to augment data analyst and engineering teams by suggesting effective monitors that improve data quality coverage and reduce time-to-detection. The optimal conditions for this feature are data warehouses and lakehouses with data sampling enabled and access to schema, query logs, and metadata.

This feature will not operate if data sampling is disabled and does not operate on top of BI layers or where lineage is missing.

How the Monitoring Agent Works

The Monitoring Agent uses AI and LLMs alongside native integrations to analyze metadata, query logs, and row level data (data sampling), in order to generate monitoring recommendations and detect anomalies. It can identify subtle data patterns (such as field formats, cross-column consistency, and statistical irregularities) and recommends precision-fit monitors with minimal manual effort.

All deployed monitoring rules are fully controlled and reviewable by the customer; customers can accept or reject suggestions.

AI Features Disabled

If AI features are disabled within your Monte Carlo instance, then the Monitoring Agent will still surface heuristic recommendations generated without the use of AI.

Data Privacy

All data processed by the Monitoring Agent is stored transiently in-memory. Upon termination of the job, all data is discarded.

This Monitoring Agent relies on AI Models provided by Amazon Bedrock (within AWS). They are noted on our Subprocessors page.

See Privacy Considerations for the Monitoring Agent for additional information.