Monitoring Agent Technical Overview

Overview

Monte Carlo's Monitoring Agent is an internally developed application that uses licensed models from Anthropic (accessed via Amazon Bedrock) and OpenAI to generate intelligent recommendations for custom data quality monitors.

We've created this section to provide a technical overview of the Monitoring Agent and answer frequently asked questions. For technical specs, such as Model Types and Versions click here.

Note: Data Sampling functionality must be enabled in order to use the Monitoring Agent.

Primary & Intended Use

The Monitoring Agent was built to augment data analyst and engineering teams by suggesting effective monitors that improve data quality coverage and reduce time-to-detection. The optimal conditions for this feature are data warehouses and lakehouses with data sampling enabled and access to schema, query logs, and metadata.

This feature will not operate if data sampling is not enabled and does not operate on top of BI layers or where lineage is missing.

How the Monitoring Agent Works

The Monitoring Agent uses AI and LLMs alongside native integrations to analyze metadata, query logs, and data samples when permitted, in order to generate monitoring recommendations and detect anomalies. It can identify subtle data patterns (such as field formats, cross-column consistency, and statistical irregularities) and recommends precision-fit monitors with minimal manual effort.

All deployed monitoring rules are fully controlled and reviewable by the customer; customers can accept or reject suggestions.

Data Privacy

All data processed by the Monitoring Agent is stored transiently in-memory. Upon termination of the job, all data is discarded.

The Monitoring Agent relies on AI Models provided by Amazon Bedrock and OpenAI. Monte Carlo has executed DPAs with these vendors and they are noted on our Subprocessors page.

See Privacy Considerations for the Monitoring Agent for additional information.