Troubleshooting Agent Technical Specs

Purpose & Functionality

Identifies root causes of data + AI reliability issues; subagents specialize in querying logs, code changes (e.g., GitHub), ETL errors (e.g., dbt/Airflow), data anomalies, and metadata lineage; returns structured recommendations for remediation; allows user to ask questions about issue, root cause, and remediation.

Model Details & Testing

Source of AI Model [Application]

Internally developed by Monte Carlo, including multi-agent design, sub-agent routing logic, evaluation workflow, and integration with lineage, query logs, GitHub, and orchestration systems.

Source of AI Model [Foundational Model]

Licensed from third-party – Anthropic; accessed via Amazon Bedrock

Model Provider

Anthropic via Amazon Bedrock

Model Type

Multi-agent LLM system

Version

Claude Sonnet 3.7 and 4.0 + Claude Haiku 3.5

Testing [Application]

Monte Carlo only uses internal self hosted data for testing purposes. No customer data is used to test or train.

Testing [Foundational Model]

Responsibility of the respective AI model provider.

Fairness and Bias

No applicable due to domain-specific, non-human context

Primary and optional use cases

System Requirements

Works with existing implementation of Monte Carlo, operating on the customer's cloud data warehouse. Data Sampling can be turned on.

Prompt/Input Requirements

Automatically gathered by Monte Carlo: anomaly details, lineage information, orchestration logs, Git commits, data samples for anomaly.

Evaluation Metrics

Cosine distance between expected and actual results during CI testing, with LLM as judge.

Continuous Monitoring Plan

Monte Carlo captures telemetry data (using OpenTelemetry) on the model behaviors; Monte Carlo monitors quality of responses in production using LLM-as-a-judge monitors; customer feedback loop informs future iteration of the agent.