Troubleshooting Agent Technical Specs
Purpose & Functionality
Identifies root causes of data + AI reliability issues; subagents specialize in querying logs, code changes (e.g., GitHub), ETL errors (e.g., dbt/Airflow), data anomalies, and metadata lineage; returns structured recommendations for remediation; allows user to ask questions about issue, root cause, and remediation.
Model Details & Testing
Source of AI Model [Application]
Internally developed by Monte Carlo, including multi-agent design, sub-agent routing logic, evaluation workflow, and integration with lineage, query logs, GitHub, and orchestration systems.
Source of AI Model [Foundational Model]
Licensed from third-party – Anthropic; accessed via Amazon Bedrock
Model Provider
Anthropic via Amazon Bedrock
Model Type
Multi-agent LLM system
Version
Claude Sonnet 3.7 and 4.0 + Claude Haiku 3.5
Testing [Application]
Monte Carlo only uses internal self hosted data for testing purposes. No customer data is used to test or train.
Testing [Foundational Model]
Responsibility of the respective AI model provider.
Fairness and Bias
No applicable due to domain-specific, non-human context
Primary and optional use cases
System Requirements
Works with existing implementation of Monte Carlo, operating on the customer's cloud data warehouse. Data Sampling can be turned on.
Prompt/Input Requirements
Automatically gathered by Monte Carlo: anomaly details, lineage information, orchestration logs, Git commits, data samples for anomaly.
Evaluation Metrics
Cosine distance between expected and actual results during CI testing, with LLM as judge.
Continuous Monitoring Plan
Monte Carlo captures telemetry data (using OpenTelemetry) on the model behaviors; Monte Carlo monitors quality of responses in production using LLM-as-a-judge monitors; customer feedback loop informs future iteration of the agent.
Updated 19 days ago