Agent Trust Security
How Agent Trust keeps your agents' traces, evaluations, and conversations under your control: architecture, trust boundaries, access controls, and shared responsibility.
Overview
Agent Trust (Agent Observability in the product) monitors the AI agents you build yourself or run on Snowflake and Databricks. It captures every agent run as a trace, evaluates output quality, alerts on regressions and drift, and traces failures back to the data and logic behind them. See the Agent Observability Overview for what it does.
Agent traces carry the content of your agents' work: prompts, completions, tool inputs and outputs, and conversation transcripts. In the customer-hosted deployment, Agent Trust is built so that this content stays in your environment. With the Monte Carlo-hosted option, trace content is stored in infrastructure Monte Carlo runs; see Monte Carlo-hosted deployment below. This page is for security reviewers. It covers where agent data lives, how Monte Carlo reaches it, who can read it, and which controls you own. For what Agent Trust processes, retention, and residency, see Agent Trust Privacy & Data Handling.
Agent Trust runs on the Monte Carlo platform under the same security program as the rest of Monte Carlo. Infrastructure Security, Application Security, and AI Security & Governance apply to the Monte Carlo-hosted side of Agent Trust.
Security at a glance
| Question | Answer |
|---|---|
| Where is trace content stored? | In the customer-hosted deployment, in your environment: the Agent Trust data platform in your own AWS, Azure, or GCP account, or your own Snowflake or Databricks account for platform-native agents. With the Monte Carlo-hosted option, in a dedicated deployment Monte Carlo runs in AWS us-east-1. |
| What does Monte Carlo receive? | In the customer-hosted deployment, span metadata and evaluation scores, which power the Agents page, dashboards, agent monitors, and alerts. Trace content is queried on demand and shown only to users whose role allows it. With the Monte Carlo-hosted option, Monte Carlo also stores and processes your trace content; see Monte Carlo-hosted deployment below. |
| Where do evaluations run? | In the customer-hosted deployment, for custom agents, inside your account, against your cloud's LLM service (Amazon Bedrock, Microsoft Foundry, or Vertex AI). Trace content isn't sent to Monte Carlo or to a Monte Carlo-hosted LLM to be evaluated. With the Monte Carlo-hosted option, evaluation calls go to Amazon Bedrock in US regions only. |
| How does Monte Carlo reach the traces? | In the customer-hosted deployment, for custom agents, through a Monte Carlo Agent you deploy in your account, as a least-privilege ClickHouse user with no write access to trace data. For platform-native agents, through your existing warehouse connection, with the grants you approve. With the Monte Carlo-hosted option, the Monte Carlo application reads your data through a function dedicated to your deployment. |
| Who can read trace content in Monte Carlo? | Users whose role grants agent-observability/content. You can share agent health and metrics without sharing content. |
| How long are traces kept? | In the customer-hosted deployment, you decide. The data platform keeps 30 days by default (clickhouse_ttl_days). Platform-native trace tables follow your Snowflake or Databricks retention. With the Monte Carlo-hosted option, trace data is kept for 30 days; see Monte Carlo-hosted deployment below for backups. |
| Can Agent Trust change my agents? | No. Reinforcement Loop proposes changes as draft pull requests, tickets, or prompts. Nothing lands until your team reviews and merges it. |
How agents connect
Custom agents: the Agent Trust data platform
Agents you build yourself, on any framework and any model, emit OpenTelemetry traces, for example with the Monte Carlo SDK, Google ADK or the OpenTelemetry Node SDK. The traces go to a data platform you deploy into your own cloud account with a Monte Carlo Terraform module:
- An OpenTelemetry Collector receives traces over OTLP with TLS, behind an internal load balancer, and writes them straight into ClickHouse. There is no intermediate object storage or warehouse.
- ClickHouse stores the traces. By default it runs as two replicas across availability zones.
- An LLM worker runs evaluations against the traces using your cloud's LLM service in your account, and writes the scores back to ClickHouse.
- The Monte Carlo Agent, a serverless workload you deploy in your account (AWS Lambda, Azure Functions, or Cloud Run), carries Monte Carlo's queries to ClickHouse over the private network.
On AWS you can run the Generic Agent as a pod in the platform cluster instead of the Lambda agent. It opens every connection outbound to Monte Carlo, so no inbound path from Monte Carlo into your account is needed.
The Agent Observability Data Platform docs cover the architecture, installation, and configuration for each cloud.
Platform-native agents: Snowflake and Databricks
Snowflake Cortex agents and Databricks agents log traces on their own platform. Monte Carlo reads them through the warehouse connection you have already configured, so no extra infrastructure is deployed and trace data stays in your Snowflake or Databricks account.
- Snowflake Cortex agents. Monte Carlo's service account needs
MONITORon each agent you want to observe, plus the grants listed in Required Snowflake permissions. WithoutREAD UNREDACTED AI OBSERVABILITY EVENTS TABLE, Snowflake redacts prompts and completions before Monte Carlo can read them. - Databricks Agent Bricks and custom Mosaic AI agents. Monte Carlo reads MLflow traces from a Unity Catalog trace table and needs read access to that table. See Agent Bricks and custom agents.
- Databricks AI/BI Genie. Genie keeps conversation history behind an API, so Monte Carlo installs a managed collector job in your workspace. The job writes the history into a
genie_tracestable in a catalog and schema you choose. It needsCAN MANAGEon each Genie space you add and captures every user's conversations in that space. Folder access is scoped to/Shared/monte_carlo/. See AI/BI Genie.
Trust boundaries
flowchart LR
classDef ext fill:#F8FAFC,stroke:#64748B,color:#0F172A;
classDef cust fill:#ECFEFF,stroke:#0891B2,color:#164E63;
classDef mc fill:#EEF2FF,stroke:#4F46E5,color:#1E1B4B;
subgraph CUST["Your cloud account"]
APPS["Your instrumented agents"]:::cust
OTEL["OpenTelemetry Collector"]:::cust
CH[("ClickHouse<br/>trace content + scores")]:::cust
LLM["LLM worker"]:::cust
LLMSVC["Your LLM service<br/>Bedrock / Foundry / Vertex AI"]:::cust
AGENT["Monte Carlo Agent<br/>deployed by you"]:::cust
end
subgraph MC["Monte Carlo platform"]
MCP["UI, agent monitors, alerting,<br/>Reinforcement Loop"]:::mc
end
APPS -->|"OTLP · TLS · internal LB"| OTEL
OTEL -->|"write traces"| CH
LLM -->|"read traces / write scores"| CH
LLM -->|"evaluation calls"| LLMSVC
MCP -->|"SQL, initiated by Monte Carlo"| AGENT
AGENT -->|"TLS · monte_carlo user"| CH
The diagram shows the customer-hosted deployment. There, trace content, evaluation prompts, and evaluator responses stay inside your account, and Monte Carlo receives span metadata, evaluation scores, and the results of queries it runs to display traces to authorized users. With the Monte Carlo-hosted option, trace content is stored and processed in your dedicated deployment in Monte Carlo's AWS environment; see Monte Carlo-hosted deployment below.
Access to the data platform
Least-privilege database users
The platform creates one ClickHouse user per access path, each scoped to the operations it performs. Monte Carlo connects as monte_carlo, which can read telemetry and append only to the job-queue, evaluation-score, and conversation-annotation tables. It has no write access to trace data. You can review the full user model in the Configuration reference for AWS, Azure, or GCP.
The module generates each user's password and stores it in your cloud's secret store, from which the External Secrets Operator syncs it into the cluster:
- AWS: AWS Secrets Manager, encrypted with a customer-managed KMS key the module creates.
- Azure: Azure Key Vault, RBAC-authorized with purge protection on.
- GCP: Secret Manager, with customer-managed encryption keys (CMEK) on by default.
Platform components authenticate to your cloud with workload identities: IRSA on AWS, Microsoft Entra Workload ID on Azure, and Workload Identity Federation on GCP.
Network
- The OTLP and ClickHouse endpoints sit behind internal load balancers in your network.
- Both endpoints use TLS: ACM certificates on AWS, and Let's Encrypt certificates issued by cert-manager on Azure and GCP, where traffic is re-encrypted from the gateway to the backends.
- Monte Carlo reaches ClickHouse only through the Monte Carlo Agent you deploy. With the Lambda, Azure Functions, or Cloud Run agent, Monte Carlo invokes the agent. With the Generic Agent on AWS, the agent connects out to Monte Carlo.
- Restrict source ranges. On AWS the internal load balancers accept traffic from any source that can route to them until you set
clickhouse_nlb_allowed_source_rangesandotel_collector_nlb_allowed_source_ranges. See Network access.
Encryption at rest
- AWS: the EBS volumes behind ClickHouse and ClickHouse Keeper are encrypted, and EKS envelope encryption of Kubernetes secrets is on for new clusters.
- Azure: new AKS clusters encrypt etcd with a customer-managed key in the module's Key Vault.
- GCP: GKE application-layer encryption of Kubernetes secrets uses CMEK by default, with Cloud KMS keys the module creates and rotates every 90 days, or keys you supply.
Access inside Monte Carlo
Agent Trust uses Monte Carlo's standard identity and access controls: SSO/SAML, SCIM, and role-based access. Two permissions separate agent health from agent content:
agent-observability/accesslets a user open Agent Observability and monitor agent health and metrics.agent-observability/contentlets a user read raw agent content: prompt and completion text, tool inputs and outputs, conversation transcripts, span attributes, LLM-judge rationale, alert samples, and exports.
Every built-in role grants access. mcd/asset-viewer and mcd/asset-editor don't grant content. To let other users monitor agents without reading what agents and their users said, deny agent-observability/content with a custom role. This permission is independent of table data sampling. See Managing access to Agent Observability content.
Reinforcement Loop
Reinforcement Loop investigates the conversations behind an agent's issues and proposes fixes.
- It never changes a running agent. Fixes arrive as a draft pull request, a ticket, or a prompt for your coding tool, and nothing lands until your team reviews and merges it.
- Its analysis runs against trace content under your existing Agent Observability settings and permissions.
- To diagnose a custom agent at the code level, it reads the repository you connect through the GitHub, GitLab, or Azure DevOps integration. For Snowflake Cortex agents, it reads the agent configuration, semantic views, Cortex Search services, and skills that Monte Carlo's service account can describe.
Shared responsibility
| Area | Monte Carlo | You |
|---|---|---|
| Monte Carlo platform | Secures the UI, API, agent monitors, alerting, and Reinforcement Loop under Monte Carlo's security program. | Assign roles, including who gets agent-observability/content. |
| Agent Trust data platform | Publishes and maintains the Terraform modules, Helm chart, and LLM worker image, with least-privilege defaults. | Deploy and operate the platform in your account: network access, upgrades, and cloud-account controls such as audit logging. Protect your Terraform state, which holds the ClickHouse credentials. |
| Trace content | Queries it on demand and shows it only to authorized users. | Decide which agents to instrument and what your instrumentation puts on spans. Set retention. |
| Evaluations | Defines the evaluation templates and orchestrates evaluation runs. | Choose and govern the evaluator model in your cloud's LLM service. |
| Platform-native agents | Reads traces through your existing warehouse connection with the grants you approve. | Grant the Snowflake or Databricks privileges, and decide whether prompts and completions are readable unredacted. |
Monte Carlo-hosted deployment
Most of this page describes the customer-hosted setup, where the Agent Trust data platform runs in your own cloud account. With the Monte Carlo-hosted option, your trace content (prompts, outputs and span data) is sent to and stored in infrastructure that Monte Carlo runs. Choosing this option means Monte Carlo stores and processes your trace content.
Current architecture: dedicated deployment per customer
- Isolation: each customer gets a dedicated deployment with its own VPC (never peered with other customers), Kubernetes (EKS) cluster and ClickHouse database. All of it runs in Monte Carlo's AWS environment in us-east-1.
- Intake: your agents send traces to a public HTTPS endpoint (TLS 1.2 or higher), authenticated with your integration key. You can limit which source IPs are accepted under Settings → Network access controls (scope
OpenTelemetry). Requests from embargoed countries are blocked. Inside Monte Carlo, traces travel over our private network to the collector. The collector writes them to an S3 handoff bucket that only your deployment uses, and objects there expire after 30 days. - Retention: trace data in ClickHouse is kept for 30 days.
- Backups: ClickHouse backups are retained until Monte Carlo's backup retention policy rollout completes. ClickHouse is backed up every 4 hours (a daily full backup plus incrementals). After the rollout, every backup from the last 7 days is kept, then one day's backups per week out to 30 days. When you offboard, backups are deleted or kept based on a decision recorded with you. Deleting removes every version.
- Encryption: data is encrypted at rest and in transit (TLS). S3 and EBS use AWS-managed keys, and backups use a dedicated per-deployment KMS key. Monte Carlo manages the keys. Customer-managed keys aren't offered for this option yet.
- Monte Carlo access: the Monte Carlo application reads your data through a function dedicated to your deployment. Its database credential is fetched at runtime and isn't stored in the Monte Carlo application. Staff reach the hosting environment through SSO with role-based permissions. Raw trace data can be read only by your deployment's own services, by engineers who have just-in-time elevated access, and by Monte Carlo's security team.
- Auditability: API activity in the hosting environment is recorded to a separate, central log archive account. Kubernetes control-plane audit logging is on for every deployment.
- AI evaluations: evaluation calls go to Amazon Bedrock in US regions only.
- Residency: US only. An EU region isn't available for this option.
Proposed architecture: shared multi-tenant cluster (coming soon)
Monte Carlo is building a multi-tenant architecture for the hosted option and expects to release it in the near future. Existing hosted customers will move to it then. In the proposed design:
- All hosted customers share one ClickHouse cluster, and each customer's data is kept in its own database.
- Traces arrive through a shared intake buffer that holds data for 3 days. Each customer's database loads only from that customer's own prefix.
We'll update this section with full security details when the new architecture is released.
Related
- Agent Trust Privacy & Data Handling
- Agent Observability Overview
- Agent Observability Data Platform
- AI Agents - Architecture & Data Handling covers the AI features Monte Carlo itself runs on AWS Bedrock.
Updated 4 days ago
