Agent Trust Security

Overview

Agent Trust (Agent Observability in the product) monitors the AI agents you build yourself or run on Snowflake and Databricks. It captures every agent run as a trace, evaluates output quality, alerts on regressions and drift, and traces failures back to the data and logic behind them. See the Agent Observability Overview for what it does.

Agent traces carry the content of your agents' work: prompts, completions, tool inputs and outputs, and conversation transcripts. Agent Trust is built so that this content stays in your environment. This page is for security reviewers. It covers where agent data lives, how Monte Carlo reaches it, who can read it, and which controls you own. For what Agent Trust processes, retention, and residency, see Agent Trust Privacy & Data Handling.

Agent Trust runs on the Monte Carlo platform under the same security program as the rest of Monte Carlo. Infrastructure Security, Application Security, and AI Security & Governance apply to the Monte Carlo-hosted side of Agent Trust.

Security at a glance

How agents connect

Custom agents: the Agent Trust data platform

Agents you build yourself, on any framework and any model, are instrumented with the Monte Carlo SDK or Google ADK and emit OpenTelemetry traces. The traces go to a data platform you deploy into your own cloud account with a Monte Carlo Terraform module:

• An OpenTelemetry Collector receives traces over OTLP with TLS, behind an internal load balancer, and writes them straight into ClickHouse. There is no intermediate object storage or warehouse.
• ClickHouse stores the traces. By default it runs as two replicas across availability zones.
• An LLM worker runs evaluations against the traces using your cloud's LLM service in your account, and writes the scores back to ClickHouse.
• The Monte Carlo Agent, a serverless workload you deploy in your account (AWS Lambda, Azure Functions, or Cloud Run), carries Monte Carlo's queries to ClickHouse over the private network.

On AWS you can run the Generic Agent as a pod in the platform cluster instead of the Lambda agent. It opens every connection outbound to Monte Carlo, so no inbound path from Monte Carlo into your account is needed.

The Agent Observability Data Platform docs cover the architecture, installation, and configuration for each cloud.

Platform-native agents: Snowflake and Databricks

Snowflake Cortex agents and Databricks agents log traces on their own platform. Monte Carlo reads them through the warehouse connection you have already configured, so no extra infrastructure is deployed and trace data stays in your Snowflake or Databricks account.

• Snowflake Cortex agents. Monte Carlo's service account needs MONITOR on each agent you want to observe, plus the grants listed in Required Snowflake permissions. Without READ UNREDACTED AI OBSERVABILITY EVENTS TABLE, Snowflake redacts prompts and completions before Monte Carlo can read them.
• Databricks Agent Bricks and custom Mosaic AI agents. Monte Carlo reads MLflow traces from a Unity Catalog trace table and needs read access to that table. See Agent Bricks and custom agents.
• Databricks AI/BI Genie. Genie keeps conversation history behind an API, so Monte Carlo installs a managed collector job in your workspace. The job writes the history into a genie_traces table in a catalog and schema you choose. It needs CAN MANAGE on each Genie space you add and captures every user's conversations in that space. Folder access is scoped to /Shared/monte_carlo/. See AI/BI Genie.

Trust boundaries

flowchart LR
  classDef ext fill:#F8FAFC,stroke:#64748B,color:#0F172A;
  classDef cust fill:#ECFEFF,stroke:#0891B2,color:#164E63;
  classDef mc fill:#EEF2FF,stroke:#4F46E5,color:#1E1B4B;

  subgraph CUST["Your cloud account"]
    APPS["Your instrumented agents"]:::cust
    OTEL["OpenTelemetry Collector"]:::cust
    CH[("ClickHouse<br/>trace content + scores")]:::cust
    LLM["LLM worker"]:::cust
    LLMSVC["Your LLM service<br/>Bedrock / Foundry / Vertex AI"]:::cust
    AGENT["Monte Carlo Agent<br/>deployed by you"]:::cust
  end

  subgraph MC["Monte Carlo platform"]
    MCP["UI, agent monitors, alerting,<br/>Reinforcement Loop"]:::mc
  end

  APPS -->|"OTLP · TLS · internal LB"| OTEL
  OTEL -->|"write traces"| CH
  LLM -->|"read traces / write scores"| CH
  LLM -->|"evaluation calls"| LLMSVC
  MCP -->|"SQL, initiated by Monte Carlo"| AGENT
  AGENT -->|"TLS · monte_carlo user"| CH

Trace content, evaluation prompts, and evaluator responses stay inside your account. Monte Carlo receives span metadata, evaluation scores, and the results of queries it runs to display traces to authorized users.

Access to the data platform

Least-privilege database users

The platform creates one ClickHouse user per access path, each scoped to the operations it performs. Monte Carlo connects as monte_carlo, which can read telemetry and append only to the job-queue, evaluation-score, and conversation-annotation tables. It has no write access to trace data. You can review the full user model in the Configuration reference for AWS, Azure, or GCP.

The module generates each user's password and stores it in your cloud's secret store, from which the External Secrets Operator syncs it into the cluster:

• AWS: AWS Secrets Manager, encrypted with a customer-managed KMS key the module creates.
• Azure: Azure Key Vault, RBAC-authorized with purge protection on.
• GCP: Secret Manager, with customer-managed encryption keys (CMEK) on by default.

Platform components authenticate to your cloud with workload identities: IRSA on AWS, Microsoft Entra Workload ID on Azure, and Workload Identity Federation on GCP.

Network

• The OTLP and ClickHouse endpoints sit behind internal load balancers in your network.
• Both endpoints use TLS: ACM certificates on AWS, and Let's Encrypt certificates issued by cert-manager on Azure and GCP, where traffic is re-encrypted from the gateway to the backends.
• Monte Carlo reaches ClickHouse only through the Monte Carlo Agent you deploy. With the Lambda, Azure Functions, or Cloud Run agent, Monte Carlo invokes the agent. With the Generic Agent on AWS, the agent connects out to Monte Carlo.
• Restrict source ranges. On AWS the internal load balancers accept traffic from any source that can route to them until you set clickhouse_nlb_allowed_source_ranges and otel_collector_nlb_allowed_source_ranges. See Network access.

Encryption at rest

• AWS: the EBS volumes behind ClickHouse and ClickHouse Keeper are encrypted, and EKS envelope encryption of Kubernetes secrets is on for new clusters.
• Azure: new AKS clusters encrypt etcd with a customer-managed key in the module's Key Vault.
• GCP: GKE application-layer encryption of Kubernetes secrets uses CMEK by default, with Cloud KMS keys the module creates and rotates every 90 days, or keys you supply.

Access inside Monte Carlo

Agent Trust uses Monte Carlo's standard identity and access controls: SSO/SAML, SCIM, and role-based access. Two permissions separate agent health from agent content:

• agent-observability/access lets a user open Agent Observability and monitor agent health and metrics.
• agent-observability/content lets a user read raw agent content: prompt and completion text, tool inputs and outputs, conversation transcripts, span attributes, LLM-judge rationale, alert samples, and exports.

Every built-in role grants access. mcd/asset-viewer and mcd/asset-editor don't grant content. To let other users monitor agents without reading what agents and their users said, deny agent-observability/content with a custom role. This permission is independent of table data sampling. See Managing access to Agent Observability content.

Reinforcement Loop

Reinforcement Loop investigates the conversations behind an agent's issues and proposes fixes.

• It never changes a running agent. Fixes arrive as a draft pull request, a ticket, or a prompt for your coding tool, and nothing lands until your team reviews and merges it.
• Its analysis runs against trace content under your existing Agent Observability settings and permissions.
• To diagnose a custom agent at the code level, it reads the repository you connect through the GitHub, GitLab, or Azure DevOps integration. For Snowflake Cortex agents, it reads the agent configuration, semantic views, Cortex Search services, and skills that Monte Carlo's service account can describe.

Shared responsibility

Related

• Agent Trust Privacy & Data Handling
• Agent Observability Overview
• Agent Observability Data Platform
• AI Agents - Architecture & Data Handling covers the AI features Monte Carlo itself runs on AWS Bedrock.