Overview

The Agent Observability data platform is a self-hosted pipeline that ingests OpenTelemetry trace data from your AI agents and stores it in ClickHouse for Monte Carlo to query. It runs entirely in your own cloud account and is deployed with a single Terraform module — one per supported cloud — that provisions everything from the OpenTelemetry Collector through to ClickHouse on a Kubernetes cluster.

Once deployed, Monte Carlo connects to the platform to power Trace Exploration and agent monitors — all queried through the Monte Carlo Agent.

Choose your cloud

Start with the Prerequisites page for your cloud:

• Prerequisites (AWS)
• Prerequisites (Azure)
• Prerequisites (GCP)

Architecture

The platform has two independent data flows:

• Ingestion — your instrumented agents send OpenTelemetry traces to the OpenTelemetry Collector, which writes them directly into ClickHouse.
• Query — the Monte Carlo platform sends all of its SQL through the Monte Carlo Agent (a serverless workload running in your account with private network access to the cluster — a Lambda function on AWS, an Azure Functions container on Azure, a Cloud Run service on GCP), which queries ClickHouse for Trace Exploration and agent monitors.

By default — as configured in the Installation guides — the platform is deployed highly available: ClickHouse runs as two replicas spread across availability zones, coordinated by a three-voter ClickHouse Keeper ensemble, so it tolerates the loss of a full zone. A single-replica variant exists for dev/testing — see High availability for AWS, Azure or GCP.

flowchart TB
  classDef ext fill:#F8FAFC,stroke:#64748B,color:#0F172A;
  classDef node fill:#FEF2F2,stroke:#DC2626,color:#7F1D1D;
  classDef net fill:#FFF7ED,stroke:#EA580C,color:#7C2D12;
  classDef agent fill:#EEF2FF,stroke:#4F46E5,color:#1E1B4B;

  APPS["Instrumented agents / apps"]:::ext
  MC["Monte Carlo platform"]:::ext
  LLMSVC["Cloud LLM service<br/>Bedrock / Foundry / Vertex AI"]:::ext

  subgraph ACCT["Your cloud account"]
    AGENT["Monte Carlo Agent · serverless, in-network<br/>Lambda / Azure Functions / Cloud Run<br/>customer-deployed, outside the cluster"]:::agent
    ING1["Internal load balancer · OTLP<br/>NLB (AWS) · Gateway API (Azure, GCP)"]:::net
    ING2["Internal load balancer · ClickHouse"]:::net

    subgraph K8S["Kubernetes cluster · montecarlo namespace<br/>EKS / AKS / GKE"]
      OTEL["OpenTelemetry Collector"]:::node
      LLM["LLM worker"]:::node
      CH[("ClickHouse · 2 replicas<br/>across zones")]:::node
      KEEPER["ClickHouse Keeper · 3 voters<br/>replication coordination"]:::node
    end
  end

  APPS -->|"OTLP 4318 · TLS<br/>(+ gRPC 4317 on AWS)"| ING1
  ING1 --> OTEL
  OTEL -->|"write traces"| CH
  MC -->|"SQL · Monte Carlo initiates"| AGENT
  AGENT -->|"forwards SQL · 8443 TLS · monte_carlo user"| ING2
  ING2 --> CH
  CH <--> KEEPER
  LLM -->|"read / write"| CH
  LLM -->|"evaluations"| LLMSVC

  style ACCT fill:#FFFBEB,stroke:#EA580C,color:#7C2D12
  style K8S fill:#ECFEFF,stroke:#0891B2,color:#164E63

Components

The Terraform module for your cloud deploys the following into your account:

The Monte Carlo Agent is not part of the module — it is deployed separately and pointed at ClickHouse. See Connect to Monte Carlo for AWS, Azure, or GCP.

Ingestion

Instrumented agents send traces to the OpenTelemetry Collector over OTLP, TLS-terminated at an internal load balancer:

• AWS — a Network Load Balancer (L4) accepting OTLP over gRPC on port 4317 or HTTP on port 4318.
• Azure and GCP — an internal Gateway API load balancer (L7) accepting OTLP over HTTP on port 4318 only (no gRPC).

The Collector batches incoming spans and writes them directly into ClickHouse using the ClickHouse exporter; there is no intermediate object storage or data warehouse. On AWS the Collector can additionally ingest OTLP trace objects from S3 buckets (notified via SQS); Azure and GCP are OTLP-push-only.

Evaluation

The platform includes an LLM worker that runs evaluations against trace data. It reads pending work from ClickHouse, invokes a model through your cloud's LLM service in your account, and writes the results back to ClickHouse:

• AWS — Amazon Bedrock. The Bedrock region defaults to your deployment region and is configurable.
• Azure — Microsoft Foundry account the Terraform module provisions (with model deployments and a least-privilege inference role).
• GCP — Vertex AI (serverless, Model-as-a-Service; the models must be enabled in your project's Model Garden — see Prerequisites (GCP)).

See the Configuration reference for your cloud (AWS, Azure, or GCP) for the evaluation-backend settings.

End-to-end setup

Getting agent traces into Monte Carlo is a four-step journey. This documentation set covers steps 2–3 (deploy and connect); the links below cover instrumenting your agents and creating monitors.

Deploying the platform

Work through these pages in order for your cloud: