Overview

The Agent Observability data platform is a self-hosted pipeline that ingests OpenTelemetry trace data from your AI agents and stores it in ClickHouse for Monte Carlo to query. It runs entirely in your own AWS account and is deployed with a single Terraform module that provisions everything from the OpenTelemetry Collector through to ClickHouse on an EKS cluster.

Once deployed, Monte Carlo connects to the platform to power Trace Exploration and agent monitors — all queried through the Monte Carlo Agent.

Architecture

The platform has two independent data flows:

• Ingestion — your instrumented agents send OpenTelemetry traces to the OpenTelemetry Collector, which writes them directly into ClickHouse.
• Query — the Monte Carlo platform sends all of its SQL through the Monte Carlo Agent (a Lambda function running in the same VPC), which queries ClickHouse for Trace Exploration and agent monitors.

flowchart TB
  classDef ext fill:#F8FAFC,stroke:#64748B,color:#0F172A;
  classDef node fill:#FEF2F2,stroke:#DC2626,color:#7F1D1D;
  classDef net fill:#FFF7ED,stroke:#EA580C,color:#7C2D12;
  classDef agent fill:#EEF2FF,stroke:#4F46E5,color:#1E1B4B;

  APPS["Instrumented agents / apps"]:::ext
  MC["Monte Carlo platform"]:::ext
  BR["Amazon Bedrock"]:::ext

  subgraph ACCT["Your AWS account"]
    AGENT["Monte Carlo Agent · Lambda, in-VPC<br/>customer-deployed, outside EKS"]:::agent
    NLB1["Internal NLB · OTLP"]:::net
    NLB2["Internal NLB · ClickHouse"]:::net

    subgraph EKS["EKS cluster · montecarlo namespace"]
      OTEL["OpenTelemetry Collector"]:::node
      LLM["LLM worker"]:::node
      CH[("ClickHouse")]:::node
    end
  end

  APPS -->|"OTLP 4317 / 4318 · TLS"| NLB1
  NLB1 --> OTEL
  OTEL -->|"write traces"| CH
  MC -->|"SQL · Monte Carlo initiates"| AGENT
  AGENT -->|"forwards SQL · 8443 TLS · otel user"| NLB2
  NLB2 --> CH
  LLM -->|"read / write"| CH
  LLM -->|"evaluations · InvokeModel"| BR

  style ACCT fill:#FFFBEB,stroke:#EA580C,color:#7C2D12
  style EKS fill:#ECFEFF,stroke:#0891B2,color:#164E63

Components

The Terraform module deploys the following into your AWS account:

The Monte Carlo Agent (Lambda) is not part of this module — it is deployed separately and pointed at ClickHouse. See Deploy the agent and connect to Monte Carlo.

Ingestion

Instrumented agents send traces to the OpenTelemetry Collector over OTLP — gRPC on port 4317 or HTTP on port 4318, both TLS-terminated at a Network Load Balancer (NLB). The Collector batches incoming spans and writes them directly into ClickHouse using the ClickHouse exporter; there is no intermediate object storage or data warehouse.

The Collector can be hosted in-cluster (deployed by this module) or, where supported, hosted by Monte Carlo with only the data store running in your account.

Evaluation

The platform includes an LLM worker that runs evaluations against trace data. It reads pending work from ClickHouse, invokes a model through Amazon Bedrock in your account, and writes the results back to ClickHouse. The Bedrock region defaults to your deployment region and is configurable.

End-to-end setup

Getting agent traces into Monte Carlo is a four-step journey. This guide covers steps 2–3 (deploy and connect); the links below cover instrumenting your agents and creating monitors.

Deploying the platform

Work through these pages in order: