Agent Observability Platform: Connect to Monte Carlo (GCP)

Connect your GCP Agent Observability data platform to Monte Carlo

☁️

This page covers GCP. Also available for AWS and Azure.

After installation, verify the deployment, deploy the Monte Carlo Agent, and create the ClickHouse integration in Monte Carlo. You carry out these steps yourself β€” work through them in order.

1. Configure cluster access

If you ran terraform apply from this same machine, you can skip this step β€” the module already ran gcloud container clusters get-credentials during apply, so the cluster's context is in your ~/.kube/config and is current. Run the command below only when connecting from a different machine:

gcloud container clusters get-credentials "$(terraform output -raw gke_cluster_name)" \
  --location <location> --project <project_id>
πŸ“˜

This requires the gke-gcloud-auth-plugin and GKE IAM on the project. The identity that created the cluster already holds cluster access through its project IAM, so the same credentials you deployed with can run the verification below.

2. Verify the deployment

All components run in the montecarlo namespace. Confirm each is healthy:

ComponentCommandHealthy when
ClickHouse operatorkubectl get pods -n montecarlo -l app.kubernetes.io/name=altinity-clickhouse-operatorPod Running
ClickHouse instancekubectl get chi -n montecarloResource otel, status Completed
ClickHouse podskubectl get pods -n montecarlo -l clickhouse.altinity.com/chi=otelOne pod per replica, all Running
Keeper ensemblekubectl get chk -n montecarloResource otel, status Completed
Keeper podskubectl get pods -n montecarlo -l clickhouse-keeper.altinity.com/chk=otelOne pod per voter, all Running
Schema migration jobkubectl get jobs -n montecarloThe schema job shows COMPLETIONS 1/1
OpenTelemetry Collectorkubectl get pods -n montecarlo -l app.kubernetes.io/name=opentelemetry-collectorPod Running
LLM workerkubectl get pods -n montecarlo -l app.kubernetes.io/component=llm-workerPod Running
Gateway & routeskubectl get gateway,httproute -n montecarloPROGRAMMED True, Gateway address is a private IP
TLS certificateskubectl get certificate gateway-otel-tls gateway-clickhouse-tls -n montecarloREADY True
Secret storekubectl get clustersecretstore gcp-secret-managerREADY True
External secretskubectl get externalsecret -n montecarloEach ao-clickhouse-*-credentials secret shows status SecretSynced

The schema migration job must reach 1/1 before the Collector and LLM worker can write data β€” both wait on it via an init container, so it is normal for them to sit in Init for a minute or two on a fresh install. Let's Encrypt certificate issuance can take a few extra minutes after external-dns publishes the DNS-01 validation records.

To check end-to-end reachability, resolve both hostnames (they should return the Gateway's private IP) and send a request from a VM inside the VPC:

curl -s -o /dev/null -w '%{http_code}\n' https://<otel_collector_domain>/v1/traces
πŸ“˜

Test from a VM or peered network, not from a pod on the cluster β€” pod-IP clients cannot hairpin the internal Gateway address.

Verify Workload Identity

The module ships a verification script that proves the secret-sync path works exactly as designed β€” it asserts the External Secrets store is Ready, syncs a throwaway secret through the external-secrets Workload Identity binding (positive check), and confirms an unbound pod is denied Secret Manager access (negative check β€” no ambient node-level leak):

# After terraform init, the module source is available locally:
.terraform/modules/ao_data_platform/scripts/verify-wif.sh \
  "$(terraform output -raw gke_cluster_name)" <location> <project_id>
πŸ“˜

If any check above doesn't reach its healthy state, see Troubleshooting & FAQ. Also confirm the Vertex AI Model Garden enablement has landed β€” the LLM worker's evaluation calls fail until it does.

3. Deploy the Monte Carlo Agent

All SQL from the Monte Carlo platform β€” for both Trace Exploration and agent monitors β€” flows through the Monte Carlo Agent. For this platform, the Agent runs as a containerized service on Cloud Run and queries ClickHouse over its internal endpoint as the monte_carlo user.

The Agent is not deployed by the terraform-google-ao-data-platform module β€” it is deployed separately with the terraform-google-mcd-agent module; see Deploying the Agent on GCP for the Agent deployment itself. The Agent Observability-specific configuration is the network path and the ClickHouse connection:

  • Network path. The ClickHouse endpoint resolves to a private IP inside the platform VPC, so the Cloud Run service must reach it through Direct VPC egress routing private ranges only, attached to a dedicated subnet in the platform VPC. The platform module's network_name and subnetwork_name outputs identify the VPC and its node subnet β€” create the Agent's egress subnet alongside them.
  • Allow-list. If you restricted the Gateway with gateway.allowed_source_ranges, add the Agent subnet's CIDR to the list and re-apply the platform module β€” the Cloud Armor policy evaluates the client IP, and the Agent's calls originate from that subnet. See Harden network access.
  • Connection. The endpoint is your clickhouse_domain, and the credentials are the monte_carlo user's Secret Manager secret, retrieved in the next step.
πŸ“˜

You deploy and register the Agent yourself β€” it is a separate deployment from the data platform, not something Monte Carlo runs in your project. Your Monte Carlo representative can help you obtain the Agent's registration values.

4. Retrieve credentials and endpoints

Monte Carlo connects to ClickHouse as the monte_carlo user β€” a least-privilege identity that reads telemetry and appends only to the evaluation job queue. Its password is stored in Secret Manager; the module exposes it as the clickhouse_monte_carlo_credentials_secret_id output (and, for convenience, the exact version as clickhouse_monte_carlo_credentials_secret_version). Retrieve it:

gcloud secrets versions access \
  "$(terraform output -raw clickhouse_monte_carlo_credentials_secret_version)"

You enter the following when you create the ClickHouse integration in the next step:

Value to provideSource
ClickHouse endpointThe clickhouse_domain you configured (e.g. clickhouse.acme.com)
ClickHouse port8443 (HTTPS)
ClickHouse databaseotel_traces
ClickHouse usernamemonte_carlo
ClickHouse passwordThe value retrieved above

The endpoint resolves to a private IP β€” Monte Carlo reaches it through the Agent you deployed in step 3, which has a network path into the VPC. Telemetry senders push to your otel_collector_domain over OTLP/HTTP (https://<otel_collector_domain>/v1/traces); the L7 Gateway does not carry OTLP/gRPC.

πŸ”’

Treat these credentials as secrets β€” avoid printing them into shared logs or terminals, and if you need to share them (for example with a teammate or your Monte Carlo representative), use a secure channel. monte_carlo is a least-privilege user: it can read telemetry and append to the evaluation job queue, but cannot write telemetry, run DDL, or manage access.

5. Create the ClickHouse integration in Monte Carlo

With the Agent deployed (step 3) and the connection values in hand (step 4), create the ClickHouse integration in Monte Carlo. Monte Carlo reaches ClickHouse through the Agent you deployed, so this is the step that ties the platform to your account. Creating the integration runs a connection check β€” when it succeeds, you have confirmation that the endpoint, port, credentials, and database are all wired correctly.

πŸ“˜

Monte Carlo's general ClickHouse integration guide covers connecting Monte Carlo to a ClickHouse warehouse. It does not strictly apply to this platform: you do not need to create a user for the connection β€” the purpose-built monte_carlo user is already provisioned with exactly the permissions the Agent needs.

Next steps


Did this page help you?