Agent Observability Platform: Connect to Monte Carlo (GCP)
Connect your GCP Agent Observability data platform to Monte Carlo
After installation, verify the deployment, deploy the Monte Carlo Agent, and create the ClickHouse integration in Monte Carlo. You carry out these steps yourself β work through them in order.
1. Configure cluster access
If you ran terraform apply from this same machine, you can skip this step β the module already ran gcloud container clusters get-credentials during apply, so the cluster's context is in your ~/.kube/config and is current. Run the command below only when connecting from a different machine:
gcloud container clusters get-credentials "$(terraform output -raw gke_cluster_name)" \
--location <location> --project <project_id>This requires the
gke-gcloud-auth-pluginand GKE IAM on the project. The identity that created the cluster already holds cluster access through its project IAM, so the same credentials you deployed with can run the verification below.
2. Verify the deployment
All components run in the montecarlo namespace. Confirm each is healthy:
| Component | Command | Healthy when |
|---|---|---|
| ClickHouse operator | kubectl get pods -n montecarlo -l app.kubernetes.io/name=altinity-clickhouse-operator | Pod Running |
| ClickHouse instance | kubectl get chi -n montecarlo | Resource otel, status Completed |
| ClickHouse pods | kubectl get pods -n montecarlo -l clickhouse.altinity.com/chi=otel | One pod per replica, all Running |
| Keeper ensemble | kubectl get chk -n montecarlo | Resource otel, status Completed |
| Keeper pods | kubectl get pods -n montecarlo -l clickhouse-keeper.altinity.com/chk=otel | One pod per voter, all Running |
| Schema migration job | kubectl get jobs -n montecarlo | The schema job shows COMPLETIONS 1/1 |
| OpenTelemetry Collector | kubectl get pods -n montecarlo -l app.kubernetes.io/name=opentelemetry-collector | Pod Running |
| LLM worker | kubectl get pods -n montecarlo -l app.kubernetes.io/component=llm-worker | Pod Running |
| Gateway & routes | kubectl get gateway,httproute -n montecarlo | PROGRAMMED True, Gateway address is a private IP |
| TLS certificates | kubectl get certificate gateway-otel-tls gateway-clickhouse-tls -n montecarlo | READY True |
| Secret store | kubectl get clustersecretstore gcp-secret-manager | READY True |
| External secrets | kubectl get externalsecret -n montecarlo | Each ao-clickhouse-*-credentials secret shows status SecretSynced |
The schema migration job must reach 1/1 before the Collector and LLM worker can write data β both wait on it via an init container, so it is normal for them to sit in Init for a minute or two on a fresh install. Let's Encrypt certificate issuance can take a few extra minutes after external-dns publishes the DNS-01 validation records.
To check end-to-end reachability, resolve both hostnames (they should return the Gateway's private IP) and send a request from a VM inside the VPC:
curl -s -o /dev/null -w '%{http_code}\n' https://<otel_collector_domain>/v1/tracesTest from a VM or peered network, not from a pod on the cluster β pod-IP clients cannot hairpin the internal Gateway address.
Verify Workload Identity
The module ships a verification script that proves the secret-sync path works exactly as designed β it asserts the External Secrets store is Ready, syncs a throwaway secret through the external-secrets Workload Identity binding (positive check), and confirms an unbound pod is denied Secret Manager access (negative check β no ambient node-level leak):
# After terraform init, the module source is available locally:
.terraform/modules/ao_data_platform/scripts/verify-wif.sh \
"$(terraform output -raw gke_cluster_name)" <location> <project_id>If any check above doesn't reach its healthy state, see Troubleshooting & FAQ. Also confirm the Vertex AI Model Garden enablement has landed β the LLM worker's evaluation calls fail until it does.
3. Deploy the Monte Carlo Agent
All SQL from the Monte Carlo platform β for both Trace Exploration and agent monitors β flows through the Monte Carlo Agent. For this platform, the Agent runs as a containerized service on Cloud Run and queries ClickHouse over its internal endpoint as the monte_carlo user.
The Agent is not deployed by the terraform-google-ao-data-platform module β it is deployed separately with the terraform-google-mcd-agent module; see Deploying the Agent on GCP for the Agent deployment itself. The Agent Observability-specific configuration is the network path and the ClickHouse connection:
- Network path. The ClickHouse endpoint resolves to a private IP inside the platform VPC, so the Cloud Run service must reach it through Direct VPC egress routing private ranges only, attached to a dedicated subnet in the platform VPC. The platform module's
network_nameandsubnetwork_nameoutputs identify the VPC and its node subnet β create the Agent's egress subnet alongside them. - Allow-list. If you restricted the Gateway with
gateway.allowed_source_ranges, add the Agent subnet's CIDR to the list and re-apply the platform module β the Cloud Armor policy evaluates the client IP, and the Agent's calls originate from that subnet. See Harden network access. - Connection. The endpoint is your
clickhouse_domain, and the credentials are themonte_carlouser's Secret Manager secret, retrieved in the next step.
You deploy and register the Agent yourself β it is a separate deployment from the data platform, not something Monte Carlo runs in your project. Your Monte Carlo representative can help you obtain the Agent's registration values.
4. Retrieve credentials and endpoints
Monte Carlo connects to ClickHouse as the monte_carlo user β a least-privilege identity that reads telemetry and appends only to the evaluation job queue. Its password is stored in Secret Manager; the module exposes it as the clickhouse_monte_carlo_credentials_secret_id output (and, for convenience, the exact version as clickhouse_monte_carlo_credentials_secret_version). Retrieve it:
gcloud secrets versions access \
"$(terraform output -raw clickhouse_monte_carlo_credentials_secret_version)"You enter the following when you create the ClickHouse integration in the next step:
| Value to provide | Source |
|---|---|
| ClickHouse endpoint | The clickhouse_domain you configured (e.g. clickhouse.acme.com) |
| ClickHouse port | 8443 (HTTPS) |
| ClickHouse database | otel_traces |
| ClickHouse username | monte_carlo |
| ClickHouse password | The value retrieved above |
The endpoint resolves to a private IP β Monte Carlo reaches it through the Agent you deployed in step 3, which has a network path into the VPC. Telemetry senders push to your otel_collector_domain over OTLP/HTTP (https://<otel_collector_domain>/v1/traces); the L7 Gateway does not carry OTLP/gRPC.
Treat these credentials as secrets β avoid printing them into shared logs or terminals, and if you need to share them (for example with a teammate or your Monte Carlo representative), use a secure channel.
monte_carlois a least-privilege user: it can read telemetry and append to the evaluation job queue, but cannot write telemetry, run DDL, or manage access.
5. Create the ClickHouse integration in Monte Carlo
With the Agent deployed (step 3) and the connection values in hand (step 4), create the ClickHouse integration in Monte Carlo. Monte Carlo reaches ClickHouse through the Agent you deployed, so this is the step that ties the platform to your account. Creating the integration runs a connection check β when it succeeds, you have confirmation that the endpoint, port, credentials, and database are all wired correctly.
Monte Carlo's general ClickHouse integration guide covers connecting Monte Carlo to a ClickHouse warehouse. It does not strictly apply to this platform: you do not need to create a user for the connection β the purpose-built
monte_carlouser is already provisioned with exactly the permissions the Agent needs.
Next steps
- Tune retention, users, TLS, and sizing in the Configuration reference.
- Hit a problem during verification or onboarding? See Troubleshooting & FAQ.
Updated 1 day ago
