Common issues during installation and after, plus frequently asked questions. This page is shared across clouds — entries that apply to a single cloud are labeled. For the deployment steps themselves, see Installation (AWS, Azure, GCP) and Connect to Monte Carlo (AWS, Azure, GCP).

Installation issues

A ClickHouse or Keeper pod is stuck in Pending

Almost always a scheduling or volume placement problem on the dedicated node groups (AWS) or node pools (Azure/GCP):

• No node in the required zone. ClickHouse and Keeper pods carry a hard one-per-zone spread constraint, and each replica/voter needs a node group or pool standing by in its zone (the module installs no autoscaler for these pods). Confirm the dedicated nodes exist and are Ready:

  Example

  kubectl get nodes -l dedicated=clickhouse -o wide
  kubectl get nodes -l dedicated=keeper -o wide
  kubectl describe pod -n montecarlo -l clickhouse.altinity.com/chi=otel

  The pod's events will name the unsatisfied constraint (zone spread, taint, or insufficient resources). On an existing (non-module-created) cluster, the per-zone node groups/pools are yours to attach — see the High availability page for your cloud (AWS, Azure, GCP).

• Keeper voters Pending on a cluster with fewer than three zones. The one-voter-per-zone spread is built into the chart and not configurable — on single-zone or dev clusters run a single voter (one entry in keeper_availability_zones on AWS / keeper_zones on Azure and GCP, or keeper.replicasCount: 1 on self-managed installs).

• (AWS) Fresh install on an existing cluster with a module-managed release. With cluster.create = false and helm.deploy_charts = true, the module renders no Keeper scheduling values on this path, so the chart's default three voters land on the main node group — which defaults to two nodes, and whose autoscaling group doesn't guarantee one node per AZ even when scaled up. With the voters' hard one-per-AZ spread, one or more voters commonly stay Pending on a bare install. The fix is the self-managed Helm route (helm.deploy_charts = false): set keeper.replicasCount and the scheduling values in your own values file — see High availability (AWS). (On Azure and GCP the module rejects module-managed HA topology on existing clusters at plan time, steering you to the self-managed route up front.)

• Availability zone mismatch. Persistent disks are zone-locked on every cloud (EBS volumes, Azure managed disks, GCP persistent disks). If a dedicated node group/pool is in a different zone than the pod's existing persistent volume, the pod can't mount it. On AWS the module guards the common case at plan time (clickhouse_availability_zones[0] must match the existing volume's zone) — if you disabled that guard or manage node groups yourself, confirm the zones line up.

A TLS certificate never becomes Ready

kubectl get certificates -n montecarlo
kubectl describe certificate -n montecarlo clickhouse-server-tls

Check, in order:

• cert-manager is installed and running — internal Collector↔ClickHouse TLS is always enabled and depends on it. If you deployed into an existing cluster, confirm you didn't skip it with helm.install_cert_manager = false while it was actually absent.

• The CA issuer is ready — the chart creates an ao-data-platform-ca issuer by default (tls.certManager.createCA = true).

• DNS-01 validation for the externally-served endpoints:

  • AWS — confirm hosted_zone_id is set and the cert-manager IRSA role can manage Route 53 records.
  • Azure — the Gateway's Let's Encrypt certs validate DNS-01 against your Azure DNS zone; confirm dns_zone (and dns_zone_resource_group_name, if the zone lives in another resource group) is correct and the cert-manager identity holds DNS Zone Contributor on the zone.
  • GCP — DNS-01 validates against your Cloud DNS zone, which must be publicly resolvable (the Let's Encrypt TXT challenge is public even though the A records point at a private IP); confirm dns_zone names the managed zone and the cert-manager Workload Identity binding exists.

  kubectl describe on the CertificateRequest/Order shows the validation error.

An ExternalSecret is not SecretSynced

kubectl get externalsecret -n montecarlo
kubectl describe externalsecret -n montecarlo ao-clickhouse-otel-credentials

The External Secrets Operator syncs the ClickHouse passwords from your cloud's secret store (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) into the cluster. If the status isn't SecretSynced:

• Confirm the ClusterSecretStore exists and is Valid.
• Confirm ESO's cloud identity can read the secret: the IRSA role and KMS decrypt on AWS, the per-secret Key Vault grants on Azure, the per-secret secretmanager.secretAccessor grants on GCP.
• Confirm the referenced secret exists in the secret store (the module's clickhouse_*_credentials_secret_* outputs).

The schema-migration job doesn't complete

kubectl get jobs -n montecarlo
kubectl logs -n montecarlo job/clickhouse-schema-<n>

The clickhouse-schema-<n> job (where <n> is the Helm release revision) creates the otel_traces database and tables and applies the TTLs. It runs as the schema_owner user (which holds the DDL grants) and waits for ClickHouse to accept its connection, so a failure here usually traces back to ClickHouse not being healthy or the schema_owner secret not being synced (see above). While waiting it logs progress ("still waiting after Ns..."), so a long wait is distinguishable from a hang in its logs. Every statement it runs is cluster-wide (ON CLUSTER), so on the HA topology a slow or still-joining replica can legitimately stretch the job's runtime — the DDL is idempotent and the job retries. The Collector and LLM worker block on this job via an init container, so if it never completes they stay in Init.

A ClickHouse replica isn't Ready, or the ClickHouse Service is empty

ClickHouse pods report readiness through a writer-safe /ready probe that fails while any of the pod's tables is read-only — the state a replica enters when it loses its connection to Keeper. See the Readiness section of your cloud's High availability page (AWS, Azure, GCP) for the full semantics. What it looks like in practice:

• One replica not Ready — the replica is read-only, usually a lost Keeper session (check the Keeper pods) or a joining replica still registering. helm --wait and kubectl wait --for=condition=Ready block during this window; that's the probe working as designed.
• All replicas not Ready at once — Keeper quorum is down; the Service is deliberately emptied (reads included, even though each replica's local data is intact) until quorum returns. On single-voter dev shapes, any Keeper outage longer than ~30 seconds does this.
• The second replica is Ready but has little data — Ready means writer-safe, not fully caught up: a joining replica turns Ready before its historical backfill finishes. Check system.replicas (queue_size draining to 0). If the replica's tables are empty and stay empty, the replica count was raised against tables that predate chart 3.0.0 without conversion — contact your Monte Carlo representative.

A node drain (or node image roll) is refused

A not-Ready replica already counts as disrupted under the operator's PodDisruptionBudget (maxUnavailable: 1), so Kubernetes refuses to evict the healthy replica while the other is read-only. Restore the unhealthy replica first, then drain. Do not add your own PDB for these pods — multi-PDB pods can't be evicted at all.

Keeper connectivity

kubectl get networkpolicy -n montecarlo

Keeper's client port (2181) speaks the ZooKeeper protocol, which has no authentication of its own. The chart isolates it with a Kubernetes NetworkPolicy that admits only the ClickHouse pods and the operator on that port (Keeper peers reach each other on the Raft port, 9444):

• Connections to port 2181 from anything else are refused. That's the NetworkPolicy working as intended — only the ClickHouse pods and the operator are admitted.
• The NetworkPolicy is only enforced if the cluster runs a NetworkPolicy engine. Without one, the policy is silently inert — Keeper's client port, unauthenticated by itself, is then reachable from any pod on the cluster network. Module-created clusters enforce it on every cloud: EKS via the VPC CNI's network policy agent, AKS via the Cilium network-policy engine, GKE via Dataplane V2. On existing clusters, confirm the policy exists with the command above and that your CNI has network policy enforcement turned on; see the Topology section of your cloud's High availability page (AWS, Azure, GCP).

(Azure / GCP, existing clusters) Terraform fails a Gateway API precondition

The module serves traffic through the managed Gateway API and verifies at plan/apply time that an existing cluster has it enabled. The module only enables these features on clusters it creates — on an existing cluster, enable them out-of-band first:

• Azure (both add-ons are required):

  Shell

  az aks update -g <resource-group> -n <cluster> --enable-gateway-api
  az aks update -g <resource-group> -n <cluster> --enable-app-routing-istio

• GCP:

  Shell

  gcloud container clusters update <cluster> --location <location> --gateway-api=standard

(Azure / GCP) The helm provider fails to configure

The modules' examples use Terraform helm provider v2 syntax (a nested kubernetes { ... } block). Helm provider v3 changed this to a top-level argument and is not yet supported — pin the provider:

helm = {
  source  = "hashicorp/helm"
  version = "~> 2.0"
}

(Azure) Foundry deployment creation fails on quota

The module creates three Claude model deployments, each defaulting to capacity 100 (100,000 tokens-per-minute, GlobalStandard). If your subscription's Claude TPM quota in the target region is lower, the apply fails — scope the footprint with the deployments / default_capacity inputs to match the quota you were granted.

I bumped the chart version but not the topology variables

On chart >= 2.3.0 the Keeper ensemble renders on every install — on AWS, bumping helm.chart_version without setting keeper_availability_zones lands the default 3-voter ensemble on the main node pool. Set the chart version and the topology variables together (see your cloud's Installation page). On Azure and GCP this can't happen on module-created clusters: keeper_zones and clickhouse_zones are required there and an empty list fails at plan time.

I'm on an older chart version

This documentation describes chart 4.0.0, which the Azure and GCP modules require — older charts predate the multi-cloud value shape and half-wire silently on those clouds. The AWS module accepts ao-data-platform chart >= 2.0.0. Landmarks in the chart's history, newest first:

• 4.0.0 — multi-cloud: gateway.provider is required whenever the Gateway serves traffic (Azure/GCP), and the LLM worker's backend is selected with llmWorker.provider (bedrock, foundry, or vertex).
• 3.0.0 — replicated table engines and cluster-wide DDL; default replica count 2; the writer-safe /ready probe and probe user. Breaking for installs with older data: see the pre-3.0.0 upgrade callout. (2.4.0 was never published.)
• 2.3.0 — Keeper support; Keeper renders on every install from here.
• 2.0.0 — the least-privilege user model (schema_owner, llm_worker, monte_carlo, restrict_grants). Pre-2.0.0 charts provision only the shared otel user.
• 1.3.0 — the dedicated tainted node-group layout; older charts use an anti-affinity rule that deadlocks with the module's node groups.
• 1.2.0 — the optional readonly_user; below this the secret exists but no SQL user is provisioned.

Pin helm.chart_version to 4.0.0 and set the HA topology variables with it.

(AWS, existing cluster) Terraform says the OIDC provider already exists

Import the existing provider before applying:

terraform import 'module.ao_data_platform.aws_iam_openid_connect_provider.cluster[0]' <arn>

kubectl / fetching cluster credentials is denied

The principal that ran terraform apply is granted cluster access automatically on every cloud. To use a different principal:

• AWS — you need eks:DescribeCluster plus an EKS access entry for the principal.
• Azure — kubectl auth goes through kubelogin (Entra ID); confirm it's installed, and grant the principal an AKS RBAC cluster role (e.g. "Azure Kubernetes Service RBAC Cluster Admin") or membership in the cluster's admin group.
• GCP — an exec plugin: gke-gcloud-auth-plugin error means the auth plugin isn't installed on the machine; otherwise grant the principal container.* access on the project and re-run gcloud container clusters get-credentials.

After installation

Traces aren't arriving in ClickHouse

If everything is healthy but no trace data appears:

• Source ranges. An overly narrow allow-list silently drops connections. On AWS, check otel_collector_nlb_allowed_source_ranges (and clickhouse_nlb_allowed_source_ranges for the query path). On Azure and GCP, check gateway.allowed_source_ranges — on GCP the deny shows up as Cloud Armor 403s, and remember the ranges your senders and the Monte Carlo Agent use must both be included. Widen the list to the correct source CIDR (do not open it to 0.0.0.0/0 in production; scope it to the sending network).
• Endpoint, protocol, and ports. On AWS, agents send OTLP over gRPC (4317) or HTTP (4318) with TLS. On Azure and GCP the Gateway is L7 HTTP(S)-only: send OTLP over HTTP(S) to the collector domain — OTLP/gRPC is not supported through the Gateway.
• Collector logs. kubectl logs -n montecarlo -l app.kubernetes.io/name=opentelemetry-collector shows receiver/exporter errors.

The Collector is running but isn't writing to ClickHouse

Check the Collector logs for ClickHouse exporter errors. Common causes: the schema-migration job hasn't completed (so otel_traces doesn't exist yet), or the otel credentials secret isn't synced. Verify both as described under Installation issues.

The LLM worker can't run evaluations

The worker calls your cloud's LLM service. Start with its logs — kubectl logs -n montecarlo -l app.kubernetes.io/component=llm-worker — then check the backend:

• AWS (Amazon Bedrock) — confirm the worker's IRSA role allows bedrock:InvokeModel for the target model, and that the model is available in the worker's region (it uses your deployment region by default; set helm.llm_worker.bedrock_region to override).
• Azure (Microsoft Foundry) — confirm the module-created Foundry account and Claude deployments exist and that your Claude TPM quota isn't exhausted (see the quota entry above). The worker reaches the account over its public endpoint with Entra auth.
• GCP (Vertex AI) — the most common cause is the one-time manual prerequisite: the Claude models must be enabled in your project's Vertex AI Model Garden (the "Enable Access" flow accepting Anthropic's terms; approval can take 24–48 hours). Also confirm the worker's Workload Identity binding holds the module-created predict role, and that helm.llm_worker.vertex_region (default "global") serves the model you need.

FAQ

Is ClickHouse highly available?

Yes, by default: two replicas spread across availability zones, coordinated by a three-voter Keeper ensemble, tolerating the loss of a full zone. See the High availability page for your cloud (AWS, Azure, GCP) — including the single-replica dev/testing variant and what it gives up.

How do I pause ingestion or the LLM worker?

• LLM worker (AWS): set helm.llm_worker.replica_count = 0 — a declarative pause that survives applies; set it back to resume. The Azure and GCP modules don't expose a replica override; on those clouds, pause by scaling the deployment directly (kubectl scale deploy -n montecarlo -l app.kubernetes.io/component=llm-worker --replicas=0) — note a later module apply restores it.
• Ingestion: setting the Collector's replica count to 0 does not work (the chart treats 0 as unset and deploys the default count). Stop ingest upstream instead: pause the OTLP senders.

Which ClickHouse user does Monte Carlo connect as?

The monte_carlo user — a least-privilege identity that reads telemetry and appends only to the evaluation job queue. Provide the ClickHouse endpoint and the monte_carlo credentials to Monte Carlo — they're in your cloud's secret store, referenced by the module's clickhouse_monte_carlo_credentials_* output. See the Connect to Monte Carlo page for your cloud (AWS, Azure, GCP).

Do I need to create a read-only ClickHouse user for Monte Carlo?

No. Monte Carlo's general ClickHouse integration guide describes creating a read-only user, but that doesn't apply here — the purpose-built monte_carlo user is already provisioned with the permissions the Agent needs. The optional readonly_user is for your own external SQL clients (e.g. DataGrip), not for the Monte Carlo connection.

How do I change trace retention?

Set clickhouse_ttl_days (default 30) — the input has the same name on all three modules. The schema job re-applies it to the telemetry tables on the next install or upgrade. See your cloud's Configuration reference (AWS, Azure, GCP).

How do I restrict who can reach ClickHouse?

These source ranges are the primary network control on every cloud. On AWS set clickhouse_nlb_allowed_source_ranges; on Azure and GCP set gateway.allowed_source_ranges (one combined list covering both the collector and ClickHouse endpoints — on GCP, module and BYO VPC ranges are always folded in so you can't lock out in-VPC clients). See the Network access section of your cloud's Configuration reference (AWS, Azure, GCP).