The platform supports a highly available topology: ClickHouse runs as two replicas spread across zones, coordinated by a three-voter ClickHouse Keeper ensemble. The Installation page shows the module inputs that deploy it; this page explains how the topology works and what it means for day-2 operations.

Topology

Replication, not sharding. The cluster is a single shard: every replica holds a complete copy of the data. Tables use ClickHouse's Replicated* engines, which coordinate through Keeper — a write accepted by either replica is replicated to the other. Monte Carlo's queries and the OpenTelemetry Collector's writes reach the healthy replicas through the internal Gateway.

Keeper is the coordination layer. ClickHouse Keeper is a Raft-based consensus ensemble that the replicated tables use to agree on the order of inserts, merges, and schema changes. Quorum requires a majority of voters, so the count must be odd — three voters tolerate the loss of one, which is what makes the platform survive a full zone outage. Keeper's client port is protected by a Kubernetes NetworkPolicy that admits only the ClickHouse pods and the operator.

Why three zones when ClickHouse only uses two. Two ClickHouse replicas in two zones already survive one zone loss — the third zone exists for the Keeper quorum, so that losing any single zone leaves two of three voters standing. A zonal control plane (location set to a zone) still supports multi-zone node pools, so the cost-minimal zonal cluster works with the HA topology; a regional location additionally makes the control plane itself HA.

Placement is explicit and pre-provisioned. Every replica and every voter is a stateful pod backed by a zone-locked persistent disk, so each one gets its own single-zone node pool, pinned by explicit zone name. The module does not autoscale these pools — capacity for every zone a stateful pod can land in is provisioned up front, which is why the topology variables list zones rather than counts. Both ClickHouse and Keeper pods also carry a hard zone-spread constraint (whenUnsatisfiable: DoNotSchedule): two replicas are never allowed to share a zone, because that would silently forfeit the single-zone-failure tolerance. Keeper pools use a distinct taint value (dedicated=keeper vs dedicated=clickhouse), so a ClickHouse node failure can never also take out a quorum voter.

Cross-zone routing. Both endpoints are served by a single regional internal Application Load Balancer (the GKE Gateway), which routes to healthy backends in any zone of the region — a routine single-replica event (pod restart, node drain) doesn't black-hole clients.

Module inputs

Set on the Terraform module (see the full example in Installation):

The HA topology converges more slowly than a single instance: the per-zone node pools scale up, the Keeper quorum forms, the replicas turn Ready, and only then can the schema job run its ON CLUSTER DDL — the module allows the Helm release up to 15 minutes for this chain.

Single-replica variant (dev/testing)

For development or testing environments where zone-failure tolerance isn't worth the node count, pin everything to a single zone with one replica. The zone lists stay required on a module-created cluster — there is no no-zones layout — so a single-entry list in each is the smallest shape:

clickhouse_zones         = ["us-central1-a"]
keeper_zones             = ["us-central1-a"]
clickhouse_replica_count = 1

What you give up:

• Node drains cause downtime again. With one replica there is no standby — node upgrades, resizes, and manual drains incur a ClickHouse outage while the pod restarts.
• Keeper has no quorum to fail over to. A single-voter Keeper outage that outlasts the readiness window turns the ClickHouse replica read-only and removes it from the Service — including for reads — until Keeper returns (see Readiness below).

On an existing cluster, this shape isn't available through a module-managed release — see Existing-cluster deployments for the self-managed equivalent.

Existing-cluster deployments

The per-zone node pools are created only on the new-cluster path (cluster.create = true). On an existing cluster the module rejects clickhouse_zones and keeper_zones at plan time — the chart's dedicated=clickhouse/dedicated=keeper node selectors would match no nodes and every replica and voter would sit Pending. A module-managed release also provides no way to override the scheduling values — so running the HA topology on an existing cluster means taking the self-managed Helm route: set helm.deploy_charts = false and manage the release yourself. To run HA there, provide the capacity and wiring yourself:

• Attach one single-zone node pool per ClickHouse zone, tainted dedicated=clickhouse:NoSchedule, and one per Keeper zone, tainted dedicated=keeper:NoSchedule.
• Wire the matching nodeSelector/tolerations (and keeper.replicasCount / clickhouse.replicasCount) through your own Helm values. See Self-managed Helm install.

Day-2 operations

Readiness: the /ready probe

ClickHouse pods report readiness through a writer-safe endpoint: /ready fails whenever any of the pod's tables is read-only — the state a replica enters when it loses its Keeper session. A replica in that state would still answer pings while inserts hang, so the probe removes it from the Service until it rejoins Keeper. Three consequences:

• Drains are refused while a replica is read-only. A not-Ready replica already counts as disrupted under the operator's PodDisruptionBudget (maxUnavailable: 1), so Kubernetes refuses to drain the healthy replica's node until the other recovers. This is correct behavior — it just changes what a stalled node roll looks like.
• helm --wait and kubectl wait --for=condition=Ready block while any pod is read-only.
• If every replica is read-only at once — a Keeper quorum loss — the Service empties and even reads are blocked, although each replica's local data is intact. This trade of read availability for writer safety is deliberate.

Ready is writer-safe, not read-complete: a replica that is joining (or rejoining) turns Ready once its replication queue is populated, before the historical data fetches finish. Gate operational waits on replication health, not on pod readiness:

SELECT table, active_replicas, queue_size
FROM system.replicas

Healthy means active_replicas equals the replica count and queue_size drains to 0.

Node and Kubernetes upgrades

The cluster's release_channel defaults to UNSPECIFIED — a static cluster. GKE never auto-upgrades the nodes, so a stateful ClickHouse or Keeper node is never drained against its PodDisruptionBudget on Google's schedule; node auto-repair stays on. The trade-off is that Kubernetes and node upgrades are customer-managed:

• Pin the control-plane version with kubernetes_version (unset = the GKE default at creation time) and bump it deliberately.
• Upgrade node pools on your own schedule (for example with gcloud container clusters upgrade / node-pool upgrades). Upgrades drain nodes through the eviction API, governed by the operator's maxUnavailable: 1 PodDisruptionBudget — on the HA topology, at most one replica or one voter is down at a time, so ClickHouse keeps serving and Keeper keeps quorum throughout.

Setting a release channel instead re-enables node auto-upgrade cluster-wide — it cannot be disabled per pool — which hands the drain schedule for the stateful pools back to GKE. Keep the static posture unless you accept that.

Verifying cluster health

In addition to the deployment verification checklist:

# Keeper ensemble reconciled
kubectl get chk -n montecarlo

# One ClickHouse pod per zone, one Keeper pod per zone
kubectl get pods -n montecarlo -l clickhouse.altinity.com/chi=otel -o wide
kubectl get pods -n montecarlo -l clickhouse-keeper.altinity.com/chk=otel -o wide

# Replication health (run via clickhouse-client on either replica)
# active_replicas = replica count, queue_size -> 0
SELECT table, active_replicas, queue_size FROM system.replicas

Next steps

• Deploy the topology: Installation
• Size the node pools and storage: Configuration reference
• Replicas not scheduling or not becoming Ready: Troubleshooting & FAQ