How the clustered ClickHouse topology works — replicas, the Keeper ensemble, and day-2 operations.

The platform's recommended production topology is highly available: ClickHouse runs as two replicas spread across availability zones, coordinated by a three-voter ClickHouse Keeper ensemble. The Installation page shows the module inputs that deploy it; this page explains how the topology works and what it means for day-2 operations.

Topology

Zone entries are Azure availability-zone numbers ("1", "2", "3"), not region names.

Replication, not sharding. The cluster is a single shard: every replica holds a complete copy of the data. Tables use ClickHouse's Replicated* engines, which coordinate through Keeper — a write accepted by either replica is replicated to the other. Monte Carlo's queries and the OpenTelemetry Collector's writes reach ClickHouse through the managed Gateway, which forwards to the healthy replicas behind the ClickHouse Service.

Keeper is the coordination layer. ClickHouse Keeper is a Raft-based consensus ensemble that the replicated tables use to agree on the order of inserts, merges, and schema changes. Quorum requires a majority of voters, so the count must be odd — three voters tolerate the loss of one, which is what makes the platform survive a full zone outage. Keeper's client port is protected by a Kubernetes NetworkPolicy that admits only the ClickHouse pods and the operator.

Why three zones when ClickHouse only uses two. Two ClickHouse replicas in two zones already survive one zone loss — the third zone exists for the Keeper quorum, so that losing any single zone leaves two of three voters standing. The AKS node subnet is regional, so spanning three zones needs no extra networking — zones are a node-pool property, not a subnet property.

Placement is explicit and pre-provisioned. Every replica and every voter is a stateful pod backed by a zone-locked Azure managed disk, so each one gets its own single-zone node pool, pinned by explicit zone number. The stateful pools have no autoscaler — capacity for every zone a stateful pod can land in is provisioned up front, which is why the topology variables list zones rather than counts. Both ClickHouse and Keeper pods also carry a hard zone-spread constraint (whenUnsatisfiable: DoNotSchedule): two replicas are never allowed to share a zone, because that would silently forfeit the single-zone-failure tolerance.

Module inputs

Set on the Terraform module (see the full example in Installation):

Both zone lists are required on the new-cluster path. A module-created cluster always runs the zoned topology — the plan fails if either clickhouse_zones or keeper_zones is empty, so ClickHouse replicas and Keeper voters always have dedicated per-zone nodes to land on. Grow the topology by appending zones and raising the replica count.

Single-replica variant (dev/testing)

For development or testing environments where zone-failure tolerance isn't worth the node count, run a single replica with a single Keeper voter:

clickhouse_zones         = ["1"]
keeper_zones             = ["1"]
clickhouse_replica_count = 1

What you give up:

• Node drains cause downtime. With one replica there is no standby — node-image upgrades, pool resizes, and manual drains incur a ClickHouse outage (~30–90s) while the pod restarts.
• Keeper has no quorum to fail over to. A single-voter Keeper outage that outlasts the readiness window (~30 seconds) turns the ClickHouse replica read-only and removes it from the Service — including for reads — until Keeper returns (see Readiness below).

Existing-cluster deployments

The per-zone node pools are created only on the new-cluster path (cluster.create = true). On an existing cluster the module rejects clickhouse_zones and keeper_zones at plan time — the chart's dedicated=clickhouse / dedicated=keeper node selectors would match no nodes and the pods would stay Pending. A module-managed release (helm.deploy_charts = true) also provides no way to override the scheduling values — so running the HA topology on an existing cluster means taking the self-managed Helm route: set helm.deploy_charts = false and manage the release yourself. To run HA there, provide the capacity and wiring yourself:

• Attach one single-zone node pool per ClickHouse zone, tainted dedicated=clickhouse:NoSchedule, and one per Keeper zone, tainted dedicated=keeper:NoSchedule.
• Wire the matching nodeSelector/tolerations (and keeper.replicasCount / clickhouse.replicasCount) through your own Helm values — the module-managed release doesn't expose them on this path. See Self-managed Helm install.

The cluster's nodes must be able to land in every zone you target — zones are a node-pool property, so no subnet changes are needed, but the region must offer the zones you list.

Day-2 operations

Readiness: the /ready probe

ClickHouse pods report readiness through a writer-safe endpoint: /ready fails whenever any of the pod's tables is read-only — the state a replica enters when it loses its Keeper session. A replica in that state would still answer pings while inserts hang, so the probe removes it from the Service until it rejoins Keeper. Three consequences:

• Drains are refused while a replica is read-only. A not-Ready replica already counts as disrupted under the operator's PodDisruptionBudget (maxUnavailable: 1), so Kubernetes refuses to drain the healthy replica's node until the other recovers. This is correct behavior — it just changes what a stalled node roll looks like.
• helm --wait and kubectl wait --for=condition=Ready block while any pod is read-only.
• If every replica is read-only at once — a Keeper quorum loss — the Service empties and even reads are blocked, although each replica's local data is intact. This trade of read availability for writer safety is deliberate.

Ready is writer-safe, not read-complete: a replica that is joining (or rejoining) turns Ready once its replication queue is populated, before the historical data fetches finish. Gate operational waits on replication health, not on pod readiness:

SELECT table, active_replicas, queue_size
FROM system.replicas

Healthy means active_replicas equals the replica count and queue_size drains to 0.

Node-image updates and Kubernetes upgrades

Node images are pinned by default: the module sets the cluster's node-OS upgrade channel to None, so AKS never auto-rolls nodes underneath the stateful pods. Updates happen on your schedule:

• Stateless pools (main node pool): set or change node_image_upgrade_token and apply — the module triggers a node-image upgrade of the pool. The token only takes effect once set, so a first apply never triggers a drain.
• ClickHouse and Keeper pools (HA topology): upgrade one zone's pool at a time. Evictions go through the eviction API, governed by the operator's maxUnavailable: 1 PodDisruptionBudget, so at most one replica or one voter is down at a time — ClickHouse keeps serving and Keeper keeps quorum throughout.
• ClickHouse pool (single-replica topology): there is no standby, so upgrade the pool's node image manually in a maintenance window — expect the ~30–90s pod restart outage.

For Kubernetes version upgrades, bump kubernetes_version and apply, one minor version at a time.

Verifying cluster health

In addition to the deployment verification checklist:

# Keeper ensemble reconciled
kubectl get chk -n montecarlo

# One ClickHouse pod per zone, one Keeper pod per zone
kubectl get pods -n montecarlo -l clickhouse.altinity.com/chi=otel -o wide
kubectl get pods -n montecarlo -l clickhouse-keeper.altinity.com/chk=otel -o wide

# Replication health (run via clickhouse-client on either replica)
# active_replicas = replica count, queue_size -> 0
SELECT table, active_replicas, queue_size FROM system.replicas

Next steps

• Deploy the topology: Installation
• Size the node pools and storage: Configuration reference
• Replicas not scheduling or not becoming Ready: Troubleshooting & FAQ