The platform's default topology is highly available: ClickHouse runs as two replicas spread across Availability Zones, coordinated by a three-voter ClickHouse Keeper ensemble. The Installation page shows the module inputs that deploy it; this page explains how the topology works and what it means for day-2 operations.

Topology

Replication, not sharding. The cluster is a single shard: every replica holds a complete copy of the data. Tables use ClickHouse's Replicated* engines, which coordinate through Keeper — a write accepted by either replica is replicated to the other. Monte Carlo's queries and the OpenTelemetry Collector's writes are load-balanced across the healthy replicas by the NLB.

Keeper is the coordination layer. ClickHouse Keeper is a Raft-based consensus ensemble that the replicated tables use to agree on the order of inserts, merges, and schema changes. Quorum requires a majority of voters, so the count must be odd — three voters tolerate the loss of one, which is what makes the platform survive a full AZ outage. Keeper's client port is protected by a Kubernetes NetworkPolicy that admits only the ClickHouse pods and the operator.

Why three AZs when ClickHouse only uses two. Two ClickHouse replicas in two AZs already survive one zone loss — the third AZ exists for the Keeper quorum, so that losing any single zone leaves two of three voters standing. The VPC the module creates spans three AZs by default, so the default HA layout needs no networking changes.

Placement is explicit and pre-provisioned. Every replica and every voter is a stateful pod backed by an AZ-locked EBS volume, so each one gets its own single-AZ node group, pinned by explicit AZ name. The module does not install a cluster autoscaler — capacity for every zone a stateful pod can land in is provisioned up front, which is why the topology variables list AZs rather than counts. Both ClickHouse and Keeper pods also carry a hard zone-spread constraint (whenUnsatisfiable: DoNotSchedule): two replicas are never allowed to share an AZ, because that would silently forfeit the single-AZ-failure tolerance.

Cross-zone NLB routing. Both NLBs (OpenTelemetry Collector and ClickHouse) have cross-zone routing enabled, so an NLB address in one AZ always forwards to healthy targets in any AZ. Without it, a routine single-replica event (pod restart, node drain) would black-hole the clients that resolve to that AZ's NLB address. Standard AWS inter-AZ data-transfer charges apply (negligible at typical telemetry volumes).

Module inputs

Set on the Terraform module (see the full example in Installation):

Two further inputs exist as migration aids for deployments that started on the older single-instance layout; most users leave the defaults:

Single-replica variant (dev/testing)

For development or testing environments where AZ-failure tolerance isn't worth the node count, run a single replica with a single Keeper voter:

clickhouse_availability_zones = ["us-east-1a"]
keeper_availability_zones     = ["us-east-1a"]
clickhouse_replica_count      = 1

What you give up:

• Node drains cause downtime again. With one replica there is no standby — EKS AMI rolls, node resizes, and manual drains incur a ClickHouse outage (~30–90s) while the pod restarts.
• Keeper has no quorum to fail over to. A single-voter Keeper outage that outlasts the readiness window (~30 seconds) turns the ClickHouse replica read-only and removes it from the Service — including for reads — until Keeper returns (see Readiness below).

On an existing cluster, this single-AZ shape isn't available through a module-managed release — see Existing-cluster deployments for the self-managed equivalent.

Existing-cluster deployments

The per-AZ node groups are created only on the new-cluster path (cluster.create = true). On an existing cluster the module rejects keeper_availability_zones at plan time, and clickhouse_availability_zones creates no node groups. A module-managed release (helm.deploy_charts = true) also renders no scheduling values here and provides no way to override them — so running the HA topology on an existing cluster means taking the self-managed Helm route: set helm.deploy_charts = false and manage the release yourself. To run HA there, provide the capacity and wiring yourself:

• Attach one single-AZ node group per ClickHouse AZ, tainted dedicated=clickhouse:NoSchedule, and one per Keeper AZ, tainted dedicated=keeper:NoSchedule.
• Wire the matching nodeSelector/tolerations (and keeper.replicasCount / clickhouse.replicasCount) through your own Helm values — the module-managed release doesn't expose them on this path. See Self-managed Helm install.

The cluster's private subnets must span every AZ you list. Note that an EKS cluster's control-plane AZ set is immutable: if the cluster was created with subnets in fewer AZs than the topology needs, append the new AZ's subnet to networking.existing_private_subnet_ids and set networking.control_plane_subnet_ids to the cluster's creation-time subnets. Expect the main node group to be replaced (create-before-destroy) and both NLBs to extend into the new AZ.

Day-2 operations

Readiness: the /ready probe

ClickHouse pods report readiness through a writer-safe endpoint: /ready fails whenever any of the pod's tables is read-only — the state a replica enters when it loses its Keeper session. A replica in that state would still answer pings while inserts hang, so the probe removes it from the Service until it rejoins Keeper. Three consequences:

• Drains are refused while a replica is read-only. A not-Ready replica already counts as disrupted under the operator's PodDisruptionBudget (maxUnavailable: 1), so Kubernetes refuses to drain the healthy replica's node until the other recovers. This is correct behavior — it just changes what a stalled node roll looks like.
• helm --wait and kubectl wait --for=condition=Ready block while any pod is read-only.
• If every replica is read-only at once — a Keeper quorum loss — the Service empties and even reads are blocked, although each replica's local data is intact. This trade of read availability for writer safety is deliberate.

Ready is writer-safe, not read-complete: a replica that is joining (or rejoining) turns Ready once its replication queue is populated, before the historical data fetches finish. Gate operational waits on replication health, not on pod readiness:

SELECT table, active_replicas, queue_size
FROM system.replicas

Healthy means active_replicas equals the replica count and queue_size drains to 0.

Node AMI updates and Kubernetes upgrades

AMI updates on the ClickHouse and Keeper node groups are rolling and availability-preserving: nodes are replaced by evicting pods through the eviction API, governed by the operator's maxUnavailable: 1 PodDisruptionBudget, so at most one replica or one voter is down at a time — ClickHouse keeps serving and Keeper keeps quorum throughout. The node groups also pin their AMI version by default, so updates happen on your schedule — set ami_release_version (or use_latest_ami_release_version = true) when you're ready — rather than as a side effect of a routine terraform apply.

When you bump the cluster's kubernetes_version, bump every pinned ami_release_version in use (ClickHouse, Keeper, and the legacy node group if present) to a matching-minor build in the same apply — a pinned node group left on the old minor fails the apply.

Verifying cluster health

In addition to the deployment verification checklist:

# Keeper ensemble reconciled
kubectl get chk -n montecarlo

# One ClickHouse pod per AZ, one Keeper pod per AZ
kubectl get pods -n montecarlo -l clickhouse.altinity.com/chi=otel -o wide
kubectl get pods -n montecarlo -l clickhouse-keeper.altinity.com/chk=otel -o wide

# Replication health (run via clickhouse-client on either replica)
# active_replicas = replica count, queue_size -> 0
SELECT table, active_replicas, queue_size FROM system.replicas

Next steps

• Deploy the topology: Installation
• Size the node groups and storage: Configuration reference
• Replicas not scheduling or not becoming Ready: Troubleshooting & FAQ