Overview

The terraform-google-ao-data-platform module provisions the GKE cluster and, by default (helm.deploy_charts = true), also deploys the ao-data-platform Helm chart — ClickHouse, the OpenTelemetry Collector, and the LLM worker — in the same terraform apply.

The kubernetes and helm providers are configured from the module's outputs, which lets Terraform defer the Kubernetes/Helm resources until after the GKE cluster exists. This is what enables a single-pass apply.

Work through the steps below in order. The required inputs are the same for both cluster paths: location, otel_collector_domain, clickhouse_domain, dns_zone, helm.chart_registry, and helm.chart_version.

1. Configure the providers

Your root module must configure the google, google-beta, kubernetes, and helm providers. The kubernetes and helm providers are wired from the module's outputs — substitute your region for us-central1:

terraform {
  required_version = ">= 1.7"
  required_providers {
    google      = { source = "hashicorp/google", version = "~> 7.0" }
    google-beta = { source = "hashicorp/google-beta", version = "~> 7.0" }
    kubernetes  = { source = "hashicorp/kubernetes", version = "~> 2.0" }
    helm        = { source = "hashicorp/helm", version = "~> 2.0" }
  }
}

provider "google" {
  project = var.project_id
  region  = "us-central1"
}

provider "google-beta" {
  project = var.project_id
  region  = "us-central1"
}

provider "kubernetes" {
  host                   = module.ao_data_platform.gke_cluster_endpoint
  cluster_ca_certificate = module.ao_data_platform.gke_cluster_ca_certificate
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "gke-gcloud-auth-plugin"
  }
}

provider "helm" {
  kubernetes {
    host                   = module.ao_data_platform.gke_cluster_endpoint
    cluster_ca_certificate = module.ao_data_platform.gke_cluster_ca_certificate
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "gke-gcloud-auth-plugin"
    }
  }
}

Two provider notes specific to this module:

• google-beta is required even though it backs a single resource — provisioning the Secret Manager service agent for CMEK, which the GA provider doesn't expose. Configure it with the same project and region as google.
• The helm provider must be v2 (~> 2.0) — the module relies on the nested kubernetes { ... } block syntax, which helm provider v3 changed.

2. Configure the module

The module's location input decides the cluster shape: pass a zone (us-central1-a) for a zonal cluster — the cost-minimal choice — or a region (us-central1) for a regional, HA control plane (node counts multiply across the region's zones). Regional sub-resources always derive their region from it.

New cluster

The module creates the VPC and GKE Standard cluster with sensible defaults (cluster name monte-carlo, private nodes with Cloud NAT egress, Dataplane V2, Workload Identity). A module-created cluster always runs the zoned ClickHouse/Keeper topology — clickhouse_zones and keeper_zones are required (non-empty) on this path. The example below deploys the standard highly available shape — two ClickHouse replicas across two zones, coordinated by a three-voter Keeper ensemble — substitute your region's zone names:

module "ao_data_platform" {
  source  = "monte-carlo-data/ao-data-platform/google"
  version = "~> 1.0"

  location = "us-central1-a"   # zone → zonal cluster; region → regional control plane

  otel_collector_domain = "otel.acme.com"
  clickhouse_domain     = "clickhouse.acme.com"
  dns_zone              = "acme-com"          # Cloud DNS zone NAME, not the DNS name

  gateway = {
    letsencrypt_email = "[email protected]"        # optional ACME contact
  }

  # Two ClickHouse replicas across two zones, backed by a 3-voter Keeper ensemble.
  clickhouse_zones         = ["us-central1-a", "us-central1-b"]
  keeper_zones             = ["us-central1-a", "us-central1-b", "us-central1-c"]
  clickhouse_replica_count = 2

  helm = {
    chart_registry = "oci://registry-1.docker.io/montecarlodata"
    chart_version  = "4.0.0"

    llm_worker = {
      image_tag = "1.1.0-gcp"
    }
  }
}

See examples/new_cluster/ for a complete, copy-paste starting point including the provider blocks.

Existing cluster

Set cluster.create = false and networking.create_vpc = false (the module rejects a module-created VPC with an existing cluster at plan time), and provide your cluster name, network and subnetwork names, the names of the node subnet's secondary ranges, and the range of your proxy-only subnet:

module "ao_data_platform" {
  source  = "monte-carlo-data/ao-data-platform/google"
  version = "~> 1.0"

  location = "us-central1-a"   # the existing cluster's zone or region

  cluster = {
    create                = false
    existing_cluster_name = "my-cluster"
  }

  networking = {
    create_vpc                   = false
    existing_network_name        = "my-vpc"
    existing_subnetwork_name     = "my-nodes-subnet"
    existing_pods_range_name     = "pods"      # secondary-range NAMES on the node subnet
    existing_services_range_name = "services"
    proxy_only_cidr              = "10.0.16.0/23"   # range of the existing proxy-only subnet
  }

  otel_collector_domain = "otel.acme.com"
  clickhouse_domain     = "clickhouse.acme.com"
  dns_zone              = "acme-com"

  helm = {
    chart_registry = "oci://registry-1.docker.io/montecarlodata"
    chart_version  = "4.0.0"

    llm_worker = {
      image_tag = "1.1.0-gcp"
    }

    # Set any controller already present in the cluster to false:
    # install_cert_manager              = false
    # install_external_secrets_operator = false
    # install_external_dns              = false
  }
}

See examples/existing_cluster/ for the full configuration, including the cluster and VPC prerequisites this path assumes (see also Prerequisites — choose your deployment path).

3. Initialize Terraform

terraform init

This downloads the module and the google, google-beta, kubernetes, and helm providers.

4. Review the plan

terraform plan

Review the planned changes before applying. On the new-cluster path the plan creates a VPC (node subnet with secondary ranges, a proxy-only subnet, Cloud Router + NAT), the GKE cluster and node pools — including one per listed ClickHouse and Keeper zone — Workload Identity IAM bindings, Cloud KMS keys, Secret Manager secrets, DNS IAM grants, and the Helm releases.

5. Apply

terraform apply

Review the plan once more, then confirm. The apply provisions the GCP infrastructure and (with helm.deploy_charts = true) deploys the chart in one pass. Most of the wall-clock time goes to the GKE cluster and node pools; on the HA topology the Helm release itself is allowed up to 15 minutes to converge (Keeper quorum forms, replicas turn Ready, then the schema job runs).

6. Confirm what was created

terraform output

You should see (among others):

The GCP infrastructure is now provisioned and the chart is deploying. Verifying that the in-cluster components (ClickHouse, the Collector, the LLM worker, the Gateway) came up healthy is the first step of Deploy the agent and connect to Monte Carlo.

ClickHouse and Keeper node pools

Every ClickHouse replica and every Keeper voter is a stateful pod backed by a zone-locked persistent disk, so each one runs on its own dedicated single-zone GKE node pool. On the new-cluster path (with helm.deploy_charts = true) the module creates them all automatically:

• Per-zone ClickHouse node pools — one per entry in clickhouse_zones, named ch-<zone-suffix> (one node each, tainted dedicated=clickhouse:NoSchedule, machine type n2-highmem-8 by default).
• Per-zone Keeper node pools — one per entry in keeper_zones (one node each, tainted dedicated=keeper:NoSchedule, machine type e2-standard-2 by default).

Both zone lists are required on this path — a single-zone list of one entry each is the smallest (dev) shape.

The module wires the matching nodeSelector/tolerations into the Helm release, so the ClickHouse and Keeper pods target their node pools exclusively — no manual configuration required. The OpenTelemetry Collector, LLM worker, and cluster controllers run on the main node pool (e2-standard-4, autoscaling between 2 and 10 nodes by default).

Harden network access

The Gateway's load balancer IP is private, but by default the module places no restriction on which sources inside your network can reach it. Before relying on the deployment, scope access with gateway.allowed_source_ranges — a single combined allow-list covering both endpoints, implemented as a Cloud Armor policy attached to the Gateway's backends:

• null (default) — no module-managed restriction.
• [] — only the VPC's own ranges (node subnet and pods) may connect.
• ["10.50.0.0/16", ...] — the VPC's ranges plus the listed CIDRs. The VPC ranges are always folded in when a restriction is in effect, so enabling it never locks out in-cluster clients.

If you deploy the Monte Carlo Agent on a dedicated subnet, include that subnet's CIDR in the list. Being an internal load balancer, a listed range is only reachable if a private path into the VPC exists.

Also consider cluster.master_authorized_networks (CIDRs allowed to reach the cluster API endpoint — must include every machine that runs Terraform or kubectl, including CI) and cluster.enable_private_endpoint (removes the public API endpoint entirely). See Network access in the Configuration reference.

Next steps

Continue to Deploy the agent and connect to Monte Carlo.