Agent Observability Platform: Configuration Reference (GCP)
Configuration reference for the Agent Observability data platform on GCP
All settings on this page are optional β the defaults produce a working deployment. Set them as inputs to the Terraform module.
Data retention (TTL)
ClickHouse retains trace data for a fixed window, after which old data is dropped automatically.
| Input | Default | Description |
|---|---|---|
clickhouse_ttl_days | 30 | Days of trace data to retain. Governs the trace tables and their conversation-derived annotation tables. Applied cluster-wide by the schema job, so retention can't drift between replicas. |
ClickHouse users
The platform provisions a least-privilege ClickHouse user model β one SQL user per access path, each scoped to exactly the operations it performs:
| User | Access | Used by |
|---|---|---|
schema_owner | Read/write on the telemetry database, plus DDL (including cluster-wide ON CLUSTER statements) and system-table maintenance | The schema-migration job (creates tables, applies TTLs) and the owner (DEFINER) of the materialized views |
otel | INSERT-only into the raw trace table | The OpenTelemetry Collector. Normalization still works because the materialized views run as schema_owner, not otel |
llm_worker | Read/append on the LLM job-queue tables | The LLM worker |
monte_carlo | Read-only on telemetry, plus append to the job-queue, evaluation-score, and conversation-annotation tables | The Monte Carlo Agent (Trace Exploration, agent monitors, queuing evaluations) β this is the user you connect Monte Carlo as |
probe | system.replicas replication health and SHOW TABLES on the telemetry database (passwordless; no row-level data reads) | The ClickHouse readiness probe β needs no secret or configuration |
readonly_user | SELECT-only (optional) | External SQL clients (e.g. DataGrip), MCP, and deployment verification |
admin | Full superuser, including access management (optional, gated) | Break-glass DBA access β reachable only over loopback (pod-exec) by default |
schema_owner, otel, llm_worker, monte_carlo, and probe are always provisioned. readonly_user and admin are opt-in.
Each user's password is generated by the module and stored in Secret Manager (CMEK-encrypted by default β see Encryption), then synced into the cluster by the External Secrets Operator through a per-secret access grant β passwords never transit Helm values. Each user has its own secret, named <cluster_name>-clickhouse-<user> and exposed as a clickhouse_<user>_credentials_secret_id output. The exception is probe, which is deliberately passwordless β its consumer is the kubelet's readiness check, which can't read a Kubernetes secret β and so has no Secret Manager entry. What probe can read is limited to replication-health metadata and table names, with no row data exposed; gateway.allowed_source_ranges (see Network access) governs who can reach it.
The stock ClickHouse
defaultsuperuser is removed during deployment as a hardening measure β every client authenticates as one of the users above.
To supply your own passwords instead of the generated ones, set the sensitive clickhouse_passwords input (per-user; any user left null is auto-generated). Note that like all Terraform-managed secrets, the values are readable in the Terraform state β protect the state accordingly.
Read-only user
To provision the optional readonly_user (SELECT-only, with per-session SET allowed so JDBC clients such as DataGrip connect cleanly):
helm = {
chart_registry = "oci://registry-1.docker.io/montecarlodata"
chart_version = "4.0.0"
clickhouse = {
readonly_user = {
enabled = true
}
}
}Its secret is exposed as the clickhouse_readonly_user_credentials_secret_id output.
Break-glass admin user
The admin superuser is opt-in and, by default, reachable only over loopback (pod-exec) β it is never on a network path. Enable it for break-glass DBA access:
helm = {
chart_registry = "oci://registry-1.docker.io/montecarlodata"
chart_version = "4.0.0"
clickhouse = {
admin = {
enabled = true
}
}
}Its secret is exposed as the clickhouse_admin_credentials_secret_id output (null when disabled).
Resource sizing
Each managed workload exposes optional Kubernetes resource requests/limits via helm.<workload>.resources. Omit a workload (or a requests/limits map) to use the chart defaults β start modest in development and tune up for production.
helm = {
chart_registry = "oci://registry-1.docker.io/montecarlodata"
chart_version = "4.0.0"
clickhouse = {
resources = {
requests = { cpu = "2", memory = "8Gi" }
limits = { cpu = "4", memory = "16Gi" }
}
}
opentelemetry_collector = {
resources = {
requests = { memory = "2Gi" }
limits = { memory = "6Gi" }
}
}
llm_worker = {
resources = {
requests = { cpu = "500m", memory = "1Gi" }
limits = { cpu = "2", memory = "4Gi" }
}
}
}requests and limits are maps keyed by Kubernetes resource name (cpu, memory, ephemeral-storage, etc.); either can be omitted independently.
There are no replica-count overrides for the Collector or the LLM worker on GCP β both run at the chart's defaults. ClickHouse replicas are controlled by clickhouse_replica_count (see High availability below).
Node pools (new-cluster path)
On the new-cluster path, you can size the node pools the workloads run on:
| Input | Default | Description |
|---|---|---|
clickhouse_node_pool.machine_type | n2-highmem-8 | Machine type for the per-zone ClickHouse node pools (memory-optimized; N2 or later is required for Hyperdisk storage). One input covers every pool; each pool's zone comes from clickhouse_zones. |
keeper_node_pool.machine_type | e2-standard-2 | Machine type for the per-zone Keeper node pools (use n2-standard-2 for strictly non-burstable β a throttled quorum voter risks spurious leader elections). |
keeper_node_pool.storage_size / .storage_class | 10Gi / standard-rwo | The Keeper persistent volume (Raft log and snapshots only). |
cluster.machine_type | e2-standard-4 | Machine type for the main node pool (Collector, LLM worker, controllers). |
cluster.main_node_pool_size | 2 | Floor of the main node pool (set 1 for a cost-optimized single node; range 1β10). |
cluster.main_node_pool_max_size | 10 | Autoscaler ceiling for the main node pool (1β100). |
The main pool autoscales between its floor and ceiling; the ClickHouse and Keeper pools do not β their capacity is provisioned up front, one node per pool (see High availability).
High availability
The topology inputs below deploy the standard HA shape β how it works, the dev/testing single-replica variant, and day-2 operations are covered on the High availability page.
| Input | Default | Description |
|---|---|---|
clickhouse_zones | [] | Zone names for the per-zone ClickHouse node pools; the list length caps the replica count. Required (non-empty) when cluster.create = true; all entries in one region. |
keeper_zones | [] | Zone names for the Keeper voter node pools β must be an odd count (3 standard, 1 for dev). Also sets the chart's Keeper replica count. Required (non-empty) when cluster.create = true; all entries in one region. |
clickhouse_replica_count | 1 | ClickHouse replicas (2 for the standard HA shape). Terraform-owned: a chart version bump alone never changes it. |
Both zone lists are required on a module-created cluster (a single-zone entry in each is the dev shape) and must be empty on an existing cluster β see High availability β existing-cluster deployments.
Storage
ClickHouse data is stored on a persistent-disk-backed volume.
| Input | Default | Description |
|---|---|---|
helm.clickhouse.storage_size | 500Gi | Size of the ClickHouse persistent volume. |
clickhouse_storage_class | clickhouse-premium | StorageClass for the ClickHouse volume. The module creates a dedicated clickhouse-premium class by default (Retain reclaim policy, WaitForFirstConsumer binding, volume expansion allowed); set to an existing class name to use your own. |
clickhouse_storage.type | pd-ssd | Disk type behind clickhouse-premium: pd-ssd (works on all machine families) or hyperdisk-balanced (production opt-in with tunable performance; requires an N2-or-later machine family). |
clickhouse_storage.provisioned_iops | 3000 | Hyperdisk only β provisioned IOPS (3,000β160,000). |
clickhouse_storage.provisioned_throughput | 140 | Hyperdisk only β throughput in MB/s (140β2,400). |
Two caveats:
- The StorageClass of an existing deployment is fixed. The ClickHouse StatefulSet's volume claim template is immutable β a deployment must stay on the class it was created with. Choose
pd-ssdvs Hyperdisk up front. - Check your quotas before raising Hyperdisk performance. Provisioned IOPS and throughput draw on per-region Hyperdisk quotas.
The ClickHouse volume survives
terraform destroy. Theclickhouse-premiumclass uses aRetainreclaim policy β deliberately, as the platform's data-loss guard β so destroying the deployment orphans the persistent disk rather than deleting it. Clean up (or re-attach) the disk manually after a destroy.
Network access
The Gateway's load balancer is internal, but by default accepts traffic from any source that can route to it. Restrict the sources to harden access:
| Input | Default | Description |
|---|---|---|
gateway.allowed_source_ranges | null (unrestricted) | CIDR ranges permitted to reach the Gateway β one combined allow-list covering both the ClickHouse and Collector endpoints. [] restricts to the VPC's own ranges; a list allows the VPC's ranges plus the listed CIDRs (the VPC ranges are always folded in, so a restriction never locks out in-cluster clients). |
cluster.master_authorized_networks | [] (unrestricted) | CIDRs allowed to reach the cluster API endpoint. Must include every machine that runs Terraform or kubectl β including CI and the module's own apply-time provisioners. |
cluster.enable_private_nodes | true | Nodes get no public IPs; egress runs through the module's Cloud NAT. |
cluster.enable_private_endpoint | false | Removes the cluster's public API endpoint entirely β all applies must then come over a private path. Can be combined with authorized networks (which then scope the private endpoint). |
The allow-list is implemented as a regional Cloud Armor security policy attached to the Gateway's backends, evaluated against the original client IP. Remember to include the Monte Carlo Agent's subnet CIDR when you restrict it.
gateway.allowed_source_rangesis the primary network control for reaching ClickHouse β the network-facing ClickHouse users accept connections from any source at the ClickHouse layer. This includes the passwordlessprobeuser (metadata-only access to replication health and table names) β one more reason to restrict the ranges. (Theadminsuperuser is the exception: it is loopback-only and never reachable through the Gateway.) Leaving the default unrestricted means anyone with a private path into the VPC can attempt to connect. Restrict the ranges especially before enablingreadonly_userfor external SQL clients.
TLS
TLS is enabled by default and is not optional on this platform. The OpenTelemetry Collector and ClickHouse endpoints are served by the internal GKE Gateway (gke-l7-rilb), which terminates publicly trusted Let's Encrypt certificates issued by cert-manager via DNS-01 validation against your Cloud DNS zone β "publicly trusted" describes the certificate chain, not exposure; the load balancer IP is private. cert-manager authenticates to Cloud DNS through its direct Workload Identity binding; no key files are involved. gateway.letsencrypt_email optionally sets the ACME contact (omitting it still auto-renews).
Behind the Gateway, traffic is re-encrypted to the backends: cert-manager maintains an in-cluster CA, and trust-manager (installed by the module) distributes it for the Gateway's backend TLS policy. Traffic between the Collector and ClickHouse inside the cluster is likewise secured with cert-manager-issued certificates.
Because the Gateway is a Layer-7 load balancer, two client-facing constraints apply: telemetry senders must use OTLP/HTTP (the Gateway does not carry OTLP/gRPC), and ClickHouse clients connect over HTTPS on port 8443 (not the native TCP protocol).
DNS-related settings:
| Input | Default | Description |
|---|---|---|
dns_zone | β (required with charts) | Cloud DNS managed-zone name serving both domains. |
dns_zone_project | cluster project | Project hosting the zone, when different β the module grants the DNS controllers zone-scoped access there. |
dns_zone_visibility | public | Let's Encrypt DNS-01 requires a public zone; the A records still point at the Gateway's private IP. |
Evaluation (Vertex AI)
The LLM worker invokes Claude models through Vertex AI. Access is serverless β the module grants the worker's Workload Identity a custom role containing the single predict permission it needs, rather than a broad Vertex role. The worker honors the model each evaluation row requests; the Claude models themselves must be enabled once in the Model Garden.
| Input | Default | Description |
|---|---|---|
helm.llm_worker.vertex_region | "global" | Vertex AI invocation region. The default routes to the global Claude endpoint; set a specific region to keep inference in-region. |
helm.llm_worker.vertex_project | deployment project | Project the Vertex calls bill to and are IAM-checked in, when different from the cluster's (the predict role and grant are created there). |
helm.llm_worker.image_repository | montecarlodata/ao-llm-worker | Override the LLM-worker image repository. |
helm.llm_worker.image_tag | latest-gcp | LLM-worker image tag. The default floats β pin to a published version for production; the current release is 1.1.0-gcp. |
Ingestion into the platform is OTLP push only β there is no object-storage (bucket) ingest path on GCP, and the Collector runs with no cloud identity at all.
Encryption (CMEK)
Customer-managed encryption keys are on by default for both sensitive stores, using Cloud KMS keys the module creates (90-day rotation) β or your own:
| Input | Default | Description |
|---|---|---|
database_encryption | { enabled = true, key = null } | GKE application-layer encryption of Kubernetes Secrets (etcd). Set key to bring your own Cloud KMS key β you then grant the GKE service agent access to it. |
secret_encryption | { enabled = true, key = null } | CMEK for the Secret Manager secrets holding the ClickHouse credentials. CMEK requires single-region secret replication, pinned to the cluster's region. |
Cluster version and upgrades
| Input | Default | Description |
|---|---|---|
release_channel | "UNSPECIFIED" | Static cluster β GKE never auto-upgrades nodes, so stateful ClickHouse/Keeper nodes only roll on your schedule. Setting a channel re-enables auto-upgrade cluster-wide. |
kubernetes_version | GKE default at create | Control-plane version pin for the static posture. |
Node and Kubernetes upgrades are customer-managed under the default posture β see High availability β node and Kubernetes upgrades.
Updated 1 day ago
