Agent Observability Platform: Self-Managed Helm Install (Azure)
Install the Agent Observability data platform Helm chart yourself on Azure
Advanced: provision infrastructure with Terraform and manage the Helm release yourself.
This is an advanced path. Most deployments should let Terraform deploy the chart in a single pass β see Installation. Use this path only if you need to manage chart upgrades and values on your own cadence, or to run the platform on an existing cluster with the HA topology.
When to use this
Set helm.deploy_charts = false to have Terraform provision the Azure infrastructure and core cluster controllers only, without deploying the ao-data-platform chart. You then install and manage the Helm release yourself.
module "ao_data_platform" {
source = "monte-carlo-data/ao-data-platform/azurerm"
version = "~> 1.0"
location = "eastus"
resource_group_name = "ao-data-platform"
helm = {
deploy_charts = false
}
}On this path Terraform still creates the AKS cluster (or targets your existing one), the VNet with NAT-gateway egress, the Key Vault with a secret per ClickHouse user, cert-manager, and the External Secrets Operator with a ClusterSecretStore named azure-keyvault wired to that Key Vault β plus external-dns when dns_zone is set. You wire the resulting values into your Helm release (see Configure values).
Several pieces the Terraform-managed path provisions are tied to the module-managed chart release and are not created on this path β you provide them yourself:
- The Gateway API and app-routing-Istio add-ons on the cluster (the module enables them only when it deploys the chart):
az aks update -g <rg> -n <cluster> --enable-gateway-api --enable-app-routing-istio.- trust-manager (with its trust namespace set to the release namespace) β the Gateway's
BackendTLSPolicyre-encrypt hop needs it.- A cert-manager Workload Identity with DNS Zone Contributor on your DNS zone, for the Let's Encrypt DNS-01 solver.
- An LLM-worker Workload Identity federated to the
llm-workerServiceAccount in the release namespace, and a Microsoft Foundry (Claude) resource with model deployments that identity may invoke β the eval backend the module would otherwise provision (see Evaluation for its shape).
The Terraform-managed path is the reference implementation: when self-managing, you reproduce the same values it would otherwise pass to the chart.
1. Apply the infrastructure
Run terraform apply with helm.deploy_charts = false, then collect the outputs you will need for the chart:
terraform outputKeep these handy β the mapping table below shows where each goes: key_vault_uri, oidc_issuer_url, montecarlo_namespace, and the Key Vault secrets behind clickhouse_otel_credentials_secret_id, clickhouse_schema_owner_credentials_secret_id, clickhouse_llm_worker_credentials_secret_id, and clickhouse_monte_carlo_credentials_secret_id (plus the optional clickhouse_readonly_user_credentials_secret_id / clickhouse_admin_credentials_secret_id).
2. Confirm cluster prerequisites
The chart expects these to be present in the cluster:
- Helm 3
- The Gateway API and app-routing-Istio add-ons (GatewayClass
approuting-istio) β enable them if you haven't:az aks update -g <rg> -n <cluster> --enable-gateway-api --enable-app-routing-istio - cert-manager β internal TLS is always enabled (
tls.enabled = true) and requires it (Terraform installs it unless you sethelm.install_cert_manager = false) - trust-manager with its trust namespace set to the release namespace (
montecarlo) - External Secrets Operator with a
ClusterSecretStorefor Azure Key Vault β Terraform installs both (store nameazure-keyvault) unless you sethelm.install_external_secrets_operator = false - A Workload Identity for cert-manager's DNS-01 solver with DNS Zone Contributor on the DNS zone
- A Workload Identity for the LLM worker (federated to ServiceAccount
llm-workerin the release namespace, via the cluster'soidc_issuer_urloutput) with inference access to your Foundry (Claude) resource
The Altinity ClickHouse operator is a bundled subchart β do not install it separately; its CRDs install with the chart.
3. Configure values
Chart 4.0.0 deploys the HA shape by default β 2 ClickHouse replicas and 3 Keeper voters, each with a hard one-per-zone spread constraint. On a cluster that can't satisfy that (single-zone, or without per-zone node pools standing by), pods stay
Pending. For a single-node deployment setclickhouse.replicasCount: 1andkeeper.replicasCount: 1. See High availability.
At minimum, point each ClickHouse user's externalSecret at the Key Vault secret holding its password (the ones Terraform created), configure the Gateway block (provider, hostnames, and the Let's Encrypt DNS-01 solver), and configure the LLM worker's Foundry backend. The chart uses a per-user externalSecret under each clickhouse.<user> key β the always-on otel, schemaOwner, llmWorker, and monteCarlo users. (The always-on probe user is passwordless and needs no wiring.) The Terraform-created values map to the chart's keys as follows:
| Terraform output / resource | Chart value |
|---|---|
clickhouse_otel_credentials_secret_id (Key Vault secret name: clickhouse-otel) | clickhouse.otel.externalSecret.remoteRef.key |
clickhouse_schema_owner_credentials_secret_id (clickhouse-schema-owner) | clickhouse.schemaOwner.externalSecret.remoteRef.key |
clickhouse_llm_worker_credentials_secret_id (clickhouse-llm-worker) | clickhouse.llmWorker.externalSecret.remoteRef.key |
clickhouse_monte_carlo_credentials_secret_id (clickhouse-monte-carlo) | clickhouse.monteCarlo.externalSecret.remoteRef.key |
The module's ClusterSecretStore (azure-keyvault, backed by key_vault_uri) | clickhouse.<user>.externalSecret.secretStoreRef.name |
| Your cert-manager Workload Identity's client ID | gateway.tls.letsencrypt.azureDNS.managedIdentityClientID |
| Your LLM-worker Workload Identity's client ID | llmWorker.serviceAccount.annotations."azure.workload.identity/client-id" |
| Your Foundry account name | llmWorker.foundry.resource |
For the Azure Key Vault provider, remoteRef.key is the secret name in the vault, not the full secret ID.
A minimal values.yaml covering the required wiring:
gateway:
enabled: true
provider: azure # derives GatewayClass approuting-istio
otelHostname: otel.acme.com
clickhouseHostname: clickhouse.acme.com
# allowedSourceRanges: [] # optional CIDR allow-list on the internal LB
tls:
source: letsencrypt
letsencrypt:
email: "" # optional ACME contact
azureDNS:
hostedZoneName: acme.com
resourceGroupName: dns # the zone's resource group
subscriptionID: <subscription-id>
managedIdentityClientID: <cert-manager-WI-client-id>
clickhouse:
storageClass: clickhouse-premium-v2 # created by the module; or your own StorageClass
storageSize: 500Gi
replicasCount: 2 # 1 for a single-node deployment
# Each always-on user reads its password from its own Key Vault secret via the
# module's azure-keyvault ClusterSecretStore; remoteRef.key is the secret NAME.
otel:
restrictGrants: true # INSERT-only ingest user (the module's default)
externalSecret:
secretStoreRef:
name: azure-keyvault
kind: ClusterSecretStore
remoteRef:
key: clickhouse-otel
schemaOwner:
externalSecret:
secretStoreRef:
name: azure-keyvault
kind: ClusterSecretStore
remoteRef:
key: clickhouse-schema-owner
llmWorker:
externalSecret:
secretStoreRef:
name: azure-keyvault
kind: ClusterSecretStore
remoteRef:
key: clickhouse-llm-worker
monteCarlo:
externalSecret:
secretStoreRef:
name: azure-keyvault
kind: ClusterSecretStore
remoteRef:
key: clickhouse-monte-carlo
# If you run dedicated ClickHouse node pools, match their label/taint:
nodeSelector:
dedicated: clickhouse
tolerations:
- key: dedicated
operator: Equal
value: clickhouse
effect: NoSchedule
keeper:
replicasCount: 3 # odd count (Raft quorum); 1 for a single-node deployment
# storageClass: managed-csi
# storageSize: 10Gi
# If you run dedicated Keeper node pools, match their label/taint:
nodeSelector:
dedicated: keeper
tolerations:
- key: dedicated
operator: Equal
value: keeper
effect: NoSchedule
tls:
enabled: true # required in gateway mode
certManager:
createCA: true # required in gateway mode
llmWorker:
provider: foundry
foundry:
resource: <foundry-account-name> # the worker's ANTHROPIC_FOUNDRY_RESOURCE
image:
repository: montecarlodata/ao-llm-worker
tag: "1.1.0-azure"
serviceAccount:
annotations:
azure.workload.identity/client-id: <llm-worker-WI-client-id>
podLabels:
azure.workload.identity/use: "true"The Gateway path is L7 HTTP(S) only: telemetry senders use OTLP over HTTP (port 4318 semantics β no gRPC), and ClickHouse is served over HTTPS on port 8443 (not 443, and not the native TCP protocol).
To enable the optional readonly_user (external SQL clients) or admin (gated break-glass superuser, loopback-only by default), set clickhouse.readonlyUser.enabled: true / clickhouse.admin.enabled: true and configure their externalSecret the same way as the always-on users above β Key Vault secret names clickhouse-readonly-user and clickhouse-admin (note the module creates those secrets only when the matching helm.clickhouse.readonly_user / helm.clickhouse.admin input is enabled).
Do not add a PodDisruptionBudget for the ClickHouse pods β the operator creates one (
maxUnavailable: 1), and a pod matched by more than one PDB cannot be evicted at all, which blocks node drains outright.
4. Install the chart
Install the published chart from Docker Hub into the montecarlo namespace. Its subchart dependencies β the Altinity ClickHouse operator and the OpenTelemetry Collector chart β are bundled in the published OCI artifact, so no separate helm dependency build is needed:
helm install ao-data-platform oci://registry-1.docker.io/montecarlodata/ao-data-platform \
--version 4.0.0 \
-n montecarlo --create-namespace \
-f values.yaml5. Verify
Confirm the components are healthy using the checklist in Connect to Monte Carlo β including the Gateway, HTTPRoute, and Gateway certificate checks β then continue with the credential handoff there.
Updated 1 day ago
