Agent Observability Platform: Self-Managed Helm Install (Azure)

Install the Agent Observability data platform Helm chart yourself on Azure

☁️

This page covers Azure. Also available for AWS and GCP.

Advanced: provision infrastructure with Terraform and manage the Helm release yourself.

πŸ“˜

This is an advanced path. Most deployments should let Terraform deploy the chart in a single pass β€” see Installation. Use this path only if you need to manage chart upgrades and values on your own cadence, or to run the platform on an existing cluster with the HA topology.

When to use this

Set helm.deploy_charts = false to have Terraform provision the Azure infrastructure and core cluster controllers only, without deploying the ao-data-platform chart. You then install and manage the Helm release yourself.

module "ao_data_platform" {
  source  = "monte-carlo-data/ao-data-platform/azurerm"
  version = "~> 1.0"

  location            = "eastus"
  resource_group_name = "ao-data-platform"

  helm = {
    deploy_charts = false
  }
}

On this path Terraform still creates the AKS cluster (or targets your existing one), the VNet with NAT-gateway egress, the Key Vault with a secret per ClickHouse user, cert-manager, and the External Secrets Operator with a ClusterSecretStore named azure-keyvault wired to that Key Vault β€” plus external-dns when dns_zone is set. You wire the resulting values into your Helm release (see Configure values).

❗️

Several pieces the Terraform-managed path provisions are tied to the module-managed chart release and are not created on this path β€” you provide them yourself:

  • The Gateway API and app-routing-Istio add-ons on the cluster (the module enables them only when it deploys the chart): az aks update -g <rg> -n <cluster> --enable-gateway-api --enable-app-routing-istio.
  • trust-manager (with its trust namespace set to the release namespace) β€” the Gateway's BackendTLSPolicy re-encrypt hop needs it.
  • A cert-manager Workload Identity with DNS Zone Contributor on your DNS zone, for the Let's Encrypt DNS-01 solver.
  • An LLM-worker Workload Identity federated to the llm-worker ServiceAccount in the release namespace, and a Microsoft Foundry (Claude) resource with model deployments that identity may invoke β€” the eval backend the module would otherwise provision (see Evaluation for its shape).

The Terraform-managed path is the reference implementation: when self-managing, you reproduce the same values it would otherwise pass to the chart.

1. Apply the infrastructure

Run terraform apply with helm.deploy_charts = false, then collect the outputs you will need for the chart:

terraform output

Keep these handy β€” the mapping table below shows where each goes: key_vault_uri, oidc_issuer_url, montecarlo_namespace, and the Key Vault secrets behind clickhouse_otel_credentials_secret_id, clickhouse_schema_owner_credentials_secret_id, clickhouse_llm_worker_credentials_secret_id, and clickhouse_monte_carlo_credentials_secret_id (plus the optional clickhouse_readonly_user_credentials_secret_id / clickhouse_admin_credentials_secret_id).

2. Confirm cluster prerequisites

The chart expects these to be present in the cluster:

  • Helm 3
  • The Gateway API and app-routing-Istio add-ons (GatewayClass approuting-istio) β€” enable them if you haven't: az aks update -g <rg> -n <cluster> --enable-gateway-api --enable-app-routing-istio
  • cert-manager β€” internal TLS is always enabled (tls.enabled = true) and requires it (Terraform installs it unless you set helm.install_cert_manager = false)
  • trust-manager with its trust namespace set to the release namespace (montecarlo)
  • External Secrets Operator with a ClusterSecretStore for Azure Key Vault β€” Terraform installs both (store name azure-keyvault) unless you set helm.install_external_secrets_operator = false
  • A Workload Identity for cert-manager's DNS-01 solver with DNS Zone Contributor on the DNS zone
  • A Workload Identity for the LLM worker (federated to ServiceAccount llm-worker in the release namespace, via the cluster's oidc_issuer_url output) with inference access to your Foundry (Claude) resource

The Altinity ClickHouse operator is a bundled subchart β€” do not install it separately; its CRDs install with the chart.

3. Configure values

⚠️

Chart 4.0.0 deploys the HA shape by default β€” 2 ClickHouse replicas and 3 Keeper voters, each with a hard one-per-zone spread constraint. On a cluster that can't satisfy that (single-zone, or without per-zone node pools standing by), pods stay Pending. For a single-node deployment set clickhouse.replicasCount: 1 and keeper.replicasCount: 1. See High availability.

At minimum, point each ClickHouse user's externalSecret at the Key Vault secret holding its password (the ones Terraform created), configure the Gateway block (provider, hostnames, and the Let's Encrypt DNS-01 solver), and configure the LLM worker's Foundry backend. The chart uses a per-user externalSecret under each clickhouse.<user> key β€” the always-on otel, schemaOwner, llmWorker, and monteCarlo users. (The always-on probe user is passwordless and needs no wiring.) The Terraform-created values map to the chart's keys as follows:

Terraform output / resourceChart value
clickhouse_otel_credentials_secret_id (Key Vault secret name: clickhouse-otel)clickhouse.otel.externalSecret.remoteRef.key
clickhouse_schema_owner_credentials_secret_id (clickhouse-schema-owner)clickhouse.schemaOwner.externalSecret.remoteRef.key
clickhouse_llm_worker_credentials_secret_id (clickhouse-llm-worker)clickhouse.llmWorker.externalSecret.remoteRef.key
clickhouse_monte_carlo_credentials_secret_id (clickhouse-monte-carlo)clickhouse.monteCarlo.externalSecret.remoteRef.key
The module's ClusterSecretStore (azure-keyvault, backed by key_vault_uri)clickhouse.<user>.externalSecret.secretStoreRef.name
Your cert-manager Workload Identity's client IDgateway.tls.letsencrypt.azureDNS.managedIdentityClientID
Your LLM-worker Workload Identity's client IDllmWorker.serviceAccount.annotations."azure.workload.identity/client-id"
Your Foundry account namellmWorker.foundry.resource

For the Azure Key Vault provider, remoteRef.key is the secret name in the vault, not the full secret ID.

A minimal values.yaml covering the required wiring:

gateway:
  enabled: true
  provider: azure                        # derives GatewayClass approuting-istio
  otelHostname: otel.acme.com
  clickhouseHostname: clickhouse.acme.com
  # allowedSourceRanges: []              # optional CIDR allow-list on the internal LB
  tls:
    source: letsencrypt
    letsencrypt:
      email: ""                          # optional ACME contact
      azureDNS:
        hostedZoneName: acme.com
        resourceGroupName: dns           # the zone's resource group
        subscriptionID: <subscription-id>
        managedIdentityClientID: <cert-manager-WI-client-id>

clickhouse:
  storageClass: clickhouse-premium-v2    # created by the module; or your own StorageClass
  storageSize: 500Gi
  replicasCount: 2                       # 1 for a single-node deployment
  # Each always-on user reads its password from its own Key Vault secret via the
  # module's azure-keyvault ClusterSecretStore; remoteRef.key is the secret NAME.
  otel:
    restrictGrants: true                 # INSERT-only ingest user (the module's default)
    externalSecret:
      secretStoreRef:
        name: azure-keyvault
        kind: ClusterSecretStore
      remoteRef:
        key: clickhouse-otel
  schemaOwner:
    externalSecret:
      secretStoreRef:
        name: azure-keyvault
        kind: ClusterSecretStore
      remoteRef:
        key: clickhouse-schema-owner
  llmWorker:
    externalSecret:
      secretStoreRef:
        name: azure-keyvault
        kind: ClusterSecretStore
      remoteRef:
        key: clickhouse-llm-worker
  monteCarlo:
    externalSecret:
      secretStoreRef:
        name: azure-keyvault
        kind: ClusterSecretStore
      remoteRef:
        key: clickhouse-monte-carlo
  # If you run dedicated ClickHouse node pools, match their label/taint:
  nodeSelector:
    dedicated: clickhouse
  tolerations:
    - key: dedicated
      operator: Equal
      value: clickhouse
      effect: NoSchedule

keeper:
  replicasCount: 3                       # odd count (Raft quorum); 1 for a single-node deployment
  # storageClass: managed-csi
  # storageSize: 10Gi
  # If you run dedicated Keeper node pools, match their label/taint:
  nodeSelector:
    dedicated: keeper
  tolerations:
    - key: dedicated
      operator: Equal
      value: keeper
      effect: NoSchedule

tls:
  enabled: true                          # required in gateway mode
  certManager:
    createCA: true                       # required in gateway mode

llmWorker:
  provider: foundry
  foundry:
    resource: <foundry-account-name>     # the worker's ANTHROPIC_FOUNDRY_RESOURCE
  image:
    repository: montecarlodata/ao-llm-worker
    tag: "1.1.0-azure"
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <llm-worker-WI-client-id>
  podLabels:
    azure.workload.identity/use: "true"

The Gateway path is L7 HTTP(S) only: telemetry senders use OTLP over HTTP (port 4318 semantics β€” no gRPC), and ClickHouse is served over HTTPS on port 8443 (not 443, and not the native TCP protocol).

To enable the optional readonly_user (external SQL clients) or admin (gated break-glass superuser, loopback-only by default), set clickhouse.readonlyUser.enabled: true / clickhouse.admin.enabled: true and configure their externalSecret the same way as the always-on users above β€” Key Vault secret names clickhouse-readonly-user and clickhouse-admin (note the module creates those secrets only when the matching helm.clickhouse.readonly_user / helm.clickhouse.admin input is enabled).

πŸ“˜

Do not add a PodDisruptionBudget for the ClickHouse pods β€” the operator creates one (maxUnavailable: 1), and a pod matched by more than one PDB cannot be evicted at all, which blocks node drains outright.

4. Install the chart

Install the published chart from Docker Hub into the montecarlo namespace. Its subchart dependencies β€” the Altinity ClickHouse operator and the OpenTelemetry Collector chart β€” are bundled in the published OCI artifact, so no separate helm dependency build is needed:

helm install ao-data-platform oci://registry-1.docker.io/montecarlodata/ao-data-platform \
  --version 4.0.0 \
  -n montecarlo --create-namespace \
  -f values.yaml

5. Verify

Confirm the components are healthy using the checklist in Connect to Monte Carlo β€” including the Gateway, HTTPRoute, and Gateway certificate checks β€” then continue with the credential handoff there.


Did this page help you?