Agent Observability Platform: Self-Managed Helm Install (AWS)

Advanced: provision AWS infrastructure with Terraform and manage the Helm release yourself

☁️

This page covers AWS. Also available for Azure and GCP.

πŸ“˜

This is an advanced path. Most deployments should let Terraform deploy the chart in a single pass β€” see Installation. Use this path only if you need to manage chart upgrades and values on your own cadence.

When to use this

Set helm.deploy_charts = false to have Terraform provision the AWS infrastructure and cluster controllers only, without deploying the ao-data-platform chart. You then install and manage the Helm release yourself.

module "ao_data_platform" {
  source  = "monte-carlo-data/ao-data-platform/aws"
  version = "2.3.0"

  region = "us-east-1"

  helm = {
    deploy_charts = false
  }
}

Terraform still creates the IRSA roles, ACM certificates, KMS key, and Secrets Manager secrets β€” and applies the ClusterSecretStore (named aws-secrets-manager) that routes ExternalSecrets to AWS Secrets Manager. You wire their values into your Helm release (see Configure values). The Terraform-managed path is the reference implementation: when self-managing, you reproduce the same values it would otherwise pass to the chart.

❗️

Several things the module provisions only when it manages the release are not created on this path β€” supply them yourself:

  • NLB source-range restriction β€” *_nlb_allowed_source_ranges are applied as chart Service annotations, so on this path they are not set. Add service.beta.kubernetes.io/load-balancer-source-ranges to the ClickHouse and Collector Services in your values, or the internal NLBs stay reachable from any in-VPC source.
  • Dedicated per-AZ ClickHouse/Keeper node groups (for the HA topology) β€” attach the tainted node groups yourself and wire the matching nodeSelector/tolerations. See High availability β€” existing-cluster deployments.
  • Bedrock model access β€” enable it per region as in Prerequisites (the worker's IRSA role is created, but model access is not).

1. Apply the infrastructure

Run terraform apply with helm.deploy_charts = false, then collect the outputs you will need for the chart:

terraform output

Keep these handy β€” the mapping table below shows where each goes:
otel_collector_irsa_role_arn, llm_worker_irsa_role_arn, otel_collector_certificate_arn, clickhouse_certificate_arn, and the Secrets Manager secrets behind clickhouse_otel_credentials_secret_arn, clickhouse_schema_owner_credentials_secret_arn, clickhouse_llm_worker_credentials_secret_arn, and clickhouse_monte_carlo_credentials_secret_arn (plus the optional clickhouse_readonly_user_credentials_secret_arn / clickhouse_admin_credentials_secret_arn).

2. Confirm cluster prerequisites

The chart expects these to be present in the cluster (Terraform installs them unless you disabled the corresponding helm.install_* flag):

  • Helm 3
  • cert-manager β€” internal Collector↔ClickHouse TLS is always enabled (tls.enabled = true) and requires it
  • External Secrets Operator with a ClusterSecretStore configured for AWS Secrets Manager (the module creates one named aws-secrets-manager, even with deploy_charts = false)
  • AWS Load Balancer Controller β€” for the NLB Services
  • Private subnets tagged kubernetes.io/role/internal-elb: 1

3. Configure values

⚠️

Chart 4.0.0 deploys the HA shape by default β€” 2 ClickHouse replicas and 3 Keeper voters, each with a hard one-per-AZ spread constraint. On a cluster that can't satisfy that (single-AZ, or without per-AZ node groups standing by), pods stay Pending. For a single-node deployment set clickhouse.replicasCount: 1 and keeper.replicasCount: 1. Upgrading an install with pre-3.0.0 data: pin clickhouse.replicasCount: 1 and contact your Monte Carlo representative before raising it β€” a second replica created against unconverted tables starts empty. See High availability.

At minimum, point each ClickHouse user's externalSecret at the AWS Secrets Manager secret holding its password (the ones Terraform created), wire in the IRSA roles and ACM certificates, set the ClickHouse hostname, and set the worker image. The chart uses a per-user externalSecret under each clickhouse.<user> key β€” the always-on otel, schemaOwner, llmWorker, and monteCarlo users. Every one of these four users needs both externalSecret.secretStoreRef.name and externalSecret.remoteRef.key set β€” the chart fails to render if any is missing. (The always-on probe user is passwordless and needs no wiring.) The chart's keys map to the Terraform outputs as follows:

Terraform outputChart value
clickhouse_otel_credentials_secret_arn (the Secrets Manager secret it points to)clickhouse.otel.externalSecret.remoteRef.key
clickhouse_schema_owner_credentials_secret_arnclickhouse.schemaOwner.externalSecret.remoteRef.key
clickhouse_llm_worker_credentials_secret_arnclickhouse.llmWorker.externalSecret.remoteRef.key
clickhouse_monte_carlo_credentials_secret_arnclickhouse.monteCarlo.externalSecret.remoteRef.key
otel_collector_irsa_role_arnopentelemetry-collector.serviceAccount.annotations."eks.amazonaws.com/role-arn"
llm_worker_irsa_role_arnllmWorker.serviceAccount.annotations."eks.amazonaws.com/role-arn"
clickhouse_certificate_arnclickhouse.service.annotations (NLB TLS termination)
otel_collector_certificate_arnopentelemetry-collector.service.annotations (NLB TLS termination)

A minimal values.yaml covering the required wiring:

clickhouse:
  hostname: clickhouse.acme.com        # adds the external-dns hostname annotation
  storageClass: clickhouse-gp3         # or your own StorageClass
  storageSize: 500Gi                   # chart default is 100Gi; the Terraform-managed path passes 500Gi
  replicasCount: 2                     # chart default; 1 for a single-node deployment
  # Each always-on user reads its password from its own AWS Secrets Manager secret.
  # secretStoreRef.name is REQUIRED per user (render fails without it); its kind defaults
  # to ClusterSecretStore. The module's store is named aws-secrets-manager. remoteRef.key
  # is the AWS Secrets Manager secret name β€” the module names them
  # <cluster_name>/clickhouse/<user>-credentials (each is also exposed as a
  # clickhouse_<user>_credentials_secret_arn output).
  otel:
    externalSecret:
      secretStoreRef:
        name: aws-secrets-manager
      remoteRef:
        key: <cluster_name>/clickhouse/otel-credentials
  schemaOwner:
    externalSecret:
      secretStoreRef:
        name: aws-secrets-manager
      remoteRef:
        key: <cluster_name>/clickhouse/schema-owner-credentials
  llmWorker:
    externalSecret:
      secretStoreRef:
        name: aws-secrets-manager
      remoteRef:
        key: <cluster_name>/clickhouse/llm-worker-credentials
  monteCarlo:
    externalSecret:
      secretStoreRef:
        name: aws-secrets-manager
      remoteRef:
        key: <cluster_name>/clickhouse/monte-carlo-credentials
  # If you run dedicated ClickHouse node groups, match their label/taint:
  nodeSelector:
    dedicated: clickhouse
  tolerations:
    - key: dedicated
      operator: Equal
      value: clickhouse
      effect: NoSchedule

keeper:
  replicasCount: 3                     # chart default; odd count (Raft quorum); 1 for a single-node deployment
  # storageClass: gp3
  # storageSize: 10Gi
  # If you run dedicated Keeper node groups, match their label/taint:
  nodeSelector:
    dedicated: keeper
  tolerations:
    - key: dedicated
      operator: Equal
      value: keeper
      effect: NoSchedule

tls:
  enabled: true                        # requires cert-manager
  certManager:
    createCA: true                     # or set existingIssuerRef to reuse an Issuer

llmWorker:
  # replicaCount is 0 or 1 only β€” 0 pauses the worker declaratively; the chart rejects >1.
  # provider defaults to "bedrock" (the AWS backend) β€” leave it unset on AWS.
  image:
    repository: montecarlodata/ao-llm-worker   # required β€” the chart has no default
    tag: "1.1.0-aws"                   # per-cloud tag; pin a release (latest-aws is the floating tag)
  aws:
    region: us-east-1                  # Bedrock region
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: <llm_worker_irsa_role_arn>

To enable the optional readonly_user (external SQL clients) or admin (gated break-glass superuser, loopback-only by default), set clickhouse.readonlyUser.enabled: true / clickhouse.admin.enabled: true and configure their externalSecret the same way as the always-on users above. Point each at the matching clickhouse_readonly_user_credentials_secret_arn / clickhouse_admin_credentials_secret_arn secret. To tighten the ingest user to INSERT-only after Monte Carlo connects as monte_carlo, set clickhouse.otel.restrictGrants: true.

πŸ“˜

Do not add a PodDisruptionBudget for the ClickHouse pods β€” the operator creates one (maxUnavailable: 1), and a pod matched by more than one PDB cannot be evicted at all, which blocks node drains outright.

4. Install the chart

Install the published chart from Docker Hub into the montecarlo namespace. Its subchart dependencies β€” the Altinity ClickHouse operator and the OpenTelemetry Collector chart β€” are bundled in the published OCI artifact, so no separate helm dependency build is needed:

helm install ao-data-platform oci://registry-1.docker.io/montecarlodata/ao-data-platform \
  --version 4.0.0 \
  -n montecarlo --create-namespace \
  -f values.yaml

5. Verify

Confirm the components are healthy using the checklist in Connect to Monte Carlo, then continue with the credential handoff there.


Did this page help you?