--- updatedAt: 2026-09-18T15:29:59.000Z agentTools: projectIndex: https://docs.getmontecarlo.com/llms.txt --- # Generic Agent: Object Storage Configure object storage for the Generic Agent on any platform The Generic Agent requires an object storage bucket for [data samples](https://docs.getmontecarlo.com/docs/configure-data-sampling) and temporary data written during operations. This storage is private and accessed only by the agent. The [AWS](https://docs.getmontecarlo.com/docs/aws-eks), [Azure](https://docs.getmontecarlo.com/docs/azure-aks), and [GCP](https://docs.getmontecarlo.com/docs/gcp-gke) Terraform modules create the bucket and grant the agent access automatically. This guide is for deployments that configure storage manually — the [Kubernetes](https://docs.getmontecarlo.com/docs/kubernetes) and [Docker Compose](https://docs.getmontecarlo.com/docs/docker-compose) guides. # Options | Storage | `storageType` | Credentials | Best for | | :-------------------------------------------------------------------------------- | :-------------- | :---------------------------------------- | :------------------------------------------------------- | | [Amazon S3](https://docs.getmontecarlo.com/docs/object-storage#amazon-s3) | `S3` | EKS Pod Identity or access keys | EKS, or any cluster with access to an AWS account | | [Azure Blob Storage](https://docs.getmontecarlo.com/docs/object-storage#azure-blob-storage) | `AZURE_BLOB` | Workload Identity, or a service principal | AKS, or any cluster with access to an Azure subscription | | [Google Cloud Storage](https://docs.getmontecarlo.com/docs/object-storage#google-cloud-storage-gke) | `GCS` | Workload Identity | GKE | | [MinIO / S3-compatible](https://docs.getmontecarlo.com/docs/object-storage#minio-and-other-s3-compatible-storage) | `S3_COMPATIBLE` | Access key and secret | On-premises clusters, development, and testing | The agent writes all objects under the `mcd/` prefix. Set `MCD_STORAGE_PREFIX` via `container.extraEnv` to change it, or to an empty string to write at the bucket root. # Amazon S3 ## 1. Create the bucket ```bash aws s3 mb s3://mcd-agent-store --region us-west-2 ``` Block public access and enable default encryption: ```bash aws s3api put-public-access-block \ --bucket mcd-agent-store \ --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true aws s3api put-bucket-encryption \ --bucket mcd-agent-store \ --server-side-encryption-configuration \ '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}' ``` ### Using a KMS key The agent never specifies an encryption method when it writes an object, so whatever you configure as the bucket's **default encryption** is what gets applied. To encrypt with a customer-managed KMS key, set it as the bucket default: ```bash aws s3api put-bucket-encryption \ --bucket mcd-agent-store \ --server-side-encryption-configuration \ '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":""},"BucketKeyEnabled":true}]}' ``` The agent's identity then also needs access to the key — add this statement to the role you create in the next step, and make sure the key policy allows that role (directly, or by delegating to IAM policies in the account): ```json { "Effect": "Allow", "Action": ["kms:Decrypt", "kms:GenerateDataKey"], "Resource": "" } ``` Both actions are required: `kms:GenerateDataKey` to write objects (including multipart uploads) and `kms:Decrypt` to read them back. The agent never copies or re-encrypts objects, so `kms:Encrypt` and `kms:ReEncrypt*` — listed in the equivalent [AWS data store](https://docs.getmontecarlo.com/docs/direct-connection-with-an-aws-data-store#can-i-use-server-side-encryption-with-aws-kms-sse-kms) policy — are not needed here. ## 2. Grant the agent access The agent needs the following permissions on the bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketPublicAccessBlock", "s3:GetBucketPolicyStatus", "s3:GetBucketAcl" ], "Resource": [ "arn:aws:s3:::mcd-agent-store", "arn:aws:s3:::mcd-agent-store/*" ] } ] } ``` Attach the policy above to a role, then choose one of the following: Requires the `eks-pod-identity-agent` add-on on the cluster. Create a role that the EKS service can assume: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": ["sts:AssumeRole", "sts:TagSession"] } ] } ``` Associate it with the agent's service account: ```bash aws eks create-pod-identity-association \ --cluster-name \ --namespace mcd-agent \ --service-account mcd-agent-service-account \ --role-arn ``` No `values.yaml` change is needed — the association is matched by namespace and service account name. Install the add-on before creating the association. The webhook injects `AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials` as soon as an association exists, but that address is served by the add-on's own pod — so with the association in place and the add-on missing, AWS calls fail as `Connect timeout on endpoint URL: "http://169.254.170.23/v1/credentials"` rather than as an access error: ```bash aws eks create-addon --cluster-name --addon-name eks-pod-identity-agent ``` If the add-on is already installed, check that it scheduled a pod onto the same node as the agent. The endpoint is node-local, so a taint or selector keeping the add-on off that node gives an identical timeout. Requires an IAM OIDC identity provider registered for the cluster. Clusters created with `eksctl create cluster --with-oidc` already have one; otherwise create it with `eksctl utils associate-iam-oidc-provider --cluster --approve`. Save the bucket policy above as `policy.json` and create it as a managed policy: ```bash aws iam create-policy --policy-name mcd-agent-s3-access \ --policy-document file://policy.json ``` Then create the role, using the ARN returned above: ```bash eksctl create iamserviceaccount \ --cluster \ --namespace mcd-agent \ --name mcd-agent-service-account \ --role-only \ --role-name mcd-agent-role \ --attach-policy-arn \ --approve ``` `--role-only` is important: without it `eksctl` also creates the service account, which then conflicts with the one the Helm chart creates. Annotate the agent's service account with the role ARN: ```yaml serviceAccount: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/mcd-agent-role" ``` If this service account previously used Pod Identity, delete the association as well. The two are not additive: the `AWS_CONTAINER_CREDENTIALS_FULL_URI` that Pod Identity injects takes precedence over IRSA's web-identity token in the AWS credential chain, so adding the annotation on its own changes nothing. Make sure `--namespace` matches the namespace you deploy the agent into — the `namespace` value in your `values.yaml`. The trust policy binds the role to that namespace and service account name, and a mismatch fails with `Not authorized to perform sts:AssumeRoleWithWebIdentity` rather than an access-denied error on the bucket. If your IAM resources are provisioned outside `eksctl` — Terraform, CloudFormation, or the console — the role needs this trust policy. The issuer host comes from `aws eks describe-cluster --name --query 'cluster.identity.oidc.issuer' --output text`, and the namespace in the `:sub` condition must be the one you deploy the agent into (`mcd-agent` below): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks..amazonaws.com/id/:sub": "system:serviceaccount:mcd-agent:mcd-agent-service-account", "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com" } } } ] } ``` For clusters outside AWS, create an IAM user with the policy above and pass its credentials as environment variables. Store them in a Kubernetes secret rather than inline: ```bash kubectl create secret generic mcd-agent-aws-credentials -n mcd-agent \ --from-literal=AWS_ACCESS_KEY_ID= \ --from-literal=AWS_SECRET_ACCESS_KEY= ``` ```yaml container: extraEnv: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: mcd-agent-aws-credentials key: AWS_ACCESS_KEY_ID - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: mcd-agent-aws-credentials key: AWS_SECRET_ACCESS_KEY - name: AWS_DEFAULT_REGION value: "" ``` `container.storageAccessKey` and `container.storageSecretKey` do **not** apply here — those values only configure `S3_COMPATIBLE` storage. With `storageType: "S3"` the agent reads the standard `AWS_*` environment variables. The agent uses the standard AWS credential chain, so any other mechanism your platform already standardizes on also works. The same role can serve both the bucket and [reading the agent's authentication credential from AWS Secrets Manager](https://docs.getmontecarlo.com/docs/secrets-management#aws-secrets-manager-direct) — add both policies to it rather than creating a second role. ## 3. Configure the agent ```yaml container: storageType: "S3" storageBucketName: "mcd-agent-store" ``` ## 4. Expire temporary objects (optional) The agent does not delete every object it writes. These lifecycle rules match what the Terraform module configures: | Prefix | Expiration | | :-------------- | :--------- | | `mcd/` | 90 days | | `mcd/tmp` | 2 days | | `mcd/responses` | 1 day | Then [verify storage access](https://docs.getmontecarlo.com/docs/object-storage#verify-storage-access). # Azure Blob Storage The agent authenticates to Blob Storage with a [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/fundamentals/whatis) identity, in one of two ways: * **Workload Identity** — a user-assigned managed identity federated with the agent's Kubernetes service account. Use this on **AKS**, where it requires no secret at all. * **A service principal** — client ID and secret supplied as environment variables. Use this anywhere else: an on-premises cluster, or a Kubernetes distribution outside Azure, where there is no managed identity to federate with. Either way the agent holds no storage account key, so the account can keep shared key access disabled. Both options need outbound access to `login.microsoftonline.com`. If your deployment cannot reach it, see [if the agent cannot reach Entra ID](https://docs.getmontecarlo.com/docs/object-storage#if-the-agent-cannot-reach-entra-id). ## 1. Create the storage account and container ```bash az storage account create \ --name \ --resource-group \ --sku Standard_LRS \ --https-only true \ --min-tls-version TLS1_2 \ --allow-blob-public-access false \ --allow-shared-key-access false ``` ```bash az storage container create \ --name mcdstore \ --account-name \ --auth-mode login ``` Blob Storage encrypts data at rest by default. If you configure a customer-managed key on the storage account, the agent needs no additional permissions — the storage account uses its own identity to reach the key. ## 2. Grant the agent access Requires the cluster's **OIDC issuer** and **workload identity** features to be enabled. Create a user-assigned managed identity: ```bash az identity create \ --name \ --resource-group ``` Assign it the **Storage Blob Data Contributor** role on the storage account: ```bash az role assignment create \ --assignee-object-id "$(az identity show --name --resource-group --query principalId -o tsv)" \ --assignee-principal-type ServicePrincipal \ --role "Storage Blob Data Contributor" \ --scope "$(az storage account show --name --resource-group --query id -o tsv)" ``` Federate the identity with the agent's Kubernetes service account, so the pod can exchange its projected token for an Entra token: ```bash az identity federated-credential create \ --name kubernetes-federated-credential \ --identity-name \ --resource-group \ --issuer "$(az aks show --name --resource-group --query oidcIssuerProfile.issuerUrl -o tsv)" \ --subject "system:serviceaccount:mcd-agent:mcd-agent-service-account" \ --audience api://AzureADTokenExchange ``` The subject must match the namespace and service account the agent runs under. The chart always names the service account `mcd-agent-service-account`; adjust the namespace if you changed `namespace` in your `values.yaml`. For clusters that are not on AKS, and so have no managed identity to federate with. Register an application and generate a client secret: ```bash az ad sp create-for-rbac \ --name mcd-agent \ --role "Storage Blob Data Contributor" \ --scopes "$(az storage account show --name --resource-group --query id -o tsv)" ``` The output contains the three values the agent needs — `appId` (the client ID), `password` (the client secret), and `tenant`. The secret is shown only once. Store the credentials in a Kubernetes secret rather than in your values file: ```bash kubectl create secret generic mcd-agent-azure-credentials -n mcd-agent \ --from-literal=tenant-id= \ --from-literal=client-id= \ --from-literal=client-secret= ``` **Storage Blob Data Contributor** is the only role required — it includes every permission the agent needs. To use a certificate instead of a client secret, mount it into the pod and set `AZURE_CLIENT_CERTIFICATE_PATH` in place of `AZURE_CLIENT_SECRET`. ## 3. Configure the agent Every deployment sets the storage account and container: ```yaml container: storageType: "AZURE_BLOB" storageAccountName: "" storageBucketName: "mcdstore" ``` Then add the configuration for the identity you created: ```yaml serviceAccount: annotations: azure.workload.identity/client-id: "" deploymentTemplateLabels: azure.workload.identity/use: "true" ``` Get the client ID with: ```bash az identity show \ --name \ --resource-group \ --query clientId -o tsv ``` Pass the credentials as the standard `AZURE_*` environment variables, read from the secret created in the previous step: ```yaml container: extraEnv: - name: AZURE_TENANT_ID valueFrom: secretKeyRef: name: mcd-agent-azure-credentials key: tenant-id - name: AZURE_CLIENT_ID valueFrom: secretKeyRef: name: mcd-agent-azure-credentials key: client-id - name: AZURE_CLIENT_SECRET valueFrom: secretKeyRef: name: mcd-agent-azure-credentials key: client-secret ``` `container.extraEnv` is rendered into the pod spec as written, which is why `valueFrom` works here — the credentials never appear in your values file. Entries are rendered last, so one named the same as a variable the chart already sets takes precedence over it. **Workload Identity only.** Both the service account annotation and the `azure.workload.identity/use` pod label are required. The label is what tells the workload identity webhook to inject the projected token and the `AZURE_CLIENT_ID` environment variable into the pod. Each one fails differently, and neither failure names the missing piece: **Annotation missing** — the webhook injects everything except `AZURE_CLIENT_ID`, so the agent fails on startup, before contacting the storage account: ``` ValueError: 'client_id' is required. Please pass it in or set the AZURE_CLIENT_ID environment variable ``` **Label missing** — nothing is injected, and the agent authenticates as the cluster's node identity instead. That identity has no role on the storage account, so the failure looks like a missing role assignment: ``` ErrorCode:AuthorizationPermissionMismatch This request is not authorized to perform this operation using this permission. ``` In either case, check the annotation and the label before revisiting role assignments. To confirm what the pod actually received: ```bash kubectl exec -n mcd-agent deploy/mcd-agent-deployment -- env | grep AZURE_ ``` A correctly configured pod lists `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, and `AZURE_AUTHORITY_HOST`. ## 4. Expire temporary blobs (optional) The agent does not delete every blob it writes. Add a lifecycle management policy on the storage account matching what the Terraform module configures: | Prefix | Expiration | | :----------------------- | :--------- | | `mcdstore/mcd` | 90 days | | `mcdstore/mcd/tmp` | 2 days | | `mcdstore/mcd/responses` | 1 day | Prefixes in a lifecycle policy include the container name, and are matched against `blockBlob` and `appendBlob` types. Then [verify storage access](https://docs.getmontecarlo.com/docs/object-storage#verify-storage-access). ## If the agent cannot reach Entra ID Both options above need outbound access to `login.microsoftonline.com` — a deployment with no public internet egress, for example, cannot use either. Use a **connection string** instead. It is a shared key with full access to the storage account, so prefer an identity wherever you can. Enable shared key access on the account, which the [account creation](https://docs.getmontecarlo.com/docs/object-storage#1-create-the-storage-account-and-container) step above disables: ```bash az storage account update \ --name \ --resource-group \ --allow-shared-key-access true ``` Many organizations disable shared key access by policy. If `allowSharedKeyAccess` is enforced as `false` on the subscription, this path is unavailable regardless of the agent configuration, and reaching Entra ID is the only option. Copy the connection string from **Storage account > Security + networking > Access keys** in the portal, or: ```bash az storage account show-connection-string \ --name \ --resource-group \ --query connectionString -o tsv ``` Store it in a Kubernetes secret: ```bash kubectl create secret generic mcd-agent-azure-credentials -n mcd-agent \ --from-literal=connection-string='' ``` Then configure the agent with no identity and no `storageAccountName` — the account name comes from the connection string: ```yaml container: storageType: "AZURE_BLOB" storageBucketName: "mcdstore" extraEnv: - name: MCD_STORAGE_CONNECTION_STRING valueFrom: secretKeyRef: name: mcd-agent-azure-credentials key: connection-string ``` Set either a connection string or an identity, not both. When `MCD_STORAGE_CONNECTION_STRING` is present the agent uses it and never attempts Entra ID, so a half-removed identity configuration is silently ignored rather than reported. Then [verify storage access](https://docs.getmontecarlo.com/docs/object-storage#verify-storage-access). # Google Cloud Storage (GKE) These steps assume the agent runs on **GKE**, with [Workload Identity Federation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) enabled on the cluster. The agent authenticates to Cloud Storage as a Google service account bound to its Kubernetes service account — no service account keys are needed. ## 1. Create the bucket ```bash gcloud storage buckets create gs://mcd-agent-store \ --project \ --location \ --uniform-bucket-level-access \ --public-access-prevention ``` Cloud Storage encrypts data at rest by default. If you configure a customer-managed encryption key (CMEK) on the bucket, the agent needs no additional permissions — Cloud Storage uses its own service agent to reach the key. ## 2. Grant the agent access Create a Google service account for the agent: ```bash gcloud iam service-accounts create mcd-agent \ --project ``` Grant it access to the bucket: ```bash gcloud storage buckets add-iam-policy-binding gs://mcd-agent-store \ --member "serviceAccount:mcd-agent@.iam.gserviceaccount.com" \ --role roles/storage.objectAdmin gcloud storage buckets add-iam-policy-binding gs://mcd-agent-store \ --member "serviceAccount:mcd-agent@.iam.gserviceaccount.com" \ --role roles/storage.legacyBucketReader ``` Both roles are required. `roles/storage.objectAdmin` covers reading, writing, and deleting objects, but grants nothing at the bucket level. The agent resolves the bucket before every operation, so without `storage.buckets.get` even a write fails: ``` 403 GET https://storage.googleapis.com/storage/v1/b/mcd-agent-store: does not have storage.buckets.get access to the Google Cloud Storage bucket ``` `roles/storage.legacyBucketReader` (**Storage Legacy Bucket Reader**) adds it. Despite the name, these roles are not deprecated — they are the granular way to grant bucket-level reads. A custom role containing only `storage.buckets.get` works equally well if you prefer to avoid the predefined role. Bind the Google service account to the agent's Kubernetes service account: ```bash gcloud iam service-accounts add-iam-policy-binding \ mcd-agent@.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[mcd-agent/mcd-agent-service-account]" ``` The member must match the namespace and service account the agent runs under. The chart always names the service account `mcd-agent-service-account`; adjust the namespace if you changed `namespace` in your `values.yaml`. ## 3. Configure the agent ```yaml container: storageType: "GCS" storageBucketName: "mcd-agent-store" serviceAccount: annotations: iam.gke.io/gcp-service-account: "mcd-agent@.iam.gserviceaccount.com" ``` ## 4. Expire temporary objects (optional) The agent does not delete every object it writes. These lifecycle rules match what the Terraform module configures: | Prefix | Expiration | | :-------------- | :--------- | | `mcd/` | 90 days | | `mcd/tmp` | 2 days | | `mcd/responses` | 1 day | Apply them with a lifecycle configuration using `matchesPrefix` conditions and the `Delete` action. Then [verify storage access](https://docs.getmontecarlo.com/docs/object-storage#verify-storage-access). # MinIO and other S3-compatible storage The agent works with any S3-compatible storage service. This is the option to use when no cloud-native service is available — on-premises clusters, or local development and testing. ## 1. Deploy MinIO If you already run an S3-compatible service, skip to [step 3](https://docs.getmontecarlo.com/docs/object-storage#3-configure-the-agent-3). A complete working example — MinIO, secrets, and a ready `values.yaml` — is available in the [mcd-public-resources](https://github.com/monte-carlo-data/mcd-public-resources/tree/main/examples/generic/kubernetes/minio) repository. The example below deploys [MinIO](https://min.io/) into the same cluster as the agent, creating a `minio` namespace with a PersistentVolumeClaim, Deployment, and Services: ```bash kubectl apply -f https://raw.githubusercontent.com/monte-carlo-data/hermes-agent/main/environments/local/minio/k8s.yaml ``` This manifest is a reference for development and testing: it runs MinIO with the default `minioadmin` credentials over plain HTTP. For production, use a cloud-native storage service, or run MinIO with your own credentials, TLS, and a storage class suited to the workload. ## 2. Create the bucket Port-forward the MinIO API and console: ```bash kubectl port-forward -n minio deploy/minio 9000:9000 9001:9001 ``` Open , log in, and create a bucket named `mcd-agent-storage`. You can also create it without the console, by running the MinIO client inside the cluster: ```bash kubectl run mc --rm -it --restart=Never -n minio --image=quay.io/minio/mc \ --env="MC_HOST_local=http://:@minio-api.minio.svc.cluster.local:9000" \ -- --config-dir /tmp mb --ignore-existing local/mcd-agent-storage ``` ## 3. Configure the agent ```yaml container: storageType: "S3_COMPATIBLE" storageBucketName: "mcd-agent-storage" storageEndpointUrl: "http://minio-api.minio.svc.cluster.local:9000" storageAccessKey: "" storageSecretKey: "" ``` `storageEndpointUrl` is the service endpoint of your storage service — the in-cluster MinIO service above, or the URL of an external S3-compatible service (use `https://` where the service supports it). `storageAccessKey` and `storageSecretKey` end up in plain text in your values file. To keep them in a Kubernetes secret instead, create the secret: ```bash kubectl create secret generic mcd-agent-storage-credentials -n mcd-agent \ --from-literal=access-key= \ --from-literal=secret-key= ``` Then leave `storageAccessKey` and `storageSecretKey` unset, and pass the equivalent environment variables through `container.extraEnv`: ```yaml container: extraEnv: - name: MCD_STORAGE_ACCESS_KEY valueFrom: secretKeyRef: name: mcd-agent-storage-credentials key: access-key - name: MCD_STORAGE_SECRET_KEY valueFrom: secretKeyRef: name: mcd-agent-storage-credentials key: secret-key ``` Set them one way or the other — not both, or the pod ends up with duplicate environment variables. Then [verify storage access](https://docs.getmontecarlo.com/docs/object-storage#verify-storage-access). # Verify storage access Restart the agent and check for storage errors in the logs: ```bash kubectl rollout restart deployment/mcd-agent-deployment -n mcd-agent kubectl logs -n mcd-agent -l app=mcd-agent --tail=50 ``` Then run the validations from **Settings > Deployments**: select your agent and click **Validate**. Storage access is one of the checks — a failure here means the agent reached Monte Carlo but could not read or write the bucket.