📝

Prerequisites

1. You are an admin in AWS.
2. You have admin permissions in your data warehouse
3. You are an Account Owner.

This guide outlines how to setup an Agent (with object storage) and the OpenTelemetry Collector in your AWS cloud using Terraform.

These FAQs answer common questions like how to review resources and what integrations are supported.

📘

If you already have the Monte Carlo Agent deployed to your cloud vendor and want to add the OpenTelemetry Collector to your deployment, skip Step 1-2 and reference this FAQ. Afterwards, continue with Step 3.

Steps

1. Deploy the Agent

Before getting started please review the Monte Carlo AWS account your collection service is hosted in.

Specifying a VPC is required to run the agent and enables certain connectivity scenarios like when you have an IP allowlist for your resource, want to peer, or deploy in your existing VPC. See more details here.

Deploy with Terraform

Monte Carlo has provided terraform modules that can be used to deploy the MC Agent and the OpenTelemetry Collector with shared storage to AWS. You can review a full example of how to use these modules in the mcd-public-resources public Github repository.

Please make sure you have the terraform CLI installed and an active session to your AWS account available in your terminal.

# Monte Carlo Agent Module
module "agent" {
  source  = "monte-carlo-data/mcd-agent/aws"
  version = "1.0.3"

  cloud_account_id  = var.cloud_account_id
  private_subnets   = var.existing_subnet_ids
  image             = var.agent_image_uri
  region            = var.region
  remote_upgradable = var.remote_upgradable
}

# OpenTelemetry Collector Module
module "opentelemetry_collector" {
  source  = "monte-carlo-data/otel-collector/aws"
  version = "0.1.1"

  deployment_name                = "Provide any name for the deployment"
  existing_vpc_id                = "Proivde a VPC ID from your AWS account"
  existing_subnet_ids            = ["Provide at least two private subnet IDs from your AWS account"]
  existing_security_group_id     = "Optional, but recommended) Provide a Security Group ID allowing your AI agents to communicate with the OpenTelemetry Collector"
  telemetry_data_bucket_arn      = module.agent.mcd_agent_storage_bucket_arn
	# These remaining variables will be modified later
  external_id                    = "N/A"
  external_access_principal      = "N/A"
  external_access_principal_type = "AWS"
}

1. Initialize Terraform:

terraform init

2. Create the Terraform plan and review the output:

terraform plan

3. Apply the Terraform plan:

terraform apply

4. Validate the deployed occurred successfully by reviewing the command output and by using the AWS Console to locate the newly created resources.

📘

Note: by default the OpenTelemetry Collector deployed with the Monte Carlo Agent will only allow ingress from resources associated with the SecurityGroup created by the CF template. You might need to provide the existing_security_group_id parameter to the stack in order for the OpenTelemetry Collector to receive incoming traces from your Agent. These SecurityGroup definition will vary depend on your network configuration.

If you wish to use an existing S3 bucket to store the OpenTelemetry trace data, specify the ARN of the existing bucket in parameter telemetry_data_bucket_arn. By default the Data Store S3 bucket created by the MC Agent deployment will be used to store the OpenTelemetry trace data unless this parameter is specified.

2. Register the Agent

After deploying the agent you can register either via the Monte Carlo UI or CLI.

And see here for examples on how to retrieve deployment output (i.e. registration input).

UI

1. Navigate to settings/integrations/agents and select the Create button.
2. Follow the onscreen wizard for the "AWS" Platform and "Data Store + Agent" Type.

CLI

Use montecarlo agents register-aws-agent to register.

See reference documentation here. And see here for how to install and configure the CLI. For instance:

montecarlo agents register-aws-agent \
  --lambda-arn arn:aws:lambda:us-east-1:123456789:function:mcd-agent-AgentLambda \
  --assumable-role arn:aws:iam::123456789:role/mcd-agent-InvocationRole-12345 \
  --external-id f3840b31-772e-4fe3-8a5f-3aa5ff7e6fec

3. Configure your Data Warehouse Ingestion Pipeline

📘

Prerequisite: Data Warehouse S3 Access Configuration

Before continuing, your data warehouse must be configured to access the AWS S3 bucket containing the OpenTelemetry trace data. If your data warehouse is not currently configured to access the S3 bucket, refer to the guides below for Monte Carlo's recommendation on how to configure S3 access in your data warehouse.

Warehouse Vendor	Guide
Snowflake	Configure Snowflake Storage Integration and Stage
Databricks	Configure Databricks External Location

Next, we need to configure your data ingestion pipeline to write the OpenTelemetry trace data from S3 to your data warehouse so it can be monitored by Monte Carlo. Follow the guide relevant to your data warehouse vendor for steps on how to configure this pipeline.

4. Configure your AI Agent

Congrats! You have now configured the Monte Carlo Agent and OpenTelemetry Collector to process traces from your AI agent and write them to your data warehouse.

The final step is to configure your AI agent to begin sending traces to the OpenTelemetry Collector.

1. Add the Monte Carlo OpenTelemetry SDK to your AI agent's source code.
2. Use the opentelemetry_collector_http_endpoint output from the Terraform deployment earlier as the URL to provide to the Monte Carlo OpenTelemetry SDK.
3. Follow the Monte Carlo OpenTelemetry SDK library's instructions to configure instrumentation.

You can now validate the deployment is working as expected by observing files being written to the S3 bucket and data being ingested into the relevant table in your warehouse.

FAQs

What if I already deployed the Monte Carlo Agent?

If you already have the Monte Carlo Agent deployed to your cloud vender, you can deploy the OpenTelemetry Collector separately alongside it via Terraform.

Before getting started please review the Monte Carlo AWS account your collection service is hosted in.

Specifying a VPC is required to run the collector and enables certain connectivity scenarios like when you have an IP allowlist for your resource, want to peer, or deploy in your existing VPC. See more details here. Be sure to use the same VPC associated to your Monte Carlo Agent.

Monte Carlo has provided a terraform module that can be used to deploy the OpenTelemetry Collector to AWS. You can review and use this module from the public Github repository.

Please make sure you have the terraform CLI installed (version >=1.9.0) and an active session to your AWS account available in your terminal. You can use the monte-carlo-data/otel-collector/aws module like this:

module "opentelemetry_collector" {
  source  = "monte-carlo-data/otel-collector/aws"
  version = "0.1.1"

  deployment_name                = "Provide any name for the deployment"
  existing_vpc_id                = "Proivde a VPC ID from your AWS account"
  existing_subnet_ids            = ["Provide at least two private subnet IDs from your AWS account"]
  telemetry_data_bucket_arn      = "Provide the ARN of an existing S3 bucket to store telemetry trace data"
  existing_security_group_id     = "(Optional, but recommended) Provide a Security Group ID allowing your AI agents to communicate with the OpenTelemetry Collector"
	# These remaining variables will be modified later
  external_id                    = "N/A"
  external_access_principal      = "N/A"
  external_access_principal_type = "AWS"
}

1. Initialize Terraform:

terraform init

2. Create the Terraform plan and review the output:

terraform plan

3. Apply the Terraform plan:

terraform apply

4. Validate the deployed occurred successfully by reviewing the command output and by using the AWS Console to locate the newly created resources.

📘

Note: by default the OpenTelemetry Collector deployed with the Monte Carlo Agent will only allow ingress from resources associated with the SecurityGroup created by the CF template. You might need to provide the existing_security_group_id parameter to the stack in order for the OpenTelemetry Collector to receive incoming traces from your Agent. These SecurityGroup definition will vary depend on your network configuration.

Continue with Step 3 above.

Can I review agent resources and code?

Absolutely! You can find details here:

*Note that due to an AWS limitation the agent image is also uploaded and then sourced from AWS ECR when executed on Lambda.

Repository: 590183797493.dkr.ecr.*.amazonaws.com/mcd-agent

Additional FAQs?

Other applicable FAQs for deploying the Monte Carlo Agent to AWS can be found here.