S3 Events - Metadata

S3 events are used to track data freshness and volume at scale for tables stored on S3.

Option 1: Use the Monte Carlo CLI [Recommended]

The easiest way to set up S3 events is to use the Monte Carlo CLI. The CLI completely automates the infrastructure creation and policy configuration. This option can only be used if you host your own Data Collector. If Monte Carlo hosts the Data Collector, proceed to Option 2.

  1. Please follow this guide to install and configure the CLI on your local machine.
  2. Run montecarlo integrations add-events with the necessary parameters.
$ montecarlo integrations add-events --help
Usage: montecarlo integrations add-events [OPTIONS]

  Setup S3 event notifications for a lake.

Options:
  --bucket TEXT                   Name of bucket to enable events for.
                                  [required]

  --prefix TEXT                   Limit the notifications to objects starting
                                  with a prefix (e.g. 'data/').

  --suffix TEXT                   Limit notifications to objects ending with a
                                  suffix (e.g. '.csv').

  --topic-arn TEXT                Use an existing SNS topic (same region as
                                  the bucket). Creates a topic if one is not
                                  specified or if an MCD topic does not already
                                  exist in the region.

  --collector-aws-profile TEXT    Override the AWS profile use by the CLI for
                                  operations on SQS/Collector. This can be
                                  helpful if the resources are in different
                                  accounts.

  --resource-aws-profile TEXT     Override the AWS profile use by the CLI for
                                  operations on S3/SNS. This can be helpful if
                                  the resources are in different accounts.

  --event-type [airflow-logs|metadata|query-logs]
                                  Type of event to setup.  [default: metadata;
                                  required]

  --auto-yes                      Skip any interactive approval.  [default:
                                  False]

  --collector-id UUID             ID for the data collector. To disambiguate
                                  accounts with multiple collectors.

  --option-file FILE              Read configuration from FILE.
  --help                          Show this message and exit.

Option 2: Use the AWS UI

The following steps are only required if setting up S3 events using the CLI is not possible.

Some steps change the data collector infrastructure, while others create infrastructure in the AWS account hosting the data. Each step is marked with "Data account" or "Collector account". If the data collector is deployed to the same account as the data, all steps are to be executed in the same account. If the data collector is managed by Monte Carlo, please reach out to your representative to execute the "Collector account" steps.

  1. Retrieve the bucket ARN (Data account)
  2. Retrieve the SQS queue ARN (Collector account)
  3. Retrieve the Data Collector AWS account ID (Collector account)
  4. Enable events (Local machine)
  5. Create a SNS Topic (Data account)
  6. Update the SQS access policy (Collector account)
  7. Create a SNS subscription (Data account)
  8. Create event notification (Data account)

📘

To complete this guide, you will need admin credentials for AWS

1. Retrieve the bucket ARN (Data account)

  1. Open the S3 Console and search for the bucket that you would like to enable events for.
  2. Select the bucket.
  3. Save the bucket ARN by selecting “Copy Bucket ARN” for later. This value will be further referred as S3_ARN.

2. Retrieve the SQS queue ARN (Collector account)

If the data collector is managed by Monte Carlo, please reach out to your representative for this value instead.

  1. Open the Cloudformation console and search for the Monte Carlo data collector.
  2. Select the stack.
  3. Select the “Outputs” tab.
  4. Save the Metadata Queue ARN for later (Key: MetadataEventQueue). This value will be further referred as EVENT_QUEUE_ARN.

3. Retrieve the Data Collector AWS account ID (Collector account)

If the data collector is managed by Monte Carlo, please reach out to your representative for this value instead.

  1. From the console, select your username in the upper right corner.
  2. Select “My Account”.
  3. Save the Account Id (without dashes) for later. This value will be further referred as COLLECTOR_ACCOUNT_ID.

4. Enable events (Local machine)

  1. Please follow this guide to install and configure the CLI on your local machine.
  2. Run montecarlo integrations toggle-metadata-events --connection-type [hive-mysql|glue|databricks-metastore].
$ montecarlo integrations toggle-metadata-events --help
Usage: montecarlo integrations toggle-metadata-events [OPTIONS]

  Toggle S3 metadata events. For tracking data freshness and volume at
  scale. Requires s3 notifications to be configured first.

Options:
  --enable / --disable            Enable or disable events. Enables if not
                                  specified.

  --connection-id UUID            ID for the connection.
  --connection-type [hive-mysql|glue|databricks-metastore]
                                  Type of the integration. This option cannot
                                  be used with 'connection-id'.

  --option-file FILE              Read configuration from FILE.
  --help                          Show this message and exit.

5. Create a SNS Topic (Data account)

  1. Open the SNS console and select “Topics”. IMPORTANT: Make sure you are in the same region as the bucket you want to add an event for.
  2. Select “Create Topic”.
  3. Choose "Standard" type.
  4. Enter a meaningful name.
  5. Select “Create Topic” and save the Topic ARN for later. This value will be further referred as SNS_ARN.
  6. The topic you just created has an access policy. Update the policy by appending the following two policy statements, replacing values in brackets with the SNS_ARN, S3_ARN and COLLECTOR_ACCOUNT_ID saved above.
  7. Save.
{
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": "SNS:Publish",
    "Resource": "<SNS_ARN>",
    "Condition": {
        "StringEquals": {
            "aws:SourceArn": "<S3_ARN>"
        }
    }
}
{ 
  "Sid": "__dc_sub",
    "Effect": "Allow",
    "Principal": {
        "AWS": "<COLLECTOR_ACCOUNT_ID>"
    },
    "Action": "sns:Subscribe",
    "Resource": "<SNS_ARN>"
}

6. Update the SQS access policy (Collector account)

If the data collector is managed by Monte Carlo these steps can be skipped by just sending the SNS Topic ARN to your representative.

  1. Open the SQS console.
  2. Search for the queue. The name follows this structure: {CF_STACK}-MetadataEventQueue-{RANDOM_STR}
  3. Select the “Access Policy” Tab and Select “Edit”.

If the access policy is empty or looks something like this:

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:<region>:<account>:<name>/SQSDefaultPolicy"
}

Paste the following, replacing the values in brackets with the SNS_ARN, COLLECTOR_ACCOUNT_ID and EVENT_QUEUE_ARN values saved above.

{
   "Version":"2008-10-17",
   "Statement":[
      {
         "Sid":"__owner",
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::<COLLECTOR_ACCOUNT_ID>:root"
         },
         "Action":"SQS:*",
         "Resource":"<EVENT_QUEUE_ARN>"
      },
      {
         "Sid":"__sender",
         "Effect":"Allow",
         "Principal":{
            "AWS":"*"
         },
         "Action":"SQS:SendMessage",
         "Resource":"<EVENT_QUEUE_ARN>",
         "Condition":{
            "ArnLike":{
               "aws:SourceArn":[
                  "<SNS_ARN>"
               ]
            }
         }
      }
   ]
}

If the access policy already has a SID with “__sender” (i.e. looks like above) append your SNS_ARN to the SourceArn list instead.

"aws:SourceArn": [
    "arn:aws:sns:::existing_sns_topic",
    "<SNS_ARN>"
]

7. Create a SNS subscription (Data account)

  1. Open the SNS console and select “Subscriptions”.
  2. Select “Create Subscription”.
  3. Select the topic ARN you saved above in the "Create a SNS Topic" subsection.
  4. Select Amazon SQS as the protocol.
  5. IMPORTANT: Be sure to select “Enable raw message delivery”.
  6. Select (or paste) the SQS ARN.
  7. Select “Create Subscription”.
  8. Validate the status is “Confirmed”.

8. Create event notification (Data account)

  1. Open the S3 Console and search for the bucket that you would like to enable events for.
  2. Select the bucket.
  3. Select the "Properties" tab.
  4. Select “Create event notification” under Event notifications.
  5. Fill in a meaningful name.
  6. Optionally specify a prefix and/or suffix.
  7. Select “All object create events” and “All object delete events” under Event types.
  8. Enter the SNS topic ARN as the Destination topic ARN. You can also choose the topic from the combo instead.

Did this page help you?