Overview

Glue is a serverless AWS offering used for data cataloging, transformation, integration, and orchestration. The Monte Carlo integration is primarily interested in the cataloging features of the Glue Metastore. The Glue Metastore provides Monte Carlo with the databases, tables, and partition information of data stored in AWS S3 Buckets.

How does Monte Carlo use Glue?

The catalog information in Glue powers the Monte Carlo Assets view providing users with

• The ability to discover and explore data assets cataloged by Glue
• Table metadata type, schema, column, and partition information
• Schema monitoring
• Query Engines (Other data lake components rely on the Glue metastore to effectively map their queries to your data assets)
• Discovery of additional Glue catalogs beyond the default AwsDataCatalog — federated catalogs and Amazon S3 Tables (s3tablescatalog) are auto-discovered from a single metadata connection (databases, tables, columns, and types), with no per-catalog configuration

What features are not enabled by a Glue integration?

Monte Carlo data lake observability is enabled by integrating the entire data lake stack. The Glue metastore is a critical piece of that stack, but does not enable the full suite of data observability features on it's own. In order to get the best coverage for your data lake you will need to include a Query Engine integration to work in conjunction with Glue. Below is a list of Monte Carlo features that are not available with Glue alone.

• Automated Freshness and Volume Monitoring
• Data monitoring and breach/anomaly alerting
• Query Logs
• Importance scores
• Insight Reports

How does Monte Carlo integrate with Glue?

Monte Carlo uses assumable IAM roles to reach into your Glue metastore. Using the Monte Carlo UI or CLI, you pass the role ARN, database names, and S3 buckets you are interested
in monitoring to Monte Carlo.

There are three options to set up the integration:

• The Monte Carlo UI (Option 1), with the optional ability to add an Athena query engine at the same time.
• The Monte Carlo CLI (Option 2) completely automates the infrastructure creation and policy configuration.
• Alternatively (Option 3), you can manually configure the IAM role and policy via the AWS console and then use the CLI to register the integration with Monte Carlo.

Option 1: Use the Monte Carlo UI

The Monte Carlo UI provides a guided setup experience for configuring your Glue integration. You can optionally add an Athena query engine during this process or add one later from the integration details page.

1. Navigate to the Integrations page in Monte Carlo, click Add
2. Under the Data Lake and Warehouses section, click AWS Glue & Athena
3. Provide your AWS account details (Account ID and Region) and configure your Glue connection:
   • Specify which databases Monte Carlo should access (all or specific databases)
   • Optionally specify S3 data buckets (if not specified, they'll be derived from your tables)
4. (Optional) Enable an Athena query engine connection to create monitors on your Glue tables
   • Provide a connection name and specify the S3 result bucket for query results
   • Optionally configure the workgroup (defaults to primary) and data catalog (defaults to `AwsDataCatalo
   • By default, Athena will use the same databases and data buckets configured for Glue. Specify different ones if necessary
   • A single Athena connection can monitor tables across all catalogs it has access to — a separate Athena not required
5. An IAM policy will be generated based on your configuration, this policy is configurable and can be modified to fit your security requirements.

   The generated policy already includes catalog-discovery permissions (glue:GetCatalogs), so additional Glue catalogs are discovered automatically. If you use Amazon S3 Tables, add the S3Tables permissions described in Collecting additional Glue catalogs to both the Glue and Athena roles.

6. Create an IAM role in AWS with the generated IAM policy (follow the steps outlined in [Creating IAM Roles
7. Provide the Role ARN and External ID (if applicable) to complete the connection

Option 2: Use the Monte Carlo CLI

1. Generate the Glue access policy
2. Create an access role
3. Provide role information to Monte Carlo

1. Generate the Glue access policy

1. Run montecarlo discovery glue-policy-gen [parameters] > glue_access_policy.json with the necessary parameters. If the data account is not the same as the collector account, use --resource-aws-region and --resource-aws-profile to pass
   the data account profile.

$ montecarlo discovery glue-policy-gen --help
Usage: montecarlo discovery glue-policy-gen [OPTIONS]

  Generate an IAM policy for Glue. After review, output of this command can
  be redirected into `montecarlo integrations create-role` or `montecarlo
  discovery cf-role-gen` if you prefer IaC.

Options:
  --database-name TEXT         Glue/Athena database name to generate a policy
                               from. Enter '\*' to give Monte Carlo access to
                               all databases. This option can be passed
                               multiple times for more than one database.
                               [required]

  --data-bucket-name TEXT      Name of a S3 bucket storing the data for your
                               Glue/Athena tables. If this option is not
                               specified the bucket names are derived (looked
                               up) from the tables in your databases. This
                               option can be passed multiple times for more
                               than one bucket. Enter '\*' to give Monte Carlo
                               access to all buckets.

  --resource-aws-region TEXT   Override the AWS region where the resource is
                               located. Defaults to the region where the
                               collector is hosted.

  --resource-aws-profile TEXT  Override the AWS profile use by the CLI for the
                               resource. This can be helpful if the resource
                               and collector are in different accounts.

  --collector-id UUID          ID for the data collector. To disambiguate
                               accounts with multiple collectors.

  --help                       Show this message and exit.

2. Create an access role

1. Run montecarlo integrations create-role glue_access_policy.json. If the data account is not the same as the collector account, use --aws-profile to pass the data account profile.
2. The command prints a role ARN and an external id, they are used in the next section.

$ montecarlo integrations create-role --help
Usage: montecarlo integrations create-role [OPTIONS] FILE

  Create an IAM role from a policy FILE. The returned role ARN and external
  id should be used for adding lake assets.

Options:
  --aws-profile TEXT  Override the AWS profile used by the CLI, which
                      determines where the role is created. This can be
                      helpful when the account that manages the asset is not
                      the same as the collector.

  --help              Show this message and exit.

3. Provide role information to Monte Carlo

1. Run montecarlo integrations add-glue with the necessary parameters.

$ montecarlo integrations add-glue --help
Usage: montecarlo integrations add-glue [OPTIONS]

  Setup a Glue integration. For metadata.

Options:
  --region TEXT        Glue catalog region. If not specified the region the
                       collector is deployed in is used.
  --role TEXT          Assumable role ARN to use for accessing AWS resources.
                       [required]
  --external-id TEXT   An external id, per assumable role conditions.
  --name TEXT          Friendly name for the created warehouse. Name must be
                       unique.
  --collector-id UUID  ID for the data collector. To disambiguate accounts
                       with multiple collectors.
  --skip-validation    Skip all connection tests. This option cannot be used
                       with 'validate-only'.
  --validate-only      Run connection tests without adding. This option cannot
                       be used with 'skip-validation'.
  --auto-yes           Skip any interactive approval.
  --option-file FILE   Read configuration from FILE.
  --help               Show this message and exit.

Option 3: Use the AWS UI

1. Create an access role

1. Follow the steps outlined in Creating IAM Roles to create a role with the policy below, replacing the values REGION, ACCOUNT_ID, S3_ARN (of the S3 bucket(s) storing the data for your Glue/Athena tables

• you can alternatively pass "*" to give access to all buckets), and DATABASE_NAME.

2. The role ARN and external ID should be saved to be used in the next step.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "<S3_ARN>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "glue:GetConnections",
            "Resource": [
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:connection/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "glue:GetCatalogs",
            "Resource": [
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "glue:GetDatabases",
            "Resource": [
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog/*",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:database/<DATABASE_NAME>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetTables",
                "glue:GetTable",
                "glue:GetPartitions",
                "glue:GetPartition"
            ],
            "Resource": [
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:catalog/*",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:database/<DATABASE_NAME>",
                "arn:aws:glue:<REGION>:<ACCOUNT_ID>:table/<DATABASE_NAME>/*"
            ]
        }
    ]
}

2. Provide role information to Monte Carlo

1. Run montecarlo integrations add-glue with the necessary parameters.

$ montecarlo integrations add-glue --help
Usage: montecarlo integrations add-glue [OPTIONS]

  Setup a Glue integration. For metadata.

Options:
  --region TEXT        Glue catalog region. If not specified the region the
                       collector is deployed in is used.
  --role TEXT          Assumable role ARN to use for accessing AWS resources.
                       [required]
  --external-id TEXT   An external id, per assumable role conditions.
  --name TEXT          Friendly name for the created warehouse. Name must be
                       unique.
  --collector-id UUID  ID for the data collector. To disambiguate accounts
                       with multiple collectors.
  --skip-validation    Skip all connection tests. This option cannot be used
                       with 'validate-only'.
  --validate-only      Run connection tests without adding. This option cannot
                       be used with 'skip-validation'.
  --auto-yes           Skip any interactive approval.
  --option-file FILE   Read configuration from FILE.
  --help               Show this message and exit.

Collecting additional Glue catalogs (Amazon S3 Tables & federated)

Monte Carlo auto-discovers all Glue catalogs from a single metadata connection — the default AwsDataCatogs and **Amazon S3 Tables** (s3tablescatalog, one child catalog per S3 table bucket). Theglue:GetCatalogsand nestedcatalog/*` permissions in the policies above enable this; without them, collection covers only the default catalog.

Amazon S3 Tables additionally requires S3 Tables permissions, because Glue federates the catalog to the S3 Tables service. These are an opt-in add-on — grant them only if you use S3 Tables. A Data Lake integration has two roles, and each
needs its own additions:

Metadata (Glue) role — to discover & collect:

{
    "Effect": "Allow",
    "Action": [
        "s3tables:GetTableBucket",
        "s3tables:ListNamespaces",
        "s3tables:GetNamespace",
        "s3tables:ListTables",
        "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation"
    ],
    "Resource": "arn:aws:s3tables:<REGION>:<ACCOUNT_ID>:bucket/*"
},
{
    "Effect": "Allow",
    "Action": "s3tables:ListTableBuckets",
    "Resource": "*"
},
{
    "Effect": "Allow",
    "Action": "lakeformation:GetDataAccess",
    "Resource": "*"
}

Athena (query) role — to monitor & query: the same as above plus s3tables:GetTableData (to read table data):

{
    "Effect": "Allow",
    "Action": [
        "s3tables:GetTableBucket",
        "s3tables:ListNamespaces",
        "s3tables:GetNamespace",
        "s3tables:ListTables",
        "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData"
    ],
    "Resource": "arn:aws:s3tables:<REGION>:<ACCOUNT_ID>:bucket/*"
},
{
    "Effect": "Allow",
    "Action": "s3tables:ListTableBuckets",
    "Resource": "*"
},
{
    "Effect": "Allow",
    "Action": "lakeformation:GetDataAccess",
    "Resource": "*"
}

Troubleshooting

Validation error

User: arn:aws:iam::<aws_account_id>:user/<user> is not authorized to perform: <action> on resource: arn:aws:_id>:<resource>

Potential Causes

• AWS policy is not attached to a Role
• Policy is misconfigured. This often happens when the policy is created by hand. Try generating the the polforming a diff between your policy and the Monte Carlo CLI generated one
• A resource policy (e.g. bucket policy) is denying access
• Other types of policies are denying access, see AWS doc, but keep in mind that this is less likely, things like permissionboundaries are not common.

CATALOG_NOT_FOUND when creating or running a monitor

CATALOG_NOT_FOUND: line 2:15: Catalog 's3tablescatalog/<bucket>' does not exist

The Athena query connection's role is missing S3 Tables access. Lake Formation hides catalogs the principal can't reach, so Athena reports them as non-existent rather than access-denied. Grant the Athena role the S3 Tables permissions in
Collecting additional Glue catalogs. Notole from the Glue metadata connection — both need the grants.

Assume role error

User: arn:aws:sts::<dc_account_id>:assumed-role/<role> is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<data_account_id>:role/<role>

Potential Causes

• The role was created under the wrong AWS account. The IAM role must exist in the same account as Glue
• The trust relationship account is wrong - the account in the trust relationship must be the data collector account
• A trust relationship is missing (step 2 in creating IAM role docs)
• The MonteCarloData tag is missing or contains a value - value must be absent (step 7 in IAM role docs)

Connection timeout to https://sts.amazonaws.com/

If you're using an AWS Remote Agent and getting this error:

Connect timeout on endpoint URL: "https://sts.amazonaws.com/"

This is probably caused by the agent running on a VPC without Internet access, you need to:

• Add a VPCE (Private Endpoint) to the VPC hosting the agent for the "STS" service (`com.amazonaws.[region]. "Private DNS" for it.

• If the URL in the error is "sts.amazonaws.com" instead of "sts.[region].amazonaws.com" you also need to add the following environment variable to the Lambda Function for the agent:

  AWS_STS_REGIONAL_ENDPOINTS=regional