PrivateLink using CloudFormation

For resources (e.g. Redshift, Tableau, etc.) that are not publicly available and you would like to privately connect with a Data Collector, as if they were in the same VPC, you can leverage an endpoint service powered by AWS PrivateLink. This is done via CloudFormation to automate the process and help manage resources as code and unlike VPC peering does not allow a free flow of traffic between VPCs.

In order to create and leverage an endpoint service, follow these steps -

1. Create a Service Provider CloudFormation stack
2. Create an Endpoint Consumer CloudFormation stack

👍

Check the region of your resource!

Endpoint services are only available within an AWS region. If your Data Collector is not deployed in the same region as Your Resource you can either (re)deploy the Data Collector, create an additional Data Collector in the matching region or peer.

If you have a region mismatch please let your Monte Carlo representative know or reach out to Support at [email protected], and we'd be happy to work with you to get you up and running!

Create a Service Provider CloudFormation stack

1. Download and review the CloudFormation template.
   https://prod-us-east-1-mcd-data-collector.s3.amazonaws.com/enablement/v0/service_provider.yaml
2. Deploy the stack in the same AWS account and region as Your Resource. Fill in the parameters from the Data Collector and Your Resource.

3. Retrieve the ServiceEndpointName from the stack outputs. This will be used in the next step.

Create an Endpoint Consumer CloudFormation stack

1. Download and review the CloudFormation template.
   https://prod-us-east-1-mcd-data-collector.s3.amazonaws.com/enablement/v0/endpoint_consumer.yaml
2. Deploy the stack in the same AWS account and region as the Data Collector. Fill in the parameters from the Data Collector and Your Resource.

3. Retrieve the ServiceEndpointDNS from the stack outputs. This will be used in onboarding.
   Note that the Endpoint DNS name will be prefixed with the hosted zone ID. Please only pass the DNS name when adding a connection. For instance, if the output is Z1HUB23UULQXV:vpce-01abc23456de78f9g-12abccd3.ec2.us-east-1.vpce.amazonaws.com you can disregard Z1HUB23UULQXV:.

FAQs

Why does the "Service Provider" stack create a Network Load Balancer (NLB)?
The NLB acts as the service front end for the VPC endpoint service, which is then granted to specific AWS principals. See details here.

Can we use a managed VPC endpoint instead?
Absolutely! If you are using Redshift with RA3 nodes and cluster relocation enabled (or can enable) you can leverage a Redshift-managed VPC endpoint instead of creating an endpoint service. See details here.

How do we find the private IP of my Network Load Balancer (NLB) for further restricting the CIDR Range?
Run the describe-network-interfaces command. For instance -

aws ec2 describe-network-interfaces --filters Name=description,Values="ELB <NLB-NAME-AND-ID>" --query 'NetworkInterfaces[*].PrivateIpAddresses[*].PrivateIpAddress'

The load balancer name and ID can be found in the the stack outputs as ServiceLbName.

How to handle an "VPC endpoint service does not support the availability zone (AZ) of the subnet" error?
When creating the Endpoint Consumer stack you might see an error as follows:

The VPC endpoint service com.amazonaws.vpce.ca-central-1.vpce-svc-123456789 does not support the availability zone of the subnet: subnet-abcd. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameter; Request ID:0987654321; Proxy: null)

This is because unlike via the VPC console, CloudFormation does not automatically perform AZ mapping and AZs can be different between AWS accounts. For instance, the subnet in us-east-1a in one account might be us-east-1b in the other.

You can run the describe-vpc-endpoint-services command in the same account and region as the Data Collector to check what availability zones the service you created is enabled for . For instance -

aws ec2 describe-vpc-endpoint-services --service-names com.amazonaws.vpce.ca-central-1.vpce-svc-123456789

How do we debug any connectivity issues with my endpoint service?
This guide might help debug reachability. Reviewing the load balancer health check is also a good starting point.