Setup Overview

1. Create a Service Account

You can create a Service Account via the Confluent CLI or the Confluent Cloud UI, their docs are here. To create via the CLI:

confluent iam service-account create monte_carlo –-description "API access for Monte Carlo data observability."

2. For accessing Kafka Cluster(s)

• Permission Service Account to Read Topics

Please follow the command or on Confluent Cloud UI to grant describe topics permissions.

confluent kafka acl create --allow --service-account {id of the service account} --operations describe --topic *

• Create a Resource API key

The Resource API allows Monte Carlo to read Kafka Topics in a given Kafka Cluster Resource.

You can create a resource API key and Secret via the Confluent CLI or the Confluent Cloud UI, their docs are here. You will need to reference the Resource ID of the Cluster To create via the CLI:

confluent api-key create --service-account {id of the service account} --resource <resource-id> –-description "API access for Monte Carlo data observability."

3. For accessing Kafka Connect Clusters(s)

• Create a Cloud API key

The Cloud API allows Monte Carlo to read Kafka Connectors.

You can create a Cloud API key and Secret via the Confluent CLI or the Confluent Cloud UI, their docs are here. To create via the CLI:

NOTE: The Cloud API key for accessing connectors may not work if it's associated with a service account. And this credentials could be used for different connect clusters in your cloud.

confluent api-key create --service-account --resource cloud

4. Create Confluent Integration in Monte Carlo

4a. Create Confluent Cloud Integration

1. Navigate to the integration settings and the Streaming Connections section.

   

2. Connect a Confluent Kafka Cluster:

   • Enter a Streaming System Name: this is a container/grouping for multiple clusters. You can reuse an existing streaming system.
   • Specify the Cluster Name (e.g., MyKafkaCluster).
   • Specify the Cluster ID, which could be obtained in the URL of the cluster.

   https://confluent.cloud/environments/{environment_id}/clusters/{cluster_id}/overvie
   

   • Specify the URL, which is the Rest endpoint in the Cluster Settings under the Cluster Overview tab.
   • Specify the API Key and Secret created in step 2.
   • Save the integration details.
   • Test the integration to ensure successful connectivity.
   

3. Connect a Confluent Kafka Connect Cluster:

   • Enter the Streaming System Name, that also contains the Kafka Cluster(s) this connect cluster associate with.
   • Specify the Confluent Cloud environment ID.
   • Specify the Confluent Cloud Kafka Cluster ID. (Both the environment ID and Cluster ID could be obtained from the cluster's URL).
   • Specify the API Key and Secret created in step3.
   • Save the integration details.
   • Test the integration to ensure successful connectivity.

Troubleshooting

If you encounter any issues during the integration process, refer to the following troubleshooting tips:

• Verify the accuracy of the provided Cluster URL, Cluster ID, API Key, and API Secret.
• Check the access permissions for the provided API Key and API Secret.
• Ensure that the Kafka Cluster is reachable from Monte Carlo.

For additional assistance, contact Monte Carlo support.

Note: This documentation provides a basic guide for integrating Confluent Kafka with Monte Carlo. Adjust the details based on your specific configuration and requirements.