Fivetran Integration
Overview
The Monte Carlo-Fivetran integration retrieves information from all connectors and their synchronization status to bring visibility into the ingestion layer. Current features include:
-
Display in lineage the upstream Fivetran sources of tables monitored by the Monte Carlo. These sources can include external systems such as Salesforce, S3, Google Sheets, or any other system supported by Fivetran.
-
Under Assets, present sync context for each upstream Fivetran source.
Follow the next steps to complete the integration.
You will need access to Fivetran API to use the integration. Note that Fivetran Starter plan does not include API access.
Setting up your Fivetran integration
To connect Monte Carlo to a Fivetran account, follow these steps:
- Install the Monte Carlo CLI
- Check if the Data Collector version you have supports Fivetran
- Obtain a Fivetran API key and secret
- Create a Fivetran integration with the Monte Carlo UI or CLI
Fivetran is supported for Data Collector versions equal or higher than 13175, so the first thing we need to do is to check the Data Collector version.
Install the Monte Carlo CLI
Please follow this guide to install and configure the CLI.
If you already have the CLI installed, it is recommended to update it to the latest version. Fivetran support starts on CLI version 0.41.0
onwards.
Check the Data Collector version
Fivetran supports starts in Data Collector version 13175, so we need to check if the Data Collector version is equals to that or higher. In order to do that we use the following CLI command:
montecarlo collectors list
This will retrieve the list of the Data Collectors configured for your account. Check the Version
column of the Data Collector you are going to use for the Fivetran integration. If the version is lower than 13175 you will have to upgrade the Data Collector before proceeding.
If you don't upgrade you will get an error as follows when setting up the integration: Error - Integration 'fivetran' not supported.
To upgrade the Data Collector follow this instructions: Upgrade Data Collector
Obtain a Fivetran API key and secret
Please follow the instructions in Fivetran documentation to create a Fivetran API key
Write down the API key and secret as they will be used as input parameters when setting up the integration with the Monte Carlo CLI
NOTE: Fivetran supports creating an API key per Administrator user and the API key will have all its permissions. It is a good practice to add another Administrator user for Monte Carlo and create the API key with that Fivetran user.
Also they have just released in their Beta version the support of API Keys for users with less privileges. So, as a best practice request Fivetran access to this Beta feature and create a readonly user for Monte Carlo. Then generate the API key for this user which will be created with the same permissions as the user.
Create Fivetran integration with the Monte Carlo UI or CLI
Monte Carlo UI
- As a Monte Carlo Account Owner, navigate to Integrations.
- Under ETL Tool Connections section, click "Create" and select "Fivetran".
- In the side modal, enter the respective details
- Name (optional) - the name you want to give to the Fivetran integration
- API Key - use the API key obtained from Fivetran
- API Secret - use the API secret obtained from Fivetran
- Click "Create"
Monte Carlo CLI
The CLI will take the following mandatory parameters to setup a Fivetran integration:
fivetran-api-key
- use the API key obtained from Fivetranfivetran-api-password
- use the API secret obtained from Fivetranfivetran-base-url
- usehttps://api.fivetran.com/v1/
collector-id
- use your collector uuid. If you do not know it you can use the CLI to obtain it by runningmontecarlo collectors list
name
- the name you want to give to the Fivetran integration
Once you have all these parameters using the CLI run montecarlo integrations add-fivetran --fivetran-api-key {fivetran_api_key} --fivetran-api-password {fivetran_api_password} --fivetran-base-url https://api.fivetran.com/v1/ --collector-id {collector_id} --name {fivetran_integration_name}
You can see all the available options in the CLI doc or by running montecarlo integrations add-fivetran --help
- Monte Carlo will perform a validation step to ensure that the Fivetran account can be successfully accessed. If validation passes, you will be prompted to create the Fivetran integration. If validation fails, please double-check that you provided the correct API key and API secret
At this point, the integration has been created. By default, Monte Carlo will poll your Fivetran account hourly to import all the connectors and sync status.
Note: Once the integration has been successfully created it may take up to one hour for connectors to initially appear.
List all the integrations
If you want to see the list of integrations run: montecarlo integrations list
. Then you can verify that your Fivetran integration was created.
Fivetran integrations will also show in the Monte Carlo UI under Integrations.
Removing an integration
Reach out to Monte Carlo Support for assistance removing a Fivetran connection.
Updated 4 months ago