Transactional Databases

Transactional databases, such as SQL Server and PostgreSQL, are foundational components of modern data architectures. They are designed to manage and store operational data, supporting a wide range of applications that require reliable and consistent data handling.

Integrating Monte Carlo with your transactional database allows you to monitor your data assets through out of the box freshness, volume and schema monitors. With the ability to execute SQL queries on those tables, Monte Carlo can run a growing number of advanced monitors to analyze the data in those tables. Support varies by database type so checkout the guides for individual transactional database types like Postgres, SQL Server and SAP HANA to learn more and see a listed of supported OOTB and custom monitor types.

Integrating Your Transactional Database

To integrate a transactional database, you will:

1. Enable network connectivity between the database and Monte Carlo.
2. Create a read-only service account with the appropriate permissions. See detailed instructions for each database type in the guides nested below this page.
3. Provide the service account credentials and database details in your Monte Carlo onboarding wizard, found here in your workspace under the Create drop down button.

4. After providing the required fields, Monte Carlo will run a series of validations to confirm the correct permissions and access has been provided. Once the validations pass you will able to add the connection. If a validation fails, you will be shown a message about what failed and suggestions on what change needs to be made.
   Note that the validations that are run will vary depending on which transactional database you are onboarding and which deployoment optionhosting option you are using.

Using the CLI to Onboard Your Transactional Database

Alternatively, you can use the Monte Carlo CLI to onboard your transactional database. Follow thisthis guide to install and configure the CLI.

To see all available integrations you can onboard, run:

montecarlo integrations --help

📘

Don't see the integration type you are looking for?

Try updating the CLI version by running

Shell

pip install -U montecarlodata

Once you have found the onboarding command for the integration you are trying to add, you can see all required and optional connection parameters by running the command with the --help flag. An example for SQL Server would be:

montecarlo integrations add-sql-server --help
Usage: montecarlo integrations add-sql-server [OPTIONS]

  Setup a SQL Server integration. For metadata, and custom SQL monitors.

Options:
  --name TEXT          Friendly name for the created integration (e.g.
                       warehouse). Name must be unique.  [required]
  --port INTEGER       HTTP port.  [default: 1433]
  --host TEXT          Hostname.  [required]
  --user TEXT          Username with access to the database.  [required]
  --password TEXT      Users password. If you prefer a prompt (with hidden
                       input) enter -1.  [required]
  --agent-id UUID      ID for the agent. To disambiguate accounts with
                       multiple agents. This option cannot be used with 'dc-
                       id'.
  --collector-id UUID  ID for the data collector. To disambiguate accounts
                       with multiple collectors. This option cannot be used
                       with 'agent-id'.
  --skip-validation    Skip all connection tests. This option cannot be used
                       with 'validate-only'.
  --validate-only      Run connection tests without adding. This option cannot
                       be used with 'skip-validation'.
  --auto-yes           Skip any interactive approval.
  --option-file FILE   Read configuration from FILE.
  --help               Show this message and exit.

Continuing this example, to onboard a SQL Server connection you would end up running a command that looks something like this:

montecarlo integrations add-sql-server \
	--name friendly-name-of-my-sql-server \
	--host sample-cluster.123456789.us-east-1.rds.amazonaws.com \
	--user montecarlo \
	--password -1

Is Lineage supported?