Overview

Databricks external tables allow you to query data stored directly in cloud object storage—such as Amazon S3, Azure Data Lake Storage (ADLS Gen2), or Google Cloud Storage (GCS)—without ingesting or copying it into the Databricks-managed storage layer. These tables reference files in your data lake (Parquet, Delta, CSV, JSON, etc.) and expose them in the metastore so they behave like regular tables while still reading from the underlying storage location. Because external tables rely on the storage system as the source of truth, Databricks simply maintains metadata such as schema, location, and table properties.

File Formats

Databricks supports creating external tables from these file formats:

• Delta
• Iceberg
• CSV
• JSON
• Avro
• Parquet
• ORC

File type support is controlled by Databricks. Please reach out to Databricks if you need additional file formats supported.

Create and Monitor Databricks External Tables

👍

Already have external tables created in Databricks?

If you already have external tables in Databricks, skip to Add Databricks integration in Monte Carlo

An example of steps you can use for creating and monitoring external tables from cloud storage in Databricks are:

1. Create a storage credential object that contains a long-term cloud credential, such as a service account, with access to the cloud storage path.
2. Create an external location object that references the storage path and the storage credential object.
3. Reference a path that is included in the external location when you create external tables, external volumes, or default managed storage locations. This can be the exact path defined in the external location or a subpath.
4. Add Databricks integration in Monte Carlo..
5. Create Table monitors for the external tables.

Below are only example queries for each of these steps for the different cloud storage provides.

Refer to the Databricks documentation for detailed instructions, configuration options, limitations, and other information.

AWS S3 Workflow

The following steps are an example of how to create external tables in Databricks from your S3 files. For detailed instructions and other info, see Databricks documentation.

-- Grant `mc_user` user permission to create external location on `my_aws_storage_cred` storage credential
GRANT CREATE EXTERNAL LOCATION ON STORAGE CREDENTIAL `my_aws_storage_cred` TO `mc_user`

-- Create an external location on the specific path to which `my_aws_storage_cred` has access
CREATE EXTERNAL LOCATION finance_loc URL 's3://depts/finance'
 WITH (CREDENTIAL my_aws_storage_cred)
 COMMENT 'finance';

 -- `mc_user` can create an external table over specific object within the `finance_loc` location
CREATE TABLE main.default.sec_filings LOCATION 's3://depts/finance/sec_filings';

Azure Storage Workflow

The following steps are an example of how to create external tables in Databricks from your Azure storage files. For detailed instructions and other info, see Databricks documentation.

-- Grant `mc_user` user permission to create external location on `my_azure_storage_cred` storage credential
GRANT CREATE EXTERNAL LOCATION ON STORAGE CREDENTIAL `my_azure_storage_cred` TO `mc_user`

-- Create an external location on the specific path to which `my_azure_storage_cred` has access
CREATE EXTERNAL LOCATION `finance_loc` 
  URL 'abfss://[email protected]/depts/finance'
  WITH (CREDENTIAL `my_azure_storage_cred`)
  COMMENT 'finance';
    
-- `mc_user` can create an external table over specific object within the `finance_loc` location
CREATE TABLE main.default.sec_filings 
LOCATION 'abfss://[email protected]/depts/finance/sec_filings';

GCS Workflow

The following steps are an example of how to create external tables in Databricks from your Azure storage files. For detailed instructions and other info, see Databricks documentation.

-- Grant `mc_user` user permission to create external location on `my_gcp_storage_cred` storage credential
GRANT CREATE EXTERNAL LOCATION ON STORAGE CREDENTIAL `my_gcp_storage_cred` TO `finance`

-- Create an external location on the specific path to which `my_gcp_storage_cred` has access
CREATE EXTERNAL LOCATION finance_loc URL 'gs://depts/finance'
  WITH (CREDENTIAL my_aws_storage_cred)
  COMMENT 'finance';
     
-- `mc_user` can create an external table over specific object within the `finance_loc` location
CREATE TABLE main.default.sec_filings LOCATION 'gs://depts/finance/sec_filings';

Add Databricks Integration in Monte Carlo

Now that you have external tables created in Databricks, we can add a Databricks integration in Monte Carlo to monitor the external tables.

First, refer to the Connecting to Databricks guide for details on creating a service user, managing permissions, and related setup (such as network configuration).

If you’ve already integrated Databricks with Monte Carlo, you can skip this step. If you’d like to confirm access to a specific asset, you can do so from the Integration Settings page.

Onboard Databricks Integration

Once you have the Databricks service user credentials created, you can add a Databricks integration using the onboarding wizard. Integrations can also be added via the CLI or API if you prefer.

Create Table Monitors on External tables

With the Snowflake integration added, you can now create table monitors for schema, freshness and volume on the external tables. See our guide on creating table monitors.

You can also create additional monitors types if you’d like (e.g., custom).