Additional Support for ADF in Lineage

Monte Carlo also offers query tagging as an additional way to surface ADF pipeline info in lineage. This is needed in cases when an ADF Script Activity is used in a pipeline, or when a Stored Procedure activity is used where users can add query tags. In those cases Monte Carlo can't provide ADF info in lineage without query tagging below.

📘

ADF Connection Required for ADF in Lineage

An ADF connection in Monte Carlo is necessary to display ADF lineage.

How to tag a query

Monte Carlo will look for the following tags in queries (uppercase or lowercase can be used):
adf_pipeline_name, adf_activity_name.

The name of the pipeline can be retrieved from the System Variable pipeline().Pipeline instead of hard-coding it.

For Snowflake, values can be passed as Snowflake Query Tags. Example:

alter session set query_tag='{"adf_pipeline_name": "@{pipeline().Pipeline}", "adf_activity_name": "my-activity"}';

INSERT INTO PUBLIC.MY_TABLE ... ;

For warehouses which do not support query tags, comments can be used instead. Example:

-- {"adf_pipeline_name": "@{pipeline().Pipeline}", "adf_activity_name": "my-activity"}

INSERT INTO PUBLIC.MY_TABLE ... ;

Multiple ADF Integrations

If you have multiple Azure Data Factory integrations, you may need to include an additional query tag to specify the integration. This is only required in the situation where there are shared Pipeline or Activity names between different ADF integrations.

Monte Carlo will look for the following tags to identify the integration (lowercase or uppercase can be used): adf_environment, adf_env.

The value of the environment tag must be the name of the integration in Monte Carlo.
Example:

-- {"adf_environment_name": "ADF-Prod", "adf_pipeline_name": "@{pipeline().Pipeline}", "adf_activity_name": "my-activity"}

INSERT INTO PUBLIC.MY_TABLE ... ;