Lineage

The following examples are queries related to lineage and nodes.

Please note that the sample queries below only show a subset of the fields available for brevity. You can use introspection to see all available fields.

Get Lineage

Lineage (i.e. upstream and downstream dependencies) can be retrieved using getTableLineage. For more information, see the official API docs.

{
  getTableLineage(
    mcon: "dummy_table_id_1"
    direction: "downstream"
    hops: 1
  ) {
    baseNode {
      displayName
      hasDownstreamNodes
      hasUpstreamNodes
      isCustom
      mcon
      objectType
    }
    connectedNodes {
      displayName
      hasDownstreamNodes
      hasUpstreamNodes
      isCustom
      mcon
      objectType
    }
    flattenedEdges {
      directlyConnectedMcons
      mcon
    }
  }
}

{
  "data": {
    "getTableLineage": {
      "baseNode": {
        "displayName": "dummy_table_id_1",
        "hasDownstreamNodes": true,
        "hasUpstreamNodes": true,
        "isCustom": false,
        "mcon": "dummy_table_id_1",
        "objectType": "table",
        "__typename": "LineageGraphNode"
      },
      "connectedNodes": [
        {
          "displayName": "dummy_bi_1",
          "hasDownstreamNodes": true,
          "hasUpstreamNodes": true,
          "isCustom": false,
          "mcon": "dummy_bi_1",
          "objectType": "looker-view",
          "__typename": "LineageGraphNode"
        },
        {
          "displayName": "dummy_bi_2",
          "hasDownstreamNodes": true,
          "hasUpstreamNodes": true,
          "isCustom": false,
          "mcon": "dummy_bi_2",
          "objectType": "looker-view",
          "__typename": "LineageGraphNode"
        }
      ],
      "flattenedEdges": [
        {
          "directlyConnectedMcons": [
            "dummy_bi_1",
            "dummy_bi_2",
          ],
          "mcon": "dummy_table_id_1",
          "__typename": "FlattenedLineageGraphEdges"
        }
      ],
      "__typename": "LineageGraph"
    }
  }
}

Add External Nodes

External Nodes Monte Carlo does not have access to can be added to Lineage and Pipelines using a combination of createOrUpdateLineageNode and createOrUpdateLineageEdge

mutation {
  createOrUpdateLineageNode(
    name: "dummyTableName"
    objectId: "database:schema.table_to_add"
    objectType: "table"
    resourceId: "dummy_warehouse_id"
  ) {
    node{
      nodeId
      displayName
    }
  }
}

mutation {
  createOrUpdateLineageEdge(
    destination: {
      objectId: "database:schema.table_to_add"
      objectType: "table"
      resourceId: "dummy_warehouse_id"
    }
    source: {
      objectId: "database:schema.table"
      objectType: "table"
      resourceId: "dummy_warehouse_id"
    }
    expireAt: "YYYY-MM-DDTHH:MM:SS.MMMZ"
  ) {
    edge{
     expireAt
    }
  }
}

{
  "data": {
    "createOrUpdateLineageEdge": {
      "edge": {
        "createdTime": null,
        "edgeId": "<edge_id>"
      }
    }
  }
}

Object Types

objectType is an freeform string field, but you can specify any of the following objectTypes in order to be included in Impact Analyses:

table  
view  
external  
wildcard_table  
temp-table  
vector-index  
dynamic  
tableau-workbook  
tableau-view  
looker-look  
looker-explore  
looker-dashboard  
looker-view  
chartio-dashboard  
periscope-chart  
power-bi-report  
power-bi-dashboard  
mode-report  
sigma-workbook-page  
custom-bi-report

Block Nodes from Lineage

You can block nodes from appearing in your lineage view based on regex at the dataset, schema, or table level. This can be convenient if you have an ETL that makes temporary tables causing your pipelines and lineage view to get cluttered quickly.

Options:

datasetRegexp: (String): Block datasets matching the regexp
projectRegexp: (String): Block projects matching the regexp
tableRegexp: (String): Block tables matching the regexp

First, you will need to grab the resource id. This is the uuid of the warehouse, bi tool, or datalake that your nodes exist in:

query getUser {
  getUser {
    email
    account {
      uuid
      name
    	tableauAccounts {
        uuid
    	}
      warehouses {
        uuid
      }
      bi {
        uuid
      }
    }
  }
}

{
  "data": {
    "getUser": {
      "email": "<email>",
      "account": {
        "uuid": "<account_uuid>",
        "name": "<account_name>",
        "tableauAccounts": [
          {
            "uuid": "<example_tableau_resource_id>"
          }
        ],
        "warehouses": [
          {
            "uuid": "<example_warehouse_resource_id>"
          }
        ],
        "bi": : [
          {
            "uuid": "<example_bi_resource_id>"
          }
        }
      }
    }
  }
}

Then you will use that in the following query:

mutation {
  createOrUpdateLineageNodeBlockPattern(
    tableRegexp: "temp_table_*"
    resourceId: "<resource_id>"
  ) {
    pattern {
      id
      uuid
    }
  }
}

{
  "data": {
    "createOrUpdateLineageNodeBlockPattern": {
      "pattern": {
        "id": "<id>",
        "uuid": "<blockage_uuid>"
      }
    }
  }
}

To verify it looks as expected, run the following query with the uuid returned above:

query {
  getLineageNodeBlockPattern(uuid: "<uuid>") {
    id
    uuid
    tableRegexp
    createdTime
  }
}

{
  "data": {
    "getLineageNodeBlockPattern": {
      "id": "<id>",
      "uuid": "<uuid>",
      "tableRegexp": "temp_table_*",
      "createdTime": "2021-07-26T18:57:45.674238+00:00"
    }
  }
}

If you are updating an existing blockage pattern, you will need to provide the uuid of the existing pattern in the following query. If this uuid is not provided, a new blockage pattern will be made:

mutation {
  createOrUpdateLineageNodeBlockPattern(
    uuid: "<uuid>"
    tableRegexp: "<new_regex>"
    resourceId: "<resource_id>"
  ) {
    pattern {
      id
      uuid
    }
  }
}

{
  "data": {
    "getLineageNodeBlockPattern": {
      "id": "<id>",
      "uuid": "<uuid>",
    }
  }
}

Insert Custom Replacement Patterns

Replacement patterns can be used to collapse multiple nodes together into a single node. Consider the following situation:

graph LR
	analytics:prod.my_table_staging_720458096 --> analytics:prod.my_table
	analytics:prod.my_table_staging_720458097 --> analytics:prod.my_table
	analytics:prod.my_table_staging_720458098 --> analytics:prod.my_table
	analytics:prod.my_table_staging_720458099 --> analytics:prod.my_table

The graph can be cluttered with these nodes that follow a consistent naming pattern. One option is to remove them completely (see Lineage: Custom filters). Alternatively, if we want to preserve lineage for these nodes, another option is to collapse these nodes together using a LineageNodeReplacementRule:

mutation {
    createOrUpdateLineageNodeReplacementRule(
        resourceId:"5d856a92-0c69-4130-8a98-badf9e7c8cbe"
        pattern:"_staging_[0-9]+$"
        replacement:"_staging_[AUTO GENERATED SUFFIX]"
    ) {
        rule {
            uuid
            resourceId
            pattern
            replacement            
        }
    }
}

In this example, the replacement pattern will transform the graph to:

graph LR
	A("analytics:prod.my_table_staging_[AUTO GENERATED SUFFIX]") --> analytics:prod.my_table

Because regex can be finicky and hard to get right, there’s an API for simulating these patterns:

query {
    simulateLineageNodeReplacementRule(
        pattern:"_staging_[0-9]+$"
        replacement:"_staging_[AUTO GENERATED SUFFIX]"
        testInputStrings: [
            "foo_table"
            "analytics:prod.my_table_staging_720458096",
            "analytics:prod.my_table_staging_720458097"
        ]
    ) {
        testInputString
        replacedString
    }
}

Returns:

{
    "data": {
        "simulateLineageNodeReplacementRule": [
            {
                "testInputString": "foo_table",
                "replacedString": "foo_table"
            },
            {
                "testInputString": "analytics:prod.my_table_staging_720458096",
                "replacedString": "analytics:prod.my_table_staging_[AUTO GENERATED SUFFIX]"
            },
            {
                "testInputString": "analytics:prod.my_table_staging_720458097",
                "replacedString": "analytics:prod.my_table_staging_[AUTO GENERATED SUFFIX]"
            }
        ]
    }
}

The pattern and replacement are used as input to the re.sub function, so any Python re.sub syntax is supported. Refer to https://docs.python.org/3/library/re.html#re.sub

🚧
Updates will not be visible right away.
Please note our Catalog view will show this change immediately, but the Pipelines view will take up to 12 hours to show this update.

Get Lineage

Add External Nodes

Object Types

Block Nodes from Lineage

Insert Custom Replacement Patterns

🚧Updates will not be visible right away.

🚧
Updates will not be visible right away.