The following examples are queries related to lineage and nodes.

Please note that the sample queries below only show a subset of the fields available for brevity. You can use introspection to see all available fields.

Get Lineage

Direct lineage (i.e. directly upstream and downstream dependencies) can be retrieved using getDirectLineage.

The nodeId is the mcon retrieved above. The upstream and downstream mcons can be used to traverse the graph.

query directLineage {
  getDirectLineage(
    dwId:"dummy_warehouse_id",
    nodeIds:["dummy_table_id_1"]) {
      downstream {
        displayName
        mcon
      }
      upstream {
        displayName
        mcon
      }
    } 
}
{
  "data": {
    "getDirectLineage": [
      {
        "downstream": [
          {
            "displayName": "looker_view",
            "mcon": "dummy_looker_id_1"
          }
        ],
        "upstream": [
          {
            "displayName": "foo:bar.qux",
            "mcon": "dummy_table_id_2"
          },
          {
            "displayName": "foo:bar.baz",
            "mcon": "dummy_table_id_3"
          }
        ]
      }
    ]
  }
}

Add External Nodes

External Nodes Monte Carlo does not have access to can be added to Lineage and Pipelines using a combination of createOrUpdateLineageNode and createOrUpdateLineageEdge

mutation{
  createOrUpdateLineageNode(
    name: "dummyTableName"
    objectId: "database:schema.table_to_add"
    objectType: "table"
    resourceId: "dummy_warehouse_id"
  ){
    node{
      nodeId
      displayName
    }
  }
}

mutation{
  createOrUpdateLineageEdge(
    destination: {
      objectId: "database:schema.table_to_add"
      objectType: "table"
      resourceId: "dummy_warehouse_id"
    }
    source: {
      objectId: "database:schema.table"
      objectType: "table"
      resourceId: "dummy_warehouse_id"
    }
    expireAt: "YYYY-MM-DDTHH:MM:SS.MMMZ"
  ){
    edge{
      createdTime
      edgeId
    }
  }
}
{
  "data": {
    "createOrUpdateLineageEdge": {
      "edge": {
        "createdTime": null,
        "edgeId": "<edge_id>"
      }
    }
  }
}

Block Nodes from Lineage

You can block nodes from appearing in your lineage view based on regex at the dataset, schema, or table level. This can be convenient if you have an ETL that makes temporary tables causing your pipelines and lineage view to get cluttered quickly.

Options:

  • datasetRegexp: (String): Block datasets matching the regexp
  • projectRegexp: (String): Block projects matching the regexp
  • tableRegexp: (String): Block tables matching the regexp

First, you will need to grab the resource id. This is the uuid of the warehouse, bi tool, or datalake that your nodes exist in:

query getUser {
  getUser {
    email
    account {
      uuid
      name
        tableauAccounts {
        uuid
        }
      warehouses {
        uuid
      }
      bi {
        uuid
      }
    }
  }
}
{
  "data": {
    "getUser": {
      "email": "<email>",
      "account": {
        "uuid": "<account_uuid>",
        "name": "<account_name>",
        "tableauAccounts": [
          {
            "uuid": "<example_tableau_resource_id>"
          }
        ],
        "warehouses": [
          {
            "uuid": "<example_warehouse_resource_id>"
          }
        ],
        "bi": : [
          {
            "uuid": "<example_bi_resource_id>"
          }
        }
      }
    }
  }
}

Then you will use that in the following query:

mutation {
  createOrUpdateLineageNodeBlockPattern(
    tableRegexp: "temp_table_*"
    resourceId: "<resource_id>"
  ) {
    pattern {
      id
      uuid
    }
  }
}
{
  "data": {
    "createOrUpdateLineageNodeBlockPattern": {
      "pattern": {
        "id": "<id>",
        "uuid": "<blockage_uuid>"
      }
    }
  }
}

To verify it looks as expected, run the following query with the uuid returned above:

query {
  getLineageNodeBlockPattern(uuid: "<uuid>") {
    id
    uuid
    tableRegexp
    createdTime
  }
}
{
  "data": {
    "getLineageNodeBlockPattern": {
      "id": "<id>",
      "uuid": "<uuid>",
      "tableRegexp": "temp_table_*",
      "createdTime": "2021-07-26T18:57:45.674238+00:00"
    }
  }
}

If you are updating an existing blockage pattern, you will need to provide the uuid of the existing pattern in the following query. If this uuid is not provided, a new blockage pattern will be made:

mutation {
  createOrUpdateLineageNodeBlockPattern(
    uuid: "<uuid>"
    tableRegexp: "<new_regex>"
    resourceId: "<resource_id>"
  ) {
    pattern {
      id
      uuid
    }
  }
}
{
  "data": {
    "getLineageNodeBlockPattern": {
      "id": "<id>",
      "uuid": "<uuid>",
    }
  }
}

🚧

Updates will not be visible right away.

Please note our Catalog view will show this change immediately, but the Pipelines view will take up to 12 hours to show this update.


Did this page help you?