Governing data products using fitness functions

Running assertions within the catalog

Modern day data catalogs like Collibra and Datahub provide hooks using
which we can run custom logic. For eg. Collibra has a feature called workflows
and Datahub has a feature called Metadata
Tests where one can execute these assertions on the metadata of the
data product.

Figure 1: Running assertions using custom hooks

In a recent implementation of data mesh where we used Collibra as the
catalog, we implemented a custom business asset called “Data Product”
that made it straightforward to fetch all data assets of type “data
product” and run assertions on them using workflows.

Running assertions outside the catalog

Not all catalogs provide hooks to run custom logic. Even when they
do, it can be severely restrictive. We might not be able to use our
favorite testing libraries and frameworks for assertions. In such cases,
we can pull the metadata from the catalog using an API and run the
assertions outside the catalog in a separate process.

Figure 2: Using catalog APIs to retrieve data product metadata
and run assertions in a separate process

Let’s consider a basic example. As part of the fitness functions for
Trustworthiness, we want to ensure that the data product includes
published service level objectives (SLOs). To achieve this, we can query
the catalog using a REST API. Assuming the response is in JSON format,
we can use any JSON path library to verify the existence of the relevant
fields for SLOs.

import json
from jsonpath_ng import parse


illustrative_get_dataproduct_response = '''{
  "entity": {
    "urn": "urn:li:dataProduct:marketing_customer360",
    "type": "DATA_PRODUCT",
    "aspects": {
      "dataProductProperties": {
        "name": "Marketing Customer 360",
        "description": "Comprehensive view of customer data for marketing.",
        "domain": "urn:li:domain:marketing",
        "owners": [
          {
            "owner": "urn:li:corpuser:jdoe",
            "type": "DATAOWNER"
          }
        ],
        "uri": "https://example.com/dataProduct/marketing_customer360"
      },
      "dataProductSLOs": {
        "slos": [
          {
            "name": "Completeness",
            "description": "Row count consistency between deployments",
            "target": 0.95
          }
        ]
      }
    }
  }
}'''


def test_existence_of_service_level_objectives():
    response = json.loads(illustrative_get_dataproduct_response)
    jsonpath_expr = parse('$.entity.aspects.dataProductSLOs.slos')
    matches = jsonpath_expr.find(response)

    data_product_name = parse('$.entity.aspects.dataProductProperties.name').find(response)[0].value

    assert matches, "Service Level Objectives are missing for data product : " + data_product_name
    assert matches[0].value, "Service Level Objectives are missing for data product : " + data_product_name

Using LLMs to interpret metadata

Many of the tests described above involve interpreting data product
metadata like field and job descriptions and assessing their fitness, we
believe Large Language Models (LLMs) are well-suited for this task.

Let’s take one of the trickier fitness tests, the test for valuable
on its own and explore how to implement it. A similar approach can be
used for the self descriptiveness fitness test and the
interoperability fitness
test for compliance with metadata registry standards.

I will use the Function calling feature of OpenAI models to
extract structured output from the evaluations. For simplicity, I
performed these evaluations using the OpenAI Playground with GPT-4 as
our model. The same results can be achieved using their API. Once you
have structured output from a large language model (LLM) in JSON format,
you can write assertions similar to those described above.

Functions:

{
  "name": "get_data_product_fitness",
  "description": "Determine if data product is fit for purpose",
  "strict": false,
  "parameters": {
    "type": "object",
    "required": [],
    "properties": {
      "valuable_on_its_own": {
        "type": "object",
        "properties": {
          "is_fit": {
            "type": "boolean",
            "description": "True if the data product is valuable on it's own, false otherwise"
          },
          "reason": {
            "type": "string",
            "description": "Reason why the data product is valuable on it's own or not"
          }
        },
        "description": "Determines if data product represents a cohesive
                        information concept in its domain. Has value on its own
                        and can be used independent of other data products"
      }
    }
  }
}

We can then send the data product metadata to the LLM to evaluate
them. Here’s a couple of results.

Customer data product: We expect this data product to pass the test
for valuable_on_its_own

User:
{
  "entity": {
    "urn": "urn:li:dataProduct:marketing_customer360",
    "type": "DATA_PRODUCT",
    "aspects": {
      "dataProductProperties": {
        "name": "Marketing Customer 360",
        "description": "Comprehensive view of customer data for marketing.",
        "domain": "urn:li:domain:marketing"
      }
    }
  }
}

Assistant:
 
 get_data_product_fitness({
  "valuable_on_its_own": {
    "is_fit": true,
    "reason": "The 'Marketing Customer 360' data product presumably provides a
               comprehensive view of customer data, which is valuable for marketing
               purposes. It could offer insights into customer behaviors, preferences,
               and engagement with marketing campaigns, all of which can be used to
               improve marketing strategies."
  }
})

Product_Id data product: We expect this data product to fail the
test for valuable_on_its_own. As product id needs to be always joined
with product or customer to have any use.

User:
{
  "entity": {
    "urn": "urn:li:dataProduct:product_id",
    "type": "DATA_PRODUCT",
    "aspects": {
      "dataProductProperties": {
        "name": "Product_Id",
        "description": "Table representing product ids per customer",
        "domain": "urn:li:domain:marketing"
      }
    }
  }
}

Assistant: 
 
 get_data_product_fitness({
  "valuable_on_its_own": {
    "is_fit": false,
    "reason": "The 'Product_Id' data product may not be valuable on its own. It simply
               represents product ids per customer and lacks contextual information
               about what those products are. For it to be meaningful, it would
               likely need to be used in conjunction with other data products that
               provide details about the products themselves."
  }
})

Publishing the results

Once we have the results of the assertions, we can display them on a
dashboard. Tools like Dashing and
Dash are well-suited for creating lightweight
dashboards. Additionally, some data catalogs offer the capability to build custom dashboards as well.

Figure 3: A dashboard with green and red data products, grouped by
domain, with the ability to drill down and view the failed fitness tests

Publicly sharing these dashboards within the organization
can serve as a powerful incentive for the teams to adhere to the
governance standards. After all, no one wants to be the team with the
most red marks or unfit data products on the dashboard.

Data product consumers can also use this dashboard to make informed
decisions about the data products they want to use. They’d naturally
prefer data products that are fit over those that are not.