What Are Data Contracts? A Beginner Guide with Examples

Semantic Data Contracts

Semantic data validations ensure that data is logically consistent and aligns with the business logic.

Semantic validations have to be explicitly enforced.

Indeed, unlike schema checks, semantic data contracts rely on business logic and must be implemented externally.

Semantics in many use case scenarios looks like an extension of schema-validated contracts. Often, it depends on business rules and represents a set of row conditions that our data model must comply with. 

An example of a semantic contract can be the following:

• Metric deviation: from the moving average or any other threshold. Indeed, we can live with the number of active users for yesterday falling below 75% of the 7-day moving average but won’t tolerate 0%.
• Business logic: Transaction monitoring alert flags and fraud prevention scores. In this scenario, the payout must be 0.
• Data lineage: This describes how data entities evolve. For instance, transaction_completed_at can’t happen before created_at.
• Referential integrity: Entity relationships are important and can be described using semantic contracts, too. Consider the dbt code below. It explains that every refund’s refund_id maps back to a valid transaction.id.

- name: refunds
   enabled: true
   description: An incremental table
 columns:
     - name: refund_id
       tests:
         - relationships:
             tags: ['relationship']
             to: ref('transactions')
             field: id

It is up to the end user to set the alert level on semantic contracts, and this is often implemented with custom data quality tests.

Various business logic rules often refer to data integrity. Illogical data scenarios may arise from errors in database and server configurations or the unintentional injection of test data into the production environment. Data integrity checks are designed to validate data against those business rules, helping to identify values that don’t look right or are illogical.

Database entities always relate to each other. This is called entity relationship and is often presented by entity relationship diagram (ERD). Inadequate referential integrity can lead to issues such as missing or incomplete data. Data contracts must address these potential pitfalls to ensure data integrity and accuracy.

For instance, a common relationship is a one-to-many association between customers and orders where a single customer may have multiple orders. In such a scenario, an order is considered valid only if it contains a valid customer_id within the customers dataset.

This type of constraint, a referential integrity constraint, ensures that the relationship between entities is accurately maintained.

Service Level Agreements (SLAs) in Data Contracts

SLAs are an extra level of data quality checks that can be implemented using data contracts.

SLAs refer to data freshness.

Since data models are regularly updated with new data, SLAs may include checks for the latest time by which new data is expected to be available or the maximum allowable delay. For instance, in dbt, it can be achieved using freshness tests:

- name: orders
   enabled: true
   description: A source table declaration
   tests:
     - dbt_utils.recency: # https://github.com/dbt-labs/dbt-utils#recency-source
         tags: ['freshness']
         datepart: day
         field: timestamp
         interval: 1

Suppose we want to generate a report on yesterday's facts; to do that, we want to ensure the data exists. Indeed, it would be odd not to see the new orders for a few days or more.

In the example below, we can test our dataset for unexpected order delays using the test config in dbt. This allows you to select and reference the test using that specific name.

version: 2
models:
 - name: orders
   columns:
     - name: status
       tests:
         - accepted_values:
             name: unexpected_order_status_today
             values: ['placed', 'shipped', 'completed', 'returned']
             config:
               where: "order_date = current_date"

By defining a custom name, you gain complete control over how the test is displayed in log messages and metadata artifacts.

Similarly, in real-time data pipelines, data is typically expected to be no more than a few hours old. SLAs are crucial for stream-processing applications where data is being processed in real time with minutes or even seconds of latency.

In streaming apps, we want to check for the maximum allowable delay for late-arriving events and metrics like Mean Time Between Failures (MTBF) and Mean Time to Recovery (MTTR). Implementing this involves meticulously tracking incidents and extracting relevant data from application monitoring and incident management tools such as PagerDuty, Datadog, and Grafana.

Data Governance Contracts

Correctly handling Personally Identifiable Information (PII) is integral to the data transformation process. For many companies, it is crucial that these datasets are at least GDPR-compliant and adhere to data privacy regulations such as HIPAA or PCI DSS.

Data governance contracts ensure adequate pseudonymization or data masking policies are in place. 

Consider the dbt code below. Contracts enforced as tests require a user_email to be a SHA256 hash (masked):

models:
 - name: customer_data
   columns:
       - name: user_email
         tests:
           - dbt_expectations.expect_column_values_to_not_match_regex:
               regex: "^(?!.*\b@\b).* # Ensure identifiers do not contain emails
               flags: i # Case-insensitive matching

On the other hand, we might want to enforce pattern matching for other tables. For instance, in the example below, the transaction_reference field must follow the [“TRX-%”, “%-2023”] pattern:

models:
 - name: transaction_data
   columns:
       - name: transaction_reference
         tests:
           - dbt_expectations.expect_column_values_to_match_like_pattern_list:
               like_pattern_list: ["TRX-%", "%-2023"]
               match_on: any

Data governance contracts can also be beneficial when they include columns with sensitive data, metadata (data owners, etc.), and user roles authorized to access a data product.

For instance, we can indicate our data model owner and model development lifecycle stage using the meta field in dbt:

# models/schema.yaml
version: 2
models:
 - name: users
   meta:
     owner: "@data_mike"
     model_maturity: in dev
     contains_pii: true
   columns:
     - name: email
       meta:
       contains_pii: true

The meta field allows you to set metadata for a resource, which is then compiled into the manifest.json file generated by dbt and can be viewed in the automatically generated documentation. 

We can use packages like dbt-checkpoint to scan it during the pull request to ensure that our mandatory meta fields are present. The hook fails if any model (from a manifest or YAML files) does not have specified meta keys.

The model’s meta keys must be in the YAML file or the manifest.

Data Contract Implementation Patterns

Contract validation can be performed in streaming data pipelines (per row) before data is ingested and after it — at the source (source data model layer).

When we ingest data “as is,” the validation process acts like a transformation step, enforcing contract rules and filtering out invalid data, i.e., into a dedicated database view or a table where data can be investigated further.

Validation checks can be applied post-facto when data is ingested directly into a raw data lake.

The primary advantage of real-time data contract validation is its ability to filter out invalid records before they reach their final destination — a data lake or a data warehouse. This approach is prevalent in event-driven or real-time data processing, such as Change Data Capture (CDC) events. In this type of validation, certain aspects of the contract are verified as data flows through the pipeline.

Data Contracts Tools

The data community is gradually recognizing the potential benefits of data contracts, a constantly evolving data engineering area. Many tools exist in this space, some still in their early stages. dbt can be considered a universal framework for data contracts.

Similar contracts can be implemented using Google’s Dataform at source (source data model layer), i.e., when the data has been successfully ingested into our data warehouse. We would want to use simple row conditions for this. 

Consider this example below. It applies certain row conditions for our table:

-- my_table.sqlx
config {
 type: "table",
 assertions: {
   nonNull: ["user_id", "customer_id", "email"]
 }
}
SELECT …

Below is another example of data contract implementation, which can be achieved using Soda.io, a niche data quality framework:

# Checks for basic validations
checks for dim_customer:
 - row_count between 10 and 1000
 - missing_count(birth_date) = 0
 - invalid_percent(phone) < 1 %:
     valid format: phone number
 - invalid_count(number_cars_owned) = 0:
     valid min: 1
     valid max: 6
 - duplicate_count(phone) = 0

Adjusting the alert settings allows you to configure a check to issue a warning instead of failing the validation. A Soda scan runs the checks specified in an agreement, checks the YAML file or inline within a programmatic invocation, and returns a result for each check: pass, fail, or error.

From my experience, the introduction of data contracts in many companies remains fragmented, depending on factors such as data pipeline design patterns (batch or real-time), data serialization choices, and the systems used for storage and processing.

Data contract implementation depends on business requirements and logic.

Great Expectations, a Python library, can be used to implement semantic-level data contracts. It can be installed using pip: pip install great_expectations. 

After running great_expectations init, we can proceed with data validation:

Using v3 (Batch Request) API
 ___              _     ___                  _        _   _
/ __|_ _ ___ __ _| |_  | __|_ ___ __  ___ __| |_ __ _| |_(_)___ _ _  ___
| (_ | '_/ -_) _ |  _| | _|\ \ / '_ \/ -_) _|  _/ _ |  _| / _ \ ' \(_-<
\___|_| \___\__,_|\__| |___/_\_\ .__/\___\__|\__\__,_|\__|_\___/_||_/__/
                               |_|
            ~ Always know what to expect from your data ~
Let's create a new Data Context to hold your project configuration.
Great Expectations will create a new directory with the following structure:
   great_expectations
   |-- great_expectations.yml
   |-- expectations
   |-- checkpoints
   |-- plugins
   |-- .gitignore
   |-- uncommitted
       |-- config_variables.yml
       |-- data_docs
       |-- validations
OK to proceed? [Y/n]:

Consider the code snippet below. It explains how to create a data check definition for the price column:

"expectation_type": "expect_column_values_to_match_regex",
"kwargs": {
 "column": "price",
 "mostly": 1.0,
 "regex": "^\\$([0-9],)*[0-9]+\\.[0-9]{2}$"
},

Data Contract Best Practices

There is no right or wrong answer here as, from my experience, a successful data contract implementation relies heavily on business requirements. To make data contracts work effectively, follow these key best practices:

• Scalability: I recommend adding mechanisms for extensibility (e.g., on_schema_change: append) and versioning, which would allow for modifying data contract terms without disrupting existing pipeline integrations. Design data contracts with an eye toward future changes and scalability. This approach ensures that the contract can adapt to evolving needs and growth.
• Clear rules: I recommend using straightforward and concise language in the data contract to prevent misunderstandings and misinterpretations. The contract should be written and named in a way accessible and comprehensible to all involved parties, regardless of their technical expertise.
• Collaboration: The collaborative effort ensures a comprehensive and well-rounded understanding of requirements, addressing various perspectives and needs. When creating data contracts, engage diverse stakeholders, including data producers, data engineers, data scientists, and representatives from business, IT, legal, and compliance departments.
• Metadata: I recommend providing data contracts with thorough documentation and metadata. Comprehensive documentation aids in the smooth application and maintenance of the contract. This should include detailed descriptions, field definitions, validation rules, and other relevant information supporting a clear understanding and effective contract implementation.
• Regular reviews: Ongoing review helps maintain the contract’s relevance and effectiveness over time. Implement a structured approach for monitoring and data contract updates. Regular reviews ensure the contract remains current, aligns with evolving business requirements, and complies with new or changing regulations.

Conclusion

There has recently been a shift toward distributed data ownership, where domain teams are responsible for their data products. This change forced organizations to redefine data quality expectations, formalized through data contracts.

Semantic data validations ensure that data is logically consistent and aligns with the business logic. They help to check data pipelines for any outliers in data, value, lineage, and referential integrity. Inadequate referential integrity can lead to missing or incomplete data, which is why it is also important. 

Data governance contracts can also be implemented in CI/CD pipelines and become very useful when they indicate columns with sensitive data, metadata (data owners, etc.), and user roles authorized to access a data product. 

A model’s metadata, along with any other mandatory metafields defined by the developer, aids in monitoring the model’s resource usage and performance. 

Service Level Agreements (SLAs) within data contracts outline specific commitments related to data freshness, completeness, and failure recovery.

Data contracts are essential to modern data modeling techniques and help ensure the data platform is prone to errors and scales well. Resolving data quality issues might be expensive. For enterprise-level companies, maximizing return on investment (ROI) from data becomes increasingly important by leveraging existing data quality tools supporting data contract validation.