Skip to main content
HomeBlogData Science

What Is a Data Mesh?

A data mesh is a decentralized data architecture where domain-specific teams own and manage their data as products, using a shared infrastructure and adhering to federated governance principles.
Jun 2024  · 10 min read

Centralized data architectures can be an effective data strategy – until they start to struggle with the growing volume and complexity of data.

As data scales, these centralized systems can face bottlenecks. They rely on a single point of control, which can become overwhelmed. The result is slow processes and difficulty extracting valuable insights efficiently.

A data mesh is an architectural system with decentralized and domain-specific data management. This approach empowers teams to take charge of their own data, leading to better data quality and faster insights.

If you want to learn more about data management, check out my other articles in this series:

Training 2 or more people?

Get your team access to the full DataCamp library, with centralized reporting, assignments, projects and more
Try DataCamp for Business

What Is a Data Mesh?

A data mesh is a modern approach to data architecture that shifts data management from a centralized model to a decentralized one.

It emphasizes domain-oriented ownership, where data management aligns with specific business areas. This alignment makes data operations more scalable and flexible, leveraging the knowledge and expertise of those closest to the data.

Core principles

At the heart of a data mesh are four key principles that guide its implementation and operation.

Domain-Oriented ownership

Data is owned and managed by the domain teams closest to the source. These teams have the best understanding of the data’s context and value, making them the ideal stewards of their own data.

Data as a product

Treating data as a product means establishing well-defined interfaces, quality standards, and documentation. This makes data easier to discover, access, and consume. This helps to ensure the data delivers value to its users.

Self-serve data infrastructure

Teams are provided with the tools and infrastructure they need to build, deploy, and manage their data products independently. This reduces reliance on central IT teams and accelerates data operations.

Federated computational governance

A federated governance model maintains data consistency, security, and compliance across the organization. It balances central oversight and domain-specific autonomy, allowing for tailored governance practices.

To learn more, check out this article on what a data federation is.

These principles collectively foster an environment where data is more accessible, reliable, and valuable within a large organization.

How a Data Mesh Works

Data products are the building blocks of a data mesh. They are reusable and discoverable assets that encapsulate domain-specific data, designed with clear interfaces and quality standards. They make data easy to integrate and use across the organization. 

For example, a sales team could create a data product that includes customer purchase history and sales trends. The marketing team can then easily access and use this data to tailor their campaigns.

The graphic shows a simplified view of an example data mesh for an organization.

The above graphic shows a simplified view of an example data mesh for an organization. Overall data governance policies guide each of the four departments: Sales, Finance, Marketing, and Product Development. Each department is responsible for its own data and creates a complete data product hosted on a shared platform. Data consumers interact with data products throughout the organization via a unified data mesh experience plane.

Data mesh architecture

A typical data mesh architecture consists of several key components that work together:

Domain data products

Data products are the heart of the data mesh. Domain teams own and manage them. These teams are responsible for the quality and maintenance of their data products and for ensuring they meet users' needs.

Data infrastructure platform

The data infrastructure platform provides a common foundation for data storage, processing, and governance. It supports the development and deployment of data products by offering the necessary tools and technologies.

Data governance

Data governance establishes policies and standards for data quality, security, and access to ensure consistent and compliant data management practices. This component is crucial for maintaining trust in the data across the organization. You can learn more in this course on Data Governance Concepts.

Data mesh experience plane

The data mesh experience plane is the user-friendly interface that enables the discovery, access, and consumption of data products. It ensures that data consumers can easily find and use the data they need. I like to think of this as a store where I can find all the data products within the organization.

Data Mesh Implementation

Let’s say we’re interested in implementing a data mesh at our organization. Where do we start? Let's walk through a few key steps.

Identify domains

Start by defining clear boundaries around business domains. This alignment helps in assigning data ownership to the right teams. Identify distinct business areas within the organization, such as sales, marketing, finance, or product development.

Establish data ownership

Assign data ownership to the relevant domain teams. These teams are accountable for the quality and management of their own data.

Build data products

Define and develop data products that cater to the needs of data consumers. These consumers may be employees in the same organization, investors, or other stakeholders. Ensure these products are discoverable, well-documented, and reusable to maximize their value. 

For example, the sales team may create a data product that includes customer purchase history and sales trends, which is then used by the marketing team, the data consumers in this example.

Federate governance

Establish a federated governance model to maintain data consistency and compliance. This approach allows for domain-specific governance practices while ensuring overall organizational standards are met. 

Our governance model should balance central oversight with domain-specific autonomy. Check out this Data Governance cheat sheet for some tips.

Implement a self-serve infrastructure

Equip teams with the tools and platforms they need to manage their data products independently. Provide domain teams with access to a common data infrastructure platform that includes tools for data storage, processing, management, and data lineage (read more in this article on what a data lineage is). This reduces dependencies on central IT teams and speeds up data operations.

Transitioning to a data mesh architecture may seem daunting, but the payoff can be substantial for large organizations.

Tools for Data Meshes

Data meshes often require tools that support domain teams in building, deploying, and managing their data products independently. These tools range from data storage and processing platforms to governance and discovery solutions.

Here are a few popular tools used in data meshes:

Tool

Description

Cloud-based

Key Features

Databricks

Unified analytics platform integrating data engineering, science, and analytics

Yes

Delta Lake for data storage, MLflow for machine learning, Databricks SQL for analytics

Snowflake

Cloud-based data platform for data warehousing, lakes, and sharing

Yes

Scalability, data sharing, secure collaboration, Snowflake Data Marketplace

Collibra Data Intelligence Cloud

Data governance and catalog platform supporting data mesh principles

Yes

Data catalog, data governance, data privacy, and data quality management

Open-source tools

Let’s now explore some popular open-source tools for data meshes:

Tool

Description

Cloud-based

Key Features

Apache Kafka

Distributed event streaming platform

Yes

Real-time data streaming, scalability

Apache Airflow

Workflow automation and scheduling system

Yes

Orchestration of complex data workflows

dbt (Data Build Tool)

Data transformation tool for analytics engineering

Yes

SQL-based transformations, version control

These tools offer a mix of capabilities that can help organizations implement a data mesh architecture effectively. It’s important to research a variety of tools to create a tailored suite of tools that meets an organization’s specific needs.

Benefits of a Data Mesh

A data mesh provides organizations with scalability by more effectively accommodating growing data volumes and complexity than centralized approaches. This scalability ensures organizations can manage and process vast amounts of data without encountering bottlenecks or performance issues.

Decentralizing data management within a data mesh promotes agility, allowing organizations to respond more quickly to changing business needs and market conditions. This flexibility enables teams to adapt their individual data strategies in real time, ensuring they can stay ahead of the curve in a rapidly evolving business landscape.

In a data mesh architecture, data quality is improved through domain-specific ownership. Domain teams, possessing the most relevant knowledge and context, are responsible for managing and maintaining their data products. This approach ensures data accuracy, reliability, and alignment with business objectives.

Additionally, a data mesh architecture promotes cross-domain collaboration and knowledge sharing. By breaking down data silos, teams can leverage each other's expertise, leading to improved decision-making and better overall outcomes.

Challenges of Adopting a Data Mesh

Adopting a data mesh comes with its own challenges that organizations must navigate.

One major hurdle is the cultural shift required. Moving towards a data mesh necessitates a fundamental change in organizational culture. Instead of a centralized data team being responsible for all the data in an organization, data meshes require decentralized decision-making and data ownership. This shift requires buy-in from all levels of the organization and may encounter resistance from those accustomed to centralized control.

The technical complexity of implementing a data mesh can also be a challenge. There will be new tools, processes, and skills, which may require significant investment in training and infrastructure. Organizations must ensure that they have the necessary resources and expertise to successfully transition to a data mesh architecture.

Striking the right balance between domain autonomy and central governance poses another challenge. While domain teams need the freedom to innovate and manage their data products independently, central governance is crucial for maintaining data consistency, security, and compliance across the organization. Achieving this balance requires careful planning and coordination to establish governance frameworks that accommodate both domain-specific needs and overarching organizational objectives.

When to Consider a Data Mesh

Large and complex organizations with extensive and varied data landscapes often find centralized approaches inadequate for scaling efficiently. In such environments, where data volumes and complexities continue to grow, a data mesh offers a decentralized alternative that can better accommodate the organization's needs.

Organizations operating in agile environments, where rapid responses to market changes or customer demands are essential, can also benefit from the flexibility of a data mesh. Its decentralized nature enables quicker adaptation to evolving business requirement. This increases agility and responsiveness.

A data mesh can be particularly beneficial for organizations with naturally distributed data ownership across different teams or departments. By aligning data management practices with the organization's existing structure, a data mesh empowers domain teams to take ownership of their own data products. This distributed ownership fosters accountability and ensures that data is managed by those with the most relevant expertise and context.

However, organizations that heavily rely on standardized, homogenized data practices across all departments or that lack distinct business domains may not benefit from a data mesh. Without clearly defined boundaries and decentralized decision-making, the advantages of domain-specific ownership and agility may be lost. These organizations may benefit more from a centralized approach to data management.

Data Mesh vs. Data Fabric

A data fabric is a more centralized approach to creating a unified data environment across an organization. It integrates various data sources and systems into a single, cohesive platform, providing users with a unified view of the data. 

Data fabrics often emphasize data integration, governance, and security to ensure consistency and reliability across the organization. You can read more in this article on what a data fabric is.

While both data mesh and data fabric address the challenges of modern data management, they do so through different approaches. A data mesh prioritizes decentralization and domain-oriented ownership, while a data fabric emphasizes centralization and integration. 

The choice between these approaches depends on factors such as organizational structure, data landscape, and business objectives.

Aspect

Data Mesh

Data Fabric

Ownership

Domain-oriented ownership; Data owned by domain teams

Centralized ownership; Data owned centrally

Data Integration

Decentralized; Integration handled by domain teams

Centralized; Integration managed by a central platform

Governance

Federated governance model; Domain-specific autonomy

Centralized governance; Standardized across the organization

Data Quality

Domain-specific accountability; Improved data quality

Centralized governance; Ensures consistent data quality

Data Access

Self-serve data infrastructure; Empowers domain teams

Centralized access control; Managed by central IT teams

Conclusion

The data mesh paradigm offers a solution to the challenges of centralized data architectures in large organizations. By decentralizing data management and aligning it with business domains, data mesh improves scalability, agility, data quality, and innovation.

To learn more about data management, check out these resources:

Training 2 or more people?

Get your team access to the full DataCamp library, with centralized reporting, assignments, projects and more

Photo of Amberle McKee
Author
Amberle McKee

I am a PhD with 13 years of experience working with data in a biological research environment. I create software in several programming languages including Python, MATLAB, and R. I am passionate about sharing my love of learning with the world.

Topics

Learn data management with these courses!

Course

Responsible AI Data Management

4 hr
1.6K
Learn the theory behind responsibly managing your data for any AI project, from start to finish and beyond.
See DetailsRight Arrow
Start Course
See MoreRight Arrow
Related

blog

What Is Data Fabric?

Data fabric is a unified data architecture that connects disparate data sources, simplifying access and management while ensuring consistency and security across the entire data landscape.
Amberle McKee's photo

Amberle McKee

16 min

blog

What Is a Data Federation?

Data federation is a data integration technique that provides a unified view of data from disparate sources without requiring physical data movement or consolidation.
Amberle McKee's photo

Amberle McKee

12 min

blog

Cloud Computing and Architecture for Data Scientists

Discover how data scientists use the cloud to deploy data science solutions to production or to expand computing power.
Alex Castrounis's photo

Alex Castrounis

13 min

blog

Democratizing Data in Government Agencies

Government agencies have access to troves of data. In order to fully harness its value, there needs to be a committed and coordinated effort towards data democratization within and across government agencies.
Kenneth Leung's photo

Kenneth Leung

8 min

blog

Scaling Data Science With Data Governance

The immense potential of data science and analytics is well recognized by businesses across all industries. But for these data science initiatives to succeed and scale, the data must first be relevant, accessible, and of high quality. This is where data governance tools come in to serve as vital enablers in the automation of governance operations and data stewardship efforts.
Kenneth Leung's photo

Kenneth Leung

8 min

blog

What Mature Data Infrastructure Looks Like

Unlocking the value of data in an organization starts with having the right data infrastructure and tooling foundations. Here’s a look at the current state and future trends of data infrastructure.
Kenneth Leung's photo

Kenneth Leung

6 min

See MoreSee More