Dive into the concept of Data Mesh with Ernout Douqué

What is data mesh? 

Strictly speaking, data mesh does not belong on the Emerging Technologies Radar because it is not a technology, but rather an organizational concept; it is a way of organizing how you deal with data in your organization. The basic idea is to think of data as a product that can be created and used by anyone in the organization. Data is captured once and can then be reused in different ways, for example in a dashboard, an AI model, and shareable data sets, and so on. These data products are created by autonomous business/ domain teams that also include data experts. Rather than a central data and analytics team creating the data products, data mesh adopts a decentralized approach. This includes a decentralized data architecture.

What are the benefits? 

The main benefit of the data mesh concept is speed. By distributing responsibility for data products throughout the organization and democratizing data, teams can develop their own data products more quickly. This means they are no longer dependent on a central data team. If your goal is to make data-driven working the norm in your organization, data mesh is a powerful concept that can help you achieve it. It works particularly well in larger organizations where there is a high demand for data products and a wide variety of data sources. Because in practice, we see that adopting an approach in which a central team is responsible for the delivery of data products can lead to bottlenecks and delays in the delivery of data products in such organizations.

Another advantage of this concept is that data teams can use their own tools if necessary, as long as they comply with architecture and data management agreements. This gives the data specialists at your organization a relatively high degree of freedom.

What makes data mesh so complex?

The most challenging aspect of such a decentralized approach is that you have to keep your eye on the big picture. You need to think about common data definitions, governance, data management, and other such tricky but vital issues for the consistent use of data. Technology is rarely the problem; convincing everyone to stick to the agreed governance principles often is.

You need to create, share, and manage common definitions. For example, how do you define a customer, and which (source) system is used to determine this? After all, a sales process may define a customer differently from a finance process. You will also need to agree which team owns which data and what that ownership entails. How do you ensure the quality of the data? How do you make sure that everyone sticks to the delivery agreements regarding data? How do you maintain the integrity of the data chain in your organization?

It goes without saying that implementing such a new way of working is not something that happens overnight. It is a change process that takes time and needs to be continuously nurtured. Otherwise, there is a good chance that, although you may quickly and very successfully deliver a few data products that are hugely popular in the first month, the change will not be sustainable and the quality and reuse of data in your organization will not improve.

Case study: Data mesh at a telecommunications provider

In almost every organization, the demand for quickly available and actionable insights is on the rise. In order to meet this demand, a telecommunications provider implemented a fundamental change in its data architecture and data management. Hot ITem Conclusion helped this company to make the transition from a centralized data department to distributed data domains at an architectural, organizational, and technological level.