The Rise of Data Mesh

Data Mesh is a set of data engineering principles coined by Zhamak Dehghani from ThoughtWorks. I highly recommend reading How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and Data Mesh Principles and Logical Architecture.

Data Mesh influenced by domain-driven design, emphasizing the importance of data ownership and shared tooling to generate, curate, and democratize data. Data Mesh principles getting good adaption in some of the leading organizations. Yelp talked about its data mesh journey, So does Netflix and Zalando.

Though the literacy around data mesh maturing, I see a few confusions around Data Mesh in a few blogs.

The sad state of data engineering

Now the fundamental question you may ask. Why Data Mesh and Why now? To understand Data Mesh, we need to understand the current state of the data engineering world. It may not directly apply to your organization, but most of the data infrastructure remains in this sad state.

Imagine you are writing a dictionary with only the words with no meaning for it. On top of it, you shuffle the words randomly and publish the dictionary without any index for it, and hire high-paid data engineers and analysts to decode the dictionary.

It is the current state of the data infrastructure. The modern data infrastructure has sophisticated systems like Kafka, Spark, and the ability to emit and process events at a petabyte-scale. Yet, the data generation process we follow is equivalent to writing a dictionary without meaning.

Wait, Don’t Data Lake solved It?

A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is without having to structure the data.

As the definition suggests, Data Lake focuses on centralized data storage to break the organization's data silo. The central repository removes entry barriers to integrate and analyze from various data sources in an organization.

However, as the data lake grows, the complexity of the data management grows.

The producer of the data generates data and sends it to the data lake.
The consumers down the line have no domain understanding of the producer and struggle to understand the data lake data.
The consumers then connect with the data producer to understand the data. At that point, the producer side's domain expertise depends on human knowledge that may or may not available.

As Data Lake grows, it becomes a technical debt rather than a strategic advantage.

How Data Mesh Solve It?

Data Mesh is an enterprise data platform principle that converges the principles from Distributed Domain Driven Architecture, Self-serve Platform Design, and Thinking Data as a Product.

Data Mesh focuses on treating data as a product. The feature team that works on the product feature has the domain understanding of the data, not the data's consumer. Data Mesh pushes the data ownership responsibility to the feature team to create, attach metadata, catalog the data, and store it for easier consumption.

The data curation and the cataloging of the data at the data creation phase bring more visibility to the data and make it easier for consumption. The process also eliminates human knowledge silo and truly democratize the data. It helps the data consumers not worry about the data discovery and focuses on producing value from the data.

How Data Lake Different From Data Mesh?

Data Lake is like a reporter writing an article for the New York Times. The reporter goes and interviews related people to write a story, fact checks it, and delivers a reporter narration to the readers.
Data Mesh is like writing a book for O'Reilly or similar publications. The publication provides a foundational infrastructure for all the authors. The authors write their views, add index, and glossary for the book and deliver their narration to the readers.

But, There Is Always A Catch

Data Mesh sounds very cool, but there is always a catch. This is a great Twitter thread summarizing the challenges in adoption.

Ananth Packkildurai @ananthdurai

@gwenshap @KishoreBytes @byte_array The data mesh approach of services owns the data is a great idea. However, there is a catch. If you don't have data discovery and links between those domains, we will be back to square 1 of good old org data silo world. A process without proper tooling is dangerous to adopt.

As I mentioned, If we blindly adopt Data Mesh principles without the proper tooling, it can easily lead us to the good old org data silo problem. As mentioned in the below Tweet, no tool can fix the problem.

Kishore Gopalakrishna @KishoreBytes

@ananthdurai @gwenshap @byte_array @ananthdurai You nailed it... putting in a process without tooling is not only dangerous but will create new set of problems that no amount of tooling can fix

The following threads narrate the observation from the industry and how to adapt the Data Mesh principles.

Gwen (Chen) Shapira @gwenshap

@KishoreBytes @byte_array We want to allow each team that owns services to also own the data store. Maybe even use S3 or Dynamo or something else.

Vinoth Chandar @byte_array

@ananthdurai @gwenshap @KishoreBytes But the thing is, this is how most #data organizations grow organically within a company. Then they try to build out something centralized and few are able to succeed there today. So this is not as radical a shift.

Kishore Gopalakrishna @KishoreBytes

@byte_array @ananthdurai @gwenshap This is definitely not new - saw this at Yahoo. I remember we had close to 8 databases at Yahoo and tons of libraries and tools doing the same thing. It has its pros - each team moves fast but highly inefficient at a company level.

Sriram Subramanian @sriramsubram

@KishoreBytes @byte_array @ananthdurai @gwenshap I have seen both. Individual teams incentivized to moving fast and building lots of microservices/DBs with 0 leverage and early teams building monoliths with tightly coupled architecture that makes it hard to scale. It is always hard to move from one to the harder

Conclusion

Data Mesh is not a technology or a storage solution instead of a set of principles to streamline its data management. As Gwen, Sriram, Kishore, and Vinoth pointed out, it is an invisible structure in most organizations and requires proper tooling to enable the Data Mesh principles.

The analogy of evolving monolithic architecture to microservices architecture fits well with the Data Mesh principles. If you are starting up, a monolithic architecture may work well for you. As you grow, focus on building tools to label, catalog, organize, and search your data, leading to the adoption of the Data Mesh principles.

Links are provided for informational purposes and do not imply endorsement. All views expressed in this blog are my own and do not represent current, former, or future employers' opinions.

Subscribe now

Leave a comment

Share

Data Engineering Weekly

Data Mesh Simplified: A Reflection Of My…