Data Mesh: Still Alive!
Long live the decentralized data strategy.
Lately I’ve been to a LOT of data conferences. Recently, I was at the ODSC East conference in Boston - they’d asked me to give a talk about Data Mesh. Lately I’ve been speaking more about AI, structured vs unstructured data, and whether the Modern Data Stack is still a thing. I’ve also made a lot of memes 😎 One question I’ve been asked a lot lately is, “what’s going on with Data Mesh?” and “is Data Mesh still a thing?” 👀 So let’s take a minute to talk about it.
Why care about Data Mesh?
I wrote a little book about Data Mesh, and I’ve been a huge proponent of it for years now. Back in the day I was a data engineering and analytics leader; even further back in the day I was a data engineer 🤓 I’ve felt first-hand the pain points that are prevalent in a centralized data organization at a growing company. I know how hard it is for data teams to be the bottleneck in the flow of data from source to value, because I’ve lived it. Do I think Data Mesh is a magic wand, that can miraculously fix all the problems? Eh. But do I think Data Mesh is a necessary and huge leap forward in data strategy? You bet I do.
Where is Data Mesh now?
In 2021-ish, Data Mesh was the hottest thing in data. People loved it or hated it, it spawned communities and conferences and books and a million blogs. Then, as with all things in the zeitgeist, we moved on. Other ideas took over in dominating the hype cycle, and while Data Mesh didn’t go away, it was no longer the belle of the ball. But a mature data strategy is more relevant than ever now.
Things on the Data Mesh front have been relatively quiet of late, and I have a theory of why this is:
It takes a long time to build out a Data Mesh. Years. So the organizations that were excited to get started in the last few years are now deep in the weeds of actually building it. While a data strategy or Data Mesh is never “done” per se, it seems a lot of organizations have received organizational support and started building their Data Mesh, but are only starting to vocalize their progress.
Have you heard of this “AI” stuff? It’s the new hot thing, and it’s the subject of all the thinkpieces out there right now. I would argue, however, that without a solid and mature data architecture and organization strategy like a Data Mesh, you’ll struggle to build out a functional AI program. The data itself is always the biggest challenge, and AI is no different. But as far as what Data People Are Talking About… right now it’s mostly AI.
The hardest part of pivoting a data strategy is the people. Change is hard and people don’t love it - especially when you’re modifying their responsibilities. It’s hard to come up with repeatable patterns for this, and this sort of change takes time and alignment at all levels. Plus it’s hard to focus on data strategy when you just laid off 1/3 of your workforce.
Data Mesh: still a thing!
So the short answer is: yes, Data Mesh is very much still a thing. My talk about Data Mesh was well-attended and well-received. Folks nodded along and afterwards told me about where they were in their journey (some were quite far along!). Myriad industries were represented and they all embraced the decentralized data strategy and journey.
Data Mesh is critical to AI
What’s more, I believe that Data Mesh is even more important in the era of AI. AI has compounded the need for treating data as a product, federated governance, domain-driven architectures, and self-serve infrastructure. These pillars of Data Mesh have become even more critical as data volumes increase exponentially to include structured, semi-structured, and unstructured data. Incredibly complex data processing coupled with the rapidly evolving regulation landscape further complicates the need for well-defined data products and federated governance.
So what’s next?
Ideally, I’m hoping that the drive toward a decentralized data strategy continues as companies and data teams realize Data Mesh alleviates existing pain points. And like I said, this is going to become much more apparent once folks realize that AI is HARD and compliance to regulations is getting harder. Data products and Data Mesh are a way to gain visibility and control over your data in these complex ecosystems. Emergent components of a data maturity such as data contracts will become essential. Over the next few months, quarters, and years, I anticipate hearing myriad additional success stories as enterprises realize the value of Data Mesh. The next few conference seasons should be interesting 🤩