I think what’s happened is people started rebranding existing features of their platforms as data contracts, to ticket a box, and that drowned out the original premise of data contracts.
So it’s great to see you publish this reminder, and hopefully restart the discussion on how data contracts can be the driver of the data platform.
Indeed, with the emerging spec-driven development and context graph, I believe the data contract will emerge stronger. Maybe, like any good science theory, it often starts with an incomplete theory and emerges with a more complete one as we iterate. I would love to connect over a call to brainstorm about this sometime early Feb, if you have time.
Take a look at DataSurface, I think it's the worlds first closed loop contract based system, spec driven... www.datasurface.com, I have had this argument with modelers for a while now. A model that is not implementable is for generating wall paper. I tend to focus on automating a set of patterns through one or more implementations. Patterns like production of data, consumption of data, materialization of existing data. Once we have consumers with specs, we can choose the best available implementation and get end to end automation.
Great article! Really appreciate you referencing our piece.
I strongly agree with your framing of data contracts as a missed opportunity when they stop at documentation.
At Soda, we’ve been thinking a lot about that exact gap between descriptive contracts and actual enforcement in pipelines. We recently published a more in-depth guide that explores the execution layer (how contracts move from YAML/specification to CI/CD validation and runtime enforcement). Thought it might be relevant to your audience as well.
Nice article, thanks for sharing. I agree it’s a missed opportunity, or at least missed by *some*, as this is exactly how we implemented data contracts (https://medium.com/gocardless-tech/implementing-data-contracts-at-gocardless-3b5c49074d13), and is how I’ve always talked about it, including in my book (https://data-contracts.com).
I think what’s happened is people started rebranding existing features of their platforms as data contracts, to ticket a box, and that drowned out the original premise of data contracts.
So it’s great to see you publish this reminder, and hopefully restart the discussion on how data contracts can be the driver of the data platform.
Indeed, with the emerging spec-driven development and context graph, I believe the data contract will emerge stronger. Maybe, like any good science theory, it often starts with an incomplete theory and emerges with a more complete one as we iterate. I would love to connect over a call to brainstorm about this sometime early Feb, if you have time.
Take a look at DataSurface, I think it's the worlds first closed loop contract based system, spec driven... www.datasurface.com, I have had this argument with modelers for a while now. A model that is not implementable is for generating wall paper. I tend to focus on automating a set of patterns through one or more implementations. Patterns like production of data, consumption of data, materialization of existing data. Once we have consumers with specs, we can choose the best available implementation and get end to end automation.
Great article! Really appreciate you referencing our piece.
I strongly agree with your framing of data contracts as a missed opportunity when they stop at documentation.
At Soda, we’ve been thinking a lot about that exact gap between descriptive contracts and actual enforcement in pipelines. We recently published a more in-depth guide that explores the execution layer (how contracts move from YAML/specification to CI/CD validation and runtime enforcement). Thought it might be relevant to your audience as well.
Sharing here in case it's useful.
https://soda.io/blog/guide-to-data-contracts