Data mesh: Difference between revisions
No edit summary |
|||
| Line 16: | Line 16: | ||
== See also == |
== See also == |
||
* [[Data management]] |
* [[Data management]] |
||
* [[Data platform]] |
|||
| ⚫ | |||
* [[Data warehouse]], a well established type of database system for organizing data in a thematic way |
* [[Data warehouse]], a well established type of database system for organizing data in a thematic way |
||
| ⚫ | |||
== References == |
== References == |
||
Revision as of 06:22, 11 August 2022
 | This article may contain excessive or inappropriate references to self-published sources. (April 2022) |
Data mesh is a sociotechnical approach to build a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design[1] and Manuel Pais’ and Matthew Skelton’s theory of team topologies.[2] The main proposition is scaling analytical data by domain-oriented decentralization.[3] With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.[4]
History
The term data mesh was first defined by Zhamak Dehghani in 2019[5] while she was working as a principal consultant at the technology company ThoughtWorks.[6][7] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[8][9] Data meshes have been implemented by companies such as Zalando,[10] Netflix,[11] Intuit,[12] VistaPrint and others.
Principles
Data mesh is based on four core principles:[13]
- Domain ownership
- Data as a product[14]
- Self-serve data platform
- Federated computational governance
In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[15] In other words, the data should be treated as a product that is ready to use and reliable.[16]
See also
- Data management
- Data platform
- Data warehouse, a well established type of database system for organizing data in a thematic way
- ETL and ELT
References
- ^ Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.
- ^ Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.
{{cite book}}: CS1 maint: location missing publisher (link) - ^ "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2022-06-13.
- ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.
{{cite book}}: CS1 maint: location missing publisher (link) - ^ "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
- ^ Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2022-02-10.
- ^ Andy Mott (2022-01-12). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2022-03-01.
- ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
- ^ Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2022-03-01.
- ^ Schultze, Max; Wider, Arif (2021). Data Mesh in Practice. ISBN 978-1-09-810849-6.
- ^ Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2022-04-29
- ^ Baker, Tristan (2021-02-22). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2022-04-29.
- ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.
{{cite book}}: CS1 maint: location missing publisher (link) - ^ "Data Mesh defined | James Serra's Blog". 16 February 2021. Retrieved 28 January 2022.
- ^ "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2021-12-29. Retrieved 2022-03-03.
- ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.