Data Engineering Weekly #50
Weekly Data Engineering Newsletter
Welcome to the 50th edition of the data engineering newsletter. This week's release is a new set of articles that focus on Benn Stancil’s the third rail, BVP’s data infrastructure roadmap, ValidIO’s DBT and analytical engineering, Zalando’s using knowledge graph to accelerate master data management, RudderStack’s real-time personalization with Redis, Anna Geller’s top 10 common pitfalls of data modeling, Zoba’s scaling more cities with Airflow, and DevianArt’s CDC journey.
Data Engineering Weekly - Brought to You by RudderStack - the Customer Data Platform for Developers
RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools.
Benn Stancil: The Third Rail
Benn Stancil writes an insightful view on the analyst role in an organization, the importance of analyst as a business decision support system, how the engineering skills overshadow the analyst function, and the lack of practical training brings the gap in the industry.
https://benn.substack.com/p/third-rail
Bessemer Venture Partners: Roadmap - Data Infrastructure
Bessemer Ventures writes the guideline for its investment strategy for the data infrastructure. The blog has a comprehensive overview of the data infrastructure trends, the adoption of the cloud, and the future direction.
https://www.bvp.com/atlas/roadmap-data-infrastructure
ValidIO: dbt and the Analytics Engineer — what’s the hype about?
The rise of the modern data stack is expanding the scope of analytical engineering. The article narrates how DBT catalyzes the change and discusses the evolution of data warehouses & data lakes.
https://medium.com/validio/dbt-and-the-analytics-engineer-whats-the-hype-about-907eb86c4938
Zalando: Knowledge Graph Technologies Accelerate and Improve the Data Model Definition for Master Data
The adoption of data lakes pushes master data management to the backseat partly because the "schema-on-write" MDM systems go against the "schema-on-read" design principles. At the same time, the adoption of microservices architecture is prone to yield inconsistent entity states. Zalando writes an exciting blog on how it tackles the MDM challenges with a graph database.
https://engineering.zalando.com/posts/2021/07/knowledge-graph-master-data-mdm.html
Sponsored - RudderStack: Real-Time Personalization with Redis and RudderStack
Nailing personalization can mean increasing revenue by 15%, but technical challenges keep many companies stuck using basic methods. RudderStack writes a step-by-step guide on designing and implementing a real-time personalization engine using Redis and RudderStack.
https://rudderstack.com/blog/real-time-personalization-with-redis-and-rudderstack
Anna Geller: 10 Common Mistakes When Building Analytical Data Models
Data modeling is the most challenging problem in data. The author narrates the most common pitfalls of data modeling, from treating data modeling as the one-of task to blindly following data modeling rules.
https://betterprogramming.pub/10-common-mistakes-when-building-analytical-data-models-814c763d1b70
Zoba: Scaling Zoba to more cities using Airflow
Zoba, the on-demand forecasting and optimization tool for mobility, writes about using Airflow to expand mobility to the newer markets. The workflow optimization to sync bootstrapping and periodic sync using Airflow's TaskFlow API is a good design reference for Airflow.
https://medium.com/zoba-blog/scaling-zoba-to-more-cities-using-airflow-1524f1941ffb
DevianArt: Change Data Capture at DeviantArt
DevianArt writes about its journey of adopting CDC infrastructure. Often the CDC narrations focusing on the choice of the CDC frameworks. The author did an excellent job focusing on describing the overall complexity of the ecosystem and troubleshooting.
https://www.wix.engineering/post/change-data-capture-at-deviantart
Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers' opinions.
valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up
https://www.sparity.com/