Data Engineering Weekly #169
The Weekly Data Engineering Newsletter
Intuit: The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform
The Data Product Builder platform is becoming increasingly important in enterprise data engineering. It offers more targeted and customized data asset building than the general-purpose data stack.
Intuit writes about the growing data mesh strategy and the strategic focus area for building data products.
Systematically organizing people, code, and data
Clearly defining ownership of each data product
Providing tools for designing, authoring, deploying, and operating data products
Lyft: Technical Learning at Lyft - Build a Strong Data Science Team
Continuous learning is a vital part of a professional career. I consider writing a newsletter to be a learning process. Lyft writes about how it enabled continuous learning with technical training, Computer Science courses for Data Science, Seminar, and Brown Bag.
https://eng.lyft.com/technical-learning-at-lyft-build-a-strong-data-science-team-a6628215513c
Netflix: Data Gateway — A Platform for Growing and Protecting the Data Tier
Netflix's Data Gateway is a platform designed to enhance and safeguard the data tier by providing a unified interface for accessing data across various sources while ensuring security and compliance. It simplifies engineers' data access and management tasks by offering a single entry point with consistent API semantics and access controls, enabling efficient and secure data handling at scale. With features like caching, load balancing, and monitoring, the Data Gateway optimizes performance and reliability, supporting Netflix's complex data ecosystem.
Sponsored: Customizable embedded dashboards and natural language AI queries drive customer analytics
Quantatec, a company based in Brazil focusing on vehicle and fleet management, needed to equip their team with an embedded analytics platform. The initial option was a static platform of hard-coded reports which offered only basic filtering options and was insufficient for the fleet managers who required more tailored features.
Dive into how Quantatec addressed this issue by integrating a semantic layer that:
- Responds to the rising data demands
- Enhances the speed of query responses
- Provides self-service customization options
- Facilitates conversational data interactions
https://cube.dev/case-studies/customizable-embedded-dashboards-and-natural-language-ai-queries
LinkedIn: Musings on building a Generative AI product
One of the major challenges for many enterprises is to find LLM usage that can significantly differentiate their product experience and enable an efficient software development process to build these features. LinkedIn writes an exciting blog about what is and is not working.
https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product
Airbnb: Airbnb Brandometer- Powering Brand Perception Measurement on Social Media Data with AI
Airbnb writes about Brandometer, which utilizes AI to gauge brand perception on social media, enabling the company to measure its online reputation effectively. By analyzing vast amounts of social media data, the Brandometer identifies key brand attributes and sentiments, providing actionable insights for enhancing Airbnb's brand image and strategy. This AI-powered tool helps Airbnb stay attuned to customer feedback and market trends, contributing to its ongoing success in the competitive hospitality industry.
DoorDash: Building DoorDash’s Product Knowledge Graph with Large Language Models
DoorDash is leveraging large language models to construct a comprehensive Product Knowledge Graph, facilitating efficient data organization and retrieval. By harnessing these models' capabilities, DoorDash aims to enhance user experience by enabling smarter search functionalities and personalized recommendations within its platform. This initiative demonstrates how advanced AI technologies can empower data engineering efforts to create more intelligent and user-centric systems in the food delivery industry.
Grab: Ensuring data reliability and observability in risk systems.
Grab's data observability article discusses the implementation of the Iris platform, which is designed to enhance decision-making through comprehensive data monitoring and analysis. Iris tackles the challenge of extracting actionable insights from complex, cross-platform metrics by routing data in real-time to systems like InfluxDB and offline to AWS-based data lakes. The platform's structure allows for detailed tracking and analysis of CPU, memory, I/O, and job lifecycle metrics, facilitating proactive management and optimization of data processes.
https://engineering.grab.com/data-observability
Venkatesh Subramanian: Lambda, Kappa, Delta Architectures for Data
The article "Data Architectures" from Venkatesh discusses various patterns in data architecture that have evolved, such as Lambda, Kappa, and Delta architectures. These architectures are designed to handle massive amounts of data by defining how data moves through systems from ingestion to processing to storage. The article delves into the strengths and applications of each architecture type, highlighting their relevance in modern data-driven environments where real-time processing and data analytics are crucial.
https://subrabytes.dev/dataarchitectures
Teads Engineering: dbt unit-test framework
Teads team writes about the dbt unit-test framework in a two-part series. The blog highlights the ability to add support multiple unit testing in the same yaml file, and the usage of ephermal model for easier testing.
Part 1: https://medium.com/teads-engineering/unit-testing-with-dbt-fb84f2ef7dd6
Part 2: https://medium.com/teads-engineering/dbt-unit-test-framework-72d9ca60c69b
Dave Flynn: So, you think you’ve got dbt test bloat?
Alert fatigue hinders developers productivity, and it is vital to understand signal from noise. Dave Flynn writes about tactics followed by Foodpanda to improve alert quality. The approach focused on
Tiered model
Weigted alerts
Clear triage responsibility and expectations
https://medium.com/inthepipeline/so-you-think-youve-got-dbt-test-bloat-37491fb330d5
All rights reserved ProtoGrowth Inc, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.
