Data Engineering Weekly #278
The Weekly Data Engineering Newsletter
Deep Dive: See how Dagster uses AI internally
What does it actually look like to use AI as part of your day-to-day engineering workflow? At Dagster, we’ve developed workflows that use AI to quickly validate concepts, identify dead ends, and focus our engineering time on what matters most.
Join our live session this week where we’ll share the prompts, processes, and lessons we’ve learned from using AI to accelerate product and engineering decisions.
Editor’s Note: Updates on leetdata.ai
Since last week, we have launched two new features in leetdata.ai
Experiences - for the data engineers to share their interview experiences anonymously to help their fellow data engineers.
Systems: An in-depth, interview-focused learning experience covering the various tools we use in data engineering. We started with Spark, Kafka, data pipeline, and data warehouse. We will expand this more in the coming releases.
Here is what the current leaderboard looks like; visit leetdata.ai
BAIR: Intelligence is Free, Now What? Data Systems for, of, and by Agents
Agents will soon become the dominant workload for data systems. An agent querying a database doesn’t behave like a person or a BI tool. These two properties open a new era of data engineering research for building data systems for, of, and by agents. A thought-provoking article and a must-read for system builders.
https://bair.berkeley.edu/blog/2026/07/07/intelligence-is-free-now-what/
Bharath Vadhoola: The language tax on AI
Another article that made me think a lot this week is the language tax on AI.
Years ago, I was fortunate to talk to someone from Guinea during an Uber ride. He talked about the invention of the Adlam script (written form) for the Fula language (Spoken form) and the social impact of having a written language for the Fulani people, and his own life.
To achieve global social impact, we must ensure equality across languages in the AI era, specifically by reducing the higher token costs across languages.
https://www.linkedin.com/pulse/language-tax-ai-bharath-vadhoola-zhghc/
Microsoft: Flint - A visualization language for the AI era
As AI starts to impact the BI & visualization layer, one of the challenges is building rich visualizations within existing BI engines. I started building an allergy to mermaid diagrams; d2 is somewhat tolerable. Flint seems like a welcome addition to try out.
https://www.microsoft.com/en-us/research/blog/flint-a-visualization-language-for-the-ai-era/
Sponsored: The Data Platform Fundamentals Guide
Learn the fundamental concepts for building a data platform within your organization, including common design patterns for data ingestion and transformation, data modeling strategies, and data quality best practices.
Netflix: Measuring the Impact of Personalized Recommendations
Both attribution and measurement are exciting challenges in product analytics. Netflix writes about building a low-rank choice model to simulate dynamic member preferences. It validates this using small-scale A/B nudge tests, demonstrating that personalization drives engagement primarily through precise targeting of mid-popularity titles rather than sheer visual exposure.
https://netflixtechblog.medium.com/measuring-the-impact-of-personalized-recommendations-4c26be3a4d96
Grab: Scaling Grab’s Data Lake: Our journey to Apache Iceberg adoption
Having a table management system on top of the object storage brings much-needed integrity and efficiency in data management at scale. Grab writes a case study on its adoption of Iceberg and compares the advantages it brings versus the data lake model for managing parquet files in object storage.
https://engineering.grab.com/our-journey-to-apache-iceberg-adoption
Lyft: From Day 1 to Production: Building Lyft’s Analytics & Rides Intelligence Assistant as Onboarding Project
Natural-language analytics only becomes useful when it survives the unglamorous production work around it. Lyft’s ARIA story shows that the gap between a LangGraph/SQL prototype and an internal data product is auth, observability, state management, and responsive streaming—not another model prompt. Shipping it as a three-week onboarding project is a smart test of whether a platform’s documentation and paved roads actually work.
AWS: MCP tool design: Practical approaches and tradeoffs
MCP servers fail less often due to protocol limits than due to treating existing APIs as agent-ready. AWS lays out the real tradeoff: richer descriptions and schemas improve tool selection, but they consume the same context that agents need to reason. The most useful design principle is progressive disclosure—defaults and constrained schemas for common paths, taxonomy or agentic control only when ambiguity justifies the latency and cost.
https://aws.amazon.com/blogs/machine-learning/mcp-tool-design-practical-approaches-and-tradeoffs/
Santosh Shinde: The Medallion Architecture Reconsidered: What It Solved, and Where It’s Cracking
Medallion’s real contribution was imposing a recoverable path from raw data to trusted analytics; the mistake was treating Bronze, Silver, and Gold as three mandatory physical destinations. The author argues convincingly that AI workloads expose their one-dimensional model: governed data products—not a longer chain of medals—must serve BI, features, vectors, and agents. Keep immutable replay and logical refinement, but let contracts and fitness-for-purpose define the boundary.
Praveen Krishnan: Direct Lake is Magic, Until It Isn’t: How We Handled 1.5 Billion Rows Without Crashing Power BI
The Direct Lake promise is compelling, but this is a familiar lakehouse lesson: query engines inherit every physical-design decision beneath their semantic layer. The author’s 1.5-billion-row case shows why capacity sizing, column discipline, partitioning, and file layout must be tested at production scale—and why silent DirectQuery fallback belongs in the release gate. Performance features are only magical when their operating mode is observable.
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.