Data Engineering Weekly #277
The Weekly Data Engineering Newsletter
Deep Dive: See how Dagster uses AI internally
What does it actually look like to use AI as part of your day-to-day engineering workflow? At Dagster, we've developed workflows that use AI to quickly validate concepts, identify dead ends, and focus our engineering time on what matters most.
Join our live session this week where we'll share the prompts, processes, and lessons we've learned from using AI to accelerate product and engineering decisions.
Editor’s Note: aidataengineer.io & leetdata.ai are now live!!!
Over the last two weeks, we launched both aidataengineer.io [aide] and leetdata.ai. DEW has so far focused on delivering the handcrafted articles for the week to the readers. However, I found two gaps when I started conversing with many data engineers.
How do I get started in a career in data engineering? leetdata.ai is the answer for it.
I’m working on my job and on how to validate my daily design compared with my peers - aide is the answer for it.
People are already spending hours on both platforms, which I’m absolutely thrilled to see. Rest assured, we are just getting started on these two platforms. We have some exciting feature additions coming to these two platforms, making them a one-stop source for all things data engineering. If you have any suggestions or feature implementation requests, please feel free to reach out to me on both LinkedIn and Substack.
Databricks: From monolith to Lakebase to LTAP: rethinking the database from storage up
Databricks publishes an in-depth article on the design thinking and high-level architecture of the LTAP system. The system builds on top of the Postgres protocol, with the replication and page server over a network. Bringing OLTP-style guarantees into the Lakehouse format is certainly appealing, especially with the rise of agent development and data engineering, not only generating data marts but also moving beyond to handle structured, unstructured, embedded, and multi-dimensional data.
https://www.databricks.com/blog/lakebase-ltap-rethinking-database-storage
Boaz Palgi: Databricks LTAP and the Unfinished Problem of Unified Data
An interesting take on the Databricks LTAP announcement, RegattaDB compares the LTAP's functioning with its design. The article raises very valid concerns about the system's concurrency guarantee and the use of copy-on-write. Anyone who has worked closely with Iceberg knows that it is pretty bad at concurrent writes and copy-on-write for fast-changing records.
https://regatta.dev/blog/databricks-ltap-storage-unification-is-not-enough
Addy Osmani: Loop Engineering
Coding agents start every run cold and fill any gap in intent with a confident guess, so durable leverage lives in the system around them. The author reframes this as loop engineering — scheduled automations, isolated worktrees, SKILL.md context, and split maker-checker sub-agents feeding an on-disk state file that survives each run. The loop engineering pattern aligns closely with the data engineering workflow, and I look forward to seeing how this will impact the data engineering function.
https://addyosmani.com/blog/loop-engineering/
Sponsored: The Data Platform Fundamentals Guide
Learn the fundamental concepts to build a data platform in your organization; covering common design patterns for data ingestion and transformation, data modeling strategies, and data quality tips.
Microsoft: Introducing Durable Functions in PostgreSQL
Postgres as a queue, especially around the job queue, is something many companies adopted without using it as intended. The “pg_durable” attempts to streamline the pattern by introducing durable functions in PostgreSQL.
Meta: Meta’s AI Storage Blueprint at Scale
Meta writes about overhauling its storage architecture to address bottlenecks that limit GPU utilization and slow down AI research. The primary bottleneck was caused by a legacy layered storage architecture that relied on stateful, slow metadata lookups and inefficient data proxies, stalling GPU training cycles. Meta’s solution was to rebuild the foundation with a unified metadata schema for faster lookups, eliminate data proxies in favor of a direct-streaming client SDK, and implement a tiered caching strategy that allows data to be hydrated on-demand across regional flash and memory.
https://engineering.fb.com/2026/07/01/data-infrastructure/metas-ai-storage-blueprint-at-scale/
Affirm: Re-architecting Affirm’s Upfunnel Platform: How We Cut Experiment Cycle Time from Months to Days
Affirm writes about migrating an upfunnel messaging service from a monolithic Python application to a standalone Kotlin microservice to improve reliability and reduce configuration bloat. By replacing a static database model with a dynamic, in-memory rule engine and implementing a robust shadow-validation pipeline, Affirm achieved independent deployability and significantly improved system performance.
Stripe: Scaling up your microservice testing with Apache Spark
Stripe writes about separating deterministic decision logic from the request path, then wrapping it in an Apache Spark harness that replays years of historical traffic through both current and candidate implementations. The harness turns production history into an executable regression asset, quantifying which records and rules change before a high-risk change ships, and anchoring it to a decision-engine boundary shared across online and offline paths.
Part 1: https://stripe.dev/blog/microservice-testing-with-apache-spark
Part 2: https://stripe.dev/blog/microservice-testing-with-apache-spark-part-2
Target: Scaling Marketing Campaign Forecasting with Generative AI
Rule-based matching and static segmentation decay as campaign types diversify, leading to false positives and manual overrides. Target writes about a RAG architecture that embeds historical campaigns into a grounding index, then has an LLM filter and rank the top three against an explicit matching hierarchy. The system lifts coverage from 75% at top-one to 100% at top-three, eliminating manual search while grounding the offer-propensity model in contextually relevant campaigns.
https://tech.target.com/blog/scaling-marketing-campaign-forecasting-ai
Expedia: Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs
Decoding the Spark SQL plan to identify the bottleneck is both an art and a science that requires considerable cognitive effort. I truly appreciate how LLMs commoditized this part of it. The author writes about feeding Spark plan metadata from an MCP server into an LLM. The grounded workflow cuts long-running job runtimes by 40–95% and compute costs by 50–95%, converting opaque plan internals into evidence-backed fixes that engineers can verify.
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.