Data Engineering Weekly #283
The Weekly Data Engineering Newsletter
How to Build a Data Platform From Scratch
We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.
In this book, you’ll learn:
- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means and how it will help you solve problems more quickly
Anthropic: Patterns and problems in emerging multiagent systems
Agent frameworks continuously draw attention, but the harder challenge is coordination. Anthropic looks at agent swarms that find vulnerabilities and struggle in shared environments. The important observation is not that agents behave unexpectedly, but that their shared incentives differ from those of people. As agents move beyond calling one another as tools, state isolation, protocol design, and human intervention will become platform primitives.
https://www.anthropic.com/research/multiagent-systems
Uber: Zero Sum by Design 10 Years of Uber’s Payments Platform
Uber writes about its payments platform and the zero-sum principles operating at scale. The immutable money order and double-entry accounting establish the core data contract before any Kafka message or microservice comes into play. Financial platforms rarely scale by starting with service boundaries; they scale by making the data model, audit trail, and consistency semantics non-negotiable.
https://www.uber.com/us/en/blog/ubers-payments-platform/
Sponsored: Drive Fund Admin Data Quality Without Scrambling
Financial services data has zero room for error. This guide breaks down why and how data engineers can drive high-quality fund admin data without scrambling every quarter. Download now for:
The six dimensions of data quality
A five-pillar framework for clean data end-to-end
Guidance on how to start reducing your quarter-end risks
Brad Coles: Data Engineering’s Shift from Imperative to Declarative - and Why It Matters More in the Age of AI
Declarative frameworks have already made progress across feature engineering and transformation systems. This article connects dbt, dynamic tables, and Spark Declarative Pipelines as the same move: declare the desired dataset and standardize dependencies, testing, and operational behavior. Notebooks still have a role, but declarative frameworks are steadily becoming the default for business logic that should be repeatable by teams—and increasingly by agents.
https://bradcoles.dev/blog/declarative-transformation.html
Uber: Running Cost-Efficient Export Workloads at Uber
Data layout has always played a bigger role than the storage format alone. Uber’s export-workload story makes the same case for Hudi: historical, selective queries can turn into expensive full scans and keep cold data hot. Sorting on the lookup key together with column statistics narrows the scan surface without a heavyweight secondary index. The interesting next step is automated selection: not every table deserves the rewrite.
https://www.uber.com/us/en/blog/running-cost-efficient-export/
Sponsored: AI Modernization Guide
AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.
Depop: How we cut our largest Spark pipeline’s cost by 52%
Pipeline cost usually grows faster than input volume because previously harmless maintenance and measurements are quietly attached to every schedule. Depop’s Spark optimization is a good case study: less frequent clustering, native write-time metrics, and append-only deduplication removed duplicated work before tuning clusters. Earlier lessons from large Iceberg tables point in the same direction—fix the query plan and data-change semantics before adding compute.
https://engineering.depop.com/how-we-cut-our-largest-spark-pipelines-cost-by-52-014faa81d179
StreamFusion: Streaming for the AI age
Columnar execution has steadily reshaped batch engines through Photon, Gluten, and Comet, but Flink has lacked a comparably open path. StreamFusion proposes one by keeping Flink’s scheduler, state, and checkpoint model while pushing compatible SQL operators through Arrow and DataFusion. The use case is compelling, but correctness and connector overhead—not just native speed—will decide whether it becomes an accelerator or another execution engine.
https://streamfusion.tech/blogs/streaming-for-the-ai-age/#streaming-for-the-ai-age
Zalando: From Homegrown to Flink: Migrating a Stateful Ad Event Join at Scale
Flink state is powerful, but the cost of its abstractions is not always visible. Zalando’s ad-event join migration makes this clear: an interval join’s RocksDB seeks and timer state did not fit a unique-key workload, so the team moved to a keyed low-level function, checkpoints, and TTL compaction. The migration reduced resources and recoverability risk, but only after extensive state and infrastructure tuning.
https://engineering.zalando.com/posts/2026/07/migrating-ad-event-processing-to-flink.html
Shiyu Chen et al.: Do Data Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval
Metadata platforms like Amundsen and Dataset Search were built because finding a table is not the same as finding usable data. This study revives the argument for agents: web retrieval can return relevant pages, yet semantic metadata more reliably identifies downloadable, FAIR-compliant assets. The appetite for agentic discovery should increase—not replace—the incentive to publish schemas, provenance, licenses, and machine-readable access.
https://arxiv.org/pdf/2605.28787
Michael Stonebraker & Peter Baile Chen: If You Think You Can Do Real-World Text-to-SQL
Text-to-SQL stories at Pinterest and Intuit demonstrate how useful these assistants can be inside a prepared environment. The Beaver benchmark supplies the missing counterweight: messy enterprise schemas, organization-specific terms, permissions, and multi-join queries make public benchmark scores far less relevant. RAG and agents help, but the real project remains semantic context, governance, and a clarifying workflow—not converting a question into SQL.
https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-world-text-to-sql/
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.