How to Build a Data Platform From Scratch
We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.
In this book, you’ll learn:
- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means and how it will help you solve problems more quickly
Editor’s Note: Leetdata.ai launching Cohorts & Ontology Podcast
Most bootcamps focus on theory and a few sample exercises, but nothing compares to learning from real-world implementations. We mine all the learning shared by companies in Data Engineering Weekly and build a cohort for learners. You can learn now on leetdata.ai- not only sql problems, but systems, modeling, and interviews- and you can share your experience too.
As you know, we are restarting the podcast, initially focusing on taxonomy, semantic modeling, ontology, and knowledge graphs. If you have domain expertise in these areas, please reach out to us. We would love to host a conversation with you.
DuckDB: A Preview of DuckDB v2.0
An embedded analytics engine becoming network-addressable is a sharper change than another batch of SQL syntax. The DuckDB 2.0 feature preview presents a broader operating model: DuckDB servers via Quack and CONNECT asynchronous reads from object storage, and an extension API for longer-lived deployments. In-process use remains central, while these additions make it easier to place lightweight analytical services closer to the data.
https://duckdb.org/2026/08/17/duckdb-20-highlights
StreamFusion: Streaming for the AI age
Columnar execution has already reshaped batch engines, and streaming is beginning to follow the same path. This article shows how Flink’s planning, checkpointing, and SQL model can remain in place while native Arrow and DataFusion operators process supported plans. Row-oriented Kafka pipelines stay important, while columnar sources and sinks create a clear opportunity for this execution model.
https://streamfusion.tech/blogs/streaming-for-the-ai-age/#streaming-for-the-ai-age
Joshua Yu: The Evolution of Ontology: From Formal Semantics to Knowledge Graphs, Decisions and AI Agents
Knowledge graphs have long helped teams organize entities and relationships; agent workflows are broadening the role of semantics. The author traces a path from formal ontology to graphs, decisions, and agents that use capabilities, policies, and effects. Business vocabularies remain the foundation, while operational semantics give agents a clearer framework for acting responsibly.
Sponsored: AI Modernization Guide
AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.
Netflix: Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II
Streaming teams have spent years bringing enrichment closer to the event path. Netflix describes extending that pattern to live advertising using a stateful Flink join, regional routing, and a Spark recovery flow for late events. Central registries continue to serve governance needs, while the stream becomes the place where time-sensitive correlation can happen at scale.
Leandro Vaz: Benchmarks against Gluten and Comet
Vectorized execution has already given Spark teams several ways to accelerate SQL workloads. The author adds FuseCore to a comparison with Gluten and DataFusion Comet, making the different execution approaches easier to study side by side. Benchmarks are most useful when they mirror the workload characteristics, and this comparison provides practitioners with another reference point for planning a pilot.
Booking.com: How we selected the next vector database at Booking.com
If you’re looking to adopt a vector database in your ecosystem, this is a good starting point as an evaluation matrix. Booking.com describes an evaluation with around 100 million embeddings, comparing performance alongside feature and operational requirements before choosing the next platform. Public benchmarks remain helpful, while production-shaped workloads give teams the richer context needed for a long-lived decision.
https://booking.ai/how-we-selected-the-next-vector-database-at-booking-com-1e738a5e3bb0
Netflix: A Tale of Two Flink Autoscalers
Autoscaling has become a core part of making streaming platforms approachable for more teams. Netflix compares a cluster-level approach with one based on per-operator true processing rates, then shows how workflow orchestration and graph-aware checks support stateful jobs. General defaults help platform adoption, while workload-specific controls make the resulting system more dependable.
https://netflixtechblog.com/a-tale-of-two-flink-autoscalers-e9f6a1b1492b
Guldmann: Data Architecture Patterns: Decisions for the AI Era
Data architecture has accumulated useful patterns across processing, storage, ownership, and transformation. The author takes an architectural view, places Lambda, Kappa, Medallion, Mesh, Lakehouse, and semantic architecture on distinct decision axes, then connects them through contracts. Each pattern retains its role, while shared meaning and governance provide a practical way for them to work together for AI.
https://guldmann.blog/2026/08/19/data-architecture-patterns-decisions-for-the-ai-era/
Chroma: wal3 - A Write-Ahead Log for Chroma, Built on Object Storage
This is a repeat read for me this week, and I did some back-of-the-envelope calculations to estimate the total cost of S3 PUT requests alone.
(26,280,000 / 1,000) × $0.005 = $131.40 per month
Not bad at all. In this case, we need to double it to account for the snapshot commit; nonetheless, given that a PUT can hold up to 64MB and a 100 ms flush rate is an interesting system design to read.
https://www.trychroma.com/engineering/wal3
Jimdoverse: How We Cut Our CDC Bill from $3,000 to $150 a Month
CDC has become a foundational pattern for moving operational changes into analytical systems. Jimdoverse reports a monthly cost reduction from $3,000 to $150 and uses it to examine the economics of a replication pipeline. Managed services remain valuable for many workloads, while matching capacity and pricing to actual change volume can create a leaner foundation.
https://www.jimdoverse.com/how-we-cut-our-cdc-bill-from-3-000-to-150-a-month-270f95495799
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.