Data Engineering Weekly #282
The Weekly Data Engineering Newsletter
How to Build a Data Platform From Scratch
We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.
In this book, you’ll learn:
- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly
Yannick Misteli: The Semantic Layer Is Not a Layer Anymore
We did this with data platforms. Open table formats gave us a standard wheel. Then every vendor added its own catalog, governance model, access layer, and control plane — the proprietary bolts that make the wheel difficult to move anywhere else.
Semantic layer conversation is everywhere. The author raises a real alarming question. Are we going to repeat the same mistake we did with the OpenTable formats?
https://y4nnick.substack.com/p/the-semantic-layer-is-not-a-layer
Dhilip Kumar: From Unstructured Chaos to Queryable Knowledge: Designing Ontology-Backed Knowledge Graphs for Agentic Applications
Traditional data engineering requires you to define your schema (ontology) before extracting data. The author argues this doesn't work for unfamiliar domains. Instead, you should use LLMs to explore unstructured data and extract candidate entities and relationships first. A human then reviews these to create a "seed ontology," allowing the schema to evolve organically from the actual data.
Oracle: What Is a Converged Database? Definition, Five Tests, and AI Use Cases
StarTree recently announced full text search support on top of Iceberg. We started seeing the Lakehouse evolving from simple columnar storage to support variants, text search, and vector indexing. Oracle terms this a converged database, differentiating it from multi-model databases.
Sponsored: AI Modernization Guide
AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.
Netflix: How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC execution API
Making a remote, distributed graph feel like an in-memory data structure requires very specific trade-offs focused on I/O management, concurrency control, and early filtering. The author writes about optimization around breadth-first traversal, an async-first concurrency model, push-down streaming filters, and volatility-aware caching.
Olx: Correlation Lied to Us - Rethinking Product Impact with Causal Inference
You cannot evaluate a product's impact just by looking at aggregate metrics, because user behavior (who chooses to use the product) skews the results. Olx writes about a classic study showing that comparing different categories of ads is often misleading, and the importance of adopting the Propensity Score Matching (PSM) method to compare similar products' performance.
Mohamed El Zein: How I managed to build and scale Flink to 300+ million RPM in a stateful exactly-once workflow, and how you could too
The author writes a battle-tested "survival guide" for engineers who are pushing Flink to its absolute limits in production environments. Scaling Flink is incredibly difficult and often involves a frustrating cycle of solving one configuration problem only to have it trigger another. To save other advanced engineers from this trial-and-error headache, the author outlines 13 specific optimizations, undocumented fixes, and pitfalls to avoid.
Netflix: TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
Netflix writes about its architectural journey of solving a massive data engineering problem: how to store cost-effectively and query petabytes of historical time-series data without sacrificing performance or adding operational complexity. It is an interesting case study to think about turning static disaster recovery backups into an active, low-latency cold storage tier—can yield massive efficiency and cost savings at scale.
Flipkart: LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search
Flipkart writes about automating the tedious and expensive process of evaluating search result relevance by replacing human annotators with a specialized LLM. The most interesting part for me is that the pipeline now acts as a diagnostic tool; because every judgment includes a reasoning trace, engineering and catalog teams can pinpoint exactly why a search result failed.
AWS: Introducing Apache Spark troubleshooting agent for Amazon EMR on EKS
One of the fundamental shifts with LLMs is that the expert operator model is vanishing. The go-to guy for xyz tool is changing, and systems thinking is making a more and more prominent impact in organizations.
The AWS Spark agent tool simplifies the expert operator model and democratizes building data pipelines.
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.