Data Engineering Weekly #281
The Weekly Data Engineering Newsletter
How to Build a Data Platform From Scratch
We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.
In this book, you’ll learn:
- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly
beamnxw: Agent Harness Engineering vs. Loop Engineering vs. Graph Engineering
Harness engineering builds the machinery around the model
Loop engineering designs the repeated work-and-feedback cycle
Graph engineering makes the workflow topology explicit: nodes, branches, joins, state transitions, and controlled cycles
Regardless of the buzzword apocalypse, it is an exciting read to understand various architectural patterns emerging in building intelligence.
https://x.com/beamnxw/status/2081022966645535079
Thoughtworks: Closing the context gap: An ontology + LLM approach to data modernization
I believe the next 5 years of data engineering will be focused on preparing data for AI-readiness, which means we need to rethink the fundamental principles of data modeling techniques.
That may elevate semantic and ontology modeling as a higher-level representation of business meaning, while dimensional modeling remains an important physical design pattern—potentially generated or maintained more automatically(like compilers) from those semantic definitions.
Netflix: GenRec - Towards LLM-Native Recommendation at Netflix
Netflix writes about GenRec — post-training a Netflix-adapted foundation LLM that verbalizes user history into prompts and scores the entire catalog in one prefill-only forward pass. The ranker beats a mature production system with 40× fewer labels, shifting effort from feature engineering to context engineering, anchored to a shared foundation backbone.
https://netflixtechblog.com/genrec-towards-llm-native-recommendation-at-netflix-f20be6f643e3
Sponsored: AI Modernization Guide
AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.
Airbnb: Eval-driven development: Lessons from evaluating GenAI at scale
Airbnb writes about eval-driven development — layering programmatic checks, LLM judges calibrated to high-80s agreement, and human review so failure modes surface before shipping. The discipline extends to agent trajectories, reconstructing traces to score tool calls and reasoning paths, anchored to golden datasets that include failures alongside successes.
Grab: Crowdsourced taxonomy verification: A feedback-driven framework for refining knowledge graph relationships via online search interactions
Along with data quality, the emerging problem in data engineering is how to maintain the accuracy and the structural integrity of the knowledge graph at a massive scale.
The authors treat unverified edges as hypotheses — injecting candidates into low-risk search slots, weighting purchases over clicks, and promoting or pruning by confidence ratio. The closed loop validates structural links instead of entities alone, pruning hallucinated relationships without manual curation, anchored to live traffic as the verification environment.
https://engineering.grab.com/crowdsourced-taxonomy-verification
Spotify: Indexing the Data Lake for Online Point Queries
The indexes must inevitably come to the Lakehouse as the workload pattern increases. Iceberg v4 spec talks about building secondary indexes; Apache Hudi, along with RLI (Record Level Indexing), is expanding to secondary indexes. Most importantly, Apache Pinot extends its indexing support to Iceberg to enable fast queries on Iceberg. Spotify's story emphasizes the need for indexing support for more point queries in the Lakehouse.
https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries
Atlassian: Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events
Atlassian writes about challenges running Kinesis at scale with the increased pressure on scaling cost, API reliability, and data retention. The blog talks about buy vs build decisions on operating Kafka and highlights challenges in adopting Kafka at scale.
Etsy: Kafka App? There’s a Skill for That
Agents work magic when the underlying infrastructure is well connected, and the developer workflow is reimagined from an agentic workflow perspective. Etsy shares such a story of a connected Kafka app builder and its developer workflow.
https://www.etsy.com/codeascraft/kafka-app-thereas-a-skill-for-that
Claude: Your index is half empty. Postgres can’t squeeze it.
Postgres’s B-tree maintenance cliff is real: VACUUM removes dead tuples, but it cannot turn half-empty leaf pages back into disk space, leaving REINDEX CONCURRENTLY as the costly reset. The author’s GSoC prototype offers a credible incremental path—merge eligible siblings, retain a tombstone until the transaction horizon passes, then let VACUUM reclaim it. The hard part is correctly supporting concurrent scans, so this is promising maintenance infrastructure, not yet a free compaction switch.
https://claude.ai/public/artifacts/8a251e82-2737-47d7-bed1-f66423cab3eb
Eduardo: All you need is PostgreSQL
The main thesis of this blog post is that modern engineering teams often overcomplicate their systems by defaulting to specialized, trendy tools (like in-memory caches or event-sourcing databases) when vanilla PostgreSQL is powerful enough to handle the vast majority of complex data requirements. The author refers to this reflex as “architectural cargo culting,” which inflates operational risk, headcount demands, and system complexity for no good reason. The author builds a highly constrained, high-performance financial transactions database (similar to a banking API) using only standard PostgreSQL features.
https://ebellani.github.io/blog/2026/all-you-need-is-postgresql/
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.