Data Engineering Weekly #280
The Weekly Data Engineering Newsletter
Editor’s Note: leetdata.ai & aidataengineer.io updates
Last week we were busy fixing some bugs and upgrading a few features.
aidataengineer.io now supports search over DEW articles & a seamless browsing experience on all the articles published so far.
We fixed bugs in subscriptions in leetdata.ai, and we're thrilled to see more paid subscribers there. Python code practices now tuned for pandas.
We are actively working on two things.
Cohorts for leetdata.ai: traditional cohort focus on a tool, but we are building an agent to build cohorts reflecting real-world design by mining the DEW articles.
Simulation Learning model, where a real-world outage/scenario is simulated, and you work to learn it. We are actively looking to partner with data tooling companies on this - more exciting development coming soon!!!
Joe Reis: To Every Agent Its Own Database
Are we prepared for the data infrastructure shake-up for the agentic workload? The author proposed a DuckDB-based solution with each agent having its own copy of the data to work with. Though I agree there are a few versioning solutions available, they are still built for humans to resolve conflicts. Data can’t afford to have multiple versions of the same facts locked by different agents. I believe this is the area where next-generation data tools will invest a lot.
https://joereis.substack.com/p/to-every-agent-its-own-database.
Datafold: The Modern Data Stack - Open-source edition
There has been a lot of debate about open source last week. I found this to be an excellent work on gathering all the open-source tools remaining in the modern data stack.
https://www.datafold.com/blog/the-modern-data-stack-open-source-edition/
OlegWock: Guide to data tools landscape for developers
Another existing playbook talking about the data tools landscape. If you’re a beginner to the data industry, this is an excellent overview to understand the landscape of data engineering.
https://sinja.io/blog/data-landscape-guide-for-developers
Housing: Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently)
I love the concept of a single source of truth. Too bad we have to wait until we are living in the singularity to actually see it work.
The author talks about the importance of having a certification process for standard metrics and the organizational alignment over it. The certification model is something that should be baked into every data product's deliverables.
Hex: The life and times of data teams in the AI era
The article reiterates the importance of data quality in the AI era and how it will play a vital role in data engineering. Getting data into people at the right time, with high quality, will still be a challenge. The challenge is multi-fold as traditional certified dashboard building is slowly fading, and headless chat focuses on intelligence emerging.
https://hex.tech/blog/data-teams-in-ai-era/
Bojan Ciric: The Knowledge Spine: Why Your Ontology Needs to Grow a Backbone
Ontologies fail when they remain documentation rather than production infrastructure. The author’s “knowledge spine” makes the useful shift: version concepts like code, map them virtually to governed sources, federate domain graphs, and make agents resolve meaning through the same queryable backbone. The architecture is persuasive, but central semantics cannot manufacture consensus; success still depends on domain ownership, tested mappings, and stewards empowered to resolve conflicting definitions.
JioHotstar: Journey of an Ad Request: The Hidden Engineering
Ad-tech is a compact lesson in real-time data systems: selecting an ad is not a ranking problem alone, but a constrained allocation problem under a 100 ms deadline. The author describes JioHotstar’s blend of eligibility filtering, frequency caps, pacing, waterfall priority, and uncertain programmatic fill. The durable data-engineering challenge sits after serving—event pipelines must reconcile every decision, impression, and conversion into trustworthy delivery and attribution metrics.
https://blog.hotstar.com/journey-of-an-ad-request-the-hidden-engineering-e7b7f9921e46
Netflix: In-House LLM Serving at Netflix
Netflix’s in-house LLM platform is compelling because it refuses to treat LLMs as a separate product category. The hard-won lesson is that model quality lives in serving details: guided decoding, batch-aware constraint logic, API compatibility, and observability determine whether a prototype can become dependable infrastructure.
https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c
Airbnb: Personalizing Airbnb search by learning from the guest journey
Airbnb writes about compressing seven years of high-signal history and 21 days of views into daily guest embeddings, then combines them with live queries at serving time. This is an exciting area of data pipelining, thinking in terms of embedding models instead of relational.
Artem Krylysov: How MVCC and Transactions Work in RocksDB
RocksDB’s LSM design supplies the raw material for MVCC—immutable versions and sequence numbers—but snapshots, SuperVersions, and compaction rules make concurrent reads safe in practice. The crucial warning is that native transactions are a local, configurable layer, not a blanket serializability guarantee: optimistic commits favor low contention, pessimistic locks favor hot keys, and range phantoms still need explicit range locking.
https://artem.krylysov.com/blog/2026/07/23/how-mvcc-and-transactions-work-in-rocksdb/
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.