And ETL is still dead.

Not dead the way Latin is dead — no one speaks it. Dead, the way landlines are dead — they still work, millions exist, but nobody builds their communication strategy around one. ETL is dead as the defining work of data engineering. Dead as the thing we hire for, build careers around, and organize teams to do. The pipelines keep running. The professional identity built around them does not survive.

The Warehouse Was Always a Metaphor. Now the Metaphor Is Breaking.

We literally called it a data warehouse. And that wasn’t just naming — we replicated the entire physical warehouse operating model into the digital world. Racks became tables. Inventory management became catalogs. Forklifts became ETL pipelines. Floor workers became data engineers. Shift supervisors became analytics leads.

Every technique we built — star schemas, slowly changing dimensions, medallion architectures, conformed dimensions — served the same purpose as aisle markers and shelf labels in a physical warehouse: help a human walk in, find what they need, and carry it out.

Data modeling organizes information so humans can discover it. Data catalogs provided wayfinding to help humans navigate them. The medallion architecture created a pick-pack-ship assembly line where humans inspected and validated data at each station. Naming conventions — fact_orders, dim_customers — acted as signage so humans could read the shelves at a glance.

Every design decision is optimized for human cognition. And then the operator changed.

What Happened When Robots Entered the Physical Warehouse

When Amazon deployed Kiva robots, they didn’t replace human tasks one-for-one. They redesigned the entire warehouse around a different operator.

Physical warehouses built for humans had wide aisles because humans need space to walk. They grouped items logically because humans need to remember where things are. They placed high-demand products at eye level because humans have ergonomic constraints. They posted signage everywhere because humans need wayfinding.

Robotic warehouses threw all of that out. Aisles shrank because robots don’t need shoulder width. Shelving went floor-to-ceiling because robots don’t have ergonomic limits. Logical grouping became unnecessary because robots navigate by coordinates, not memory. Signage disappeared because robots don’t read signs — they read instructions.

But the biggest gains weren’t physical. They were cognitive. Human warehouse workers carried an enormous cognitive load — remembering locations, making routing decisions, prioritizing picks, and mentally handling exceptions. Robots eliminated that cognitive burden entirely. The warehouse didn’t just move faster. It became a fundamentally different system that could handle complexity no human floor operation could manage.

The Data Warehouse Is Still Designed for Human Forklift Operators

Now look at our data warehouse through this lens.

Star schemas and dimensional modeling exist so a human analyst can visualize how tables relate. A human needs to see the star — the fact table at the center, dimensions radiating outward. An agent doesn’t need a star. It needs a validated semantic definition of what each entity means and how entities connect.

Data catalogs are digital signage. We built them because humans need to browse and discover what’s in the warehouse. An agent doesn’t browse a catalog the way a human walks an aisle. It queries for a validated meaning.

The medallion architecture — Bronze to Silver to Gold — is an assembly line designed for human inspection at each station. Raw data lands, gets progressively cleaned, and arrives ready for consumption. Each station assumes a human will inspect, validate, and pass the data forward. And at each handoff, context erodes — the original meaning collapses a little more, like a game of telephone played silently in the pipeline.

We optimized every layer of the data warehouse for human cognitive constraints. And just like the physical warehouse, those very optimizations become limitations when the operator changes.

Where the Analogy Holds — and Where It Breaks

I want to be precise about this, because imprecise analogies are how our industry ends up with decade-long hype cycles built on half-truths.

The analogy holds powerfully for navigation and discovery. Physical warehouses organized shelves for human wayfinding. Data warehouses organize tables for human querying. Robots don’t need aisle signs. Agents don’t need star schemas to find data. That part maps cleanly.

But here’s where it breaks: physical goods don’t change meaning based on how you store them. A box of shoes is a box of shoes, whether it sits on shelf A3 or shelf Z9. Data is different. How you structure data shapes what questions you can ask of it. A normalized schema enables different analytical patterns than a denormalized one. A slowly changing dimension preserves the temporal context that a snapshot table destroys.

Structure still matters for agent-operated data. It just serves a different purpose. Instead of organizing for human navigation — “how do I find the data?” — you organize for agent operation — “what data and context does this agent need for this task?” Think about how AI tools work with a scoped working folder. You don’t reorganize your filesystem into an agent-friendly layout. You give the agent a well-scoped boundary, and it operates within it. The structure shifts from navigational to operational — from shelf labels to access boundaries.

The Thinking Survives. The Format May Not

I took the last class Ralph Kimball taught before his retirement. I remember the vivid conversation around HBase (which was popular at the time) and the notion of versioning to handle slowly changing dimensions. I’ve internalized dimensional modeling deeply enough to know which parts are permanent and which parts are artifacts of their era.

Kimbal didn’t start the training with the star schema and slowly changing dimensions. Kimball’s dimensional modeling process starts with two steps: identify the business process and select the grain. These steps ask the most fundamental questions in data engineering — what does the business actually do, and at what level of detail does it matter? Only after answering those do you design the dimensions, the facts, and the star schema.

Steps one and two are context architecture. They always were. Identifying the business process means understanding the semantic reality of what the organization does. Selecting the grain means choosing the level of meaning that matters. That thinking is more relevant today than it was in 1996.

Steps three and four — the star schema, the dimension tables, the fact tables — were a rendering choice. They were the best output format for the consumer of that era: a human analyst writing SQL against a relational database. The star schema serialized business understanding into a structure that humans could query using the available tools.

The consumer has changed or is changing. The rendering should too. When the consumer is an AI agent, the same analytical thinking about business processes and grain produces a Context Store entry — a validated, versioned, queryable semantic definition — not a fact table. The thinking survives. The format may not.

Dismissing dimensional modeling entirely would be ignorant. Clinging to its output format when the consumer has fundamentally changed would be equally so.

The Pendulum

Every era of data architecture has tried to solve the same tension: semantic precision versus operational flexibility.

The relational era chose precision. ERDs, primary keys, foreign keys, referential integrity, constraints — the schema was the semantic contract. Bill Inmon’s Corporate Information Factory formalized this into an enterprise architecture. It worked. It encoded business meaning directly into the physical structure. But it was rigid. I remember interviewing at a company in the pre-Hadoop era and asking what their current priority was. The interviewer told me they were working on implementing a schema change in a day rather than a month. That was the state of the art — a month to add a column, because the semantic contracts were so tightly welded to the physical structure that touching one meant touching everything.

Hadoop’s answer was brute force. Sheer machine power, schema-on-read, commodity hardware — throw everything in and figure it out later. It broke the operational rigidity overnight. And it also broke every semantic contract the relational era had built. We traded meaning for speed and went too far. The data lake became a data swamp because nobody could remember what anything meant — the constraints that encoded that meaning were gone.

The lakehouse tried to find a middle ground. Iceberg, Delta, Hudi — the flexibility of the lake with some structure of the warehouse. Better. But the semantic layer remained an afterthought.

catalogs, documentation, and governance overlays that nobody maintained because nobody’s career depended on them being right.

Even recent efforts like Snowflake’s Open Semantic Interchange initiative acknowledge the gap — the industry is only now trying to standardize how semantic meaning travels between tools.

Each swing of the pendulum traded one problem for another. Rigidity for meaninglessness. Meaninglessness for a partial structure. What none of them achieved was decoupling — semantic precision that doesn’t require physical rigidity. Context that travels alongside the data but isn’t welded to the table structure. Change the schema in seconds. The context updates through the Contextualize pipeline. The meaning stays current without the rigidity.

That decoupling is what ECL provides. It’s the first architecture that doesn’t force you to choose between knowing what your data means and being able to change it.

The Graveyard of Good Intentions

I know what the skeptics are thinking, because I’ve thought it myself: we’ve heard this before.

Bill Inmon literally wrote the book on this in 2007 — Business Metadata: Capturing Enterprise Knowledge — which covers semantics, ontologies, business rules, and the capture of tacit knowledge. He laid out a complete methodology for capturing it. The methodology was sound. The economics weren’t there yet.

Business glossaries in the 2000s promised to capture institutional knowledge. They became static documents that nobody updated. Semantic layers in the 2010s promised a unified layer of meaning. They became another piece of middleware to maintain. Data catalogs promised discoverability and governance, but soon proved to be useless. Many became expensive shelfware. Enterprise knowledge graphs promised connected meaning. Most never made it past the proof-of-concept stage.

Every generation of data practitioners has pointed at the same north star: capture business meaning as a first-class artifact. Every generation has underestimated the organizational gravity that pulls teams back to “just get the data there, and we’ll figure out what it means later.”

So what makes this time structurally different? One thing: the consumer changed from forgiving to unforgiving.

When the consumer was a human analyst, missing context was inconvenient. The analyst would Slack a colleague, read the dbt code, ask in standup, and check the wiki. Humans are remarkably good at filling semantic gaps through social channels. Bad metadata produced frustrated analysts, not system failures.

When the consumer is an AI agent, missing context produces systematic errors at scale. The agent doesn’t Slack anyone. It doesn’t read tribal knowledge. It sees a column called rev_adj, makes its best inference, and acts — confidently, consistently, and potentially wrong across every downstream decision. Bad context doesn’t produce frustration. It produces hallucination at an enterprise scale.

For the first time, the cost of missing context exceeds the cost of maintaining it. That economic inversion is what none of the previous attempts had. Business glossaries failed because humans bore the cost of maintaining them, while the benefit was diffuse. The Context Store succeeds or fails based on whether agents produce reliable results — and that feedback loop is immediate, measurable, and impossible to ignore.

The graveyard is real. But the economics changed.

What Replaces It

ETL asked: Did the data land? ECL asks: Can the data be trusted? I introduced the ECL framework in my earlier article on data engineering after AI.

Extract remains. Data still moves from source systems to analytical environments. That work still requires engineering judgment about reliability, latency, and failure modes. AI handles more of the mechanical construction. Humans make the architectural decisions.

Contextualize is the new center of gravity. A dedicated, agentic pipeline that builds and maintains a living store of semantic context. It isn’t documentation. It isn’t a catalog. It’s an engineering artifact with its own trigger model, validation layer, and storage — the Context Store.

The Context Store holds two types of objects. Context objects capture long-lived semantic definitions — what “revenue” means, who validated that definition, when, and at what confidence level. These compounds increase in value over time. Decision objects capture what agents produce when they act on context — which definitions they used, what they inferred, and what they recommended. These create the audit trail.

Link connects entities across the data landscape — and emerging standards like Model Context Protocol (MCP) are starting to standardize how agents access data without moving it. Not just table joins — semantic relationships between business entities across systems. A customer in CRM is linked to a user in your product, linked to a session in your support tool. Whether you implement that as a graph, a mapping table, or a markdown file matters less than whether the linkage is validated and the semantic relationship is explicit.

And because data is inherently social in nature, you don’t build this all at once. You start with one business flow. One critical table. Early bind where you control the data and can hold producers accountable for meaning. Late bind where data comes from outside your accountability boundary — third-party feeds, undocumented internal systems, legacy data where the person who knew what the fields meant left five years ago. Even one table, well contextualized, starts compounding as you connect it to the next one, and the next one.

Long Live the Context Architect

The physical warehouse workers who resisted robotics didn’t save their jobs. They delayed their own transition. Those who moved into robotics coordination, system design, and exception architecture found themselves more valued, more strategic, and more central to the operation than they were when driving forklifts.

Data engineers who built their identity around moving data from one bucket to another have felt that identity under pressure for a while now. That pressure isn’t going away. AI will write your Spark jobs. AI will generate your dbt models. AI will build more pipelines in a year than your team could build in a decade.

But AI cannot decide what “revenue” means for your organization. It cannot negotiate data contracts between producing and consuming teams. It cannot design the appropriate level of context for an agent addressing a specific business problem. It cannot build the organizational agreements that make semantic definitions stick. That work requires institutional knowledge, cross-functional coordination, and architectural judgment. That work is context architecture.

The data engineer’s value migrates from pipeline reliability to semantic reliability. From “the job ran” to “the meaning is right.” From operating the warehouse floor to designing the system that makes robotic operation trustworthy.

The frontier is genuinely open. Nobody has this figured out yet. The practitioners who invest in the architecture of meaning — not just the mechanics of movement — will define this discipline for the next decade.

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Best practices for orchestrating Databricks at scale

As Databricks deployments scale, a familiar pattern emerges: multiple workspaces, multiple teams, and no reliable way to manage the dependencies between them.

In this hands-on deep dive, we'll show you how to build a cross-workspace control plane using Dagster on top of your existing Databricks environment. Demo-heavy and practitioner-focused, you'll leave with working patterns you can apply to your own platform the same day.

Save your spot now

underCurrent: A one-day conference for data engineers and architects

Confluent is hosting a free one-day conference with a catch: there’s no catch. It’s a single-track event with no sponsors and no product pitches—just technical talks for data engineers and architects.

🎙️ Speakers include Joe Reis, Holden Karau, and Max Beauchemin
🚫 No vendors. No sales pitches
✨ 100% free to attend
📅 March 26
📍 San Francisco
🎟️ Limited to 100 seats — register for free here

Vinoth Govindarajan: OpenClaw Architecture

Production AI agents fail at scale because uncontrolled state mutations corrupt execution and create unpredictable behavior. In “The Agent Stack,” Vinoth Govindarajan outlines OpenClaw’s architecture, in which isolated execution contexts and strict invariants prevent state leakage, while sessions enable async pause-resume semantics. The pattern standardizes how teams decouple short-term context from persistent state, ensuring agents reliably rehydrate their mental model and enforce authorization boundaries that gate tool access to user privilege levels.

Part 1, Part 2, Part 3.1, Part 3.2, Part 4

Pinterest: Unified Context-Intent Embeddings for Scalable Text-to-SQL

Navigating sprawling data warehouses forces analysts to choose between slow manual exploration and unreliable keyword-based search. Pinterest Engineering built a production Analytics Agent that embeds historical SQL queries as semantic intent signatures, injecting business glossary terms and extracting structural patterns (join keys, filters, usage signals) to retrieve contextually relevant tables at scale. The system reached 40% internal adoption within two months by standardizing discovery through an asset-first pattern, converting years of institutional SQL knowledge into a searchable, governance-aware library.

https://medium.com/pinterest-engineering/unified-context-intent-embeddings-for-scalable-text-to-sql-793635e60aac

Francesca Lazzeri: AI evals platforms: A comparative guide for production AI systems

Production AI systems fail silently in ways demos never expose, forcing teams to replace manual testing with automated evaluation as the enterprise LLM market scales toward $71.1 billion by 2034. A comparative analysis of six leading eval platforms reveals a consolidation around open standards (OpenTelemetry, OpenInference) and specialized architectures—Microsoft AI Foundry embeds red-teaming agents into Azure workflows, while Galileo replaces expensive LLM judges with smaller consensus models (Luna) to reduce eval latency. The shift standardizes safety as a structural property of development, enabling teams to catch jailbreaks and data leaks early while choosing platform fit based on stack priorities: simulation-first, research rigor, or ecosystem depth.

https://medium.com/data-science-at-microsoft/how-do-you-know-your-ai-actually-works-b1a380a07825

Sponsored: The AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Netflix: MediaFM - The Multimodal AI Foundation for Media Understanding at Netflix

Understanding content at scale requires machine-readable representations that capture narrative structure, not just visual features—a challenge intensified as streaming catalogs exceed tens of thousands of titles. Netflix built MediaFM, a tri-modal transformer that fuses video frames, audio (wav2vec2), and subtitles into shot-level embeddings using Masked Shot Modeling, with a [GLOBAL] token injecting title-level context (synopsis, genre) to ground each segment. The model powers ad placement, clip ranking, content tagging, and cold-start recommendations by contextualizing shots within narrative sequence, outperforming external benchmarks and enabling machine-readable understanding across Netflix's entire catalog.

https://netflixtechblog.com/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d

Nabin Debnath: Building a Least-Privilege AI Agent Gateway for Infrastructure Automation with MCP, OPA, and Ephemeral Runners

AI agents in infrastructure automation bypass traditional guardrails by making runtime decisions without human validation, risking silent resource destruction or credential exfiltration at scale. The author writes about the Agent Gateway to treat the agents as untrusted requesters, layering Model Context Protocol (MCP) for tool discovery, Open Policy Agent (OPA) for intent-based authorization, and ephemeral Kubernetes runners for isolated execution. The pattern enforces least privilege by mediating all API calls through policy code, validates plan integrity against immutable hashes, and surfaces decision reasoning via OpenTelemetry—standardizing agent governance with SLO targets (100ms policy decisions, 5s runner startup) that prevent silent bypasses.

https://www.infoq.com/articles/building-ai-agent-gateway-mcp/

Dropbox: Using LLMs to amplify human labeling and improve Dash search relevance

Enterprise search ranking requires massive labeled datasets, but traditional human annotation is prohibitively slow and cannot scale to sensitive content across billions of internal documents. Dropbox Dash uses LLMs as labeling force multipliers by calibrating a small human-labeled set to generate millions of relevance judgments offline, then training lightweight production models (XGBoost) on synthetic labels at scale. The pattern standardizes judgment consistency by pairing contextual research tools (for acronyms and ambiguous queries) with programmatic prompt optimization (DSPy), enabling continuous ranking improvements while keeping human oversight as the ground truth rather than replacing it.

https://dropbox.tech/machine-learning/llm-human-labeling-improving-search-relevance-dropbox-dash

Zalando: Why We Ditched Flink Table API Joins: Cutting State by 75% with DataStream Unions

Declarative SQL joins in Flink multiply state across operators, forcing teams to choose between snapshot overhead or operational instability—a scaling bottleneck for pipelines enriching millions of real-time product records. Zalando replaced chained Table API joins with a custom KeyedProcessFunction that unions all streams into a single keyed DataStream, storing each product’s enriched state once in RocksDB instead of redundantly across join operators. The shift cut state size by 75% (235GB to 56GB), reduced snapshot time by 77% (11 minutes to 2.5 minutes), and lowered AWS costs by 13%—demonstrating how imperative control over stream topology recovers efficiency when declarative abstractions misalign with physical execution.

https://engineering.zalando.com/posts/2026/03/why-we-ditched-flink-table-api-joins-cutting-state.html

Aihua Xu & Andrew Lamb: Variant Type in Apache Parquet for Semi-Structured Data

Semi-structured data in columnar formats forces a choice between slow JSON parsing or rigid schemas that block evolution, creating friction in pipelines handling heterogeneous records. Apache Parquet’s new Variant type uses binary-encoded metadata plus value fields, enabling direct nested field access without full-document parsing while preserving native types (timestamps, integers) that JSON loses. The type standardizes schema flexibility through “shredding”—extracting hot fields into strongly-typed columns for predicate pushdown and pruning—allowing heterogeneous records to coexist in one column, reducing migration overhead and accelerating adoption across DuckDB, Spark 4.0, and Snowflake.

https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/

Pranav Mehta: Silent Data Loss in ClickHouse: 3 Reasons Your Distributed Queue Keeps Growing

ClickHouse distributed inserts silently fail when coordination services downtime, execution timeouts, or concurrency limits block the async flush pipeline, leaving data trapped in on-disk queues while clients receive no error signals. The author identifies three failure modes: Keeper/ZooKeeper downtime forcing ReplicatedMergeTree read-only, oversized insert blocks exceeding max_execution_time that cork sequential queue processing, and exhausted user concurrency slots starving background INSERT workers. The pattern demands proactive monitoring of DistributedFilesToInsert (alert at 50+ files), debugging via system.distribution_queue.last_exception, and inode-aware filesystem choice (XFS over ext4) to prevent silent data loss and system crashes from queue explosion.

https://medium.com/@pranavmehta94/silent-data-loss-in-clickhouse-3-reasons-your-distributed-queue-keeps-growing-9bf6b8af88e5

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

AI is moving fast. Is your data platform ready?

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the AI Modernization Guide

underCurrent: A one-day conference for data engineers and architects

Confluent is hosting a free one-day conference with a catch: there’s no catch. It’s a single-track event with no sponsors and no product pitches—just technical talks for data engineers and architects.

🎙️ Speakers include Joe Reis, Holden Karau, and Max Beauchemin
🚫 No vendors. No sales pitches
✨ 100% free to attend
📍 San Francisco 📅 March 26
🎟️ Limited to 100 seats — register for free here

Netflix: DataJunction as Netflix’s answer to the missing piece of the modern data stack

Metric inconsistency and definition sprawl across distributed teams create onboarding bottlenecks and fragment analytics workflows. Netflix built DataJunction, an open-source semantic layer that decouples metric definitions from compute through a graph-based metadata model and SQL generation engine. This standardizes metrics across the experimentation platform, reducing onboarding from weeks to hours, while enabling expansion across all business verticals and LLM integration for auditable metric lineage.

https://netflixtechblog.medium.com/datajunction-as-netflixs-answer-to-the-missing-piece-of-the-modern-data-stack-92af926b40a5

Benoit Pimpaud: Specs Should Be Equations, Not Essays

As AI automates code generation, the engineering bottleneck shifts from writing implementation to defining precise specifications. the author argues that natural language specifications create compounding ambiguity when parsed by LLMs and proposes layered specifications that combine text, diagrams, and mathematical notation as constraint definitions for AI iteration. Mathematical specs eliminate interpretation drift, enabling AI agents to generate correct programs by satisfying invariants rather than reconstructing intent from prose.

https://fromanengineersight.substack.com/p/specs-should-be-equations-not-essays

Notion: Balancing cost and reliability for Spark on Kubernetes

Spark clusters on Kubernetes face a fundamental tension between aggressive cost optimization through spot instances and job reliability during capacity interruptions. Notion reduced compute costs by 60–90% through EKS migration with Karpenter bin-packing, then open-sourced Spot Balancer—a Kubernetes webhook that enforces stable spot-to-on-demand ratios per job, preventing cascade failures during AWS termination windows. Spot Balancer abstracts infrastructure trade-offs into developer-friendly stability tiers, enabling teams to optimize costs without sacrificing job completion rates.

https://www.notion.com/blog/balancing-cost-and-reliability-for-spark-on-kubernetes

Sponsored: Building a Cross-Workspace Control Plane for Databricks

As Databricks deployments scale, a familiar pattern emerges: multiple workspaces, multiple teams, and no reliable way to manage the dependencies between them.
In this hands-on deep dive, we'll show you how to build a cross-workspace control plane using Dagster on top of your existing Databricks environment. Demo-heavy and practitioner-focused, you'll leave with working patterns you can apply to your own platform the same day.

Register now

Apache Iceberg: Introducing the Apache Iceberg File Format API

It is indeed an exciting development in Iceberg to support a plugable file format API spec. As we increasingly handle unstructured data, this will significantly enhance data management practices through unified governance and compliance. Interestingly, Apache Hudi’s RFC-100 is, in fact, the feature request to support the Lance File Format.

https://iceberg.apache.org/blog/apache-iceberg-file-format-api/

Delta Lake: The next evolution of Delta - Catalog-Managed Tables

We went through the full cycle, from exposing the files directly through Hadoop to Snowflake-style cloud data warehouses, to Iceberg-style direct file access, back to catalog-managed tables.

Nonetheless, it will be interesting to watch DuckLake-style catalog-managed tables vs object-store-style managed tables.

https://delta.io/blog/2026-02-02-delta-catalog-managed-tables/

Microsoft Fabric: Under the hood: an introduction to the Native Execution Engine for Microsoft Fabric

The Apache Gluten project is continually making an impact on the Spark ecosystem, bringing unique optimization and efficiency. Microsoft Fabric writes an under-the-hood story of adopting Apache Gluten in its Fabric platform.

https://blog.fabric.microsoft.com/en-us/blog/under-the-hood-an-introduction-to-the-native-execution-engine-for-microsoft-fabric/

Pinterest: Piqama - Pinterest Quota Management Ecosystem

As companies scale, manual and static quota systems become bottlenecks, forcing engineers to choose between over-provisioning resources and managing brittle enforcement logic. Pinterest developed Piqama, a unified quota platform that dynamically right-sizes limits using historical data stored in Apache Iceberg, then applies custom enforcement strategies across batch schedulers and online services. Piqama centralizes resource governance across hardware and service metrics, enabling teams to optimize capacity allocation while linking consumption directly to financial costs.

https://medium.com/pinterest-engineering/piqama-pinterest-quota-management-ecosystem-dc7881433bf5

LinkedIn: Engineering LinkedIn’s job ingestion system at scale

Ingestion systems struggle to scale source onboarding—hard-coded extraction logic creates engineering bottlenecks that slow integration of new data partners. LinkedIn shifted extraction logic from code to configuration files called Sitemaps, enabling AI tools and browser plugins to onboard sources without engineering deployments. At the same time, a transactional state machine enforces precise failure boundaries across parallel mining tasks. The configuration-driven approach reduces onboarding time from weeks to hours, allowing LinkedIn to ingest 20TB daily across thousands of global sources.

https://www.linkedin.com/blog/engineering/infrastructure/engineering-linkedins-job-ingestion-system-at-scale

Shopify: The generative recommender behind Shopify’s commerce engine

Recommendation systems traditionally treat purchases as isolated events, missing the temporal and causal structure that shapes buyer journeys across millions of products. Shopify transitioned to an autoregressive sequence model that treats commerce journeys as token sequences, implementing RoPE-inspired rotary encoding combined with relative attention bias to capture temporal gaps and seasonality across its catalog. The time-aware attention mechanism drove +0.94% order growth and +0.71% conversion lift while achieving 7.3x training speedup through optimized CUDA kernels, enabling Shopify to integrate richer context into a unified generative framework.

https://shopify.engineering/generative-recommendations

Alibaba: PostgreSQL Blink-tree Implementation

As we increasingly use AI to code, understanding database internals is more critical than ever. Alibaba Cloud engineers break down how PostgreSQL utilizes the Blink-tree architecture to achieve massive concurrency. By adding link pointers to sibling nodes and high keys to mark boundaries, PostgreSQL allows searches to proceed without lock-coupling. This enables the system to gracefully handle concurrent page splits—following links when data exceeds old boundaries—and significantly outperforms the more rigid lock-subtree approach used in MySQL’s InnoDB.

https://www.alibabacloud.com/blog/postgresql-blink-tree-implementation_602913

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

The poll wasn’t specifically about data engineering, but the answer it yielded applies directly to us. When AI can generate a pipeline as fluently as a senior engineer, the question isn’t whether our toolbox is changing — it clearly is. The question is: what kind of thinking has always been too important to automate, and why we let it get buried under the more mechanical work in the first place.

My answer is that the irreducible work was never about moving data. It was always about meaning. And the framework we’ve been using — ETL — was never really designed to capture meaning.

The ETL Era and Why It’s Ending

Extract, Transform, Load made sense as a job description for a specific historical moment. Source systems were siloed, formats were inconsistent, and somebody had to write the code that moved data from where it lived to where it could be used. The data engineer was that somebody.

But if we’re honest, the transformation step was always the most brittle part. Teams encoded business rules as SQL logic or Python functions, buried them in pipeline code, version-controlled them alongside infrastructure, but rarely treated them with the same rigor as application code. When the definition of “active user” changed — and it always changed — someone had to find every place that definition lived and update it, hoping they caught them all.

AI is now competent at generating this kind of code. Not perfect, but competent enough that the mechanical work of pipeline construction is no longer a meaningful differentiator. If your professional identity is built around being good at writing transformation logic, that identity is under pressure.

But this isn’t a story about loss. It’s a story about clarity. The mechanical work was always obscuring the more important work underneath it. AI forcing that reckoning is, in a strange way, a gift.

Introducing ECL — Extract, Contextualize, Link

The framework emerging as a replacement isn’t a technical architecture so much as a reorientation of purpose. Instead of Extract, Transform, Load, think Extract, Contextualize, Link.

Extract remains. Data still needs to move from source systems to analytical environments, and that work still requires engineering judgment — about reliability, latency, volume, and failure modes. AI will increasingly handle the mechanical parts, but the architectural decisions about what to extract, when, and how belong to people who understand both the source systems and the downstream consequences.

Contextualize is where the real shift happens. This is the work of giving data semantic meaning — understanding that “revenue” is calculated differently by Finance and Sales, that a timestamp in a clickstream event means something different than a timestamp in a billing record, that a null value in one system represents the absence of information while in another it represents an explicit user choice. AI can draft this work at scale — inferring field definitions, classifying entities, and mapping relationships across a data landscape that no human team could manually annotate in full. What AI cannot do is be accountable for itself. The judgment of whether an inference is correct, the organizational authority to declare a definition, the decision to formalize a discovered pattern into an enforced contract — that belongs to humans. Contextualize is where AI inference and human judgment meet, structured by a pipeline built specifically for that purpose.

Link is about entity relationships across the data landscape — connecting a customer record in your CRM to a user record in your product database, linking an event in your analytics system to a session in your support tool. As AI generates more of the code that consumes data, the ability to reason about how entities relate across systems becomes more valuable, not less. Linkage is what makes context portable — what allows the meaning built in one part of the landscape to be grounded in its relationships to the rest.

The rest of this article discusses how ECL works at the architectural level, not as three abstract concepts, but as three concrete pipelines — and why you need all of them.

Early Binding — Contracts as Executable Constraints

The first technique is early binding: capturing semantic intent at the point of data production, before the data moves.

Data contracts are the practical implementation of this idea. At their core, contracts are agreements between data producers and their consumers — specifying schema, data quality expectations, ownership, and the semantic meaning of each field.

Data Engineering Weekly identified this gap precisely in their piece Data Contracts: A Missed Opportunity. While the data industry was debating what contracts were and drafting governance frameworks to describe them, software engineering had quietly converged on a different organizing principle: treating specifications as executable constraints with real failure semantics. The data industry treated contracts as documentation. Software engineers treated them as interfaces — things that could break, that had versioning implications, that enforced behavior rather than merely describing it.

A data contract that lives in a wiki and gets updated when someone remembers is the documentation. A data contract that is enforced at the point of production — that fails a pipeline when a schema changes without notice, that alerts a consumer when quality thresholds are violated, that an AI agent can reason about deterministically — that is architecture.

This matters more in an AI-heavy world, not less. When AI agents generate transformation code, bad contracts are amplified at scale. The agent will faithfully implement whatever logic it’s given; if the contract governing its inputs is ambiguous or unenforced, the errors it produces will be systematic rather than isolated. Early binding is the mechanism by which human intent gets formalized into something AI can actually work with.

But early binding alone has a fundamental limitation. And understanding that limitation is what makes the Contextualize pipeline necessary.

The Problem Early Binding Alone Can’t Solve

Consider what happens to a well-contracted dataset as it moves through a modern Medallion architecture.

At the Bronze layer, data lands close to its source — raw, minimally transformed, the contract’s guarantees still largely intact. Silver applies conformance rules: deduplication, type casting, and light standardization. By the time data reaches Gold, the pipeline has made a series of editorial decisions on the data’s behalf. Aggregations collapse granular events into metrics. Engineers bake business logic into the shape of the table. The Gold layer is an artifact optimized for a specific set of questions — the ones that seemed important when the pipeline was built.

Early binding contracts help at the source, but they can’t prevent this erosion at every subsequent hop — especially when those contracts are treated as descriptive rather than executable. If there’s no enforcement mechanism preventing meaning from drifting across transformations, the telephone game plays out silently in your pipeline. By the time a consumer queries the Gold layer, they’re working with an artifact whose original intent may be several editorial decisions removed from the contract.

This is the problem that early binding alone cannot solve. Each transformation layer progressively collapses the context captured at the source. You need a complementary approach—one that preserves the ability to recover context when it’s actually needed.

Late Binding — The Agentic Contextualized Pipeline

Traditional late binding deferred the application of business rules to query time. What it didn’t defer was the definition of those rules — domain experts still had to specify them upfront, just applied through a semantic layer rather than baked into a physical table. In complex domains, that knowledge engineering process was its own bottleneck.

The more forward-looking approach is to defer definition itself — and hand that work to a dedicated pipeline.

The Contextualize pipeline is a separate, agentic pipeline that runs alongside your data infrastructure. Its job is singular: build and maintain a living, validated store of semantic context. It isn’t part of the Extract pipeline. It isn’t a query-time process. It’s a first-class engineering artifact with its own triggering model, validation layer, and storage.

The trigger is event-driven, not scheduled. Every new dataset that lands automatically kicks off the pipeline. For existing datasets, continuous profiling monitors for meaningful changes — a new column appears, a column is dropped, a data distribution shifts in ways that suggest something changed upstream. Any of these events re-triggers the pipeline for the affected entities. Semantic context isn’t a one-time annotation exercise. It tracks the data as it evolves.

The pipeline itself is agentic. An AI agent analyzes the incoming data — schema, sample values, statistical profiles, lineage — and infers semantic meaning. What does this field represent? What business entity does it belong to? What relationships exist between it and other data in the landscape? It produces structured, versioned context artifacts: inferences about meaning that didn’t require a domain expert to pre-specify every scenario.

Those inferences don’t automatically commit. They route to a validation layer that works like a labeling workflow — because structurally, it is one. An LLM-as-Judge validates high-confidence inferences before any human review triggers. Medium-confidence ones surface to domain experts for labeling. The pipeline flags low-confidence or contested inferences for deeper investigation. The humans aren’t reviewing every artifact; they’re reviewing the uncertain ones. Every labeling automation technique that works in ML pipelines applies here.

Validated artifacts land in a Context Store — a dedicated, versioned, queryable store of semantic definitions, entity classifications, and relationship maps. This is the new infrastructure component that ECL requires. Downstream agents don’t query raw data and infer meaning on the fly. They query the Context Store first, ground their understanding in validated context, and then query the data. The context is stable, reusable, and auditable — the opposite of ephemeral query-time inference.

Early Binding vs Late Binding — When to Choose What

The decision criterion isn’t about semantic maturity or how well-understood a domain is. It’s about where the data comes from relative to your accountability boundary.

When a dataset originates within a controlled environment — produced by a team or system within your organization’s sphere of accountability — early binding is the right tool. The producer and consumer share an organizational context. Contracts can be negotiated, enforced, and held to. The producing team can be made accountable for the schema they declare and the semantics they commit to. Prescribed context is possible because the relationship that makes it enforceable exists.

When a dataset originates outside that boundary — third-party feeds, partner data, public datasets, marketplace sources — that relationship doesn’t exist. You cannot hold an external provider to a data contract. The schema can change without notice. The semantics are inferred, not declared. This is where the Contextualize pipeline earns its place. Discovered context is the only kind available.

But the boundary is not purely organizational. Poorly governed internal data — produced by a team with no accountability to its consumers, with undocumented schemas and inconsistent definitions — is effectively uncontrolled even if it sits within the same organization. The real test is not position on an org chart. It is accountability. Early bind where accountability exists. Let the Contextualize pipeline discover where it doesn’t.

The feedback loop holds in both directions. Discovered context built up through repeated profiling, inference, and validation can graduate into a prescribed context over time. An external dataset that your organization ingests consistently enough to profile, validate, and republish as an internal data product crosses the boundary from uncontrolled to controlled at that point. The Contextualize pipeline is what makes that transition possible — and makes the resulting contract trustworthy rather than assumed.

A data environment that treats all data as early-bindable is brittle. It can only contract what it already understands, and it has no mechanism for the uncontrolled data that makes up a growing share of the analytical landscape. A data environment that treats all data as requiring discovery never formalizes what it learns into enforceable guarantees. The architecture that works reads the accountability boundary correctly and applies the right technique on both sides.

Context Propagation — The Relay, Not the Pipeline

With three pipelines now in play, the question becomes: how does context actually travel through the architecture without getting lost?

The conventional mental model is wrong. Context doesn’t travel through the data pipeline—if it did, it would be lost at every transformation step, which is precisely the Medallion erosion problem. Context travels alongside the pipeline, as metadata, lineage records, and contract provenance. The transformations change the data; the metadata preserves the meaning.

The relay works like this. Early binding stamps prescribed context at the point of origin — schema, field-level semantics, producing team ownership, quality thresholds — as an executable contract living in metadata, not column values. Lineage tooling propagates this through Bronze, Silver, and Gold, maintaining a record of the transformations applied and the contract that governed the data at each stage. The Contextualize pipeline reads that lineage as part of its inference process — understanding not just what a field looks like today, but also the history of how it arrived and the commitments made about it at the source. Validated inferences land in the Context Store, which becomes the relay’s destination: a durable, queryable record of what the data means, grounded in both original contract and accumulated lineage.

The analogy that makes this concrete is git. A file can be modified heavily across dozens of commits — refactored, renamed, moved, rewritten — but the context of how it got there is never lost, because it lives in the commit history, not in the file itself. The Gold layer is the latest commit. The lineage graph is the git log. The Context Store is the understanding you build by reading that log systematically rather than hoping the current file tells the whole story.

This reframe — from pipeline to relay — changes what data engineers are actually responsible for building. The transformations are increasingly automatable. The metadata infrastructure, the lineage graph, the Contextualize pipeline that reads it, the Context Store that accumulates from it — that is the engineering surface that requires sustained human judgment.

The Context Store as the New Engineering Surface

Which brings us to where the most consequential engineering work has migrated.

The Context Store is where business definitions live — not as documentation in a wiki, not as logic engineers have baked into a Gold table, but as versioned, validated artifacts that downstream systems can query and trust. This is where the validation workflow resolves the competing interpretations of “revenue” from Finance and Sales — not organizational politics, but a confidence-based process that determines which inference earns formalization. Where AI consumers find the grounded, stable context they need to act reliably rather than reverting to ad hoc inference.

This surface distinguishes queryable data from trustworthy data. A table can be perfectly partitioned, indexed, and replicated while being semantically wrong — built on a definition that drifted from its source contract three transformations ago and never caught because no Contextualize pipeline was watching. The Context Store is where that failure mode gets closed.

As AI generates more transformation code and AI agents consume more data at scale, the stakes of this surface rise. An agent operating on a stale or conflicting context artifact produces systematic errors rather than recoverable ones. The engineering work that governs trustworthiness — designing the trigger model for the Contextualize pipeline, structuring the labeling workflow, deciding what validation confidence threshold earns formalization, and versioning context artifacts as definitions evolve — requires human judgment at every step.

Practitioners are still working out the patterns for doing this at scale. The tooling is maturing. How organizations govern ownership of the Context Store, adjudicate conflicts between teams, and manage the graduation from discovered to prescribed context are genuinely open questions. This is where the frontier actually is.

The New Data Engineer — Context Architect

Return to the poll. 53% said architecture and trade-offs are what remain irreducibly human. In the data engineering context, ECL is what that looks like in practice.

The data engineer of the next decade owns the architecture of meaning. They design the contractual foundations at the source—executable, enforceable, versioned. They build the lineage infrastructure that carries context through every transformation layer without losing it. They design and govern the Contextualize pipeline and the Context Store — the infrastructure where inferences get built, validated, and formalized into the definitions that everything downstream depends on. They understand when to prescribe context upfront and when to let it be discovered, and they build the systems that make both possible.

But this is not only a technical role. Context erosion is as much an organizational failure as a technical one. Teams don’t share semantic definitions because no ownership model incentivizes them to do so. Nobody enforces contracts because producing teams have no accountability to the consumers they serve. In this new frame, the data engineer is the person who builds both the technical system and the organizational agreement that holds it together. They sit at the intersection of architecture and coordination — the two things the poll respondents correctly identified as irreducibly human.

The title “Data Engineer” may need an update. What we are actually describing is a Context Architect — someone whose primary material is not data movement but data meaning, not pipelines but provenance, not transformation logic but the semantic infrastructure that makes transformation logic trustworthy.

An Open Frontier

I want to be honest about what ECL is and what it isn’t. It is a reorientation — a way of thinking about what the work actually is, now that AI is handling more of what the work used to look like. It is not a finished methodology. The tooling that links early binding contracts to the Contextualize pipeline and Context Store is still maturing. The organizational patterns for governing who owns the Context Store, how conflicts between teams get adjudicated, and how discovered context earns formalization don’t yet have established templates. Practitioners are working out the engineering patterns for building contextual pipelines that operate reliably at scale in production environments right now, figuring it out as they go.

That’s precisely what makes this moment worth paying close attention to. The frontier is genuinely open. The practitioners who invest in the architectural and organizational work of context — who treat contracts as executable infrastructure, who build lineage as a first-class engineering concern, who govern the Contextualize pipeline and Context Store as seriously as they once owned the ETL pipeline — will define the discipline for the decade ahead.

The 53% who said architecture and trade-offs are irreducibly human were right. We didn’t yet know which architecture, or which trade-offs.

Now we do.

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

AI is moving fast. Is your data platform ready?

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the AI Modernization Guide

Garry Tan: Half the AI Agent Market Is One Category. The Rest Is Wide Open

AI Agents thrive in RL environments with a verifiable target and quick feedback. Software manufacturing is a perfect model for such an environment, but the challenge persists in other categories. It will be an exciting decade for software engineering as we build new infrastructure that we never imagined.

https://garryslist.org/posts/half-the-ai-agent-market-is-one-category-the-rest-is-wide-open

LangChain: How to Use Memory in Agent Builder

Agents fail to improve over time when they treat every conversation as stateless and discard learned preferences or workflows. The article explains how LangChain’s Agent Builder implements short-term and long-term memory as a filesystem of Markdown files, enabling persistent instructions and reusable skills. Explicit memory updates, modular skill loading, and direct file editing enable agents to reliably evolve behavior without increasing core prompt complexity.

https://blog.langchain.com/how-to-use-memory-in-agent-builder/

LinkedIn: Scaling LLM-Based ranking systems with SGLang at LinkedIn

LLM-based ranking systems face strict latency and concurrency constraints because they score thousands of items per query without requiring text generation. The article explains how LinkedIn optimized SGLang for prefill-only ranking through batching improvements, scoring-only execution paths, prefix KV reuse, and Python runtime parallelization.

https://www.linkedin.com/blog/engineering/ai/scaling-llm-based-ranking-systems-with-sglang-at-linkedin

Sponsored: The Scaling Data Teams Guide

More datasets. More pipelines. More AI demands. The old way of doing things doesn’t work at this scale.
This free eBook walks through how teams actually scale sustainably with roles, responsibilities, automation, and patterns that work.

Get the guide now

Spotify: Our Multi-Agent Architecture for Smarter Advertising

Fragmented decision logic across buying channels prevented Spotify from translating high-level campaign goals into unified execution plans. The article explains how Spotify built Ads AI, a multi-agent orchestration layer with intent routing, specialized resolution agents, and data-grounded media planning using real-time tool integration. The architecture reduced campaign setup time from minutes to seconds, simplified user inputs, and grounded recommendations in historical performance data.

https://engineering.atspotify.com/2026/2/our-multi-agent-architecture-for-smarter-advertising

Uber: Database Federation: Decentralized and ACL-Compliant Hive™ Databases

Monolithic Hive warehouses create shared-fate outages, resource contention, and weak governance when thousands of datasets share a single database. The article explains how Uber implemented database federation by reorganizing datasets into domain-specific units, updating Hive Metastore pointers to avoid data duplication, and deploying both real-time and batch synchronizers to maintain consistency. The decentralized architecture improves ACL compliance, strengthens resource isolation, and reclaims storage while enabling zero-downtime migration at the petabyte scale.

https://www.uber.com/en-IN/blog/database-federation/

Anton Borisov: AutoMQ: Shared Storage Architecture Deep Dive

Kafka’s shared-nothing architecture imposes high replication costs, slow failover, and tight coupling between storage and compute. The article explains how AutoMQ replaces local disk replication with S3-backed shared storage, using layered abstractions, WAL batching, metadata-driven ownership, and epoch fencing to enable stateless brokers and zero-copy failover. AutoMQ design eliminates the 3x replication tax and simplifies scaling to “add compute,” while accepting higher cold-read latency from object storage.

https://medium.com/fresha-data-engineering/automq-shared-storage-architecture-deep-dive-043c5226847e

Pinterest: Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest

OOM in Spark jobs is an infamous issue across data processing, creating operational overhead and inefficient cluster utilization. The article explains how Auto Memory Retries dynamically adjusts executor resources by retrying failed tasks with higher CPU allocation or larger executors through modified Spark resource profiles. The elastic strategy reduced OOM failures by 96%, lowered infrastructure costs by avoiding over-provisioning, and improved overall pipeline reliability.

https://medium.com/pinterest-engineering/drastically-reducing-out-of-memory-errors-in-apache-spark-at-pinterest-c55d7dac2257

StarTree: Consistent, Scalable Compaction for Real-Time Upserts in Apache Pinot

Near-Real-Time upsert is my favorite subject to study, and I've worked with many OLAP engines. Apache Pinot always stands out for its flexible indexing and fast upsert capabilities. The article explains how StarTree’s SegmentRefreshTask compacts segments in the background by merging only valid records and ensuring atomic visibility with bitmap-based consistency controls. The approach reduces storage costs, supports sustained high ingestion rates, and maintains predictable query latency at a billion-key scale.

https://startree.ai/resources/upserts-compaction-in-apache-pinot-startree/

Zepto: Debezium at Scale: An Open Source CDC Story from Zepto

High-velocity CDC pipelines can overwhelm downstream databases due to redundant updates and MVCC-induced write amplification. The article explains how Zepto optimized Debezium by introducing an in-memory reduction buffer to collapse duplicate updates and a Postgres UNNEST-based batching strategy to reduce parsing overhead. These improvements stabilized CPU and I/O usage, eliminated replication lag during peak traffic, and ensured the database processes only final record states.

https://blog.zeptonow.com/debezium-at-scale-an-open-source-cdc-story-from-zepto-aa4b12e32bf7

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Dagster Running Dagster

In this upcoming session, find out how Dagster's data team has increased its capacity, along with best practices for data modeling that work well with AI assistants. We'll also demo a real case where our Compass Dagster+ integration identified the root cause of a Postgres-to-Snowflake pipeline that was failing 40-50% of the time.

Register now

Ben Lorica: Your agents need runbooks, not bigger context windows

AI agents struggle to scale in operational environments because they rely on large, transient context windows that reset after each task and incur repeated planning costs. The article discusses a Context File System (CFS) that separates reasoning from persistent procedural memory, enabling agents to mount task-specific runbooks, reuse indexed tools, and replay proven workflows.

https://gradientflow.substack.com/p/the-missing-layer-in-todays-agent

Netflix: High-Throughput Graph Abstraction at Netflix - Part I

High-throughput OLTP graph workloads demand low-latency traversal, strong typing, and global consistency at scale. The article explains how Netflix built a Graph Abstraction service on existing KV, TimeSeries, and caching infrastructure, using a property graph model with partitioned namespaces, optimized edge indexing, and write- and read-aside caching. The architecture processes millions of operations per second with single-digit millisecond latency while maintaining strict eventual consistency across regions.

https://netflixtechblog.medium.com/high-throughput-graph-abstraction-at-netflix-part-i-e88063e6f6d5

Reliable Data Engineering: Data Contracts in Practice - What 50 Production Implementations Actually Look Like

I recently wrote about Data Contract - a missed opportunity, the recent Twitter conversation, and the recent OSI spec from Snowflake reflecting the traces of Data Contract. The author did a solid job summarizing various data contract patterns and their implementations.

https://medium.com/@reliabledataengineering/data-contracts-in-practice-what-50-production-implementations-actually-look-like-f1c953336bf2

Sponsored: The AI Modernization Guide

Learn how to build a data platform that enables AI-driven development, reduces pipeline failures, and cuts complexity.

- Transform from Big Complexity to AI-ready architecture
- Real metrics from organizations achieving 50% cost reductions
- Introduction to Components: YAML-first pipelines that AI can build

Get the guide now

Netflix: Scaling LLM Post-Training at Netflix

Production LLM deployment requires a scalable post-training infrastructure that handles complex data pipelines, distributed GPUs, and evolving fine-tuning strategies. The article explains how Netflix built a unified post-training framework on its ML platform to support efficient SFT and RL workflows, modular data and model abstractions, and tight integration with open-source ecosystems. Custom optimizations, such as sequence packing and hybrid RL orchestration, increase token throughput and enable researchers to focus on modeling rather than infrastructure.

https://netflixtechblog.com/scaling-llm-post-training-at-netflix-0046f8790194

Abhishek Goswami: From Prompts to Production: A Playbook for Agentic Development

Enterprise AI agents fail in production when teams rely on prompt experimentation without structured lifecycle and behavioral controls. The article introduces an Agentic SDLC that separates deterministic and agentic components, applies reusable orchestration patterns, and enforces versioned prompts, tool manifests, and MCP-based integrations. Behavioral testing with golden trajectories, layered validation, and human-in-the-loop oversight enables reliable, scalable deployment of agents beyond prototypes.

https://www.infoq.com/articles/prompts-to-production-playbook-for-agentic-development/

Zepto: How We Built High-Precision, Low-Latency Semantic Search in Production

Keyword-based search fails on short, misspelled, and tail queries that lack lexical overlap with product catalogs. The article explains how Zepto built a dual-encoder semantic retrieval system trained with weak supervision, synthetic data, and InfoNCE loss to learn intent-aware embeddings under strict latency constraints. The approach delivered a 35% uplift on impacted queries, improved downstream ranking quality, and enabled semantic retrieval for both search and ads use cases.

https://blog.zeptonow.com/how-we-built-high-precision-low-latency-semantic-search-in-production-75a6c61dee25

Atlas9: The challenges of soft delete

Deleting data is one of the hardest problems; it is easy to write, but hard to delete. The standard approach is soft deletion, and it has several complications. The article covers the nuances of soft deletes, architectural patterns, and best practices for handling delete system design at scale.

https://atlas9.dev/blog/soft-delete.html

Apache Parquet: Native Geospatial Types in Apache Parquet

It is exciting to see Apache Parquet evolve with the addition of more types and indexing support. The flexibility of a data format that allows you to select data types and the indexing patterns associated with them improves data management efficiency. I wish Apache Parquet/Lakehouse formats such as Hudi, Iceberg, and Delta Lake offered the flexibility of Apache Pinot’s types and indexing patterns.

https://parquet.apache.org/blog/2026/02/13/native-geospatial-types-in-apache-parquet/

Dalto Curvelano: Introduction to PostgreSQL Indexes

Speaking of types and indexing, it always comes back to basics and a solid foundation. The article is an excellent overview of PostgreSQL indexing support and its use.

https://dlt.github.io/blog/posts/introduction-to-postgresql-indexes/

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

What assets do best: The Dagster Children's Book

We’re excited to share something a little unexpected from the Dagster team: What assets do best, a children’s book about data assets! Perfect for kids and data-loving grown-ups alike, you’ll learn how assets work together, adapt to change, and give teams a complete view of their data.

Watch the narrated story, find out where you can snag a free book IRL, and print & play puzzles!

Check out the book & other activities

Alexander Shereshevsky: Graph RAG in 2026 - A Practitioner’s Guide to What Actually Works

Graph RAG adoption stalled after early hype because high indexing costs and unclear performance trade-offs limited production use. The article explains when Graph RAG outperforms vector search, how teams reduce costs with selective graph construction, and why hybrid vector–graph architectures deliver the best results.

https://medium.com/@shereshevsky/graph-rag-in-2026-a-practitioners-guide-to-what-actually-works-dca4962e7517

Mark Rittman: So, Just How Relevant is Multi-Touch Attribution for Marketers in 2026?

Marketing attribution struggles in 2026 as privacy controls, regulations, and cookie loss remove large portions of user-level data. The article explains how teams adapt by combining deterministic identity for logged-in users, server-side tracking, and triangulation across MTA, MMM, and incrementality testing. Prioritizing authentication, tracking micro-conversions, and owning raw event data enables more reliable attribution in a privacy-first environment.

https://blog.rittmananalytics.com/how-relevant-is-multi-touch-attribution-for-marke-275a71a36d5e

Pinterest: Next Generation DB Ingestion at Pinterest

Legacy batch-based ingestion pipelines created high latency, operational complexity, and compliance gaps across Pinterest’s data ecosystem. The article explains how Pinterest built a unified CDC-based ingestion framework using Kafka, Flink, Spark, and Iceberg to stream database changes and efficiently upsert them into analytical tables, reducing data latency from days to minutes while lowering compute costs and improving reliability at the petabyte scale.

https://medium.com/pinterest-engineering/next-generation-db-ingestion-at-pinterest-66844b7153b7

Sponsored: Dagster Running Dagster

In this upcoming session, find out how Dagster's data team has increased its capacity, along with best practices for data modeling that work well with AI assistants. We'll also demo a real case where our Compass Dagster+ integration identified the root cause of a Postgres-to-Snowflake pipeline that was failing 40-50% of the time.

Register now

Netflix: The Data Canary: How Netflix Validates Catalog Metadata

Although this article is not specifically about the data warehouse, it demonstrates how data corruption can occur without a code change and still disrupt the system. The article describes how Netflix built a data canary system that validates new catalog metadata using side-by-side clusters, chaos-based testing, and customer-centric behavioral metrics. By detecting corruption within minutes and blocking the release of unsafe data, Netflix applies code-level deployment rigor to high-velocity data pipelines.

https://netflixtechblog.medium.com/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36

Uber: Introducing uFowarder - The Consumer Proxy for Kafka Async Queuing

Kafka consumer services struggle to scale reliably when direct protocol access introduces head-of-the-line blocking, inefficiency, and operational complexity. The article explains how Uber built uForwarder, a gRPC-based Kafka consumer proxy that resolves head-of-line blocking, improves hardware utilization, isolates traffic, and supports delayed processing. By abstracting Kafka internals behind a push-based interface, uForwarder increases reliability and efficiency across thousands of consumer workloads.

https://www.uber.com/en-IN/blog/introducing-ufowarder/

Pierce Lamb: Agentic Search over Graphs of Long Documents (or LAD-RAG++)

Possibly one of the best reads for this week for me. Vanilla RAG struggles with long, structured documents because static chunking loses layout, relationships, and cross-page context. The author reviews LAD-RAG++, which constructs layout-aware document graphs and employs agentic retrieval to explore structural and semantic connections dynamically. Engineering improvements in memory control, graph pruning, and cost-efficient processing make graph-based RAG practical for high-recall question answering over dense professional documents.

https://pierce-lamb.medium.com/agentic-search-over-graphs-of-long-documents-or-lad-rag-1264030158e8

Halodoc: Halodoc’s Layered Data Validation Strategy for Building Trust in the Lakehouse

Data quality issues in healthcare analytics demand stronger guarantees than generic validation frameworks can provide. The article explains how Halodoc built a custom, configuration-driven validation pipeline with four-layered checks that combine time-bound reconciliation, AI-generated structural tests, and business-rule enforcement across the Lakehouse. By integrating LLMs into validation and surfacing failures in real time, the system reduces incidents, increases trust in analytics, and supports reliable clinical decision-making.

https://blogs.halodoc.io/halodocs-layered-data-validation-strategy/amp/

Booking.com: Beyond Prompt Engineering: How We Used Supervised Fine-Tuning for Travel Recommendations

Prompt-based LLMs struggle to deliver fast, privacy-safe, and personalized travel recommendations at production scale. The article explains how Booking.com used supervised fine-tuning on an open-weight model with parameter-efficient techniques, contextual inputs, and carefully designed labels to combine conversational understanding with behavioral signals.

https://booking.ai/beyond-prompt-engineering-how-we-used-supervised-fine-tuning-for-travel-recommendations-91e8f4711e4b

Pranav Mehta: Clickhouse Internals: A Deep Dive into ClickHouse Distributed Connection Pooling

ClickHouse operators may misinterpret connection-retry warnings as leaks when distributed queries encounter transient network errors. The article explains how ClickHouse reuses pooled TCP connections for distributed tables and why idle timeouts in spiky workloads produce harmless “Broken pipe” warnings.

https://medium.com/@pranavmehta94/clickhouse-internals-a-deep-dive-into-clickhouse-distributed-connection-pooling-d9e956b5eb57

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Dagster Running Dagster dives into AI analytics.

In this upcoming session, Analytics Lead Anil walks through how Compass has increased the Dagster data team's capacity, shares best practices for data modeling that work well with AI assistants (hint: nested columns and wide tables are your friends), and demos a real case where our Compass Dagster+ integration identified the root cause of a Postgres-to-Snowflake pipeline that was failing 40-50% of the time.

Save your spot now.

OpenAI: Unrolling the Codex agent loop

The explanations of AI agents often obscure how local tools, model inference, and user interaction are orchestrated in practice. The article breaks down the Codex CLI agent loop, detailing how prompts, tool calls, iterative inference, context compaction, and prompt caching work together to execute software tasks efficiently. By combining stateless operation, automatic context management, and flexible tool integration via MCP, Codex achieves secure, performant local agent execution without server-side session retention.

https://openai.com/index/unrolling-the-codex-agent-loop/

OpenAI: Inside OpenAI’s in-house data agent

OpenAI writes about its internal data agent, which uses a closed-loop, self-correcting process and multiple context layers to translate natural language into reliable queries across hundreds of petabytes of data. By grounding meaning in code, minimizing tool complexity, and enforcing pass-through permissions with continuous evaluation, the system delivers fast, secure, and reliable data access for employees at scale.

https://openai.com/index/inside-our-in-house-data-agent/

Preset: The Semantic Layer Is Back. Here’s What We’re Doing About It.

The article reads like a pitch for the present, but what I liked most is the clear analogy for what a semantic layer is and why it has failed, even though legacy tools like Business Objects do support it. Overall, I’m excited about the agents as an interface for insights and the renewed interest in the semantic layer.

https://preset.io/blog/semantic-layer-is-back/

Sponsored: How to build a data platform that's ready for AI

Traditional data platforms are becoming the biggest bottleneck when companies experiment with AI. Learn how to build a unified control plane that enables AI-driven development, reduces pipeline failures, and cuts complexity.

- Transform from Big Complexity to AI-ready architecture
- Real metrics from organizations achieving 50% cost reductions
- Introduction to Components: YAML-first pipelines that AI can build

Get the free guide now

LangChain: Context Management for Deep Agents

Large AI agents risk context rot when long-running tasks exceed LLM memory limits, degrading reasoning quality. The article explains how the Deep Agents SDK actively manages context using tool input and output offloading, filesystem-backed pointers, and structured summarization to stay within token limits. Targeted evaluations ensure agents can recover critical details from compressed context and maintain task intent over extended workflows.

https://www.blog.langchain.com/context-management-for-deepagents/

Dropbox: Engineering VP Josh Clemm on how we use knowledge graphs, MCP, and DSPy in Dash

Building a universal search and agentic workspace is difficult because work data spans many tools, formats, and contexts, while LLMs face latency and context limits. The article explains how Dropbox Dash uses an index-based retrieval system, a context engine with multimodal processing, knowledge bundles, and MCP-based super tools, combined with LLM-as-a-judge and DSPy-driven prompt optimization.

https://dropbox.tech/machine-learning/vp-josh-clemm-knowledge-graphs-mcp-and-dspy-dash

Whatnot: Lessons learned from scaling data scientists with AI

AI-driven analytics struggle to generate correct queries because raw tables lack explicit business meaning and consistent relationships. The article explains how semantic views encode business logic, table relationships, and approved data scope to give LLMs precise, machine-readable context for SQL generation. By standardizing definitions and constraining access to vetted datasets, semantic views improve query accuracy, reduce hallucinations, and make AI-assisted analytics safer and more reliable.

https://medium.com/whatnot-engineering/lessons-learned-from-scaling-data-scientists-with-ai-e7aa7b3235b4

Netflix: Data Bridge: How Netflix simplifies data movement

Fragmented data movement tooling creates operational overhead, inconsistent governance, and tightly coupled implementations across large data ecosystems. The article describes how Netflix built Data Bridge as a unified control plane that separates user intent from execution, centralizes governance, and orchestrates existing data movement systems through standardized interfaces.

https://netflixtechblog.medium.com/data-bridge-how-netflix-simplifies-data-movement-36d10d91c313

LinkedIn: Contextual agent playbooks and tools: How LinkedIn gave AI coding agents organizational context

AI coding agents struggle to operate effectively without access to company-specific context, tools, and workflows. The article describes LinkedIn’s CAPT framework, which uses MCP, executable playbooks, and scalable meta-tools to connect agents to internal systems while controlling context and tool discovery. By packaging CAPT as a zero-friction local service, LinkedIn enables agents to automate debugging, incident response, data analysis, and issue triage, reducing investigation time by up to 70%.

https://www.linkedin.com/blog/engineering/ai/contextual-agent-playbooks-and-tools-how-linkedin-gave-ai-coding-agents-organizational-context

Netflix: The AI Evolution of Graph Search at Netflix: From Structured Queries to Natural Language

Enterprise search systems struggle when users must express complex filters through rigid, technical query languages. The article explains how Netflix evolved Graph Search by using LLMs to translate natural language into validated, schema-aware DSL queries with field-level RAG and AST-based verification. By visualizing AI-generated logic and supporting explicit entity selection, the platform lets users query federated data intuitively while maintaining correctness and trust.

https://netflixtechblog.com/the-ai-evolution-of-graph-search-at-netflix-d416ec5b1151

Modern Data 101: Modeling Semantics: How Data Models and Ontologies Connect to Build Your Semantic Foundations

AI-driven systems struggle without explicit semantic context to ground reasoning and reduce hallucinations. The article argues that data modeling and ontologies both capture entities and relationships and should serve as core methods for discovering and structuring organizational knowledge. By combining industry standards, conceptual modeling, and AI-assisted enrichment, teams can build a unified semantic foundation that improves both human understanding and AI accuracy.

https://medium.com/@community_md101/modeling-semantics-how-data-models-and-ontologies-connect-to-build-your-semantic-foundations-3a9a0664e3ff

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Join The Great Data Debate to get answers to questions the data & AI industry is so curious about right now:

• Where does context materialize in practice?

• Semantic layers, ontologies, context graphs - what should data teams build in 2026?

• Who owns context as meaning evolves?

• Where should that context live: in the warehouse, inside agents, or in a dedicated context layer?

Register Here

Why Data Engineers Must Think in Graphs, Not Just Tables

If you have been following the “Systems of Record” debate on tech Twitter, you likely saw the clash between the “Agents kill SaaS” camp and the “Long live the Database” camp. But for data engineers, the reality is more nuanced—and far more interesting.

As we move from dashboards to autonomous agents, we are hitting a wall. It turns out that knowing the state (what happened) is not the same as knowing the reasoning (why it happened).

Drawing on recent insights from Foundation Capital, Jamin Ball (Altimeter), OpenAI’s internal engineering team, and the TrustGraph manifesto, this post explores the emergence of the Context Graph. This missing architectural layer will likely redefine how we build data platforms in the agentic era.

The Problem: State Machines vs. Decision Traces

For the past decade, our role as data engineers has been to centralize data in the warehouse (or Lakehouse). We built ETL pipelines to move data from Salesforce, NetSuite, and Zendesk into a “Single Source of Truth.”

However, traditional Systems of Record (SoR) effectively act as “state machines.” They record the final output: the organization closed a deal, applied a discount, and escalated a ticket. But they fail to capture the decision traces.

As Foundation Capital notes, the reasoning behind a decision—the Slack threads, the cross-system synthesis, the VP’s verbal override of a policy—is rarely captured in the database. A CRM might show a “20% discount,” but it won’t tell an AI agent why that exception was granted (e.g., “Customer represents a strategic entry into the APAC market”).

Without these traces, agents fly blind. They have the rules (”Do not give discounts >10%”), but they lack historical context on when and why they were violated.

The Solution: The Truth Registry and the Context Graph

To address this, we observe a bifurcation in the modern data stack, as illustrated by the Hybrid Agentic Architecture (see Figure 1 below).

This architecture consists of two distinct but integrated planes:

1. The Warehouse as the “Truth Registry.”

Jamin Ball argues that systems of record aren’t dying; they are becoming “boring, rock-solid sources of truth”. In an agentic world, the warehouse must evolve into a “Truth Registry” that encodes semantic contracts.

Agents are fragile. If an agent hallucinates the definition of “Churn,” it can automate disastrous decisions. Therefore, we must clean and canonize data before the agent sees it. In the architecture above, the flow is from Raw (Variant) to Silver (Extracted) to Gold (Canonical Model).

• Engineering Takeaway: You cannot feed agents raw JSON blobs. Extracting variant columns into typed, named columns in the Silver layer is critical. It transforms “available data” into “governed data,” preventing agents from guessing schemas at runtime.

2. The Context Graph as the “Reasoning Layer.”

While the warehouse handles facts, the Context Graph handles relationships. TrustGraph defines a context graph as a “triples-representation of data (Subject → Predicate → Object) optimized for AI”.

Why a graph? Because structure is information. When you feed an LLM structured data (like RDF or Cypher), the structure itself encodes meaning. This allows agents to traverse relationships that SQL joins struggle to represent—stitching together a user’s support ticket, their billing status, and their web activity into a single, queryable context.

Case Study: Inside OpenAI’s Data Agent

OpenAI recently reported that standard metadata was insufficient for their internal data agent. They had to build a custom “Context Layer” that closely resembles the architecture above.

Their agent failed when it relied solely on table schemas. To fix this, they added:

1. Human Annotations: Curated descriptions of what tables actually mean (e.g., “This table excludes logged-out users”).

2. Code Enrichment: They used “Codex” to crawl their own codebase, understanding data lineage not just by metadata, but by reading the pipelines that produced the data.

This confirms a major trend: The metadata is the model. Providing agents with a semantic ontology (machine-readable definitions of terms) is just as important as the data itself.

The “Front Door” is Moving

The implications for the industry are massive. Historically, if you owned the System of Record (like Salesforce), you owned the “Front Door” (the UI).

But as agents take over workflows, the UI is unbundling from the data. Jamin Ball compares this to the travel industry: GDS systems (Sabre, Amadeus) remained the backend source of truth, but Online Travel Agencies (Expedia, Booking) captured the front door—and the value.

In our new stack, the Agents become the OTAs. They are the new interface. The Warehouse/Lakehouse becomes the GDS—the invisible, essential infrastructure layer.

What This Means for Data Engineers

1. Stop Hoarding State, Start capturing Traces: We need to instrument our systems to emit “decision traces” on every run. If an agent (or human) makes a decision, record the inputs and the logic used, not just the result.

2. The Rise of the “Gold” Layer: Your dbt models are no longer just for dashboards. They are the safety rails for autonomous agents. Strict typing, “Gold” tables, and canonical definitions are non-negotiable.

3. Graph Literacy: You don’t need to be a Neo4j expert, but understanding the basics of triples (Subject-Predicate-Object) and ontologies is becoming a core DE skill.

4. Extract Your Semi-Structured/Unstructured Data: As shown in the architecture diagram, leaving data in unstructured blobs is a liability. Agents need explicit structure to reason safely.

As agents grow more capable, the infrastructure beneath them must evolve. The Context Graph offers a powerful new foundation—not just for smarter agents, but for more transparent, explainable, and aligned systems. It’s time for data teams to build not just pipelines, but reasoning engines.

References

https://x.com/KirkMarple/status/2003944353342149021

https://x.com/KirkMarple/status/2005443843848856047

https://foundationcapital.com/context-graphs-ais-trillion-dollar-opportunity/

https://trustgraph.ai/news/context-graph-manifesto/

https://openai.com/index/inside-our-in-house-data-agent/

Clouded Judgement

Clouded Judgement 12.12.25 - Long Live Systems of Record

Every week I’ll provide updates on the latest trends in cloud software companies. Follow along to stay up to date…

Read more

3 months ago · 203 likes · 12 comments · Jamin Ball

Clouded Judgement

Clouded Judgement 12.19.25 - The System of Record's Front Door

Every week I’ll provide updates on the latest trends in cloud software companies. Follow along to stay up to date…

Read more

3 months ago · 61 likes · 10 comments · Jamin Ball

LLMs are transforming software development, but integrating them into real projects can be tricky when models don’t understand your codebase, pipelines, or conventions.

Join Dagster on Tuesday, January 27th, for a practical look at data engineering best practices, common pitfalls, and live demos of LLM developments.

Register now

Debate Alert: Ontology vs Context Graphs vs Semantic Layers: What Will AI Need in 2026?

What are Context Graphs, and why are builders, VCs, and operators calling it the next $1T opportunity? Join The Great Data Debate to get answers to questions the data & AI industry is so curious about right now:

• Where does context materialize in practice? Who owns context as meaning evolves?

• Semantic layers, ontologies, context graphs - what should data teams build in 2026?

• Where should that context live: in the warehouse, inside agents, or in a dedicated context layer?

Join Bob Muglia(former CEO, Snowflake), Karthik Ravindran(GM, Microsoft), Tony Gentilcore(Co-founder, Glean), Prukalpa Sankar(Co-founder, Atlan), and Jaya Gupta(Foundation Capital) for an open discussion on what data teams should actually build next.

Register: Feb 5 · Virtual · 11 AM ET

Mark Rittman: Why We’ve Tried to Replace Data Analytics Developers Every Decade Since 1974

Perhaps the recurring dream of replacing data analytics developers isn’t a mistake. Perhaps it’s a necessary optimism that drives tool creation.

Developers are always hungry for the next big abstractions. The article is an excellent reminder that Semantic layers didn’t make metric definition trivial, but they made it more maintainable. Self-service BI didn’t remove IT from the equation, but it expanded who could explore data.

https://blog.rittmananalytics.com/why-weve-tried-to-replace-data-analytics-developers-every-decade-since-1974-5c0de5a05088

Alibaba: AI Trends Reshaping Data Engineering in 2026

Alibaba writes that data engineering is evolving from data movement to building intelligent, autonomous systems in which data serves as a continuously learning capability. The field is getting reshaped by unified data–AI platforms, self-healing operations, context engineering for AI agents, real-time and multimodal pipelines, and privacy-first approaches such as synthetic data and federated learning.

https://www.alibabacloud.com/blog/ai-trends-reshaping-data-engineering-in-2026_602816

Thoughtworks: The state of data mesh in 2026: From hype to hard-won maturity

As with any scientific theories, it starts with a half-finished, widely debated theory and matures over time. Data Mesh and Data Contract are two such theories; though widely adopted by many companies internally, they still have much room to mature. ThoughtWorks writes an excellent article about the current state of data mesh adoption.

https://www.thoughtworks.com/insights/blog/data-strategy/the-state-of-data-mesh-in-2026-from-hype-to-hard-won-maturity

Sponsored: The AI Modernization Guide

While 75% of enterprises experiment with AI, traditional data platforms are becoming the biggest bottleneck. Learn how to build a unified control plane that enables AI-driven development, reduces pipeline failures, and cuts complexity.

- Transform from Big Complexity to AI-ready architecture
- Real metrics from organizations achieving 50% cost reductions
- Introduction to Dagster Components: YAML-first pipelines that AI can build

Get the guide now

Anthropic: Demystifying evals for AI agents

AI agent development faces evaluation gaps as systems evolve from single-turn prompts to autonomous, multi-step tool use. The article defines agent evals around task outcomes rather than transcripts, using repeated trials, hybrid grading methods, and reliability metrics such as pass@k and pass^k to capture non-deterministic behavior. Starting with small, failure-driven task sets and layering automated evals with production monitoring and human review enables teams to iterate faster, adopt new models safely, and maintain consistent agent reliability.

https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Booking.com: AI Agent Evaluation - practical tips at Booking.com

Evaluating autonomous AI agents is challenging because simple prompt-based metrics fail to capture whether user intents are reliably fulfilled. The article presents a dual approach combining black-box evaluation focused on task completion using LLM-as-a-judge with glass-box evaluation that inspects tool selection, syntax, and intermediate agent decisions. Benchmarking agents against simpler baselines and measuring consistency across repeated queries helps teams justify added complexity, control costs, and assess production readiness.

https://booking.ai/ai-agent-evaluation-82e781439d97

LinkedIn: Reimagining LinkedIn’s search tech stack

Traditional keyword-based search struggles to understand user intent, handle natural language queries, and bridge vocabulary gaps at scale. The article describes LinkedIn’s shift to a semantic search stack built on LLM-based query understanding, embedding-based retrieval, and small-language-model ranking, supported by LLM judges, model distillation, and continuous relevance measurement.

https://www.linkedin.com/blog/engineering/search/reimagining-linkedins-search-stack

Stefan Kecskes: Kafka Dead Letter Queue Triage: Debugging 25,000 Failed Messages

The DLQ pattern in Event processing is well-known and somewhat widely adopted. However, I’m delighted to read about what happens after a message enters a failed state for the first time. Discard or fix it? The author provides practical tips for analyzing DLQ messages, cautions, and resiliency practices before rebroadcasting the message.

https://skey.uk/post/kafka-dead-letter-queue-troubleshooting-guide/

Teads: The End of the Dashboard as We Know It: Designing for Insight in the Age of AI

Static dashboards fail to support decision-making because they require users to interpret large volumes of data manually. The article argues for AI-driven dashboards that act as proactive, conversational assistants by focusing on scenarios, personalized insights, and transparent recommendations rather than fixed screens. Measuring success by decisions enabled instead of data displayed reframes dashboards as adaptive systems that guide action and improve human–AI collaboration over time.

https://medium.com/teads-engineering/the-end-of-the-dashboard-as-we-know-it-designing-for-insight-in-the-age-of-ai-fec16bddf677

Flipkart: High-Risk, High-Scale: Guaranteeing Ad Budget Precision at 1 Million Events/Second

Real-time ad budget enforcement at scale is risky because latency can cause advertiser overspend, revenue loss, and delayed mobile events to be miscounted. The article describes Flipkart Ads’ architecture that separates real-time enforcement from batch settlement using Apache Flink, stateful deduplication with RocksDB, and event-time processing with watermarking. Prioritizing availability over strict consistency enables the system to process nearly one million events per second while ensuring accurate budget capping and final financial reconciliation.

https://blog.flipkart.tech/high-risk-high-scale-guaranteeing-ad-budget-precision-at-1-million-events-second-cc23977796d7

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

There was no shortage of activity around the topic. Definitions were proposed and refined. Conceptual boundaries were drawn and redrawn. Data contracts were compared to APIs, governance frameworks, data mesh primitives, and ideas teams already “sort of” implemented in practice.

The discussion was energetic and well-intentioned, but it tended to stay at the level of classification rather than construction.

What was largely absent was sustained engagement with the engineering consequences of taking data contracts seriously. Questions about enforcement, evolution, compatibility, and failure modes appeared only briefly before the conversation moved on. The result was an industry consensus that data contracts were “interesting,” without a shared understanding of what it would actually mean to build platforms around them.

In hindsight, this matters—not because the debate was unproductive, but because of what happened in parallel.

The Shift Happening Elsewhere

While the data community was debating what data contracts were, the software engineering world was converging—quietly and pragmatically—on a different organizing principle: specifications as the primary unit of system design.

This wasn’t a philosophical shift so much as an operational one. As systems became more distributed, more automated, and more interdependent, informal agreements stopped scaling. Documentation drifted. Assumptions diverged. Human coordination became the bottleneck.

The response was not more process, but more precision.

APIs began with schemas rather than code. Infrastructure moved from scripts to declarative specifications. Compatibility rules were automatically encoded and enforced. In these systems, the specification was no longer an artifact produced alongside the system—it was the system.

More recently, AI agents have accelerated this trend. Agents do not operate on intent, convention, or context. They operate on explicit, machine-readable, and verifiable data. Where specifications exist, agents can reason deterministically. Where they do not, agents approximate—and approximation is rarely acceptable in core infrastructure.

This is where the connection to data contracts becomes unavoidable.

Data Contracts as Specifications, Not Concepts

Viewed through the lens of spec-driven development, data contracts stop looking like a data-specific innovation and become a familiar pattern applied to a different domain.

A properly implemented data contract is a specification:

• It defines structure, semantics, and invariants

• It establishes compatibility guarantees over time.

• It is versioned, validated, and enforced programmatically.

• It creates a stable interface between independently evolving systems.

This is exactly what spec-driven development has optimized for in software engineering.

The difference is not conceptual—it is operational. Software engineering treated specifications as executable constraints. The data industry often treated contracts as descriptive artifacts. As a result, contracts were discussed as governance tools or communication mechanisms, rather than as interfaces with failure semantics.

That framing limited how far the idea could go.

Where the Two Worlds Diverged

Spec-driven systems force early clarity around hard problems.

• How does change propagate?

• What is allowed to evolve independently?

• What breaks compatibility, and how is that detected?

• Where does enforcement occur, and what happens when it fails?

In software systems, these questions are answered in code and tooling. In data systems, they were often answered socially. Producer–consumer agreements existed, but they lived in tickets, meetings, and tribal knowledge rather than in executable form.

Many teams compensated by building partial solutions: strong schemas, upstream quality checks, and informal SLAs. These patterns worked, but they relied heavily on human intervention. They were resilient, but not legible to machines.

As long as humans were the primary integrators, this was manageable. As soon as AI agents enter the workflow, it becomes a constraint.

Why Spec-Driven Thinking Changes the Next Phase

AI agents make an implicit demand of data platforms: make your rules explicit.

Agents can generate schemas, propose transformations, reason about compatibility, and enforce policy—but only if the platform exposes contracts in a form they can execute against. Without that, agents revert to inference, which introduces uncertainty precisely where determinism is required.

This is the practical implication of spec-driven development for data engineering. It’s not about adopting a new paradigm. It’s about recognizing that the platform already behaves like a system of interfaces—and formalizing those interfaces accordingly.

Teams that have already internalized contract discipline will find this transition incremental. Teams that have not will experience it as friction.

What We Should Do Next

At this point, the terminology matters less than the mechanics.

Whether we call them data contracts, data interfaces, or executable schemas, the path forward is the same:

• Treat schemas as specifications, not documentation

• Encode quality, semantics, and compatibility as executable rules

• Enforce contracts at clear system boundaries, preferably early.

• Version data interfaces with the same rigor as APIs

• Make ownership and accountability explicit and machine-readable.
  

This is not about adding process. It is about making systems legible to other systems.

Closing Thought

The original data contracts conversation wasn’t wrong. It just stopped too early.

Spec-driven development has shown that explicit, enforceable interfaces are not optional in complex, automated systems. Data platforms are now at that same inflection point.

Data contracts were never the destination.

They were the missing layer that would have made everything else easier to build.

The opportunity is still there—but only if we’re willing to treat contracts as infrastructure, not ideas.

References

https://www.thoughtworks.com/insights/podcasts/technology-podcasts/data-contracts-what-why

https://airbyte.com/data-engineering-resources/data-contracts

https://soda.io/blog/what-are-data-contracts

https://atlan.com/data-contracts/

https://en.wikipedia.org/wiki/Spec-driven_development

https://medium.com/software-architecture-in-the-age-of-ai/why-interfaces-and-contracts-are-not-the-same-and-why-that-matters-with-10-examples-408524f6d17c

https://arxiv.org/abs/2507.21056

Modernize your data platform for the age of AI.

While 75% of enterprises experiment with AI, traditional data platforms are becoming the biggest bottleneck. Learn how to build a unified control plane that enables AI-driven development, reduces pipeline failures, and cuts complexity.

- Transform from Big Complexity to AI-ready architecture
- Real metrics from organizations achieving 50% cost reductions
- Introduction to Dagster Components: YAML-first pipelines that AI can build

Get the guide

Lance Martin: Effective Agent Design

An effective agent design largely boils down to context management. The author proposes design patterns to effectively build an agent, including providing filesystem and shell access to the agents, using a multi-layer action space, and offloading memory to a filesystem rather than keeping everything in the context window.

https://x.com/RLanceMartin/status/2009683038272401719

Médéric Hurier: Architecting the AI Agent Platform: A Definitive Guide

The industry is shifting from simple LLMs and RAG (Retrieval-Augmented Generation) to AI Agents. The author proposes a 7-layer logical container architecture to build an AI agent platform. The 7-layer container architecture organizes the AI agent platform into logical levels—Interaction, Development, Core, Foundation, Information, Observability, and Trust—to manage the complexity of building production-grade systems. The structure enforces a separation of concerns, ensuring that user interfaces, execution engines, data management, and security governance are handled independently yet cohesively.

https://mlops.community/architecting-the-ai-agent-platform-a-definitive-guide/

Tidepool: Stop using natural language interfaces

The user experience of chatbot-driven enterprise application flow is taking center stage in the product design. The author argues that pure natural language interfaces are inefficient due to the high latency of LLMs (often taking tens of seconds to respond). Instead, the author proposes a hybrid approach in which the LLM dynamically generates structured Graphic User Interfaces (GUIs)—such as popups with checkboxes, sliders, and forms—to interact with the user.

https://tidepool.leaflet.pub/3mcbegnuf2k2i

Sponsored: Best practices for LLM development

LLMs are transforming software development, but integrating them into real projects can be tricky when models don’t understand your codebase, pipelines, or conventions.

Join Dagster on January 27th for a practical look at data engineering best practices, common pitfalls, and live demos of LLM developments.

Save your spot

Microsoft: SQL Telemetry & Intelligence – How we built a Petabyte-scale Data Platform with Fabric

Microsoft writes about how the SQL Telemetry & Intelligence (T&I) team built a 10+ petabyte Data Lake using Microsoft Fabric, processing real-time data from global SQL Server engines. The focus on CI/CD pipelines, testing optimization, local development, and data quality & observability is an interesting system read.

https://blog.fabric.microsoft.com/en-us/blog/sql-telemetry-intelligence-how-we-built-a-petabyte-scale-data-platform-with-fabric

Vikram Sreekanti & Joseph E. Gonzalez: Data is your only moat

The ease of adopting a tool enables data collection, which in turn creates a defensive advantage hard for competitors to replicate. The authors make a solid argument that, for enterprise applications, the moat isn’t just about volume but about specificity. By deeply integrating with a company’s legacy systems, a product gathers data on exactly how that specific customer works. This creates “stickiness”—replacing the tool becomes difficult because a new competitor wouldn’t have that accumulated knowledge of the company’s unique workflows.

https://frontierai.substack.com/p/data-is-your-only-moat

Uber: Apache Hudi™ at Uber: Engineering for Trillion-Record-Scale Data Lake Operations

Uber writes about the criticality of Apache Hudi in their overall data lake operations, enabling the management of trillion-record ingestion. Uber highlighted the addition of record indexes, Which Enable O(1) record lookups and allow efficient updates on tables with hundreds of billions of rows. Personally, this is a pretty cool feature from Apache Hudi.

https://www.uber.com/en-IN/blog/apache-hudi-at-uber/

Etsy: How Etsy Uses LLMs to Improve Search Relevance

Etsy writes about upgrading its search capabilities by using LLMs to focus on semantic relevance, which prioritizes understanding a buyer's true intent over simple click data. Etsy uses high-quality human and LLM annotations train a lightweight "student" model that runs in real time. This model actively filters and ranks search results, successfully increasing the percentage of fully relevant listings shown to shoppers.

https://www.etsy.com/codeascraft/how-etsy-uses-llms-to-improve-search-relevance

AWS: How Slack achieved operational excellence for Spark on Amazon EMR using generative AI

Slack writes about reaching operational excellence by replacing manual debugging with a custom monitoring framework that captures over 40 granular metrics from its EMR clusters. Slack exposed this data to generative AI models via Amazon Bedrock and a Model Context Protocol (MCP) server, enabling tools like Claude Code to analyze performance and suggest optimal configurations automatically. This automated system reduced compute costs by 30–50% and slashed developers' time spent tuning jobs by over 90%.

https://aws.amazon.com/blogs/big-data/how-slack-achieved-operational-excellence-for-spark-on-amazon-emr-using-generative-ai/

Agoda: How Agoda Enhanced the Uptime and Consistency of Financial Metrics

Agoda writes about addressing inconsistencies in its financial reporting by consolidating multiple disjointed data pipelines into a single Financial Unified Data Pipeline (FINUDP) built on Apache Spark. Agoda talks about approaches to ensure reliability and accuracy, including automated freshness monitoring, shadow testing for all code changes, and strict data contracts with upstream providers.

https://medium.com/agoda-engineering/how-agoda-enhanced-the-uptime-and-consistency-of-financial-metrics-ef7d54c4e4f0

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Best practices for LLM development

LLMs are transforming software development, but integrating them into real projects can be tricky when models don’t understand your codebase, pipelines, or conventions.

Join Dagster on January 27th for a practical look at data engineering best practices, common pitfalls, and live demos of LLM developments.

Reserve your spot now.

Foundation Capital: AI’s trillion-dollar opportunity: Context graphs

Agents are cross-system and action-oriented. The UX of work is separating from the underlying data plane. Agents become the interface, but something still has to be canonical underneath.

This will be a core construct of the next evolution of data engineering. A scalable data infrastructure that gives a unified view of the system of records and the analytical data, past decision traces, and a system of record that accepts high concurrent modifications. The promise of agents holds, but I don’t think our underlying infrastructure is ready for it.

https://foundationcapital.com/context-graphs-ais-trillion-dollar-opportunity/

ThoughtWorks: How to build the organizational muscle needed to scale AI beyond PoCs

Thoughtworks argues that AI initiatives fail to scale beyond pilots because organizations hit compliance hurdles, data silos, and lack stakeholder engagement—problems that require building "organizational muscle" rather than buying technology solutions. The article recommends a "thin slice" approach that addresses five building blocks simultaneously for a single use case: starting with clear business outcomes instead of technology, building tech platforms incrementally based on concrete needs, creating repeatable MLOps paths to production through cross-functional product teams, and investing in AI literacy and human-collaborative tool design to drive sustained adoption.

https://www.thoughtworks.com/insights/articles/how-to-build-organizational-muscle-needed-to-scale-AI

Sharon Campbell-Crow: Multi-Agent Systems: The Architecture Shift from Monolithic LLMs to Collaborative Intelligence

Developers are moving away from monolithic LLM “God Prompts” toward multi-agent systems because single models suffer from context limits and lack built-in self-critique. Multi-agent systems use specialized, sometimes adversarial agents, improving factual accuracy by up to 23%.

The article describes four architectures—LangGraph for graph-based control and auditability, AutoGen for event-driven distributed agents, CrewAI for role-based content workflows, and OpenAI Swarm for stateless, high-scale routing—along with production patterns such as planner–executor separation, memory streams for relevance, and deferred execution to manage cost and latency.

https://www.comet.com/site/blog/multi-agent-systems/

Sponsored: The Scaling Data Teams Guide

Building and scaling a data platform has never been more important or more challenging. Whether you’re just starting to build a data platform or leading a mature data organization, this guide will help you scale your impact, accelerate your team, and prepare for the future of data-driven products.

Learn how real data teams, from solo practitioners to enterprise-scale organizations, build.

Get the guide now

Ly: Building a multi-agent pipeline for NL-to-SQL analytics

LY Corp writes about migrating from a monolithic MCP-based NL-to-SQL system to a five-agent pipeline after encountering execution coupling, single-point-of-failure debugging, and oversized prompt contexts.

The new design adopts a Swarm-style orchestration model in which specialized agents handle routing, intent parsing, validation, SQL generation, query execution, and result presentation, using strict JSON interfaces and a tightly scoped context.

Preprocessed domain-specific data marts with normalized action units further reduce hallucinations by helping agents reliably map intents to the correct tables and columns.

https://techblog.lycorp.co.jp/en/building-a-multi-agent-pipeline-for-nl-to-sql-analytics

Vinted: Building a Global, Event-Driven Platform: Our Ongoing Journey

Vinted Engineering describes migrating from a monolithic system handling 150k requests per second to a global, event-driven platform processing over 300k requests per second. The redesign applies Domain-Driven Design across nearly 300 domains and uses Saga-based orchestration to coordinate multi-step workflows, centralizes writes, and globally replicates read-only projections via event streams. Separating read and write paths enables low-latency features such as feeds and search to be close to users, but requires teams to design for eventual consistency, retries, and out-of-order events rather than assuming immediate consistency.

https://vinted.engineering/2026/01/09/building-global-event-driven-platform-part-1/

https://vinted.engineering//2026/01/09/building-global-event-driven-platform-part-2/

Lyft: Lyft’s Feature Store: Architecture, Optimization, and Evolution

Lyft writes about a centralized Feature Store that maintains consistency between offline training and online inference across batch, streaming, and real-time serving. Batch features run on Spark SQL with auto-generated Airflow DAGs and Hive storage, streaming features use Apache Flink with Kafka and Kinesis, and online serving relies on DynamoDB with a ValKey write-through cache for low-latency access.

https://eng.lyft.com/lyfts-feature-store-architecture-optimization-and-evolution-7835f8962b99

Zeta Global: Zeta’s Lakehouse Journey: A Composable, Scalable, and Federated Architecture

Zeta Global writes about moving to a composable, federated Lakehouse architecture to integrate a highly heterogeneous data landscape that traditional warehouses could not unify. The platform standardizes on object storage with Apache Iceberg for transactional guarantees. It uses AWS S3 Tables with AWS Glue as the control plane, allowing Spark, Snowflake, and Trino to operate on shared datasets.

https://medium.com/@zeta-decoded/zetas-lakehouse-journey-a-composable-scalable-and-federated-architecture-df0ab5f19c3a

Google: Developer’s guide to multi-agent patterns in ADK

Similar to the typical Enterprise Integration Pattern, the multi-agent integration pattern is emerging as agent architecture becomes more widely adopted. Google lists about 8 patterns emerging in multi-agent systems.

1. Sequential Pipeline

2. Coordinator/Dispatcher

3. Parallel Fan-Out/Gather

4. Hierarchical Decomposition

5. Generator and Critic

6. Iterative Refinement

7. Human-in-the-loop

8. Composite Patterns

https://developers.googleblog.com/developers-guide-to-multi-agent-patterns-in-adk/

Ashpreet B: Memory: How Agents Learn

Most AI agents do not truly learn because they reset after each session.

The author presents a “GPU-poor continuous learning” approach in which agents store and retrieve successful patterns from databases rather than retrain models, demonstrated using the agno library with SQLite for session context, a memory manager for user data, and vector databases with human review to curate high-quality learned memories.

https://www.ashpreetbedi.com/articles/memory

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

“Latency is not just a performance characteristic; it is a fundamental part of correctness.” — Designing Data-Intensive Applications

In Designing Data-Intensive Applications, Martin Kleppmann makes a subtle but critical point: the CAP theorem omits latency, yet in real systems, latency often determines whether a system is usable at all. A system that is correct but slow is, in practice, incorrect.

This observation is directly applicable to the Apache Iceberg REST Catalog specification. While the specification achieves semantic clarity, it fails to define the operational realities that enable distributed systems to remain predictable at scale. The result is a standard that is formally correct, yet operationally fragile.

Semantic Interoperability Without Predictability

Over the past two years, the Iceberg REST Catalog specification has emerged as the de facto standard for metadata access in the Iceberg ecosystem. We have seen the outburst of the catalog war around the REST spec. It promises a universal interface that allows engines such as Trino, Spark, Flink, and StarRocks to interact with Iceberg tables via a common REST abstraction, independent of the underlying catalog implementation.

At the semantic level, this promise largely holds. The specification rigorously defines metadata structures: tables, schemas, snapshots, and namespace operations. A LoadTable or CreateNamespace request looks identical across implementations. This semantic interoperability has been critical to Iceberg’s rapid ecosystem adoption.

However, semantic interoperability alone is insufficient. The specification defines what metadata operations mean, but it avoids specifying how they must behave in real-world conditions, such as concurrency, latency sensitivity, and cross-catalog synchronization.

This gap—between semantic interoperability and operational interoperability—is where systems begin to fail in production.

The Core Problem: No Operational SLA, No Predictability

The Iceberg REST Catalog specification is intentionally silent on performance guarantees. There are no latency expectations, no throughput baselines, and no service-level objectives. While this flexibility lowers the barrier to implementation, it creates an ecosystem where:

• Two catalogs can both be “compliant” yet differ by orders of magnitude in response time.

• Clients cannot reason about metadata latency during query planning.

• Synchronization behavior across catalogs becomes unpredictable.

In distributed data systems, predictability matters more than raw performance. Without a strict operational SLA—or at least defined behavioral constraints—clients are forced into defensive, retry-heavy designs that amplify load and increase tail latency.

The “List Tables” Problem: Cross-Catalog Sync Failure

The ListTables endpoint (GET /v1/namespaces/{namespace}/tables) is semantically straightforward. It allows clients to enumerate tables within a namespace and supports pagination through pageSize and pageToken.

The primary issue is not pagination itself. The real failure emerges when the same Iceberg tables are registered in multiple catalogs, a pattern that is increasingly common in hybrid and multi-platform deployments.

A Realistic Scenario

• An Iceberg table is registered in Catalog A and Catalog B

• Both catalogs point to the same underlying metadata and object storage.

• One catalog is used by ingestion and streaming workloads.

• Analytics engines or BI tools use the other.

The Sync Pathology

When a client connects to Catalog B and issues a metadata discovery operation—such as listing tables or syncing namespace state—the catalog must:

1. Enumerate all tables

2. Resolve metadata pointers

3. Validate access permissions

4. Reconcile the state with the underlying storage.

Because the REST specification defines no operational expectations:

• There is no SLA for how long this sync should take

• There is no distinction between a “lightweight” listing and a fully validated listing.

• There is no mechanism to express intent (e.g., names only, no ACL validation)

As table counts grow into the tens of thousands, synchronization latency grows non-linearly. In practice, sync operations can take minutes—or fail—causing engines to stall, time out, or repeatedly retry.

The result is not merely slow metadata access. It is system-wide unpredictability. Query engines cannot determine whether a delay is transient, systemic, or catastrophic.

Latency Is Treated as an Implementation Detail—But It Is a Contract

The REST Catalog specification implicitly treats latency as an implementation concern. From a standards perspective, this is understandable. But in data-intensive systems, latency is part of the correctness contract.

The specification does not define:

• Upper bounds on metadata retrieval latency

• Maximum metadata payload sizes

• Limits on metadata fan-out operations

• The number of round-trip required to plan a query

As a result, a compliant catalog may require megabytes of JSON metadata and dozens of HTTP calls just to validate a single query plan. Engines appear slow and unstable, even though the root cause lies in an underspecified protocol.

This is precisely the class of problem Kleppmann warns about: correctness without latency guarantees is operationally meaningless.

Commit Semantics Under Contention: Undefined and Unfair

Iceberg relies on optimistic concurrency control. When multiple writers attempt to commit simultaneously, conflicts are expected and resolved through retries.

The REST specification defines the 409 Conflict response, but stops there. It does not define:

• Backoff expectations

• Retry fairness

• Starvation prevention

In a multi-engine environment, this creates asymmetric outcomes. A high-frequency streaming writer with aggressive retries can permanently starve batch compaction jobs that follow conservative retry policies. Over time, table health degrades due to file explosion and unbounded metadata growth.

Once again, the issue is not semantic correctness. It is the absence of operational guarantees.

Caching Without a Freshness Model

While HTTP caching is permitted, it is not part of the correctness model. Support for conditional requests, ETags, or freshness validation is optional.

This forces clients into a pessimistic stance: always re-fetch, always revalidate, always assume staleness. The REST protocol degenerates into a chatty, high-latency control plane that negates its own architectural benefits.

Without a standardized freshness contract, caching becomes a gamble rather than a reliability tool.

Behavioral Conformance Is Missing

The Iceberg ecosystem has strong conformance testing for table formats. It lacks an equivalent for catalog behavior.

Today, “REST Catalog compliant” means:

• The endpoints exist

• The JSON schema is correct.

• The happy path works.

It does not mean:

• Predictable latency under load

• Stable pagination during concurrent updates

• Graceful overload signaling

• Bounded retry amplification

Without behavioral conformance tests, compliance guarantees syntax, not operability.

Underspecification Is Still a Design Decision

The absence of operational constraints is not accidental. It reflects a deliberate choice to prioritize adoption and flexibility.

However, in distributed systems, underspecification pushes complexity downstream. It burdens clients, operators, and platform teams with the need to implement compensating logic. As Iceberg becomes core infrastructure rather than experimental tooling, this trade-off increasingly limits its reliability.

Semantic agreement without behavioral agreement leads to fragile systems.

Toward Operational Interoperability

Operational interoperability does not require rigid SLAs or centralized control. It requires acknowledging that latency, retries, and fairness are part of the interface.

Concrete improvements could include:

• Defined operational profiles with minimum latency and concurrency expectations

• Lightweight metadata views to avoid synchronization amplification

• Standardized retry and backoff semantics for conflict scenarios

• Explicit freshness and caching contracts

Semantic interoperability enabled Iceberg’s success. Operational interoperability will determine whether it remains dependable at scale.

Until then, the Iceberg REST Catalog remains a textbook example of why semantic specifications alone are not enough.

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Best practices for LLM development

LLMs are transforming software development, but integrating them into real projects can be tricky when models don’t understand your codebase, pipelines, or conventions.

Join Dagster on January 27th for a practical look at data engineering best practices, common pitfalls, and live demos of LLM developments.

Reserve your spot now.

Editor’s Note: The Edition of Predictions!! Well, Mostly

It is always exciting to read the predictions and look back on 2025. I put a lot of effort into collecting some of these predictions and bundling them in this edition. At DEW, we also reached a few existing milestones. We just published our 250th edition and reached 50,000 Substack followers. It is remarkable growth, considering how lazy I am on LinkedIn and how little I promote DEW. I’m looking to improve on it, and over the holidays, I tried a bit of Agent building on top of DEW, which I’m hoping to launch soon. I wish all the DEWers a prosperous 2025 and thank you for your continued support.

Ananth Packkildurai: DEW - The Year in Review 2025

Why not start with our own year-in-review and a bit of predictions? Agent Engineering is undoubtedly becoming a discipline of its own in engineering, similar to the rise of data scientists. Both, funny enough, run into data inconsistency issues, and everyone does data engineering eventually. (Hello Context Engineering)

I wrote a bit about how the catalog becomes the new database as the adoption of Apache Iceberg and Knowledge Engineering increases. I’ve a lot of concern about the Iceberg Rest Catalog, which I will write about as a separate blog this week. Stay tuned.

https://www.dataengineeringweekly.com/p/dew-the-year-in-review-2025

Sebastian Raschka: The State Of LLMs 2025: Progress, Problems, and Predictions

The State of LLMs 2025 examines how DeepSeek R1's breakthrough demonstrated that reasoning models can be trained for approximately $5 million using Reinforcement Learning with Verifiable Rewards (RLVR), fundamentally shifting industry cost assumptions and development approaches. The field has pivoted from pure pre-training scale to inference-time scaling—where models allocate more compute during generation for complex tasks—while grappling with "benchmaxxing," which undermines evaluation reliability, and recognizing that LLMs enhance rather than replace human expertise in coding and writing.

https://magazine.sebastianraschka.com/p/state-of-llms-2025

Ian Cook: 10 Predictions for Data Infrastructure in 2026

The article forecasts the maturation of open standards like Apache Arrow and ADBC replacing legacy protocols (ODBC/JDBC) while Apache Iceberg transitions from hype to mainstream adoption. Organizations are shifting from monolithic data warehouses to composable, multi-engine architectures built on open formats and components like DuckDB and DataFusion, enabling analytical systems to power operational applications and provide AI agents with fast, governed access to structured data.

https://columnar.tech/blog/2026-predictions/

Sponsored: The Scaling Data Teams Guide

Building and scaling a data platform has never been more important or more challenging. Whether you’re just starting to build a data platform or leading a mature data organization, this guide will help you scale your impact, accelerate your team, and prepare for the future of data-driven products.

Learn how real data teams, from solo practitioners to enterprise-scale organizations, build.

Get the guide now

Ben Lorica: Data Engineering in 2026: What Changes?

The article argues that agent-native platforms must move beyond tabular data to handle multimodal assets like text, images, and video, while treating documentation as versioned, queryable context stores that agents can understand. The platform shift requires pipeline-native safety guarantees, using write-audit-publish patterns in which agents write to isolated branches before merging, fundamentally changing data engineers' roles from manual ETL work to supervising agent fleets and defining governance policies.

https://gradientflow.substack.com/p/data-engineering-for-machine-users

Stanford HAI: Stanford AI Experts Predict What Will Happen in 2026

Stanford experts predict AI will shift from evangelism to evaluation in 2026, with organizations deflating the hype bubble by reporting failed projects and measuring actual Return on Investment rather than chasing AGI. The report highlights breakthroughs in medical AI enabled by self-supervised learning that eliminate the need for expensive expert-labeled data, alongside nations building sovereign infrastructure to reduce dependence on US providers.

https://hai.stanford.edu/news/stanford-ai-experts-predict-what-will-happen-in-2026

HuggingFace: Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Hugging Face's Transformers v5 redesigns tokenizers by separating the architecture from the trained vocabulary, similar to how PyTorch separates the model structure from the weights. The update consolidates confusing Python and Rust implementations into a single Rust-backed version while allowing users to inspect tokenizer internals directly and train custom tokenizers from scratch on their own data.

https://huggingface.co/blog/tokenizers

Netflix: Towards Generalizable and Efficient Large-Scale Generative Recommenders

Netflix explains that generative recommendation models scale differently than standard LLMs because each task has an inherent unpredictability limit, which changes how compute should be allocated. The blog explains how it reduced training costs by using sampled softmax and projected output heads to handle extremely large catalogs with millions or billions of items.

https://netflixtechblog.medium.com/towards-generalizable-and-efficient-large-scale-generative-recommenders-a7db648aa257

Vinted: Orchestrating Success

The blog explains how decentralized data pipelines created coordination bottlenecks, shifting dependencies from code to endless meetings, and running dbt layers via Airflow lacked granularity, causing entire layers to fail when a single model broke. The team built an automated DAG generator that reads the dbt manifest to create task-per-model setups in Airflow, paired with an Asset Registry that manages cross-domain dependencies using ExternalTaskSensor and automatically marks waiting sensors as satisfied when upstream tasks finish late, eliminating manual restarts.

https://vinted.engineering//2025/12/29/orchestrating-success/

Tim Castillo: How I Structure My Data Pipelines

The author proposes resolving Medallion Architecture's ambiguity by strictly combining it with Kimball dimensional modeling and Semantic Layers, mapping each methodology to specific pipeline layers. The approach defines Bronze for mechanical normalization and type casting without business logic, Silver for Kimball Facts and Dimensions representing authoritative business logic, and Gold as a first-class Semantic Layer using governed Metric Views that serve pre-calculated metrics to business users while allowing data scientists direct SQL access to Silver's full-grain data.

https://loglevelinfo.substack.com/p/how-i-structure-my-data-pipelines

All rights reserved, Dewpeche Pvt Ltd, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

The Scaling Data Teams Guide

Building and scaling a data platform has never been more important or more challenging. Whether you’re just starting to build a data platform or leading a mature data organization, this guide will help you scale your impact, accelerate your team, and prepare for the future of data-driven products.

Learn how real data teams, from solo practitioners to enterprise-scale organizations, build.

Get the guide now

Thoughtworks: The Model Context Protocol’s impact on 2025

Thoughtworks writes about Model Context Protocol's (MCP) transformative impact on 2025 software development, highlighting its role in accelerating agentic AI adoption by simplifying connections between AI systems and external data sources .The blog identifies emerging techniques including context engineering for systematic LLM information optimization, AI-powered UI testing via Playwright-mcp and mcp-selenium servers, and anchoring coding agents to reference applications to prevent code drift, while cautioning against security vulnerabilities (tool poisoning, cross-server shadowing) and antipatterns like naive API-to-MCP conversion.

https://www.thoughtworks.com/insights/blog/generative-ai/model-context-protocol-mcp-impact-2025

Uber: Powering Billion-Scale Vector Search with OpenSearch

Uber Engineering migrated from Apache Lucene’s HNSW to Amazon OpenSearch for billion-scale vector search, addressing algorithm inflexibility and GPU support limitations when handling 1.5 billion items with 400-dimension embeddings for personalized recommendations and fraud detection. The implementation reduced indexing time by 79% (from 12.5 hours to 2.5 hours) through Spark batch ingestion, optimized flush/merge policies, while achieving 52% lower P99 latency (250ms to 120ms at 2K QPS) via shard-to-node ratio tuning, replica scaling, in-memory KNN graph optimization, and blue/green deployments.

https://www.uber.com/en-IN/blog/powering-billion-scale-vector-search-with-opensearch/

Andrew Hoblitzell: QConAI NY 2025 - Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery

The blog narrates the QCon AI NYC 2025 presentation on treating agentic AI as an engineering problem requiring deterministic boundaries around probabilistic components, arguing that reliability improves when models handle interpretation and classification while deterministic systems execute actions and enforce constraints. The speaker outlined practical patterns including reducing schema complexity to limit query generation errors, using role-specialized agents, constraining tool catalogs to prevent "paradox of choice" degradation, and anchoring agents to deterministic runbooks for repeatable operations rather than allowing runtime process invention.

https://www.infoq.com/news/2025/12/qcon-nvidia-platform/

Sponsored: The data platform playbook everyone's using

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating undering a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your free copy now.

Monday: The Power of Structured Data: Inside monday.com’s Data Entities

Monday.com writes about Data Entities as a semantic layer to address fragmentation caused by boards functioning as unstructured spreadsheets, where business objects like projects, tickets, and deals lacked standardized definitions that hindered cross-board reporting and AI agent context understanding. The implementation uses field-and-policy-based models with hierarchical inheritance, Managed Columns for field standardization, and a new "data-entity" app feature in monday's open platform that enables entity-level updates to propagate across all associated boards.

https://engineering.monday.com/the-power-of-structured-data-inside-monday-coms-data-entities/

Meta: Python Typing Survey 2025: Code Quality and Flexibility As Top Reasons for Typing Adoption

Python becomes the defacto langiage for data programming, and have a strong typing system brings more reliability to the codebase. The 2025 Typed Python Survey reveals that 86% of Python developers regularly use type hints, with adoption highest among developers with 5-10 years of experience (93%). Key challenges include third-party library support gaps (NumPy, Pandas, Django), complexity of advanced features (generics, TypeVar, decorators), tooling fragmentation between type checkers (Mypy at 58% usage, emerging Rust-based checkers at 20%), and lack of runtime enforcement.

https://engineering.fb.com/2025/12/22/developer-tools/python-typing-survey-2025-code-quality-flexibility-typing-adoption/

Alibaba: Say Goodbye to Manual Tracking! An In-Depth Analysis of Non-Intrusive Data Collection for Android Apps

Data tracking a critical data engineering skill which the industry won’t talk or write about a lot. I’m glad to see this in-depth analysis on data collection from Alibaba. Alibaba writes about a non-intrusive Android monitoring solution using Gradle plugins, AGP API, and ASM bytecode instrumentation to automate APM data collection without manual SDK initialization or code modifications. The implementation uses dynamic AGP version adaptation, conflict-prevention strategies (blacklists for system packages and third-party APM tools, whitelist filtering, idempotent instrumentation), and defensive programming in injected agents (SDK initialization checks, exception isolation, silent returns when uninitialized), while supporting diverse collection scenarios including user behavior tracking via proxy listeners, network monitoring through OkHttp/HttpURLConnection hooks.

https://www.alibabacloud.com/blog/say-goodbye-to-manual-tracking-an-in-depth-analysis-of-non-intrusive-data-collection-for-android-apps_602758

PyTorch: Deploying Smarter: Hardware-Software Co-design in PyTorch

People who are really serious about software should make their own hardware." — Alan Kay

Even if we don’t often build software and hardware together, it is vital to understand the software behaviour under certain hardware configurations. Arm released a collection of Jupyter notebook tutorials demonstrating hardware-software co-design techniques in PyTorch for efficient on-device AI, addressing the limitations of traditional post-training quantization that applies uniform precision across all neural network layers.

https://pytorch.org/blog/deploying-smarter-hardware-software-co-design/

DuckDB: Iceberg in the Browser

DuckDB introduced browser-based access to Iceberg REST Catalogs through DuckDB-Wasm, enabling users to query and edit Iceberg tables directly from a browser tab without installing software or managing infrastructure. The implementation unified DuckDB's HTTP networking layer across native and WebAssembly environments through three changes: redesigning the core HTTP interface for consistent extension access, creating a JavaScript network wrapper for DuckDB-Wasm, and routing all Iceberg networking through this common layer, with initial Amazon S3 Tables support where all computations run locally in the browser.

https://duckdb.org/2025/12/16/iceberg-in-the-browser

All rights reserved, Dewpeche Pvt Ltd, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

If 2023 was the year of “Shock” and 2024 was the year of “Hype,” 2025 will be remembered as the year of Engineering.

For the past decade, our industry has been obsessed with the mechanics of movement. We argued about “ETL vs. ELT.” We fought “Format Wars” over table specifications. We optimized commit protocols and debated the merits of various orchestrators. We were, fundamentally, digital plumbers ensuring the water reached the tap.

But in 2025, the mandate changed. The business no longer wants “data”; it demands “intelligence.” It demands systems that reason, agents that act, and infrastructure that guarantees truth in a non-deterministic world. The “Big Data” era of managing volume formally ended, replaced by the “Context Era” of managing meaning.

We are no longer just Data Engineers. We are the architects of the cognitive layer.

Here are the seven patterns that defined Data & AI Engineering in 2025.

1. Agent Engineering: The Inevitable Evolution of the Pipeline

The most significant shift of 2025 was the industry’s realization that “Agents” are not just fancy chatbots—they are the new compute engine. In 2024, we treated LLMs as text generators. In 2025, we started treating them as reasoning engines that execute logic we previously wrote in Python or SQL.

This birthed a new discipline: Agent Engineering.

We moved beyond the chaotic “vibes-based” coding of early experiments into structured, rigorous engineering. We stopped asking “Can AI write code?” and started asking “How do we architect a system where AI reliably executes complex workflows?”

The Rise of Context Engineering

The bottleneck for intelligent systems shifted from model capacity to context management. We realized that an agent is only as smart as the context you feed it.

Anthropic defined the year with their masterclass on Effective Context Engineering, framing it as a discipline focused on managing the “attention budget” of models. It wasn’t enough to dump documents into a prompt. Engineers at Manus demonstrated that we must curate, compress, and dynamically retrieve tokens during inference to sustain coherent behavior over long horizons in their piece on Context Engineering for AI Agents.

We learned that “Context” is an information management problem. We saw teams optimizing “KV-cache hit rates” and treating context windows like precious RAM. The winning architecture wasn’t the one with the biggest model; it was the one that engineered the most relevant context.

The USB-C of Intelligence: Model Context Protocol (MCP)

History will likely view the introduction of the Model Context Protocol (MCP) as the moment agents became viable enterprise software. Before MCP, connecting an LLM to a database or API was a bespoke, brittle integration task.

In 2025, MCP standardized this connection. It became the “USB-C for Agents,” allowing developers to build a connector once and have it work across any MCP-compliant model or application, as detailed in Alibaba’s comprehensive analysis of MCP features. However, the rollout wasn’t without caution; TigerData engineers noted that while MCP solved interoperability, it introduced new attack vectors, arguing that security is its Achilles heel.

From Chatbots to Colleagues

The proof was in the production deployments. Uber revealed Genie, an internal agent that didn’t just answer questions but also acted as a “near-human” subject-matter expert. LinkedIn unveiled its Hiring Assistant, an agent that handled complex recruiting workflows using speculative decoding to accelerate inference.

These weren’t toys. They were engineered systems with rigorous orchestration, state management, and error handling. The industry formalized patterns like “Prompt Chaining,” “Routing,” and “Parallelization.” We stopped treating agents as magic boxes and started treating them as software components that required a new kind of engineering.

2. “Evals” Are The New Unit Tests

If Agent Engineering was the engine of 2025, Evaluation (Evals) was the brakes—and the steering wheel.

The “Vibe Coding” era—where we judged models by looking at a few outputs and saying “looks good to me”—died a hard death. In 2025, organizations realized they could not ship non-deterministic software without rigorous, deterministic measurement.

The “Judge-LLM” Pattern

How do you test a system that gives a different answer every time? You build a machine to grade the machine.

The industry standardized around the Judge-LLM framework. Booking.com offers practical tips for LLM evaluation, using a “stronger” model (trained on a “Golden Dataset” of human-verified answers) to grade the outputs of “weaker,” cheaper production models. Pinterest followed suit with its LLM-powered relevance assessment, replacing costly manual labeling with fine-tuned LLMs that achieved high agreement with human experts.

This wasn’t just about checking for “correctness.” We developed specific metrics for Hallucination Rate, Instruction Following, and Tone Consistency. Uber built Requirement Adherence systems that extracted rules from standard operating procedures (SOPs) and enforced them in real-time, reducing post-labeling audits by 80%.

Evaluation-Driven Development (EDD)

“Test-Driven Development” (TDD) evolved into Evaluation-Driven Development (EDD). Engineers learned that you cannot optimize what you cannot measure.

Infrastructure teams integrated these evals directly into CI/CD pipelines. Databricks shared how they “shifted left” on reliability, scaling database reliability by embedding schema scorers into their build processes to catch data quality issues before they hit production.

The takeaway for every data engineer in 2025 was clear: If you don’t have an eval for it, it doesn’t exist. You aren’t “prompt engineering” until you have a metric that tells you if your changes made things better or worse.

3. The Streaming-Lakehouse Merger: The End of Lambda Architecture

For fifteen years, we lived with the “Lambda Architecture”—maintain a fast streaming path (Kafka/Flink) and a slow batch path (Hadoop/Spark), and pray they match. In 2025, we finally merged the lanes.

The barrier between “Stream” and “Table” dissolved. We entered the era of the Streaming Lakehouse.

Stream-Table Duality

The concept of “Stream-Table Duality”—long preached by Kafka’s creators—became a reality in storage. New engines like Apache Paimon and Apache Fluss emerged to bridge the gap.

Alibaba championed Apache Paimon as a lake format designed specifically for real-time updates, offering the high-throughput ingestion of a stream with the query capabilities of a lakehouse table. Jack Vanlightly’s deep dive into Understanding Apache Fluss revealed a system that combines log tablets with KV tablets, effectively creating a database that exposes its own changelog as a first-class citizen.

We stopped debating “Stream vs. Batch” and started designing architectures that ingest data once and make it immediately available for both real-time operational lookups and historical analytical queries.

The Zero-Copy Debate

However, this merger wasn’t without conflict. The buzzword of the year was “Zero-Copy”—the promise that you could point your data warehouse at your Kafka topic and query it without moving bytes.

But seasoned engineers pushed back. WarpStream argued the case for an Iceberg-native database, claiming that coupling your operational message bus directly to your analytical engine violates separation of concerns.

The consensus that emerged? “Zero-Copy” is great for ad-hoc exploration, but for production, Materialization (making a copy) is still the price of performance and isolation.

Diskless Kafka and the Cloud-Native Log

Even Kafka itself couldn’t escape the modernization wave. The community rallied around KIP-1150 (Diskless Topics), a proposal to re-architect Kafka for the cloud era.

We realized that in a world of S3 Express and high-speed networking, storing data on local broker disks was an expensive relic. The future of streaming is “Tiered Storage by Default,” where the broker is just a caching layer on top of infinite object storage. This shift promises to slash costs and make “infinite retention” streams a standard architectural pattern.

4. The Efficiency Counter-Revolution: Small Data and Rust

While the AI teams were burning cash on GPUs, the Data Infrastructure teams were leading a quiet counter-revolution. After years of defaulting to massive, expensive distributed clusters (Spark/Hadoop) for every problem, 2025 was the year we right-sized our compute.

We realized that “Big Data” tools are overkill for “Medium Data” problems.

The Single-Node Renaissance

“Does this really need a 50-node cluster, or just a bigger laptop?”

That question dismantled pipelines across the industry. Tools like DuckDB and Polars graduated from “analyst favorites” to “production workhorses.” Decathlon shared a viral case study about being Ready to Play with Polars, replacing massive Spark clusters with Polars scripts for datasets under 50GB and slashing infrastructure costs to near zero.

Benchmarks from Daniel Beach confirmed this, showing that for 650GB of Data (Delta Lake on S3), single-node engines often beat distributed clusters simply by avoiding network overhead. We stopped being ashamed of “vertical scaling” and started embracing it as a FinOps victory.

The Rust Rewrite

When we did need performance, we turned to Rust.

Agoda explained why they bet on Rust to supercharge their Feature Store, achieving a 5x increase in traffic capacity.

The lesson was clear: The “Java Tax” is real. For critical, low-latency infrastructure, Rust’s safety and performance are worth the learning curve. We are entering a new era where the foundational tools of data engineering are being rebuilt, brick by brick, in Rust.

FinOps as Architecture

Efficiency wasn’t just about code; it was about architecture. Wix documented how they slashed Spark costs by 60% not by rewriting code, but by migrating workloads from managed services (EMR) to Kubernetes (EKS).

We learned that “Serverless” often means “Wallet-less” if you aren’t careful. The smartest teams in 2025 were those who aggressively optimized their compute substrates, moving workloads to Spot instances, ARM processors, and single-node containers whenever possible.

5. Lakehouse 2.0: The Catalog is the New Database

The “Format Wars” (Iceberg vs. Delta vs. Hudi) that dominated the early 2020s largely settled into a peace treaty in 2025. With the rise of interoperability layers, we stopped caring about data folder structure.

The battleground shifted “up the stack” to the Catalog. We realized that the “Lakehouse” is just a database turned inside out, and the Catalog is its operating system.

The Catalog Wars

In 2025, the Catalog stopped being just a “list of tables” and became the active control plane for the enterprise.

New concepts like DuckLake challenged the status quo, proposing that we use DuckDB itself as a catalog and metadata layer, replacing the heavy, complex Hive Metastore with a lightweight, transactional database.

Hyperscalers (AWS, Snowflake, Databricks) all converged on Managed Iceberg services. As Simon Späti noted in The Open Table Format Revolution, the major players stopped fighting the open format and started trying to own the metadata layer that manages it. The value proposition shifted from “we store your data” to “we govern your transactions.”

6. The “Context” Supply Chain: Unstructured Data & Knowledge Graphs

For decades, Data Engineering was about rows and columns. In 2025, we had to get good at text, images, and relationships. To support the GenAI revolution, we had to build the Context Supply Chain.

We aren’t just moving data anymore; we are moving meaning.

Knowledge Graphs Return

The most surprising comeback of 2025 was the Knowledge Graph. As we struggled with LLM hallucinations, we realized that probabilistic models need deterministic facts to ground them.

Netflix details its Unified Data Architecture and how it is Unlocking Entertainment Intelligence with Knowledge Graph to provide a “source of truth” for its AI models.

We learned that “RAG” (Retrieval-Augmented Generation) isn’t just about vector search. It’s about “GraphRAG”—using the relationships between data points to provide richer, more accurate context to the model.

The Embedding Pipeline

Data Engineers had to master a new type of transformation: the Embedding.

We moved beyond simple “Word2Vec” tutorials. Milvus released guides on how to choose the right embedding model for RAG, helping teams select between LLM2Vec, BGE-M3, and others for specific domains. We built pipelines that chunked documents, generated embeddings, and stored them in vector databases, treating “semantic distance” as a first-class data type.

Unstructured Data Management

We formally recognized Unstructured Data Management at Scale as a core competency. Piethein Strengholt argued that it wasn’t enough to dump PDFs into an S3 bucket; we needed to parse, clean, chunk, and govern that data with the same rigor we apply to our financial tables.

The “Medallion Architecture” (Bronze/Silver/Gold) was adapted for unstructured data. We started talking about “Raw Documents” (Bronze), “Parsed & Chunked Text” (Silver), and “Curated Embeddings” (Gold).

7. Governance 2.0: The Safety Brake for Autonomous Agents

As AI agents began taking actions—booking interviews, executing code, modifying databases—Governance stopped being a “compliance checkbox” and became a “safety brake.”

In 2024, if a dashboard was wrong, a manager made a bad decision. In 2025, if an agent is wrong, it might delete a production database or leak PII to a competitor.

Privacy-Aware Infrastructure

Meta writes about discovering data flows via lineage at scale. They didn’t just write policies; they wrote code that tracked data lineage and enforced purpose limitations at the exabyte scale. Meta introduced Policy Zones to ensure that data collected for one purpose (e.g., safety) couldn’t be used for another (e.g., ad targeting) without explicit permission.

This level of granularity is the future. We are moving toward systems where every piece of data carries its own “passport” of permissions, and every agent must present a “visa” to access it.

Shadow AI and the New Perimeter

The rise of “Shadow AI”—engineers spinning up local LLMs or using unapproved APIs—forced data teams to harden the perimeter.

We saw the emergence of Data Contracts as a defense mechanism. Grab implemented real-time data quality monitoring with strict contracts for its Kafka streams, ensuring that bad data was rejected before it could poison the downstream AI models.

Governance in 2025 is about Observability. It’s about knowing exactly which agent accessed which document, why, and what it did with it.

Conclusion: The Era of the Context Engineer

Looking back at 2025, it is clear that the role of the Data Engineer has fundamentally changed.

We are no longer just building pipelines to move data from Point A to Point B. We are building the enterprise’s Cognitive Nervous System.

• We are Agent Engineers, designing the workflows that allow AI to reason and act.

• We are Eval Architects, building the metric systems that keep AI honest.

• We are Context Curators, ensuring that the “meaning” of our data is preserved and accessible.

• We are Efficiency Experts, maximizing the ROI of every compute cycle in a world of expensive GPUs.

The tools will continue to change. Spark might yield to Polars; Kafka might yield to object storage. But the discipline of engineering—of rigor, measurement, and architecture—is stronger than ever.

All rights reserved, Dewpeche Pvt Ltd, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

How to scale your data team

Building and scaling a data platform has never been more important or more challenging. Whether you’re just starting to build a data platform or leading a mature data organization, this guide will help you scale your impact, accelerate your team, and prepare for the future of data-driven products.

Learn how real data teams, from solo practitioners to enterprise-scale organizations, build.

Get the guide now

Andrej Karpathy: 2025 LLM Year in Review

One year seems a decade in the LLM era. Gemini was still effectively at version 1.5, image models routinely failed at basic text rendering, and credible video generation had not yet arrived. DeepSeek R1 did not exist; o1 was only beginning to introduce test-time inference. The author highlights the significance of 2025 and the paradigm shift that altered the landscape.

https://karpathy.bearblog.dev/year-in-review-2025/

LangChain: State of Agent Engineering

LangChain has published a survey of 1300 professionals on Agent Engineering in the industry. Key highlights for me,

1. Customer service and productivity products dominate the AI adoption.

2. Quality of the output is the biggest barrier to entry for AI Agents.

3. The open-source model has an equal market share between Gemini and Claude.

https://www.langchain.com/state-of-agent-engineering

Google: Introduction to AI Agents

Just like a Data Scientist, Agent Engineering is becoming the hottest job in 2026. If you’re looking to get started in this space, Google publishes an excellent overview of an Introduction to AI Agents.

https://drive.google.com/file/d/1C-HvqgxM7dj4G2kCQLnuMXi1fTpXRdpx/view

Sponsored: The data platform playbook everyone's using

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating undering a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your free copy now.

Ben Lorica: “World Model” is a mess. Here’s how to make sense of it.

The article examines how the term "world model" has become an overloaded marketing label across AI vendors, encompassing seven distinct interpretations. The analysis highlights that the “world model” term captures a fundamental shift from token prediction to modeling geometry, physics, and long-horizon dynamics, and recommends that teams evaluate specific implementations based on actual inputs, maintained state, and decision-making outputs rather than accepting vendor terminology at face value.

https://gradientflow.substack.com/p/world-model-is-a-mess-heres-how-to

LanceDB: From BI to AI: A Modern Lakehouse Stack with Lance and Iceberg

Are Iceberg and LanceDB complementary formats? The blog highlights LanceDB's ability to add new columns without a full table rewrite, native multimodel data storage, built-in text and vector indexing, and the potential to change the BI landscape, bringing it closer to AI.

https://lancedb.com/blog/from-bi-to-ai-lance-and-iceberg/

Dropbox: Inside the feature store powering real-time AI in Dropbox Dash

Dropbox describes a hybrid feature store for Dash's real-time AI ranking system, combining Feast for orchestration with a custom Go-based serving layer and Dynovault for storage, achieving sub-100ms latency requirements. The system processes thousands of concurrent feature lookups per search query through a three-part ingestion pipeline—batch processing with intelligent change detection (reducing writes from 100M+ to under 1M records), streaming for real-time signals, and direct writes for precomputed features—delivering p95 latencies of 25-35ms while maintaining feature freshness within minutes of user actions.

https://dropbox.tech/machine-learning/feature-store-powering-realtime-ai-in-dropbox-dash

Uber: How Uber Indexes Streaming Data with Pull-Based Ingestion in OpenSearch™

Uber writes about its native pull-based ingestion contribution to OpenSearch 3.0 to power their multi-region search platform, replacing traditional push-based indexing with Kafka-backed streaming ingestion that eliminates translog overhead and enables durable replay for shard recovery. The system maps each OpenSearch shard one-to-one with Kafka partitions, uses a no-op translog by treating Kafka as the source of truth, and supports two ingestion modes—segment replication, where only primaries ingest, and all-active, where all shards consume independently—delivering consistent global views across regions.

https://www.uber.com/en-IN/blog/how-uber-indexes-streaming-data-with-pull-based-ingestion-in-opensearch/

Grab: How Grab is accelerating growth with real-time personalization using Customer Data Platform scenarios.

Grab writes about CDP Scenarios, a self-serve real-time personalization platform that processes user-initiated events via Apache Flink pipelines, combined with historical profile data from StarRocks and predictive models, to enable sub-15-second latency targeting. The system powers over 12 production use cases, achieving a 3% uplift in subscriber conversions for the Grab Unlimited signup abandonment scenario by delivering personalized nudges within 15 minutes of users dropping off the registration flow.

https://engineering.grab.com/cdp-scenarios

Decathlon: Polars at Decathlon: Ready to Play?

Apache Spark is an undisputed leader in batch processing, and we are starting to see native distributed dataframe processing engines like Polars gaining adoption. Decathlon writes about adopting Polars to process datasets under 50 GiB, replacing Apache Spark for smaller workloads and achieving near-zero infrastructure costs compared to their typical 180 GiB, 24-core Spark clusters.

https://medium.com/decathlondigital/polars-at-decathlon-ready-to-play-6abc4328d06c

Zalando: Contributing to Debezium: Fixing Logical Replication at Scale

Zalando writes about implementing a mechanism to flush LSNs via JDBC keepalives, even when no data changes occur, to prevent disk-exhausting WAL growth in low-traffic databases. By pairing this with a strategy that trusts the replication slot's position over stored Kafka offsets, the design eliminated the "offset mismatch" errors that previously triggered expensive and unnecessary full database re-snapshots.

https://engineering.zalando.com/posts/2025/12/contributing-to-debezium.html

All rights reserved, Dewpeche Pvt Ltd, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

The Scaling Data Teams Guide

The latest eBook in our popular series is now available.

Building and scaling a data platform has never been more important or more challenging. Whether you’re just starting to build a data platform or leading a mature data organization, this guide will help you scale your impact, accelerate your team, and prepare for the future of data-driven products.

Learn how real data teams, from solo practitioners to enterprise-scale organizations, build.

Get the guide now

Jason Gorman: The Gorman Paradox: Where Are All The AI-Generated Apps?

I had this very conversation recently. A friend of mine claimed that it is now easy to build a CRM and that everyone will build their own, thereby making all SaaS companies obsolete in the near future. It’s cheaper to manufacture software now, but SaaS companies don’t win on manufacturing. They win on distribution, trust, and operational burden. Building a CRM, perhaps commodized; operating one as a durable product is not.

https://codemanship.wordpress.com/2025/12/14/the-gorman-paradox-where-are-all-the-ai-generated-apps/

Gunnar Morling: You Gotta Push If You Wanna Pull

Balancing between Query over Data (pull) at Rest and Query (push) over a Stream is often a challenging part of system design. The author points out that the balance is that push is an efficient way to keep the state fresh through incremental data processing, making pull more efficient.

https://www.morling.dev/blog/you-gotta-push-if-you-wanna-pull/

LangChain: Agent Engineering - A New Discipline

Data Scientist → Analytical Engineer → Agentic Engineer. As the technology landscape evolves, we see new roles emerging that require specific skill sets. The blog lays the foundation of the emerging agentic engineering discipline in software engineering.

https://blog.langchain.com/agent-engineering-a-new-discipline/

Sponsored: The data platform playbook everyone’s using

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating undering a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your free copy now.

Mark Rittman: An Homage to Oracle Warehouse Builder, 25 Years Ahead of its Time in all its Java Thick-Client Glory

What goes around… Comes around is so true in data engineering. The blog is an excellent reminder of the quote, and, in fact, the author is making a subtle point: Oracle Warehouse Builder is still way ahead of modern data tools.

https://medium.com/@mark_37168/an-homage-to-oracle-warehouse-builder-25-years-ahead-of-its-time-in-all-its-java-thick-client-48d11eede8a2

Julien Le Dem: Column Storage for the AI Era

Apache Parquet is the de facto file format in data engineering. The blog offers an excellent, pragmatic view of Parquet, from its origins and the format's wins and adoption over the years, to some of its shortcomings, such as the underlying storage and compute semantics, and what the future holds for Apache Parquet. One of my favorite quotes in the blog is

The cheapest data to transfer is still the data you skip entirely.

https://sympathetic.ink/2025/12/11/Column-Storage-for-the-AI-era.html

Uber: Blazing Fast OLAP on Uber’s Inventory and Catalog Data with Apache Pinot™

Uber writes about adopting Apache Pinot to power real-time analytics on over 10 billion catalog entities across their INCA (Inventory and Catalog) system, replacing slow batch processing with sub-second query latencies. The blog narrates how it leveraged Pinot’s upsert capabilities, handling hundreds of thousands of updates per second, combined with optimizations such as UUID primary key compression, Java 17 runtime upgrades, and the Small Segment Merger minion task, to achieve 75% latency reduction and 40% storage savings.

https://www.uber.com/en-IN/blog/blazing-fast-olap-on-ubers-inventory-and-catalog-data-with-apache-pinot/

Pinterest: LLM-Powered Relevance Assessment for Pinterest Search

Pinterest writes about fine-tuned open-source multilingual LLMs (XLM-RoBERTa-large) on human-annotated data to automate search relevance assessment, replacing costly manual labeling that previously limited measurement capabilities. The LLM-based system, which processes Pin text features through a cross-encoder architecture and employs stratified sampling by query interest and popularity, reduced minimum detectable effects from 1.3-1.5% to ≤0.25% while achieving 73.7% exact match with human labels and enabling 150,000 relevance predictions in 30 minutes on a single A10G GPU.

https://medium.com/pinterest-engineering/llm-powered-relevance-assessment-for-pinterest-search-b846489e358d

Agoda: How Agoda Enhanced the Uptime and Consistency of Financial Metrics

Agoda writes about consolidating multiple fragmented financial data pipelines into a centralized Financial Unified Data Pipeline (FINUDP) built on Apache Spark to eliminate inconsistencies caused by duplicate sources, conflicting transformation logic, and inconsistent data quality controls across teams. The unified pipeline, which processes millions of daily booking transactions, achieved 95.6% uptime through three-tier alerting (email, Slack, GoFresh monitoring), shadow testing in merge requests, automated data quality checks, data contracts with upstream teams, and ML-based anomaly detection, while reducing end-to-end runtime from five hours to 30 minutes through query optimization and infrastructure adjustments.

https://medium.com/agoda-engineering/how-agoda-enhanced-the-uptime-and-consistency-of-financial-metrics-ef7d54c4e4f0

Flipkart: Apache Spark Optimisations

If you’re deep into data pipeline optimization, the blog is an excellent resource. The author identifies frequent patterns that can cause Spark performance issues and proposes an approach to address them.

https://blog.flipkart.tech/apache-spark-optimisations-c3464f71bd38

McDonald’s: Built to Scale: How a Config-Driven ETL Engine Is Powering Environmental, Social, and Governance Data Innovation

McDonald’s writes about building a reusable, config-driven ETL/ELT engine to address scalability and performance limitations in traditional manual ETL development, which couldn’t keep pace with growing ESG data requirements. The Python-based framework, which uses YAML configuration files instead of custom code and includes a Visual YAML Configuration Generator web tool, enables teams to build pipelines across multi-cloud environments with support for diverse transformations (column renaming, type conversion, aggregations, SQL functions), automated notifications, and comprehensive auditing while significantly reducing development time compared to traditional ETL methods.

https://medium.com/mcdonalds-technical-blog/built-to-scale-how-a-config-driven-etl-engine-is-powering-environmental-social-and-governance-d0cd2383554f

All rights reserved, Dewpeche Pvt Ltd, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

How to build trustworthy AI analytics

If your team is relying on AI-driven insights, this upcoming webinar will show you how to make those insights more dependable, transparent, and explainable. In this 12/9 Deep Dive with our friends at Cube, you’ll learn:

- Why AI analytics fails without governance (and what that actually means)
- How semantic layers provide the guardrails AI needs to be trustworthy
- Technical implementation: how Compass + Cube work together to prevent hallucinations
- Live demo: governed self-service analytics that data teams can actually trust

Save your spot now

DoorDash: Beyond Single Agents: How DoorDash is building a collaborative AI ecosystem

DoorDash highlights the challenge of extracting reliable insights from fragmented knowledge systems and the limitations of single agents constrained by context, determinism, and long-horizon reasoning. The article details an evolutionary architecture that progresses from deterministic workflows to adaptive agents, hierarchical deep-agent systems with shared memory, and exploratory swarm-based A2A collaboration, all built on a unified platform featuring hybrid search, schema-aware SQL generation, multi-stage validation, and integrated guardrails.

https://careersatdoordash.com/blog/beyond-single-agents-doordash-building-collaborative-ai-ecosystem/

LinkedIn: The evolution of the Venice ingestion pipeline

LinkedIn writes about the challenge of scaling Venice ingestion to support massive bulk loads, hybrid Lambda-style stores, partial updates, and active/active replication while avoiding bottlenecks in producing, consuming, persisting, and compaction. The article details the end-to-end evolution of the ingestion pipeline, including partition scaling, shared consumer and writer pools, SST-based ingestion, RocksDB tuning with leveled compaction, BlobDB, Fast-Avro adoption, parallelized DCR processing, and adaptive throttling for deterministic latency.

https://www.linkedin.com/blog/engineering/infrastructure/evolution-of-the-venice-ingestion-pipeline

Dropbox: How Dash uses context engineering for smarter AI

Dropbox writes about the transition from a traditional RAG search system to an agentic AI that must reason, plan, and act without being overwhelmed by excessive context, tool proliferation, or accuracy degradation in long-running tasks. The article details three core context-engineering strategies:

1. Consolidating retrieval into a single universal search tool

2. Filtering context through a unified index and knowledge graph for relevance.

3. Delegating complex workflows, such as query construction, to specialized agents with focused prompts.

The approach improves reasoning quality, reduces token and decision overhead, and enables faster, more accurate agentic execution across Dash’s growing AI capabilities.

https://dropbox.tech/machine-learning/how-dash-uses-context-engineering-for-smarter-ai

Sponsored: The guide to scaling your data team

Scaling Data Teams, the latest in our popular eBook series, is now available. Building and scaling a data platform has never been more important or more challenging. Whether you’re just starting to build a data platform or leading a mature data organization, this guide will help you scale your impact, accelerate your team, and prepare for the future of data-driven products.

Learn how real data teams, from solo practitioners to enterprise-scale organizations, build.

Get the guide now

LinkedIn: FishDB - a generic retrieval engine for scaling LinkedIn’s feed

LinkedIn writes about replacing its legacy Java-based feed infrastructure with FishDB, a custom Rust-based retrieval engine designed to eliminate Garbage Collection latency and reduce memory overhead by ~5x compared to JVM equivalents. By leveraging a lambda architecture for ingestion and graph-based document references for query execution, FishDB cut hardware usage by 50% while maintaining a strict 40ms p99 latency. The query language is very interesting, improving expressiveness, and it suits the retrieval engine well.

https://www.linkedin.com/blog/engineering/infrastructure/fishdb-a-generic-retrieval-engine-for-scaling-linkedins-feed

Netflix: Integrating Netflix’s Foundation Model into Personalization applications

Netflix writes about integrating the personalization Foundation Model into diverse production systems that have different latency constraints, feature pipelines, and appetites for model complexity. The article details three integration patterns: using stable daily embeddings via the Embedding Store, embedding the Foundation Model as a fine-tunable subgraph within downstream models to provide fresh representations, and fully fine-tuning the Foundation Model to power product-specific objectives directly.

https://netflixtechblog.medium.com/integrating-netflixs-foundation-model-into-personalization-applications-cf176b5860eb

Lyft: LyftLearn Evolution: Rethinking ML Platform Architecture

Lyft writes about the challenge of scaling its Kubernetes-only LyftLearn ML platform as operational complexity, state management overhead, and cluster-level capacity tuning become bottlenecks for thousands of daily training jobs and notebooks. The article details a hybrid LyftLearn 2.0 architecture that keeps low-latency online serving on Kubernetes while moving offline compute to AWS SageMaker, enabled by cross-platform base images, a SageMaker manager service, and SQS/EventBridge–based state tracking.

https://eng.lyft.com/lyftlearn-evolution-rethinking-ml-platform-architecture-547de6c950e1

Grab: Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test.

Grab writes about implementing a data contract in a real-time streaming engine with syntactic and semantic tests. The blog is essentially a testimonial for data contract frameworks and their role as an integral part of data pipeline engineering.

https://engineering.grab.com/real-time-data-quality-monitoring

BlaBlaCar: Why We Built “BlaBlaCar Data Copilot”: Shifting Data Analysis Left

BlaBlaCar writes about the bottleneck created by a hard boundary between software engineers and data analysts, where engineers avoid the warehouse, and analysts drown in ad hoc questions and fragile SQL, slowing feedback on product data and undermining true data ownership. The article presents BlaBlaCar Data Copilot, an AI-powered “junior analyst” inside the IDE that tunnels into BigQuery, uses curated queries and table previews via a lightweight zero-infrastructure RAG pattern, generates SQL/Python plus data health cards, and turns analyses into tested scripts reviewed via pull requests and stored as long-term repo memory.

https://medium.com/blablacar/why-we-built-blablacar-data-copilot-shifting-data-analysis-left-b4cc246faf52

Vinted: Dense Retrieval

The blog describes how low recall keyword searches on a highly visual, multilingual e-commerce catalog led to missed business opportunities and made it hard to safely roll out dense retrieval at scale under tight latency and consistency constraints. The article details a CLIP-based two-tower dense retrieval model trained with large-scale contrastive learning, exported to ONNX and embedded into a Vespa-based architecture, plus a long list of optimizations including index sharding by market, tuned ANN thresholds, approximate→exact retry strategies, global-phase reciprocal rank fusion to cap NN matches, and JVM tuning with GraalVM and ZGC.

https://vinted.engineering/2025/11/18/dense-retrieval/

All rights reserved, Dewpeche Pvt Ltd, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.