Data Engineering Weekly

Data Engineering Weekly #273

Ananth Packkildurai — Mon, 08 Jun 2026 02:44:02 GMT

How to Build a Data Platform

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating under a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your free copy now

Fei-Fei Li: A Functional Taxonomy of World Models

The World Model is one of the most overloaded terms today. The author writes about the taxonomy of the world model into three functions,

Renderer: A renderer outputs observations in the form of pixels meant for human eyes, and the quality that matters most is visual fidelity.
Simulator: A simulator outputs a state: a geometrically, physically, or dynamically faithful representation of the world that humans and computer programs can both compute on and interact with.
Planner: A planner outputs actions.

From a data warehouse perspective, the metric tree is an attempt to build a simpler version of a business simulator. We need to think about extending the simulation of the entire business as it emerges.

https://drfeifei.substack.com/p/a-functional-taxonomy-of-world-models

Gorgias: When Event Time Meets Reality: Lessons from Building Billing on Apache Flink

In real-time pipelines, we assume that the watermark configuration alone preserves ordering, but reprocessing historical data with keyBy operators breaks source-level alignment downstream. The author aligns deduplication and consolidation using the same customer-user key, removes intermediate repartitioning, and extends consolidation by 1 day when event lag exceeds 15 minutes. The combined fix cuts overlapping consolidation windows by 10x without delaying real-time billing, anchored to lag-aware timer registration with explicit cleanup.

https://medium.com/gorgias-engineering/when-event-time-meets-reality-lessons-from-building-billing-on-apache-flink-581ff895c60d

Cloudflare: Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse.

Cloudflare writes that Multi-tenant ClickHouse partitioning hides query-planner coupling at scale — the part-list mutex and vector copy serialize across concurrent queries, even when each query reads few parts. Cloudflare patches ClickHouse’s MergeTreeData hotspot with three upstream fixes — replacing the exclusive lock with std::shared_lock, deferring the parts-vector copy, and binary-searching the sorted namespace prefix.

https://blog.cloudflare.com/clickhouse-query-plan-contention/

Helpshift: Migrating from a Monolithic Orchestrator to Apache Airflow

Helpshift writes about migrating a monolithic Clojure-based scheduler system to Apache Airflow. The team writes about its journey and the overall impact on metrics such as reliability, availability, performance, data quality, & the developer productivity.

https://medium.com/helpshift-engineering/migrating-from-a-monolithic-orchestrator-to-apache-airflow-30fde94bcdc0

Indeed: Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search

We have long relied on distillation to compress complexity. But while humans reason within finite cognitive dimensions, foundation models now navigate data at an order-of-magnitude greater scale. We are moving from a paradigm of static, structural data modeling to a new era where the embedding model itself has become our primary, high-dimensional data model. - Ananth

This case study perfectly illustrates the thesis: Indeed’s shift from traditional feature engineering to a unified, transformer-based UBM architecture demonstrates the evolution from static, structural data design to the embedding model as the primary, high-dimensional data model for reasoning.

https://engineering.indeedblog.com/blog/2026/06/distilling-long-tail-user-behavior-into-scalable-embeddings-for-job-search/

Netflix: Dynamic Repartitioning for Time Series Workloads

Netflix discusses how static Cassandra partitioning breaks down under evolving time-series workloads when a small percentage of IDs accumulate disproportionate event volumes, driving wide partitions that cause tail-latency timeouts. Netflix layers two runtime adjustments onto its TimeSeries Abstraction — a background worker returning future Time Slice partitions, and per-ID dynamic splitting that routes reads via Bloom filters.

https://netflixtechblog.com/dynamically-splitting-wide-partitions-in-cassandra-for-time-series-workloads-0eded064f456

Jack Vanlightly: Broker-Visible vs Client-Local Parallelism

Messaging system parallelism has a hidden cost — where the unit of parallelism lives determines whether scaling stays cheap or explodes into broker-managed TCP connections and state. The author compares broker-visible parallelism with client-local parallelism using virtual threads, showing that the latter scales a 60K msg/sec workload from 60,000 consumers down to 60. Client-local parallelism shifts coordination cost off the broker but adds client complexity, leaving share groups in need of a parallel-processing library layer, anchored to per-message acknowledgment semantics.

https://jack-vanlightly.com/blog/2026/6/3/broker-visible-vs-client-local-parallelism

Criteo: Introducing CLEPR, our model for semantic understanding

Criteo writes about building CLEPR, a two-tower contrastive embedding model with separate keyword and product encoders — training on session-stitched pseudo-clicks and deduplicating repeated pairs to strip popularity bias. CLEPR scores 10 billion keyword-product pairs daily as a semantic guardrail before performance ranking, lifting CTR by 6%, anchored to embeddings that generalize to human-judged relevance.

https://medium.com/criteo-engineering/introducing-clepr-our-model-for-semantic-understanding-d3984eed84c8

WMG: Why we shrank our TimescaleDB chunks from 30 days to 7

Storage configuration choices made when a database is small calcify silently as ingest grows, turning yesterday’s reasonable defaults into compression failures and backfill storms. WMG Lab shrinks TimescaleDB chunk intervals from 30 days to 7 across its hot hypertables — using set_chunk_time_interval to retune future chunks only, without rewrites or locks. The smaller chunks let compression finish cleanly and reduce backfill to a single 7-day decompression instead of a month, anchored to the 25% active-chunk memory rule.

https://tech.wmg.com/why-we-shrank-our-timescaledb-chunks-from-30-days-to-7-07cab8afefc5

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.

Data Engineering Weekly #272

Ananth Packkildurai — Mon, 01 Jun 2026 03:18:34 GMT

How to Build a Data Platform

Download your free copy now

Netflix: High-Throughput Graph Abstraction at Netflix

One emerging data model envisions the organization model as a property graph abstraction, though the underlying data may be stored in a relational model, key/value store, or document store. GraphQL is one abstraction, but it leads to many complications while integrating the api. Netflix writes about its high-throughput graph abstraction and explains how it handles read-aside cache and write-aside cache.

https://netflixtechblog.com/high-throughput-graph-abstraction-at-netflix-part-i-e88063e6f6d5

Slack: Slack AI - The Path to Multi-Cloud

Building AI Infrastructure for a multi-tenant stack with enterprise security requirements is challenging in itself. Slack writes about evolving from SageMaker-managed endpoints to Bedrock provisioned throughput, Bedrock on-demand spillover, and a multi-cloud Vertex AI architecture with normalized APIs, model hierarchies, circuit breakers, health-aware routing, and feature-level model selection. The platform focuses on reducing the infrastructure lock-in, enabling same-day model migration, improving the quality of complex reasoning by about 10%, cutting latency for low-token workloads by about 67%, and increasing resilience against regional and provider-wide failures.

https://slack.engineering/slack-ai-the-path-to-multi-cloud/

Uber: Solving the Identity Crisis for AI Agents

The article seems so timely, as I recently finished a similar design for AI Identity. The conventional way to start the design is to assign the user role to the agent to work on their behalf, but the model soon fails. The services-centric IAM role also fell apart, since the same service can potentially assume multiple roles. Uber provides an in-depth account of how it handles these challenges.

https://www.uber.com/us/en/blog/solving-the-agent-identity-crisis/

Mehul Batra: A field journal on Ray Data and Daft for multimodal data lake

Multimodal data lake pipelines on Kubernetes require engines that can handle mixed CPU, GPU, and async inference workloads, along with cataloging and checkpointing, across text, PDF, image, audio, video, and LLM metadata. The author compares Ray Data 2.55.1 and Daft 0.7.13 on the same KubeRay, DigitalOcean, Gravitino, Iceberg, Lance, and H100 setup, measuring wall time, GPU occupancy, memory, code size, native primitives, concurrency behavior, and completion reliability across eight use cases. Ray Data scored 56 versus Daft’s 47 out of 70 because both engines tied on most multimodal batch workloads, Daft delivered stronger native media ergonomics, and Ray Data won the decisive async LLM inference workload by completing the 50k-email job with enforced concurrency while Daft failed to finish.

https://mehulbatra.medium.com/a-field-journal-on-ray-data-and-daft-for-multimodal-data-lake-e0c26839f5b5

LakeOps: Routing Multiple Query Engines with Iceberg

Multi-engine Iceberg deployments let Spark, Trino, Flink, DuckDB, Athena, Snowflake, and StarRocks share the same tables. Still, they leave query placement, cost control, dialect compatibility, and workload isolation to each client team. LakeOps writes about QueryFlux, a Rust-based SQL routing proxy that supports existing database protocols, selects engine groups via routing rules, translates SQL dialects with sqlglot, enforces concurrency limits, and dispatches queries based on cost, latency, throughput, or health.

https://lakeops.dev/blog/routing-multiple-query-engines-with-iceberg

Rion Williams: Prepare for Launch: Enrichment Strategies for Apache Flink

Flink enrichment jobs become fragile when they depend on referential data that is not already present in state, forcing teams to choose between slow per-record lookups, incomplete early results, or complex state preloading. The article compares external enrichment, CDC-backed gradual enrichment, State Processor API–based two-phase bootstrapping, and gated enrichment as ways to manage the launch-time pain of warming the state before live traffic depends on it.

https://rion.io/2026/01/27/prepare-for-launch-enrichment-strategies-for-apache-flink/

Agoda: How Agoda Simulates Booking Flows to Test Flight Integrations

Agents perform relatively well when there is a faster feedback loop to tune, but unfortunately, in many business cases, the transactions are sensitive. Simulation is the most critical aspect of engineering components, not only for testing but also as an RL environment for Agents. Digital twins are emerging as a critical component of simulation. Agoda describes a simulation system for booking flows to test flight integrations.

https://medium.com/agoda-engineering/how-agoda-simulates-booking-flows-to-test-flight-integrations-204ec4f2e128

Databricks: Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs

PySpark Pandas UDFs improved Python execution with Arrow serialization and batching, but still incur Pandas/Arrow conversion costs, add memory copies, and struggle with complex data types. Databricks writes about the introduction of native Arrow UDFs, Arrow aggregate functions, Arrow UDTFs, and DataFrame-level mapInArrow and applyInArrow APIs that operate directly on PyArrow arrays, record batches, and tables. Arrow UDFs preserve columnar execution end-to-end, run about 10% faster than Pandas UDFs, use about 40% less memory, and support complex datatype and table-in/table-out transformations with familiar Python and SQL interfaces.

https://www.databricks.com/blog/introducing-arrow-udfs-pyspark-faster-leaner-replacement-pandas-udfs

Giannis Polyzos: When Tables Became the Language of Time

Batch and streaming architectures create duplicated logic, fragmented corrections, and inconsistent state when systems treat logs as the source of truth instead of modeling how data changes over time. The author reframes streams and tables as two views of the same history, using Flink for streaming-first compute, lakehouse commits for durable memory, and Apache Fluss as table-first streaming storage with native updates, deletes, schemas, snapshots, and columnar access. Table-centric streaming collapses batch jobs, replays, late corrections, and real-time dashboards into different temporal views of evolving state, making unification an architectural consequence rather than a glue layer.

https://ipolyzos.substack.com/p/when-tables-became-the-language-of

Data Engineering Weekly #271

Ananth Packkildurai — Mon, 25 May 2026 16:05:00 GMT

How to Build a Data Platform

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating undering a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your free copy now

Netflix: The Evolution of Cassandra Data Movement at Netflix

The Change Data Capture (CDC) from the operational store is often expensive and involves multiple staging hops and an expensive merge operation in Iceberg. Netflix writes one such case study with its Cassandra, the challenges with capturing operational data into Iceberg tables, and its solution to avoid partition skew with a layered approach.

https://netflixtechblog.medium.com/the-evolution-of-cassandra-data-movement-at-netflix-6e13329c80a1

Grab: The Hugo evolution: Engineering Grab’s unified, one-click data ingestion platform with Apache Flink

Grab narrates a similar challenge with the CDC, highlighting the disintegration of data ingestion across multiple operational data stores with schema management and ingestion issues. The unified pipeline, with the Flink pipeline, auto-detects schema changes and ingests the data back into the Hive Tables.

https://engineering.grab.com/one-click-data-ingestion-platform-with-apache-flink

Meta: A Blueprint for Valuing Content When A/B Tests Are Not an Option

Content is a primary driver of the Quest ecosystem. With the recent announcement at Google I/O about seamless shopping integration with content, it is evident that content-driven commerce has reached the mainstream. How do you value the contents when there is no A/B testing option available? Meta writes about implementing the DoubleML method to tackle the challenge.

https://medium.com/@AnalyticsAtMeta/meta-a-blueprint-for-valuing-content-when-a-b-tests-are-not-an-option-7880bac721f1

Uber: Scaling Real-Time Traffic Forecasting with a Graph-Aware Transformer

Uber writes about rebuilding the traffic forecasting stack, DeepETT, a real-time traffic forecasting system. DeepETT approaches forecasting as a fixed-input graph-aware transformer that combines pre-aggregated segment, road-graph, regional, historical, real-time, and event features with continuous Flink-based calibration.

https://www.uber.com/us/en/blog/scaling-real-time-traffic/

Airbnb: Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure

Counting and Finding Unique Users are the two hard problems in Data Engineering.

One of the long-standing questions in data engineering is: since many real-world systems are fundamentally about connections, why can’t we model them using the graph data model? Airbnb highlighted the reasons for the scalability issues with Graph and its adoption of JanusGraph, using DynamoDB as a backend.

https://medium.com/airbnb-engineering/scaling-airbnbs-identity-graph-with-a-unified-knowledge-graph-infrastructure-ebac467b7836

Pinterest: Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

The user journey/user sequence of actions is one of the most important signals for analyzing user behavior. Pinterest publishes a comprehensive case study on how to approach user sequence data as a product and its architectural patterns.

https://medium.com/pinterest-engineering/making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use-2a56a928cae1

Yelp: How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%

Usage-driven data retention & storage class optimization is a must-have tool for your Lakehouse management, given the growing need to ingest more data. Yelp applies the art and science of table management by collecting usage metrics at the table-partition level to optimize storage.

https://engineeringblog.yelp.com/2026/05/partition-access-visualizations.html

LinkedIn: Crosscheck: Benchmarking AI Models in the Real World

Static AI benchmarks lose signal as models optimize toward them, collapsing role-, industry-, and task-specific performance into one number that answers no professional’s actual question. LinkedIn writes about Crosscheck, which extends the Bradley-Terry comparison model with time-decay weighting, low-data regularization, and confidence-aware ordinal tiering — surfacing only differences supported by 95% statistical evidence.

https://www.linkedin.com/blog/engineering/ai/crosscheck-benchmarking-ai-models-in-the-real-world

Jack Vanlightly: Introducing Dimster, a performance benchmarking tool for Apache Kafka

Kafka performance benchmarks rarely travel — results lack the configuration, hardware, and version metadata that another engineer needs to reproduce or trust them. The author builds Dimster, a Kafka benchmarking tool centered on dimensional testing — sweeping config axes like batch.size or consumer type while emitting self-contained result bundles. Dimster runs explore, drain-backlog, and correctness modes on Kubernetes as a portable runtime, making benchmark campaigns reproducible across any cloud or laptop, anchored to traceable result artifacts.

https://jack-vanlightly.com/blog/2026/5/20/introducing-dimster-a-performance-benchmarking-tool-for-apache-kafka

Data Engineering Weekly #270

Ananth Packkildurai — Mon, 18 May 2026 01:54:53 GMT

The Data Platform Fundamentals Guide

Download your free copy now

Airbnb: Viaduct 1.0 and the future of Airbnb’s data mesh

The term Data Mesh focuses on the platform & services, Viaduct aims to be the multi-tenant GraphQL system, unlike the data mesh as we know it in data engineering. However, the first principle of Viaduct is something the data teams should drive towards to bring a more connected experience across the business functions.

One global schema. Independent teams contribute schema and resolvers

https://medium.com/airbnb-engineering/viaduct-1-0-and-the-future-of-airbnbs-data-mesh-6bab4ec98b89

Netflix: Data Projects: Managing Data Assets at Netflix Scale

Netflix writes about the operational challenges of enforcing ACLs at the individual user level. When pipelines run under an On-Behalf-Of permission model, access frequently breaks when employees change teams or leave the organization. To address this, Netflix adopts the concept of Data Projects, replacing user-bound identities with durable, team-owned application identities.

https://netflixtechblog.medium.com/data-projects-managing-data-assets-at-netflix-scale-7ca25888591e

Meta: Migrating Data Ingestion Systems at Meta Scale

Meta writes about successfully migrating tens of thousands of mission-critical pipelines at the petabyte scale. The blog focused on how it ensures the shadow testing mode to enable end-to-end robustness of the migration process. The partition-level metadata flags are an interesting approach. If a partition was marked as “bad,” the system automatically halted new delta landings and forced merges with older, known-good partitions, stopping the bleeding instantly.

https://engineering.fb.com/2026/05/12/data-infrastructure/migrating-data-ingestion-systems-at-meta-scale/

Databricks: The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available

The data infrastructure went through a full lifecycle, from Hive metastore to S3 as the primary metastore, and back to catalogs. Databricks writes about the convergence of table formats and the catalogs, with how Unity Catalog and Delta Lake formats interplay.

https://www.databricks.com/blog/convergence-open-table-formats-and-open-catalogs-catalog-commits-generally-available

Eric Sun: A Query Proxy for Analytical and Fast Data

The author writes about the Query Proxy, which fronts Snowflake, Databricks, StarRocks, and Iceberg via gRPC, with service-credential-based RBAC, and returns async query IDs and presigned Parquet URLs to clients. The proxy collapses per-engine clients into a single interface, federates hot Postgres data with warm Iceberg partitions, and tracks query-shape telemetry, all anchored to a stateless gateway model.

https://eric-sun.medium.com/a-query-proxy-for-371907878996

The Craft: How we rebuilt search ranking at Faire with deep learning

Faire writes about replacing XGBoost with a Deep and Cross Network trained with session-normalized listwise cross-entropy, then accelerates new-ranker development by fine-tuning Brand Page models from Product Search. The rebuild increases Product Search order volume by 2.14% in North America and 1.54% in Europe while cutting new-surface launch cycles by half through a shared multi-task representation.

https://craft.faire.com/how-we-rebuilt-search-ranking-at-faire-with-deep-learning-14f080679c83

Alibaba: When MySQL Meets the Columnar Storage Engine DuckDB in the AI Era

AliSQL writes about integrating DuckDB as a pluggable storage engine, transforming MySQL into a lightweight Hybrid Transactional and Analytical Processing (HTAP) database. Motherduck recently released the client-server protocol DuckDB, as it is slowly moving to a warehouse-style engine on its own.

https://www.alibabacloud.com/blog/when-mysql-meets-the-columnar-storage-engine-duckdb-in-the-ai-era_603117?spm=a2c65.11461433.0.0.2e5d5355O7iepw

Marc Bowes: Aurora DSQL: Meet Coupler

The author writes about the design of Coupler, a highly scalable CDC solution for Aurora DSQL. Traditionally, CDC is hard on databases and degrades performance. Because DSQL’s architecture already decouples its read/write paths using a low-latency journal fan-out, Coupler acts as another subscriber, making it highly performant.

https://marc-bowes.com/dsql-coupler.html

Pete: Full-Text Search with DuckDB

The author writes about DuckDB’s Full-Text Search (FTS) extension for data workflows, such as exploratory text mining and document archive analysis. The development is worth watching, as I have always found cases that balance both free-text analytics and typical exploratory analysis. Apache Pinot & ParadeDB are the two systems I know of that try to handle these cases.

https://peterdohertys.website/blog-posts/full-text-search-w-duckdb.html

Data Engineering Weekly #269

Ananth Packkildurai — Mon, 11 May 2026 02:50:23 GMT

The Data Platform Fundamentals Guide

Download your free copy now

Meta: How We Built an AI Second Brain for 60K Knowledge Workers

Meta writes about building an AI Second Brain that combines a PARA-based workspace model, authenticated infrastructure access via MCPs and CLIs, agentic execution loops, and reusable markdown-defined skills that continuously organize, retrieve, and act on structured work context across tools and projects. The article is a classic case of data engineering evolving into context engineering, making a broader impact in the industry.

https://medium.com/@AnalyticsAtMeta/how-we-built-an-ai-second-brain-for-60k-knowledge-workers-78c507dd795b

Salesforce: How Informatica Built a Multi-Agent AI System to Reduce Data Workflows from Months to Days

Enterprise data workflows break down when discovery, governance, data quality, and orchestration span disconnected systems, requiring repeated human coordination across tools, teams, and execution stages. Informatica writes about the CLAIRE platform, which addresses fragmentation through a multi-agent architecture that combines orchestration agents, semantic context modeling, deterministic tool routing, and specialized execution agents to coordinate workflows involving 50–60 model calls with validation checkpoints and adaptive planning.

https://engineering.salesforce.com/how-informatica-built-a-multi-agent-ai-system-to-reduce-data-workflows-from-months-to-days/

Netflix: Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

ML platforms fragment across registries, orchestrators, feature stores, and experimentation systems, forcing practitioners to traverse multiple tools to trace lineage or reuse assets across domains. Netflix writes about its Metadata Service, which ingests events via a unified AIP URI scheme and materializes a Model Lifecycle Graph in Datomic, connecting models, features, pipelines, and experiments through asynchronous enrichment. The graph collapses multi-system investigations into a single GraphQL traversal from models to features to pipelines to A/B tests, standardizing cross-domain discovery anchored in a normalized entity model.

https://netflixtechblog.com/democratizing-machine-learning-at-netflix-building-the-model-lifecycle-graph-5cc6d5828bb1

Confluent: Stream Processing vs. Real-Time OLAP: Flink, ClickHouse & Pinot Compared

Real-time data platforms routinely collapse when teams treat stream processors and OLAP engines as substitutes, exposing dashboards to stateful pipelines or routing continuous ETL through scan engines. Confluent writes about drawing the boundary at computation timing — stream processors evaluate data in motion through event-time windows and watermarks. In contrast, OLAP engines evaluate data at rest through columnar scatter-gather scans.

https://www.confluent.io/blog/stream-processing-vs-real-time-olap-flink-clickhouse-and-pinot-compared/

Kirill Bobrov: The Power of Data Sketches: A Comprehensive Guide

Exact answers to cardinality and quantile queries become infeasible at the billion-event scale because COUNT DISTINCT shuffles every unique value across the cluster rather than aggregating partially. The author walks through Apache DataSketches — Theta, HyperLogLog, CPC, KLL, and Frequent Items — replacing full-data shuffles with kilobyte hash summaries that merge across partitions. Sketches shift analytics from query-time scans to precomputed dimensional cubes, trading bounded approximation for parallel ingestion and late-data merges, anchored in a fixed-memory streaming model.

https://luminousmen.com/post/the-power-of-data-sketches-a-comprehensive-guide/

Whatnot: The ML Feature Pipeline That Got Slower and No One Noticed

Hourly pipelines leave a shorter window to catch errors, compressing the failure window and exposing silent degradation — zero-row regressions, runtime drift, and cadence slippage that never trigger threshold alerts. Whatnot writes about handling such pipeline failures by splitting monitoring into two tiers — a zero-row signals page on-call, and distribution shifts routed to Slack — backed by a 2-day Redis TTL that absorbs missed updates.

https://medium.com/whatnot-engineering/the-ml-feature-pipeline-that-got-slower-and-no-one-noticed-8e90c224eae3

Pinterest: Enhancing Ad Relevance: Integrating Real-Time Context into Sequential Recommender Models

Sequential recommender models trained on offline behavioral history fail on contextual surfaces because the user embedding ignores what the viewer is currently browsing. Pinterest writes about injecting a context layer into the two-tower query tower — concatenating Transformer history with subject-Pin features and training with synthetic context from positive labels. The hybrid flow precomputes the Transformer offline and runs the context layer online, lifting Recall@K by 3-10x and ROAS by 0.7%, anchored to dynamic user embeddings.

https://medium.com/pinterest-engineering/enhancing-ad-relevance-integrating-real-time-context-into-sequential-recommender-models-bc3a2f9b682e

Grab: Enhancing Flink deployment with shadow testing.

Staging environments cannot reproduce production-specific failure modes in streaming applications — checkpoint incompatibility, traffic shape, and environment drift only surface under live traffic. Grab writes about adding a Shadow Testing stage to its Flink deployment pipeline — running a parallel shadow job in an isolated Kubernetes namespace with prefixed Kafka consumer groups and dedicated shadow sinks. The pre-deployment shadow run absorbs the failure window that caused 10-minute rollback outages, raising Deployment Frequency and cutting Change Failure Rate, anchored to connector-level isolation.

https://engineering.grab.com/enchancing-flink-shadow-testing

Halodoc: Building Self-Healing Data Pipelines at Halodoc

Generic retry-everything pipeline recovery breaks down at scale because CDC restarts, OOM failures, orphaned warehouse queries, and cascading backfills each demand distinct checkpoint logic. Halodoc layers six targeted recovery mechanisms — CDC restart with three-gate eligibility checks, file-size-aware mini-batching, OOM-classified retry scaling, and watermark-based warehouse lock cancellation. The platform cuts CDC recovery to under 5 minutes, drops on-call alerts by 80%, and collapses backfill setup to 15 minutes, anchored to per-failure-mode contracts.

https://blogs.halodoc.io/building-self-healing-data-pipelines-at-halodoc/

Data Engineering Weekly #268

Ananth Packkildurai — Mon, 04 May 2026 01:09:52 GMT

Free Course: AI-Driven Data Engineering

Get started now

Event Highlight: Don't Miss AI Council

- The technical conference for humans who ship

Join the people actually building AI & data infrastructure - and hear them share what’s working, what broke in prod, and what’s coming next. May 12–14 in San Francisco.

Speakers include: The co-inventor of ChatGPT. Creator of DuckDB. Creator of Codex. Engineers from ClickHouse, Databricks, Datadog, and LangChain.

→ Save 20% on your ticket with code DATAEW20 through 5/5

Grab: Data Mesh at Grab Part II: The Foundational Tools behind Certification

Scaling a data mesh across hundreds of thousands of assets demands enforceable trust, consistent governance, and reliable quality without centralized bottlenecks. Grab operationalizes certification through three integrated platforms — Hubble for metadata and event-driven certification, Genchi for continuous quality validation, and a Data Contract Registry enforcing producer-consumer agreements. The system converts data mesh principles into a metadata-driven workflow, reducing dataset sprawl and driving convergence toward certified, reusable assets anchored to analytics and AI workloads.

https://engineering.grab.com/data-mesh-2

Doug Turnbull: Can agents replace the search stack?

Search APIs depend on layered pipelines for query understanding, retrieval, and reranking, creating complexity that struggles to adapt across heterogeneous user intents. The author writes about testing GPT-5 and GPT-5-mini agents with basic BM25 and e5 embedding tools on Amazon ESCI, lifting NDCG from 0.289 to 0.453 — exploration prompts and duplicate-query rejection further closed the gap on smaller models. The findings reframe retrieval for product-style “finding things” workloads as an agent-driven loop. However, knowledge-gap tasks like MSMarco still favor traditional embedding stacks anchored in dense-retrieval quality.

https://softwaredoug.com/blog/2026/04/28/search-apis-replaced-by-agents

Pinterest: Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer

Root-leaf ML serving architectures unlock GPU specialization but bottleneck on network bandwidth when fanning out feature payloads to partitioned model inference. Pinterest writes about building Feature Trimmer, a “Send What You Use” system that treats the model signature as the source of truth — version-aware allowlists sync with bundle deployments through the same staged rollout, fallback, and concurrency safeguards as model configs.

https://medium.com/pinterest-engineering/optimizing-ml-workload-network-efficiency-part-i-feature-trimmer-ae20beb08d69

Sponsored: The AI Modernization Guide

Will your data platform accelerate your AI initiatives or become their biggest bottleneck? Learn how to build a data platform that's ready for AI:

- Transform from Big Complexity to AI-ready architecture
- Real metrics from organizations achieving 50% cost reductions
- Introduction to Components: YAML-first pipelines that AI can build

Get the guide now

Pinterest: From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest

Optimizing ad retrieval for offsite conversions requires modeling sparse, noisy, delayed signals that engagement-based candidate generators were never designed to surface. Pinterest writes about built a two-tower shopping conversion retrieval model that unifies conversions and click-duration-weighted engagement under a single multi-task head, paired with a parallel DCN v2 and MLP cross-layer architecture, and an advertiser-level loss to stabilize sparse Pin-level supervision.

https://medium.com/pinterest-engineering/from-clicks-to-conversions-architecting-shopping-conversion-candidate-generation-at-pinterest-04cae5e1455b

FiveTran: How we accelerated transpilation by compiling SQLGlot with mypyc

Fivetran writes about compiled SQLGlot with mypyc, transpiling type-annotated Python into C extensions, contributing five upstream string primitives, and inlining hot paths like sentinel tokens, native i64 integers, and pre-built dispatch dictionaries — all while preserving the pure Python install path. The compiled sqlglot[c] package accelerates parsing by 5x, SQL generation by 2.5x, and optimizer passes by 2-2.5x, anchored to a dual-distribution model that keeps SQL transpilation portable across multi-engine data lake architectures.

https://www.fivetran.com/blog/how-we-accelerated-transpilation-by-compiling-sqlglot-with-mypyc

Robin Moffatt: Materialized Tables in Apache Flink

Streaming SQL frameworks split table definitions from the population logic, leaving INSERT jobs orphaned across restarts and forcing operators to manage schema evolution and lifecycle as separate concerns. The author walks through Flink 2.2’s Materialized Tables, which bind the refresh query to the table definition and support CONTINUOUS or FULL refresh modes, partition-scoped reloads, suspend/resume via savepoints, and unified batch-streaming semantics through a single FRESHNESS parameter. The construct collapses three legacy patterns — CREATE/INSERT, CTAS, and external schedulers — into a single durable object. However, catalog support beyond Paimon and the embedded scheduler remains anchored in early-stage maturity gaps.

https://rmoff.net/2026/04/28/materialized-tables-in-apache-flink/

Alexey Makhotkin: 5NF and Database Design

Traditional database normalization tutorials present 5NF through contrived table-splitting exercises that obscure the underlying business logic and leave practitioners unable to apply it in practice. The author reframes 5NF design around two logical patterns — the AB-BC-AC triangle for independent M: N relationships across three anchors, and the ABC+D star pattern, where a fourth entity binds three 1:N links — thereby driving table construction directly from business requirements rather than from post hoc decomposition. The approach replaces 5NF reasoning with a deterministic logical-to-physical workflow, anchored to anchor-link modeling that produces normalized schemas without invoking decomposition theorems.

https://kb.databasedesignbook.com/posts/5nf/

ultrathink: SQLite in Production: Lessons from Running a Store on a Single File

Single-file embedded databases promise operational simplicity, but their filesystem-level locking model breaks down when modern container orchestration introduces concurrent writers across overlapping deploys. Ultrathink writes about running a production e-commerce store on Rails 8 with four SQLite databases on a shared Docker volume, diagnosing lost orders through sqlite_sequence after eleven rapid Kamal blue-green deploys caused overlapping containers to corrupt WAL writes despite successful Stripe charges.

https://ultrathink.art/blog/sqlite-in-production-lessons

Capital One: Spark tuning: executor optimization for performance

Spark applications often underperform when executors are configured by default, leaving CPU cores and memory underutilized while introducing fault-tolerance risks and network overhead across worker nodes. Capital One Tech walks through executor sizing trade-offs — fat executors maximize data locality but concentrate failure risk, thin executors improve parallelism but flood the network — and codifies an optimal configuration recipe reserving cores and memory for OS overhead, capping executors at 3-5 cores, and accounting for the 384 MB or 10% memory overhead. The framework converts executor tuning from guesswork into a deterministic sizing exercise, anchored to balanced parallelism, fault tolerance, and resource utilization across distributed Spark clusters.

https://medium.com/capital-one-tech/spark-tuning-executor-optimization-for-performance-c757b39f0efe

Data Engineering Weekly #267

Ananth Packkildurai — Mon, 27 Apr 2026 00:52:38 GMT

Free Course: AI-Driven Data Engineering

Enroll today

Monzo: A “meshy” approach to Data: Enabling 100+ teams to build Data Models

Decentralized data ownership breaks down when cross-team dependencies remain implicit and upstream schema changes silently cascade through downstream models. Monzo introduces Interfaces—explicitly declared, tested dbt models that serve as governed data contracts—stabilizing cross-domain consumption across its 12,000-model warehouse. The migration has already reduced processing costs by 40% and accelerated data landing times by 25%, proving that formalized contracts scale distributed data modeling.

https://monzo.com/blog/a-meshy-approach-to-data

Aparna Dhinakaran: Context Management in Agent Harnesses

Long-running AI agents degrade as context windows fill with unbounded tool outputs, stale conversation history, and redundant file reads. The author analyzes five agent frameworks—Pi, OpenClaw, Claude Code, Letta, and Arize's Alyx—revealing convergence on hard file caps, token-triggered compaction, and isolated sub-agents. These patterns mirror the classical memory hierarchy—registers, cache, and swap—suggesting that context management is maturing into an invisible system-level discipline.

https://x.com/aparnadhinak/status/2048492731929149929

Spotify: Flow generation through natural language: An agentic modeling approach

LLMs struggle to reason over deeply nested domain-specific schemas that lack representation in pretraining data. Shopify builds a bidirectional transpiler that converts its Flow automation JSON into Python—improving syntactic correctness by 22% and semantic correctness by 13% for its fine-tuned Qwen3-32B model. The approach delivers a Sidekick assistant that runs 2.2x faster and 68% cheaper than the closed-source frontier model it replaces.

https://shopify.engineering/fine-tuning-agent-shopify-flow

Sponsored: The AI Modernization Guide

Download the free guide

Pratish Yadava: Data agents - When enterprise analytics learns to reason

Traditional dashboards answer predefined questions but struggle to diagnose root causes or recommend actions within live business workflows. The author outlines continuous data agents that interpret intent and make bounded decisions—anchored in governed semantic layers and modular, domain-specific orchestrators. This operating model moves analytics from passive reporting into decision-adjacent automation with explicit guardrails and escalation paths.

https://medium.com/data-science-at-microsoft/data-agents-when-enterprise-analytics-learns-to-reason-13345ec8998e

Pinterest: Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest

Content platforms waste significant compute re-fetching identical pages, disguised by URL variations introduced by tracking tags, session tokens, and click identifiers. Pinterest engineers MIQPS—a data-driven algorithm that renders pages with and without each query parameter to empirically classify content-changing signals from noise. The system strips redundant parameters at runtime via precomputed offline maps, reducing duplicate fetches and improving catalog deduplication at scale.

https://medium.com/pinterest-engineering/smarter-url-normalization-at-scale-how-miqps-powers-content-deduplication-at-pinterest-4aa42e807d7d

Meltwater: Doing More With Less: Rethinking Entity-Level Sentiment at Scale

Entity-level sentiment analysis scales linearly when systems re-encode the same document once per entity, multiplying inference costs without improving accuracy. Meltwater extracts per-entity embeddings from a single shared Transformer forward pass, proving that local mention context carries sufficient sentiment signal. The approach reduces inference costs by 45.5% and improves accuracy by 3.02%, converting linear per-entity scaling into near constant-time processing.

https://underthehood.meltwater.com/blog/2026/04/23/rethinking-entity-level-sentiment-at-scale/

Halodoc: Implementing Apache Yunikorn on EMR on EKS at Halodoc

Kubernetes-native Spark workloads trigger aggressive node scaling when the default scheduler evaluates pods independently—causing cost whiplash from rapid scale-outs followed by immediate underutilization. Halodoc adopts Apache YuniKorn's bin-packing strategy to fill existing nodes before provisioning new ones, paired with hierarchical queues that govern cross-team resource boundaries. Node utilization reaches 96%, with a 10% reduction in EC2 costs and increased Spot instance adoption due to improved scheduling predictability.

https://blogs.halodoc.io/implementing-apache-yunikorn-on-emr-on-eks/amp/

Netflix: Scaling Camera File Processing at Netflix

Media production pipelines struggle to manage massive daily camera footage when raw metadata remains unconformed and unsearchable across downstream workflows. Netflix integrates FilmLight's API into its Media Production Suite to parse and normalize camera metadata at ingest—conforming it to a standardized schema that enables automated retrieval and pipeline validation. The system deploys as stateless serverless functions on CPU-only instances, scaling elastically to handle spiky VFX plate generation without dedicated GPU infrastructure.

https://netflixtechblog.com/scaling-camera-file-processing-at-netflix-6dab2b1e80be

Z1: Airflow DAG Bundles: Managing DAGs Across Teams Without Helm Upgrades

Onboarding new data pipelines in Airflow typically requires Helm upgrades, pod restarts, and infrastructure tickets—turning every DAG addition into a deployment bottleneck. The author leverages Airflow 3.x DAG bundles with an S3-backed sidecar sync pattern that hot-reloads pipeline configurations without downtime or centralized repository dependencies. New DAGs appear in the Airflow UI within 30 seconds of a commit, decentralizing the entire pipeline lifecycle to self-service.

https://blog.platform.zerotoone.ai/blog/airflow-dag-bundles-without-helm-upgrades/

Data Engineering Weekly #266

Ananth Packkildurai — Mon, 20 Apr 2026 03:09:39 GMT

Free Course: AI-Driven Data Engineering

Enroll today

Animesh Kumar: AI-Ready Data vs. Analytics-Ready Data

Analytics and AI systems fail when teams treat data readiness as a single maturity path instead of two distinct axes optimized for different consumers. The author distinguishes analytics-ready data—designed for human interpretation through aggregation, stability, and explainability—from AI-ready data, which requires contextual completeness, timeliness, and semantic richness often lost in aggregation pipelines.

https://medium.com/@community_md101/ai-ready-data-vs-analytics-ready-data-f67ef0804341

Whatnot: The model is the easy part: Building the LLM Platform at Whatnot

Production LLM systems fail primarily in the surrounding infrastructure rather than the model layer, where non-deterministic outputs, unconstrained inputs, and missing feedback loops undermine reliability and trust. Whatnot writes about structuring its LLM platform around three pillars—velocity, reliability, and trust. It uses the post-exposure A/B logging to isolate divergent outputs, a reusable tool registry, and LLM-as-a-judge calibration workflows to detect production drift early.

https://medium.com/whatnot-engineering/the-model-is-the-easy-part-building-the-llm-platform-at-whatnot-ec8730fa9bdf

Slack: Managing context in long-run agentic applications

Long-running multi-agent systems lose investigative coherence as stateless LLM APIs force context accumulation, degrading reasoning quality and amplifying hallucinations across rounds. Slack writes about addressing this with three context channels—a Director’s Journal for working memory, a Critic’s Review that scores findings on a five-level credibility rubric, and a Critic’s Timeline that prunes incoherent findings by enforcing narrative consistency.

https://slack.engineering/managing-context-in-long-run-agentic-applications/

Sponsored: The AI Modernization Guide

Download the free guide

Atlassian: Engineering the Forge Billing Platform for Reliability and Scale

The billing pipeline, especially the usage-based billing pipeline, is often very complex, given the system's sensitivity to correctness. Atlassian describes Forge Billing as a deterministic pipeline—routing 300M daily usage events through StreamHub and UTS for deduplication and schema validation, then splitting them into cold-tier raw storage and StarRocks hot-tier aggregations for fast developer queries. The architecture handles counter and gauge metrics through idempotency keys and last-write-wins windowing, enabling full charge traceability from Developer Console back to raw events while scaling to 1B daily events within a year.

https://www.atlassian.com/blog/atlassian-engineering/engineering-the-forge-billing-platform-for-reliability-and-scale

Giannis Polyzos: From Events To Real-Time Profiles On Apache Fluss

Real-time entity decisions fail when profile state lives in a separate OLAP layer, introducing ingestion lag between event arrival and visibility. The approach builds risk profiles directly in Apache Fluss by mapping identifiers to integers, encoding group membership as Roaring Bitmaps, and merging updates at write time using the Aggregation Merge Engine without stateful Flink jobs. This design removes the need for a separate profile store, reduces state latency from hours to seconds, and ensures correct recovery through replay-safe inverse operations in the UndoRecoveryOperator.

https://ipolyzos.substack.com/p/from-events-to-real-time-profiles

Thiago Baldim: The journey to Agentic BI

Agentic BI tools amplify data quality issues—ambiguous schemas, undocumented columns, and fragmented sources of truth—rather than resolving them at query time. SafetyCulture writes about rebuilding its data platform on Kimball architecture with SCD Type 2 dimensions, over 90% dbt test and documentation coverage, and column-level ownership aligned to business stakeholders, reducing pipeline execution time from 14 hours to 1.5 hours.

https://medium.com/@thiagobaldim/the-journey-to-agentic-bi-617975c106b7

Pinterest: Scaling Recommendation Systems with Request-Level Deduplication

Scaling recommendation models creates infrastructure pressure because request-level data—especially long user action sequences—is redundantly stored and processed once per candidate item across storage, training, and serving. Pinterest writes about applying request-level deduplication across the ML lifecycle using request-sorted Iceberg datasets, SyncBatchNorm, and user-level masking to preserve training correctness, and a Deduplicated Cross-Attention Transformer that caches user context for reuse across ranked items.

https://medium.com/pinterest-engineering/scaling-recommendation-systems-with-request-level-deduplication-93bd514142d9

Just Eat: Daedalus and the Data Labyrinth

Data governance fails when organizations share data without sharing definitions, ownership, lineage, and reusable business logic across the systems humans and AI agents rely on. Just Eat frames modern governance as a layered navigation system—combining a business glossary, data catalog, metadata, data quality signals, lineage, and a semantic layer—to connect business language with trusted data assets and machine-usable definitions. These governance layers turn complex data platforms from opaque labyrinths into navigable systems, enabling consistent analytics, more reliable AI-generated queries, and a lower risk of conflicting metrics.

https://medium.com/justeattakeaway-tech/daedalus-and-the-data-labyrinth-2c166b1d9866

Teads: We Let AI Agents Orchestrate Our ML Experiments

Teads writes about extending its Datakinator platform with agentic orchestration by exposing APIs through MCP, enriching them with dataset probing and error-retrieval tools, and adding cost guardrails that estimate and gate expensive runs. This approach scales experiment throughput from hundreds to thousands, enables autonomous retry and correction of failed runs, and delivers 5–10% model improvements that translate to nearly $1M in margin gains despite increased cloud usage.

https://medium.com/teads-engineering/we-let-ai-agents-orchestrate-our-ml-experiments-fc8606816fde

A Response to Our Reader Survey

Ananth Packkildurai — Wed, 15 Apr 2026 16:26:18 GMT

Thank you to every reader who took a few minutes to fill out the survey. Your feedback is the clearest signal we have — and this round gave us a lot to act on.

One piece of feedback came through clearly enough that we’d be wrong to gloss over it: several of you feel we’ve leaned too heavily into AI coverage. That’s a fair critique. So we did what data engineers do: we audited ourselves.

We reviewed 233 articles published across 25 issues, from July 2025 through April 2026. The audit made one thing clear — while the majority of our content has remained relevant to data engineers, we have not been as precise as we should be. That precision gap showed up in 44 articles, or 18.9% of the total, which we now classify as outside the newsletter’s editorial scope: prompting tutorials, generic agent framework walkthroughs, model-release commentary, and LLM inference economics that follow the AI news cycle more than the data engineer’s day-to-day reality.

The survey also returned encouraging numbers. Data Engineering Weekly earned a Net Promoter Score of +17.3, and 79.6% of respondents rated their experience as Satisfied or Very Satisfied. We don’t take those numbers for granted. They reflect real signals from practitioners who are selective about where they invest their attention. But the critique is what drove this post, and the critique is what we are acting on.

What the Data Says

Across 233 articles published in 25 issues: Category Articles Share of the total 233 articles

Core DE 99 42.5%
Adjacent but Relevant 60 25.8%
Not DE 44 18.9%
Context Engineering (DE extension) 30 12.9%

The 44 out-of-scope articles are the problem we are fixing. The 189 in-scope articles are the standard we are holding ourselves to.

The survey revealed two truths at once: readers were right to call out drift, and we still believe that some of the work at the intersection of data infrastructure and AI belongs squarely within the future of data engineering. The rest of this post explains exactly where we draw that line — and what changes now.

You can read the complete analysis here
https://docs.google.com/spreadsheets/d/1kLq_touW4Z0SXDjCWEg4zt1DZzBoZFyTM2d2KJAR_tQ/edit?usp=sharing

Our Editorial Categories

Our test is straightforward: if a working data engineer cannot reasonably connect a piece to the systems they build, operate, govern, or evolve, it does not belong in the newsletter.

We use four categories to apply that test. They are not new instincts — they formalize the editorial judgment we have always tried to apply, and they give us, and you, a shared framework for accountability.

Core DE

Classical data engineering topics: ingestion, storage, orchestration, batch/streaming, table formats, query engines, data modeling, quality, governance, and platform reliability.

This is the foundation. Articles in Core DE cover the systems data engineers’ own every day — pipeline architecture, storage layer decisions, table format trade-offs, query engine internals, data quality frameworks, and the operational realities of running data infrastructure at scale.

Context Engineering (DE extension)

Context/semantic layers, ontologies, knowledge graphs, NL-to-SQL, data agents, and other systems that turn enterprise data into governed machine-usable context.

One category deserves a more explicit explanation, because it sits at the center of both the feedback we received and the direction this field is heading.

The reader survey did not change my view that Context Engineering belongs in Data Engineering Weekly. It forced a more disciplined articulation of that view — and a much stricter standard for what qualifies.

If you have read ETL is Dead or Data Engineering After AI, you know where I stand. AI coding agents are reshaping how we work with data, and that shift is not temporary. The data engineer’s value is migrating from pipeline reliability to semantic reliability — from “the job ran” to “the meaning is right.” ETL is dead, the way landlines are dead: the pipelines still run, but nobody builds their data strategy around them anymore. The new leverage lies in designing systems that make AI-generated code trustworthy and grounded in a governed, machine-usable context. Extract, Contextualize, Link rather than Extract, Transform, Load.

I would not be doing justice to our Data Engineering Weekly community if I excluded the engineering questions that come with that shift. The issue was never whether DEW should cover AI-adjacent work. The issue was whether each piece remained grounded in the systems, semantics, governance, and infrastructure that data engineers actually own.

That is the line. In scope: systems that make enterprise data structured, governed, semantic, and machine-readable. Out of scope: prompting tactics, agent framework tutorials, model release commentary, and generic application-layer AI workflows.

Adjacent but Relevant

AI/ML platform, feature store, search/retrieval infrastructure, eval/observability, or AI governance topics that materially touch DE skills or infrastructure without being pure DE.

Data engineers do not work in isolation. The systems they build feed ML platforms, power search infrastructure, and underpin AI governance frameworks. We include this category selectively — only when the infrastructure implications are concrete, the engineering depth is substantial, and the connection to a working data engineer’s responsibilities is immediate rather than abstract. An article on feature store architecture belongs here. A think-piece on the AI market does not.

Not DE

Prompting, generic agent frameworks, model-news / benchmark/market commentary, app-layer AI orchestration, pure ML modeling, or other pieces with no substantial data-infrastructure center of gravity.

This is the content we are eliminating. It may be technically credible and widely read, but it is not written for data engineers and does not serve your work.

What Changes Now

Here is what is different, starting with the next issue:

Zero Not DE articles per issue. Every article will be evaluated against the four categories above before publication. If it does not clear the bar, it does not go in.

A stricter standard for Adjacent but Relevant. The infrastructure angle must be concrete, the engineering substance must be real, and the relevance to a working data engineer must be direct — not abstract or aspirational.

Strict vendor neutrality. Data Engineering Weekly will remain strictly vendor-neutral. We generally exclude content published primarily to promote a product, platform, or company, with rare exceptions for work of exceptional technical depth and broad practitioner value. The content we select is written by practitioners for practitioners and published on personal blogs, company engineering blogs, or open platforms, where the goal is to share real experience, not to drive adoption.

Periodic transparency reports. We will share our category breakdown across recent issues so you can hold us to this standard.

Help Us Build the Signal

Data Engineering Weekly is a community effort. The best issues are shaped by readers who surface work worth sharing.

If you have read something — a post, an engineering deep-dive, a company blog — that fits clearly within Core DE, Context Engineering, or Adjacent but Relevant, submit it. Contributions are reviewed weekly, and we read everything that comes in. The process is lightweight: open a pull request with the article title, link, author details, and topic tags using our Contributing Guide. The same editorial categories and vendor-neutrality standard described here apply to all submissions.

As we advance, each issue should feel sharper, more grounded, and more consistently useful to working data engineers. That is the standard we are setting for ourselves, and the standard we invite you to hold us to.

If you have feedback on this policy or on individual articles, reply directly to any issue. We read every response.

Data Engineering Weekly #265

Ananth Packkildurai — Mon, 13 Apr 2026 03:24:30 GMT

This week: Multi-Tenancy for Modern Data Platforms

Join Brooklyn Data Co. and Dagster Labs for a live deep dive on multi-tenancy for modern data platforms. We’ll cover:

- Code location isolation and project structure patterns
- Managing dependencies across tenants (including AI models)
- Operational strategies that scale with your organization
- Lessons learned from real production implementations

Save your spot for practical guidance that you can apply immediately.

Reserve your spot now

dbt: Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

AI will write all the code generated in data engineering. It is the fundamental shift we all have to prepare for. dbt published its benchmark update, claiming GPT-5.3-Codex with Semantic Layer acheives 100.0% accuracy.

https://docs.getdbt.com/blog/semantic-layer-vs-text-to-sql-2026?version=1.10

Rill: Introducing Metrics SQL: A SQL-based semantic layer for humans and agents

Staying on the metrics and the semantic layer, Rill introduces Metrics SQL to define logic once in YAML and expose it through standard SQL, automating aggregations, enforcing row-level security, and serving governed definitions to AI agents via an MCP server without exposing raw schemas. Deterministic metric resolution eliminates inconsistencies across consumers, while a semantic pushdown roadmap targets native MEASURE support in OLAP engines like ClickHouse and Snowflake.

https://www.rilldata.com/blog/introducing-metrics-sql-a-sql-based-semantic-layer-for-humans-and-agents

Meta: How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines

AI coding assistants fail in proprietary codebases because tribal knowledge—implicit design decisions and legacy constraints—remains absent from training data and documentation. Meta Platforms deploys a swarm of 50 specialized agents to map a 4,100-file pipeline into concise context artifacts, using tiered explorer, analyst, critic, and fixer roles with automated decay detection. This system increases context coverage from 5% to 100%, captures over 50 non-obvious patterns, reduces tool-call volume by 40%, and cuts codebase research time from two days to 30 minutes.

https://engineering.fb.com/2026/04/06/developer-tools/how-meta-used-ai-to-map-tribal-knowledge-in-large-scale-data-pipelines/

Sponsored: The AI Modernization Guide

Download the free guide

Netflix: Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Real-time dashboards with rolling windows invalidate traditional caches because shifting intervals cause repeated misses on otherwise unchanged historical data. Netflix builds an interval-aware caching proxy that decomposes Druid queries into one-minute buckets, serves historical segments from Cassandra, and fetches only the uncached tail from Druid using exponential TTLs ranging from 5 seconds to 1 hour. The system achieves 82% partial cache hit rates, reduces Druid query volume by 33%, improves P90 latency by 66%, and shifts the bottleneck from compute-heavy Druid to low-cost Cassandra storage.

https://netflixtechblog.com/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-22fadc9b840e

Booking.com: Scaling Experimentation Quality at Booking.com

Underpowered experiments, premature peeking, and inconsistent reporting degrade decision quality as experimentation scales without statistical rigor. Booking.com embeds experimental quality across design, execution, and decision-making through data science ambassadors, peer-review practices, and tooling such as a Quality Tab that enforces power calculations and pre-registered hypotheses in real time. These changes increase the share of high-quality experiments, with the largest gains in design, where proper power improves the reliability of results and decision confidence.

https://booking.ai/scaling-experimentation-quality-at-booking-com-726152ee4ef0

Andros Fenollosa: From zero to a RAG system: successes and failures

Legacy engineering knowledge locked in unstructured simulation files and technical documents remains inaccessible when confidentiality requirements prohibit sending proprietary data to external LLM APIs. The author writes about building a local RAG system using Ollama, LlamaIndex, and ChromaDB—filtering out non-text files to reduce indexable load by 54% and serving source documents from Azure Blob Storage while staying within a 100GB disk constraint. The architecture delivers confidential retrieval over 1TB of legacy engineering data while establishing batch checkpointing and error-tolerant ingestion as the critical patterns for production RAG deployments at scale.

https://en.andros.dev/blog/aa31d744/from-zero-to-a-rag-system-successes-and-failures/

All Things Distributed: S3 Files and the changing face of S3

The introduction of S3 Files certainly generated a lot of interest on my reading list last week. I’m still studying the impact of S3 files on data pipeline engineering. One thing to note, S3 Files indeed breaks the read-on-write (Write in S3 Files, but read in S3) consistency model. I wonder if the data infrastructure really wants to go back to that world; nonetheless, this is an exciting blog to read to understand the thought process behind S3 Files.

https://www.allthingsdistributed.com/2026/04/s3-files-and-the-changing-face-of-s3.html

Apache Kafka: KIP-848: The Next Generation of the Consumer Rebalance Protocol

Whether you like or dislike Apache Kafka, its KIPs are among the best learning materials for distributed systems. Consumer rebalancing is one of the hottest debated topics in the Kafka world. KIP-848 moves rebalance logic from consumer clients to the Group Coordinator—introducing a ConsumerGroupHeartbeat API, three-layered epochs for group, assignment, and member state, and server-side Range and Uniform assignors that drive incremental partition reconciliation without global synchronization barriers.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol

Data Engineering Weekly #264

Ananth Packkildurai — Mon, 06 Apr 2026 01:55:25 GMT

How data teams are solving multi-tenancy

As data teams grow and serve multiple teams, clients, or business units from a shared platform, maintaining isolation and velocity without sacrificing either becomes a defining architectural challenge.

In this Deep Dive, Dagster Labs and Brooklyn Data Co. will cover the patterns, trade-offs, and real-world implementations behind multi-tenant data platforms built on Dagster. Attendees will leave this session with practical guidance they can take back to their own teams.

Reserve your spot now

Editorial Note: Help Us Make Data Engineering Weekly Better

We’re working to make Data Engineering Weekly more useful, more relevant, and more worth your time every Sunday. If you have 2 minutes, please share your feedback through this short survey. Your input will directly shape what we cover, how we write, and where we improve next.

https://forms.gle/cgeww7czFAVBiVmV7

Meta: Inside Meta’s Home-Grown AI Analytics Agent

Routine analytical queries dominate enterprise data science workloads, yet agents fail as warehouse scale grows without a bounded, structured context. Meta Platforms seeds per-user memory from historical query logs and organizes domain knowledge into cookbooks, recipes, and ingredients that encode validated analyst logic. This approach drives 77% weekly adoption within six months as community-authored recipes expand coverage across domains

https://medium.com/@AnalyticsAtMeta/inside-metas-home-grown-ai-analytics-agent-4ea6779acfb3

Michel Tricot: Beyond ETL - The Case for Context

Agentic data infrastructure exposes a meaning gap that traditional ETL never addressed, as autonomous agents propagate poor context across queries at scale. The author validates the ECL framework through real-world failures and reframes the Context Store as a materialized view that pre-replicates SaaS data into versioned semantic structures for agent consumption. Existing data engineering primitives—incremental replication, schema normalization, and tenant isolation—support this model, shifting the data engineer’s role from data movement to context architecture.

https://agentblueprint.substack.com/p/beyond-etl-the-case-for-context

Chris Gambill: Medallion Architecture Isn’t As New As You Think

AI tools amplify data-quality failures at scale when pipelines lack clear boundaries among the raw capture, transformation, and business-consumption layers. The author reframes Medallion Architecture as a disciplined evolution of staging and reporting models: Bronze preserves raw audit trails, Silver enforces schema contracts, and Gold delivers business-ready KPIs. This separation provides a reliable context for AI systems and reduces the downstream cost of bad data beyond incremental storage overhead.

https://gambilldataengineering.substack.com/p/medallion-architecture-isnt-as-new

Zapier: Lessons from using the outbox pattern at scale

High-throughput event pipelines require durable buffering between producers and brokers to prevent data loss during failures or maintenance windows. Zapier implements a transactional outbox in its Go-based Events API using sharded SQLite on EBS-backed Kubernetes StatefulSets, with WAL mode, 50 shards per pod, and per-shard mutexes sustaining 15,000 events per second during Kafka outages. Operational limits from static sharding and StatefulSet constraints push a shift toward a sidecar pattern that writes to S3 on failure and replays via SQS.

https://zapier.com/blog/lessons-from-using-outbox-pattern-at-scale/

Lyft: Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees

Sparse contextual data causes standard ML models to overfit and generate unstable predictions across long-tail combinations of location, time, and demand. Lyft models rider conversion using a Bayesian Tree that organizes context hierarchically and applies Gaussian priors with L2 regularization to balance sparse leaf signals against stable parent trends. This approach delivers localized accuracy in dense data and degrades to broader signals in sparse regions, while enforcing monotonicity constraints to ensure consistent, interpretable predictions.

https://eng.lyft.com/predicting-rider-conversion-in-sparse-data-environments-with-bayesian-trees-07227ff92789

LinkedIn: Building LinkedIn’s CTV Ads: Scaling professional reach to the big screen

B2B advertisers struggle to reach professional audiences on connected TV while maintaining the targeting precision and measurement fidelity of digital environments. LinkedIn extends its identity graph to CTV through private marketplace supply, cross-device household mapping, and transcoding pipelines that meet CBR encoding and native frame rate standards. The platform delivers 99% brand-safe inventory, achieves 11x cost efficiency over linear TV, and scales from manual deals to self-serve inventory via Campaign Manager.

https://www.linkedin.com/blog/engineering/marketing/building-linkedins-ctv-ads

Netflix: Synchronizing the Senses: Powering Multimodal Intelligence for Video Search

Video search across large productions requires unifying outputs from multiple ML models into a low-latency retrieval system that editors can query in real time. Netflix pipelines multimodal annotations through Cassandra for high-throughput ingestion, Kafka for temporal bucketing into one-second intervals, and Elasticsearch for hierarchical indexing that combines character, scene, and dialogue signals. The system enables semantic vector search via HNSW, supports match-phrase dialogue queries, and applies union–intersection logic to reconstruct scene boundaries across billions of data points.

https://netflixtechblog.com/powering-multimodal-intelligence-for-video-search-3e0020cf1202

Salesforce: Inside Informatica’s Spark-Based Data Integration Platform: Running 250K Enterprise Pipelines Daily

Enterprise data integration platforms struggle at the petabyte scale when single-node execution engines lack distributed compute and automated resource optimization. Informatica migrates CDI to Spark++ on Kubernetes, preserving graphical mappings while introducing row-level fault isolation, ephemeral, VPC-bound clusters, and automated FinOps tuners that optimize infrastructure and Spark parameters based on historical workloads. The distributed system supports 5,500 enterprise clients across 250,000 daily tasks, reduces infrastructure costs by 1.65x, and maintains 99.9% control plane availability.

https://engineering.salesforce.com/inside-informaticas-spark-based-data-integration-platform-running-250k-enterprise-pipelines-daily/

ZeroToOne: Taming S3 Shuffle at Scale

S3-based Spark shuffle suffers from quadratic scaling of GET requests, prefix throttling, and executor hangs, which drive high API costs and instability at production scale. ZeroToOne reduces shuffle costs by 95% by coalescing map tasks and expanding S3 prefixes from 10 to 500, then hardens the shuffle plugin with ConcurrentHashMap-based atomic locking and prefetch iterator timeouts to eliminate race conditions and deadlocks. These changes stabilize spot instance execution and reduce per-stage API costs from $72 to near zero in large backfill workloads.

https://blog.platform.zerotoone.ai/blog/taming-s3-shuffle-at-scale/

Radim Marek: Production query plans without production data

Just the other day, I was in a design discussion about building a routing engine for SQL query execution based on the query plan, and how to back this up in the CI pipeline to catch expensive queries earlier. It is one of the critical problems that I wish all data warehouses and Lakehouses would provide out of the box.

https://boringsql.com/posts/portable-stats/

The Missing Interface in Data Platform Engineering

Ananth Packkildurai — Thu, 02 Apr 2026 03:36:07 GMT

A familiar pattern plays out inside many platform organizations. A data platform team ships what it sees as a milestone: a self-service stack with governed datasets, reusable pipelines, access automation, lineage, templates, and documentation. Leadership sees leverage. The platform team sees scale.

Then requests start arriving.

Can someone help model the first few datasets? Can someone validate the ownership setup? Can someone walk us through the abstractions? Can the platform team handle the initial rollout for this one use case?

Platform teams often call that resistance to self-service. The real problem is simpler: the interface is incomplete.

The tooling exists. The technical path exists. But the consumer team still cannot tell where responsibility begins, where support ends, what failure looks like, or what the team must operate independently.

The platform team sees a reusable capability. The consumer team sees a system that still depends on human interpretation.

Both teams are acting rationally. The platform team has built a technical interface. The consumer team is still looking for an operating interface.

That gap accounts for more platform friction than most platform strategies acknowledge.

Data platform engineering often gets framed as a systems problem: storage layers, compute engines, orchestration, metadata, governance, access control, and developer tooling. Those components matter. Once a platform becomes shared infrastructure, however, the harder problem shifts. The platform becomes a dependency surface across teams, applications, workflows, and operational responsibilities.

At that point, the key question shifts from “What did we build?” to “How should other teams depend on it?”

That question defines the real interface in data platform engineering.

Data Platform Engineering is Coordination Engineering

A mature data platform is not just a collection of capabilities. A mature data platform creates a shared system that other teams must trust, integrate with, and operate against. Every dependency on that platform carries assumptions: what stays stable, what can change, who responds when something breaks, how fast a team can expect support, what a consumer must understand, and what remains the platform team’s responsibility.

Teams carry those assumptions whether they write them down or not. When teams leave them implicit, engineers reconstruct them through tickets, Slack threads, tribal knowledge, escalations, and repeated misunderstandings. When teams make them explicit, the assumptions become part of the platform’s operating interface.

The operating interface has two parts.

One part defines the explicit rules that govern the relationship: schemas, APIs, freshness guarantees, ownership boundaries, compatibility expectations, escalation paths, and adoption responsibilities.

The other part defines the communication pattern through which teams use those rules: reactive ticketing, temporary embedding, joint execution, self-service federation, or community contribution.

Most platform failures begin when teams underdesign one or both parts.

The platform team thinks it has published a reusable capability. The consumer team experiences an ambiguous boundary. We could document the schema, but not the operational expectations. The self-service path may exist, but the adoption model does not. The API may be stable, but teams still negotiate failure semantics socially every time they matter.

Platform maturity, then, depends on more than better tooling. Platform maturity depends on how well teams design the dependency boundary between the platform and the groups that rely on it.

A contract is only one layer of the operating interface

Platform discussions often stall because engineers use the word contract too narrowly. In data work, many engineers hear “contract” and immediately think of a data contract: schema shape, field semantics, compatibility rules between producer and consumer, and perhaps a validation mechanism.

That category matters. That category does not cover the problem.

A stack of interface layers governs a data platform dependency, and each layer answers a different question.

1. Technical interface

The technical interface is the layer most teams already know how to discuss. It includes APIs, schemas, tables, events, payloads, SDKs, versioning rules, authentication mechanisms, and compatibility expectations. The technical interface defines the shape of interaction.

When people say a platform has a clear interface, they often mean only that layer.

Teams can still fail operationally even when the technical interface is clear.

2. Operational contract

The operational contract defines runtime expectations. How fresh should the data be? What latency matters for a given workflow? How should retries behave? What happens when a dependency degrades? Which failures does the platform absorb, and which failures propagate to consumers? Which SLOs, error budgets, or maintenance windows apply?

The operational contract separates descriptive interoperability from dependable interoperability.

Two teams may agree on a schema and still disagree completely on whether a six-hour delay is acceptable, whether we tolerate the stale reads, or whether a breaking change in behavior requires a coordinated rollout.

3. Ownership model

The ownership model defines authority and accountability. Who approves interface changes? Who owns backward compatibility? Who responds during incidents? Who decides when a consumer must migrate? Who can reject a new use case because it violates platform constraints?

Many recurring platform conflicts are ownership failures disguised as technical disputes.

A consumer team says, “The platform changed under us.” The platform team says, “You were never supposed to rely on that behavior.” In most cases, unclear ownership boundaries create the conflict long before the disagreement surfaces.

4. Adoption model

The adoption model defines what a consuming team must do to use the platform successfully. Is the platform truly self-service? Does first adoption require embedding? Must the consumer own pipeline logic, operational monitoring, data quality checks, and incident response? How much platform literacy must a team build before independence becomes realistic?

Most platform design documents ignore that layer even though it often determines whether adoption succeeds.

A workflow is not self-service because a platform engineer no longer types the commands. Self-service begins when a consumer team can understand, operate, and recover within the platform’s boundaries independently.

5. Communication pattern

Every platform also has a practical communication mode. Teams may collaborate through tickets, pairing, embedded work, shared planning, interfaces, or contribution models. Those patterns are not secondary to the platform. Those patterns determine how the platform behaves in practice.

When teams do not consciously design that layer, habits and local workarounds define it by default.

Together, those layers form the platform’s operating interface: the real boundary through which teams depend on one another.

Every Platform Already has a Communication Model

Platform teams often speak as though better tooling will eventually eliminate communication. Tooling never eliminates communication. Tooling only changes its shape.

A ticket queue is a communication system. An embedding is a communication system. An API with onboarding guides and escalation rules is a communication system. An internal RFC process also serves as a communication system.

For that reason, a platform maturity model is also a communication maturity model. The model describes not only what the platform team has built, but also how dependency information moves between the platform and its consumers.

Communication may remain human-mediated, partially codified, or increasingly interface-led. No single mode is always superior. The right mode depends on the capability’s maturity, the consumer’s readiness, and the complexity of the work.

Five ways platform teams and dependent teams actually work

No single collaboration pattern fits every internal platform. The right mode depends on the capability in question, the consuming team, and the type of dependency with them. Strong platform organizations usually operate across several modes at once.

Teams rarely fail because they sit at the “wrong” level. Teams fail because they assume every consumer can interact with the platform through the same interface, when reality clearly shows otherwise.

Level 1: Reactive — The service desk

At Level 1, the platform team operates primarily through request fulfillment. Teams file tickets. Platform engineers provision resources, troubleshoot access, define ingestion patterns, or implement parts of the first workflow manually—knowledge about how the platform works lives mostly in people’s heads.

Many people dismiss that mode as immaturity. That judgment misses the point. Level 1 is where many new platform capabilities should begin.

When a capability is still emerging, the platform team does not yet know what the stable interface ought to be. Manual repetition helps the team discover the pattern worth codifying. The first few onboarding efforts reveal which inputs remain stable, which edge cases arise frequently, which assumptions break under real-world workloads, and which abstractions are premature.

The real danger is not Level 1 itself. The real danger is staying there after repetition becomes obvious.

Level 1 fails when demand scales linearly. Every new consumer increases direct demand on the platform team. The team becomes a fulfillment bottleneck. Consumers experience the platform as a queue instead of a leverage point. Platform engineers spend more time context-switching than building reusable capabilities.

Teams should move beyond Level 1 when repetition becomes predictable. Once the same task repeats enough times to reveal a stable pattern, some part of the operating interface should move out of people’s heads and into a reusable form.

Level 2: Coordinated — The embedding

At Level 2, the platform team transfers capability through direct collaboration. A platform engineer temporarily works with a consumer team to bootstrap adoption, interpret abstractions, and help the team operate inside the intended boundary.

Level 2 is not just support. Level 2 is a deliberate adoption model.

Embedding works when the platform capability is ready enough to be reused but still requires high-context interpretation. Embedding also works when the platform team needs to learn from consumers before it can fully stabilize the interface. The interaction runs in both directions: the platform teaches the intended path, and the consumer exposes where the path is incomplete.

Level 2 fails when dependency persists. The embedded engineer becomes the permanent translator. The team learns to route every ambiguity to a familiar person rather than building independent platform fluency. Once the embedding ends, the team slides back into ticketing.

Teams should move beyond Level 2 when understanding becomes repeatable. Once the same questions keep recurring, lack of exposure is no longer the main problem. Interface clarity is the problem. The platform then needs better runbooks, better failure handling, clearer ownership boundaries, or a more legible self-service path.

Level 3: Partnership — The joint mission

At Level 3, platform and consumer teams align around a shared objective for a bounded period. Level 3 is not request fulfillment, nor is it simple enablement. Level 3 is a temporary joint execution model.

Level 3 works when the dependency boundary itself is part of the problem. Teams often need Level 3 when they launch a new real-time product feature that requires changes across ingestion, serving, governance, and application behavior; when they stand up an experimentation platform that affects both platform architecture and domain logic; or when they build a new cross-cutting data product whose responsibilities cannot yet be separated cleanly.

Level 3 creates speed under complexity. Instead of negotiating everything through a queue, the teams create a shared execution context.

Level 3 fails when the temporary mission becomes a permanent entanglement. What was supposed to be a time-boxed collaboration becomes a staffing model. The platform roadmap drifts toward one team’s local priorities. The consumer team stops building independent ownership.

Teams should move beyond Level 3 when reusable patterns emerge. Once joint work starts producing structures that other teams will need, the organization should ask which parts belong in a generalized operating interface rather than in a persistent bespoke relationship.

Level 4: Federation — The self-service operating interface

At Level 4, teams collaborate primarily through explicit interfaces rather than constant human mediation. The platform publishes technical interfaces, operational expectations, ownership rules, onboarding guidance, and support boundaries clearly enough that consuming teams can adopt capabilities independently.

Level 4 is where platform economics start to work.

The marginal cost of onboarding new teams drops because the interface does more of the teaching. The platform team shifts away from request fulfillment and toward maintaining compatibility, reliability, documentation, tooling, and interface evolution.

Many organizations fool themselves at Level 4.

A team can publish an API, a portal, or a template and claim self-service while leaving the actual operating interface incomplete. The consumer can create the resource, but does not know how to handle failure. The documentation describes the happy path, but not the migration path. The schema is versioned, but the escalation model is still social. The ownership boundary exists in theory, but not in behavior.

That condition is not federation. That condition is a ticket queue with better branding.

Level 4 fails when self-service arrives too early. The platform exposes an interface before it has done enough repeated work to understand which parts are stable and which parts still require human judgment. Consumers adopt the easy 60% and escalate the hard 40%, forcing the platform team to run Level 1 and Level 4 simultaneously.

A healthy Level 4 shows more than usage. A healthy Level 4 shows independent operation. A consuming team should be able to adopt the capability, reason about normal failure, understand the support model, and make routine changes without renegotiating the relationship each time.

Level 5: Ecosystem — The internal commons

At Level 5, teams not only consume the platform. Teams extend it. The platform becomes a stewarded commons with contribution pathways, governance, standards, RFCs, and shared maintenance expectations.

That operating model looks attractive in strategy decks because it suggests scale through internal open-source behavior. In practice, organizations struggle to sustain it.

Contribution requires more than technical maturity. Contribution requires governance maturity. Teams need clarity on how to adopt the contribution, how to review its quality, who maintains the artifact over time, how support obligations are assigned, and how the platform distinguishes production-grade extensions from abandoned experiments.

Level 5 fails when the commons turns into unmanaged sprawl. Shared repositories fill with unevenly maintained components. The boundary between the core platform and the contributed surface becomes blurry. Consumers cannot distinguish between governed and incidental capabilities.

For many organizations, Level 4 is the durable steady state. Level 5 becomes valuable only when culture, incentives, and governance can support shared stewardship.

The missing variable is contract literacy

Most discussions of platform maturity focus on the platform side. That view is incomplete.

A platform can be highly mature in its own design and still fail in practice because the consuming team is not ready to operate against that interface. A Level 4 platform paired with a Level 1 consumer often behaves like a Level 1 system.

Consumer readiness matters, but teams should define the term more precisely than “platform familiarity” alone.

Consumer readiness is really a form of contract literacy. Consumer readiness measures a team’s ability to understand the operating interface, interpret the support boundaries, reason about failure modes, absorb ownership, and use the self-service path without relying on informal rescue.

A mature platform with a new team often needs Level 2 interaction first. The capability may be stable, but the team lacks the context to operate within it.

An early platform with a strong consumer base may benefit from a Level 3 partnership. The capability is yet to enter production, but the team is strong enough to co-develop the future interface.

A mature platform with a mature consumer can operate effectively through Level 4 and, in some cases, Level 5.

An early platform with a new consumer should not pretend to be anything other than Level 1 for a while.

The diagnostic question is not “How mature is the platform?” The better question is, “How mature is the dependency relationship, given both sides of the interface?”

Why platforms fail even when the interface exists

Many platform incidents seem surprising only when the platform team mistakes the technical interface for the whole interface.

A schema can remain stable even as the operational contract breaks down. A consumer receives the expected fields but cannot tolerate the freshness lag introduced by the new implementation.

An API can be correct while the ownership model remains unclear. Both teams assume the other team is responsible for migration sequencing, and the rollout fails in the gap.

A portal can be self-service while the adoption model remains incomplete. The consumer can provision the resource, but does not know which observability, alerting, backfill policy, or quality checks now belong to the team.

Documentation can be extensive while the communication pattern remains reactive. The written material explains the happy path, but the only reliable way to get edge-case answers is still to message a platform engineer directly.

Each example points to the same problem. The platform appears mature on paper and unstable in practice because one layer of the operating interface is missing.

That pattern also explains why many arguments about “data contracts” feel unsatisfying. A schema contract may remove one class of ambiguity while leaving the dependency relationship fundamentally underdesigned. Platforms do not scale on descriptive clarity alone. Platforms scale when the operating interface becomes explicit enough for teams to coordinate predictably.

Why does this matter more in an agentic enterprise

As organizations move toward AI-mediated operations, autonomous workflows, and increasingly automated decision loops, the cost of implicit interfaces rises sharply.

Human teams can absorb ambiguity through judgment, relationships, escalation habits, and informal context. Human teams can often compensate for missing rules because they know whom to ask, which unwritten assumptions are in place, and when a local exception is acceptable.

Automated systems do not compensate in the same way. Automated systems require explicit state, explicit boundaries, and explicit expectations. A platform boundary that depends on tribal knowledge, undocumented ownership, or socially negotiated failure handling already strains human teams. That same boundary becomes structurally limiting when an organization wants automation to operate consistently across it.

No company needs a grand operating model of the enterprise to benefit from that insight. Organizations do need a simpler discipline. Teams need to make the dependency interface between systems and teams legible enough to operate without constant interpretation.

Maturity is reversible

We view the maturity models as a ladder. Real organizations behave more like shifting systems.

A platform that reached Level 4 can slide back toward Level 1 when documentation rots, examples stop working, and teams stop trusting the interface. An embedding that once worked can decay when the engineers who learned the system leave, and the knowledge never becomes fully externalized. A clear ownership model can collapse after a reorganization. A rearchitecture can reset operational assumptions so thoroughly that teams must return to manual coordination until the new boundary stabilizes.

Those events are not unusual. Those events are the normal dynamics of organizational life.

A maturity model is useful not because it promises to eliminate regression. A maturity model is useful because it helps teams name regression quickly and respond deliberately.

If a self-service platform has quietly reverted to a ticket queue, the problem is not just support volume. Some part of the operating interface has decayed: the technical surface, the operational contract, the ownership model, the adoption model, or the communication pattern.

Once teams name that decay, they can turn frustration back into design work.

The real platform interface is organizational.

The hardest part of a data platform is rarely the infrastructure itself. The hardest part is designing the boundary between the platform and the teams that depend on it.

That boundary cannot be reduced to schemas or APIs alone. The boundary includes operational expectations, support models, ownership rules, adoption assumptions, and the communication pattern through which those expectations become real.

When teams leave those layers implicit, they do not avoid design work. They push the work into tickets, escalations, workarounds, and repeated misunderstandings. When teams make those layers explicit, the platform becomes easier to trust, adopt, and scale.

Data platform engineering is not just infrastructure engineering. Data platform engineering is interface design at the level of teams, systems, and responsibilities.

A maturity model should not rank organizations morally or insist that every platform must reach some final stage. A maturity model should give teams a vocabulary for the dependency relationships they actually have, the interfaces they are really exposing, and the ones they need to design next.

The missing interface in data platform engineering is not another layer of tooling. The missing interface is the operating interface that defines how dependent teams rely on one another.

Until teams make that interface explicit, most platform scale remains performative.

Once teams make that interface explicit, platform scale becomes real.

Data Engineering Weekly #263

Ananth Packkildurai — Mon, 30 Mar 2026 02:18:13 GMT

How data teams are solving multi-tenancy

Reserve your spot now.

Aurimas Griciūnas: State of Context Engineering in 2026

LLM reasoning degrades with oversized context, forcing developers to manage attention through structured context engineering rather than scaling model size. The author outlines five patterns—progressive disclosure, compression, routing, agentic RAG, and tool management—that control how context is selected and applied. Layered orchestration across discovery, activation, and execution enables complex agent behavior within fixed context limits while preserving reasoning quality.

https://www.newsletter.swirlai.com/p/state-of-context-engineering-in-2026

Joe Reis: AI Is Here, But The Hard Parts Haven’t Changed

AI is accelerating coding velocity, but it’s also exposing structural weaknesses that data teams have ignored for years—legacy systems, misaligned leadership, and poor business context modeling. Data from Joe Reis’s March 2026 survey reinforces the gap: teams are shipping code faster, yet many still lack clarity on production value, while data modeling and semantic layers are emerging as the next critical frontier. Data engineering now faces a reset moment—improving end-to-end delivery efficiency matters more than optimizing isolated pipelines, a direction I’ve been exploring in “Data Engineering After AI” and “ETL is Dead.”

https://joereis.substack.com/p/ai-is-here-but-the-hard-parts-havent

Hamel Husain: The Revenge of the Data Scientist

LLM API accessibility enables rapid AI feature development but obscures reliability requirements grounded in evaluation and experimental design. The author argues modern AI development reuses core data science practices: analyzing production traces, validating LLM-as-judge with precision and recall, grounding test sets in real data, and using domain experts to define criteria. Teams that avoid synthetic benchmarks and over-automation focus on inspecting data to identify failure modes, reinforcing the role of data scientists as reliability gatekeepers.

https://hamel.dev/blog/posts/revenge/

Figma: Redefining impact as a data scientist

Data science impact in mission-critical systems like billing depends on domain expertise and observability rather than experimentation, shifting focus from models to correctness and clarity. The author describes Figma’s full-stack approach, where data scientists build consistency checks, create applications that explain system behavior, and define correctness criteria. Embedding these practices into operational systems scales their impact through tools rather than reports.

https://www.figma.com/blog/redefining-impact-as-a-data-scientist/

BlaBlaCar: Beyond the dashboard: how BlaBlaCar PMs use AI to self-serve data

Data analyst bottlenecks in fast-moving organizations require enabling non-technical users to self-serve without compromising data integrity or introducing hallucinations. BlaBlaCar evolves its approach from generic LLM usage to structured JSON schema documentation and few-shot learning on expert query histories, teaching the system to map natural language to business rules. A three-zone autonomy framework—safe, risky, and dead zones—combined with SQL literacy training for PMs reduces error rates from 32% to 15% and shifts analysts from reactive ticket handling to strategic work.

https://medium.com/blablacar/beyond-the-dashboard-how-blablacar-pms-use-ai-to-self-serve-data-95ccd33ab1f9

Expedia: Operating Trino at Scale With Trino Gateway

Managing Trino at scale requires isolating workloads to prevent resource contention across analytical, ETL, and BI queries. Expedia writes about operating Trino Gateway—a fork of Lyft's Presto Gateway—as a single-endpoint proxy that routes queries to dedicated clusters using configurable rules. This design eliminates noisy-neighbor failures, supports zero-downtime deployments, and provides real-time visibility into cluster health.

https://medium.com/expedia-group-tech/operating-trino-at-scale-with-trino-gateway-41824af788de

LangChain: How we build evals for Deep Agents

Building reliable AI agents requires evals that target specific production behaviors rather than optimizing for aggregate benchmark scores. LangChain's Deep Agents harness defines behavior-first evals sourced from production errors, BFCL, and hand-written unit tests — then scores agents on correctness and Ideal Trajectory ratios for step and tool-call efficiency. Teams run tagged eval subsets via pytest in GitHub Actions and trace every run in LangSmith to isolate failure modes and control evaluation cost.

https://blog.langchain.com/how-we-build-evals-for-deep-agents/

LinkedIn: The LinkedIn Generative AI Application Tech Stack: Personalization with Cognitive Memory Agent

AI agents lose personalization across sessions because they lack structured memory that separates conversational, episodic, semantic, and procedural signals. LinkedIn’s Cognitive Memory Agent ingests activity traces through streaming and batch pipelines, then uses an LLM-based orchestrator to retrieve and reason across all four memory layers. This architecture enables the Hiring Assistant to auto-populate role requirements and generate recruiter-specific insights from historical hiring activity.

https://www.linkedin.com/blog/engineering/ai/the-linkedin-generative-ai-application-tech-stack-personalization-with-cognitive-memory-agent

Ilia Gusev: Change Data Capture: Stop Copying 50M Rows to Move 5K Changes

Nightly full-table copies introduce fragility and place increasing load on source databases as data volumes scale. The article contrasts timestamp, trigger, and log-based CDC approaches, recommending Debezium with Postgres WAL or MySQL binlog as the production standard for near-real-time replication without impacting OLTP performance. Log-based CDC captures hard deletes, handles DDL changes, and decouples replication throughput from transactional write load.

https://podostack.com/p/change-data-capture-cdc-intro

Micheal Lanham: The Markdown File That Beat a $50M Vector Database

Agentic workflows expose the cost and operational overhead of managed vector databases when used for single-threaded memory and state management. The author shows how Manus, OpenClaw, and Claude Code converge on Markdown files as the primary memory layer, leveraging KV-cache efficiency, filesystem hierarchy for scoped retrieval, and sqlite-vec for lightweight semantic search. This file-first architecture reduces token costs by nearly 10x and defers vector database adoption to scenarios that require multi-user concurrency.

https://medium.com/@Micheal-Lanham/the-markdown-file-that-beat-a-50m-vector-database-38e1f5113cbe

Data Engineering Weekly #262

Ananth Packkildurai — Mon, 23 Mar 2026 01:31:45 GMT

This week: Orchestrating Databricks across multiple workspaces

In this hands-on deep dive, you'll learn how to build a cross-workspace control plane for Databricks using Dagster — connecting multiple workspaces, dbt, and Fivetran into a single observable asset graph with zero code changes to get started.

Register now

Pinterest: Building an MCP Ecosystem at Pinterest

Agent tooling at scale requires decentralizing context across domain-specific servers while maintaining security, discoverability, and governance across production systems. Pinterest’s MCP ecosystem deploys specialized servers (Presto, Spark, Airflow) behind a central registry, routes requests via JWT end-user tokens and SPIFFE mesh identities, and enforces human approval for sensitive actions such as data overwrites. The system handles 66,000+ monthly invocations while saving engineers 7,000 hours monthly, validating decentralized tooling as the production pattern for agentic workflows.

https://medium.com/pinterest-engineering/building-an-mcp-ecosystem-at-pinterest-d881eb4c16f1

Julien Simon: Still Missing Critical Pieces

Tool standardization protocols face re-fragmentation when architectural constraints—token overhead, stateless scaling, weak auth—force enterprises toward custom implementations for production workloads. The author argues that MCP won adoption but lacks enterprise readiness: Cloudflare's native MCP costs 244,000 tokens versus 1,000 in "Code Mode"; sticky routing defeats load balancers; and missing governance leaves security to individual teams. Companies like Perplexity and Cloudflare are abandoning MCP's tool-calling layer in favor of direct APIs and code generation, signaling that production-scale enterprises require deterministic execution patterns that MCP cannot provide.

https://julsimon.medium.com/still-missing-critical-pieces-7a78077235e5

Databricks: Breaking the Microbatch Barrier: The Architecture of Apache Spark Real-Time Mode

Real-time analytics infrastructure traditionally required separate engines for throughput (Spark) and sub-100ms latency (Flink), leading to duplicated tooling and operational complexity. Apache Spark 4.1's Real-Time Mode eliminates this trade-off by using longer epochs with boundary checkpointing, concurrent map-reduce stages, and non-blocking operators that emit results continuously rather than buffering them. The unified engine handles both massive ETL and low-latency fraud detection while preserving Spark's lineage-based fault tolerance, consolidating the data stack for single-engine architectures.

https://www.databricks.com/blog/breaking-microbatch-barrier-architecture-apache-spark-real-time-mode

Sponsored: The AI Modernization Guide

Download the free guide

Etsy: Making Ads Count: Using MMoE and Auxiliary Tasks to Better Connect Buyers & Sellers

Multi-objective ranking in marketplaces faces metric conflicts—optimizing for clicks often degrades conversions—thereby requiring task-specific expert routing while managing data sparsity across event hierarchies. Etsy's MMoE architecture routes CTR and purchase prediction tasks through specialized experts with gated selection, then bridges sparse purchase signals using auxiliary add-to-cart tasks that correlate strongly with intent. The system achieved a 3.5% lift in purchase AUC and a 1% lift in click AUC while reducing inference cost through model pruning, enabling more accurate auto-bidding for sellers.

https://www.etsy.com/codeascraft/making-ads-count-using-mmoe-and-auxiliary-tasks-to-better-connect-buyers--sellers

Meta: Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovation

ML model iteration at production scale requires balancing hypothesis generation, resource constraints, and infrastructure resilience across multi-day experiment cycles. Meta’s Ranking Engineer Agent combines a Dual-Source Hypothesis Engine (historical experiments + novel ML research proposals) with autonomous debugging and cost-aware planning to execute long-horizon ranking workflows without human supervision. The system doubled average model accuracy across six models while enabling three engineers to maintain eight production models—a 5x productivity gain over traditional team structures.

https://engineering.fb.com/2026/03/17/developer-tools/ranking-engineer-agent-rea-autonomous-ai-system-accelerating-meta-ads-ranking-innovation/

Rahul Garg: Context Anchoring

AI-assisted development degrades over long sessions as models lose reasoning context ("why") despite retaining technical choices ("what"), trapping developers in single conversations to avoid context loss. The author proposes Context Anchoring—externalizing decision rationale, rejected alternatives, and constraints into lightweight Feature Documents outside the chat interface. Teams adopting external anchoring achieve warm starts in seconds, reduce token costs by 98%, enable multi-developer alignment, and validate logic through forced documentation, eliminating session anxiety as a design anti-pattern.

https://martinfowler.com/articles/reduce-friction-ai/context-anchoring.html

Dropbox: How we optimized Dash’s relevance judge with DSPy

Relevance scoring at scale faces a model-cost-quality trade-off: premium models like o3 are accurate but expensive, while cheaper open-weight models degrade without model-specific prompt tuning. Dropbox's DSPy-based optimization automates prompt refinement against human judgments using NMSE metrics and GEPA feedback loops, reducing manual tuning from weeks to 1–2 days. The system cut relevance error by 45%, eliminated JSON formatting failures by 97%, and enabled 10–100x data scaling by shifting to cheaper models while maintaining quality through systematic prompt compilation.

https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevance-judge-with-dspy

Zalando: Search Quality Assurance with AI as a Judge

Search quality assurance in a new domain often lacks historical user data, forcing teams to rely on manual testing and reactive fixes post-launch. Zalando's framework automates evaluation by generating NER-clustered queries translated into target languages, then routes the results through GPT-4o as a multimodal judge that assesses product metadata and images against a 0–4 relevance scale. The system evaluates 1,500 search segments (37,500 results) in 3–5 hours for $250, enabling proactive root-cause identification across languages and ensuring Day 1 quality without local market expertise.

https://engineering.zalando.com/posts/2026/03/search-quality-assurance-with-llm-judge.html

Andrey Novitskiy: Volga - A Rust Rewrite of a Real-Time AI/ML Data Engine (DataFusion, Arrow, SlateDB) with a Chronon + OpenMLDB–Style Architecture

Real-time ML feature computation requires unified streaming-batch execution with low-latency serving, forcing platforms to choose between specialized engines (Flink for latency, Spark for batch, Redis for serving). Volga's Rust implementation pairs DataFusion SQL execution with SlateDB (an embedded LSM on object storage) and Request Mode, embedding serving logic directly into operator state to eliminate external cache round trips. The system handles month-year windows via tiling, includes native ML aggregations (top-k, categorical sums), and achieves compute-storage separation by consolidating the feature pipeline, batch training, and real-time serving into a single Rust binary.

https://volgaai.substack.com/p/volga-a-rust-rewrite-of-a-real-time

Max Halford: Lower your warehouse costs via DuckDB transpilation

Cloud warehouse compute costs escalate during development and testing cycles, incentivizing hybrid approaches that separate cheap storage from expensive query execution. Max Halford’s “Quack Mode” transpiles warehouse SQL (BigQuery → DuckDB) using SQLGlot, pulls only upstream dependencies into local DuckDB instances, and executes transformations at near-zero cost, optionally pushing results back. The pattern dramatically reduces development compute spend while maintaining warehouse portability, though pulling tables >100GB remains a bottleneck until zero-copy solutions like Iceberg are available.

https://maxhalford.github.io/blog/warehouse-cost-reduction-quack-mode/

Data Engineering Weekly #261

Ananth Packkildurai — Mon, 16 Mar 2026 00:49:14 GMT

How to Orchestrate Databricks Across Multiple Workspaces

As Databricks deployments scale, a familiar pattern emerges: multiple workspaces, multiple teams, and no reliable way to manage the dependencies between them.

In this hands-on deep dive, we'll show you how to build a cross-workspace control plane using Dagster on top of your existing Databricks environment. Demo-heavy and practitioner-focused, you'll leave with working patterns you can apply to your own platform the same day.

Register now

Editor’s Note: Introducing Data Engineering After AI Podcast Series

Lately, I’ve been thinking a lot about the intersection of data architecture and AI. To dig deeper into this, I’m launching a new podcast series called Data Engineering After AI.

I’m looking for guests who are in the trenches. If you have strong opinions on where the industry is heading, or if you are actively building solutions in this space (either in-house or as a product), let’s talk.

Please note: my goal is to foster an authentic discussion about how AI is reshaping data engineering from the ground up. This isn’t a space for promotional product pitches, and I want to keep the conversation strictly focused on the technology, the challenges, and the architectural shifts.

If you are passionate about the future of our field and want to share your insights, DM me on LinkedIn.

Joseph M. Hellerstein: AI and the Mixed-Consistency Future

In my recent article, ETL is dead, I projected that the data modeling techniques that got us here may not be sufficient for the AI era. The consistency model is one of the biggest gaps in the emerging file-based system design around the AI Agent. We have seen this shift from the Hadoop file system to the Lakehouse model. The author suggests that we may be entering the Mixed-Consistency future.

https://jhellerstein.github.io/blog/ai-mixed-consistency/

Milan Mosny: Ontology, Taxonomy, Data Model, Context Graph & Friends

Context Engineering is the hot topic in the industry. I found the author did an excellent recap on ontology, taxonomy, data model & context graph. As the famous saying goes, it is all data engineering.

https://medium.com/response42/ontology-taxonomy-data-model-context-graph-friends-56a605e14355

Jason Cui & Jennifer Li: Your Data Agents Need Context

Contextual grounding—standardized terminology, data lineage, operational semantics—determines whether natural language agents answer analytics questions reliably. The authors propose a “Context Layer” combining LLM-powered metadata construction with human refinement to map business knowledge onto warehouse schemas. Organizations adopting context-aware agent architectures unlock self-serve analytics without brittleness, enabling agents to reason consistently across disparate schemas.

https://www.a16z.news/p/your-data-agents-need-context

Sponsored: The AI Modernization Guide

Download the free guide

Robin Moffatt: Claude Code isn’t going to replace data engineers (yet)

We see some degree of success with the Claude Code in software engineering. Is it ready for the prime data engineering? The author noted the gap in trust & accuracy, silent data loss, non-determinism, technical flaws, and maintenance. There is a data engineering gap in building an efficient sandbox environment to bridge it, which is a must for brownfield projects.

https://rmoff.net/2026/03/11/claude-code-isnt-going-to-replace-data-engineers-yet/

Snap: Agent Format: A Declarative Standard for AI Agents

Speed and Correctness in execution always have their own trade-off. Snap writes about how different teams adopted different AI frameworks to move fast and focus on standard interface design to make everything work together. I believe as long as the pendulum swings between speed and efficiency, the software engineering is safe. We will always build the next best abstraction.

https://eng.snap.com/agent-format

LinkedIn: Engineering the next generation of LinkedIn’s Feed

Feed personalization at a massive scale requires unifying disparate retrieval signals into semantic representations while maintaining sub-second latency across billions of users. LinkedIn's architecture consolidates keyword matching, collaborative filtering, and engagement signals into a dual-encoder LLM retrieval paired with a Generative Recommender transformer that sequences 1,000+ historical interactions to capture professional trajectories. Custom infrastructure—Flash Attention variants, GPU-optimized data loaders, decoupled nearline pipelines—enables semantic ranking at sub-second latency for 1.3 billion members while reducing training memory by 37%.

https://www.linkedin.com/blog/engineering/feed/engineering-the-next-generation-of-linkedins-feed

Spotify: Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights

Generating personalized narratives at a billion-scale requires balancing creative consistency, latency constraints, and data fidelity without requiring human review. Spotify's Wrapped Archive distills frontier LLM outputs into smaller production models via DPO, grounds narratives in heuristic-ranked "remarkable days" from distributed pipelines, and uses layered prompts to enforce tone while preventing hallucinations. Column-oriented storage with per-day qualifiers, pre-scaled compute, and automated Judge-model sampling of 165,000 reports enables 1.4 billion unique narratives at launch latency while catching systemic failures such as timezone bugs.

https://engineering.atspotify.com/2026/3/inside-the-archive-2025-wrapped

LinkedIn: Driving data enhancement & recruitment success with LinkedIn’s unified integrations

Recruitment data fragmentation—disparate ATS schemas, semantic conflicts, and partner integration overhead—blocks AI agents from reliably reasoning across hiring pipelines. LinkedIn's unified platform standardizes partner data into canonical schemas via hybrid push/pull models (BuildIn for speed, BuildOut with Temporal orchestration for reliability), assigns stable Integration IDs to decouple identity, and reconciles multi-source conflicts into single-truth serving layers. The system cut onboarding from 12 months to 4, expanded job field coverage 1.8x, and dropped resume gaps below 10%, enabling agents to reason and act consistently across enterprise hiring systems.

https://www.linkedin.com/blog/engineering/talent/driving-data-enhancement-and-recruitment-success-with-linkedins-unified-integrations

Uber: Transforming Ads Personalization with Sequential Modeling and Hetero-MMoE at Uber

Ads ranking at scale requires capturing sequential user intent over long behavioral histories while simultaneously optimizing competing objectives such as clicks and conversions. Uber's system pairs target-aware transformers with Multi-Head Latent Attention (reducing sequence complexity from O(N²) to O(N×L)) to compress engagement histories, then routes the compressed signals through Hetero-MMoE—blending DCN and CIN experts to capture low- to high-order feature interactions across multimodal inputs. Online experiments yielded +0.93% AUC on predicted CTR and +0.66% AUC on predicted click-to-order, validating sequential modeling at the ranking scale.

https://www.uber.com/en-EG/blog/transforming-ads-personalization/

Databricks: LogSentinel: How Databricks uses Databricks for LLM-Powered PII Detection and Governance

PII discovery and compliance monitoring at a data warehouse scale requires automating label classification across schema evolution without manual audit cycles. Databricks’ LogSentinel orchestrates multiple LLM “experts” in parallel—augmented with Vector Search context and AI-generated column comments—to classify data across 100+ granular, hierarchical, and residency labels, selecting predictions by confidence voting. The system achieves 92% precision and 95% recall while reducing manual review cycles from weeks to hours, enabling real-time governance as schemas drift.

https://www.databricks.com/blog/logsentinel-how-databricks-uses-databricks-llm-powered-pii-detection-and-governance

ETL is Dead

Ananth Packkildurai — Wed, 11 Mar 2026 14:42:11 GMT

More ETL pipelines will run in 2027 than in any year in history. AI will generate more extraction jobs, more transformation logic, and more loading routines than any team of data engineers could write by hand. The volume of ETL will explode.

And ETL is still dead.

Not dead the way Latin is dead — no one speaks it. Dead, the way landlines are dead — they still work, millions exist, but nobody builds their communication strategy around one. ETL is dead as the defining work of data engineering. Dead as the thing we hire for, build careers around, and organize teams to do. The pipelines keep running. The professional identity built around them does not survive.

The Warehouse Was Always a Metaphor. Now the Metaphor Is Breaking.

We literally called it a data warehouse. And that wasn’t just naming — we replicated the entire physical warehouse operating model into the digital world. Racks became tables. Inventory management became catalogs. Forklifts became ETL pipelines. Floor workers became data engineers. Shift supervisors became analytics leads.

Every technique we built — star schemas, slowly changing dimensions, medallion architectures, conformed dimensions — served the same purpose as aisle markers and shelf labels in a physical warehouse: help a human walk in, find what they need, and carry it out.

Data modeling organizes information so humans can discover it. Data catalogs provided wayfinding to help humans navigate them. The medallion architecture created a pick-pack-ship assembly line where humans inspected and validated data at each station. Naming conventions — fact_orders, dim_customers — acted as signage so humans could read the shelves at a glance.

Every design decision is optimized for human cognition. And then the operator changed.

What Happened When Robots Entered the Physical Warehouse

When Amazon deployed Kiva robots, they didn’t replace human tasks one-for-one. They redesigned the entire warehouse around a different operator.

Physical warehouses built for humans had wide aisles because humans need space to walk. They grouped items logically because humans need to remember where things are. They placed high-demand products at eye level because humans have ergonomic constraints. They posted signage everywhere because humans need wayfinding.

Robotic warehouses threw all of that out. Aisles shrank because robots don’t need shoulder width. Shelving went floor-to-ceiling because robots don’t have ergonomic limits. Logical grouping became unnecessary because robots navigate by coordinates, not memory. Signage disappeared because robots don’t read signs — they read instructions.

But the biggest gains weren’t physical. They were cognitive. Human warehouse workers carried an enormous cognitive load — remembering locations, making routing decisions, prioritizing picks, and mentally handling exceptions. Robots eliminated that cognitive burden entirely. The warehouse didn’t just move faster. It became a fundamentally different system that could handle complexity no human floor operation could manage.

The Data Warehouse Is Still Designed for Human Forklift Operators

Now look at our data warehouse through this lens.

Star schemas and dimensional modeling exist so a human analyst can visualize how tables relate. A human needs to see the star — the fact table at the center, dimensions radiating outward. An agent doesn’t need a star. It needs a validated semantic definition of what each entity means and how entities connect.

Data catalogs are digital signage. We built them because humans need to browse and discover what’s in the warehouse. An agent doesn’t browse a catalog the way a human walks an aisle. It queries for a validated meaning.

The medallion architecture — Bronze to Silver to Gold — is an assembly line designed for human inspection at each station. Raw data lands, gets progressively cleaned, and arrives ready for consumption. Each station assumes a human will inspect, validate, and pass the data forward. And at each handoff, context erodes — the original meaning collapses a little more, like a game of telephone played silently in the pipeline.

We optimized every layer of the data warehouse for human cognitive constraints. And just like the physical warehouse, those very optimizations become limitations when the operator changes.

Where the Analogy Holds — and Where It Breaks

I want to be precise about this, because imprecise analogies are how our industry ends up with decade-long hype cycles built on half-truths.

The analogy holds powerfully for navigation and discovery. Physical warehouses organized shelves for human wayfinding. Data warehouses organize tables for human querying. Robots don’t need aisle signs. Agents don’t need star schemas to find data. That part maps cleanly.

But here’s where it breaks: physical goods don’t change meaning based on how you store them. A box of shoes is a box of shoes, whether it sits on shelf A3 or shelf Z9. Data is different. How you structure data shapes what questions you can ask of it. A normalized schema enables different analytical patterns than a denormalized one. A slowly changing dimension preserves the temporal context that a snapshot table destroys.

Structure still matters for agent-operated data. It just serves a different purpose. Instead of organizing for human navigation — “how do I find the data?” — you organize for agent operation — “what data and context does this agent need for this task?” Think about how AI tools work with a scoped working folder. You don’t reorganize your filesystem into an agent-friendly layout. You give the agent a well-scoped boundary, and it operates within it. The structure shifts from navigational to operational — from shelf labels to access boundaries.

The Thinking Survives. The Format May Not

I took the last class Ralph Kimball taught before his retirement. I remember the vivid conversation around HBase (which was popular at the time) and the notion of versioning to handle slowly changing dimensions. I’ve internalized dimensional modeling deeply enough to know which parts are permanent and which parts are artifacts of their era.

Kimbal didn’t start the training with the star schema and slowly changing dimensions. Kimball’s dimensional modeling process starts with two steps: identify the business process and select the grain. These steps ask the most fundamental questions in data engineering — what does the business actually do, and at what level of detail does it matter? Only after answering those do you design the dimensions, the facts, and the star schema.

Steps one and two are context architecture. They always were. Identifying the business process means understanding the semantic reality of what the organization does. Selecting the grain means choosing the level of meaning that matters. That thinking is more relevant today than it was in 1996.

Steps three and four — the star schema, the dimension tables, the fact tables — were a rendering choice. They were the best output format for the consumer of that era: a human analyst writing SQL against a relational database. The star schema serialized business understanding into a structure that humans could query using the available tools.

The consumer has changed or is changing. The rendering should too. When the consumer is an AI agent, the same analytical thinking about business processes and grain produces a Context Store entry — a validated, versioned, queryable semantic definition — not a fact table. The thinking survives. The format may not.

Dismissing dimensional modeling entirely would be ignorant. Clinging to its output format when the consumer has fundamentally changed would be equally so.

The Pendulum

Every era of data architecture has tried to solve the same tension: semantic precision versus operational flexibility.

The relational era chose precision. ERDs, primary keys, foreign keys, referential integrity, constraints — the schema was the semantic contract. Bill Inmon’s Corporate Information Factory formalized this into an enterprise architecture. It worked. It encoded business meaning directly into the physical structure. But it was rigid. I remember interviewing at a company in the pre-Hadoop era and asking what their current priority was. The interviewer told me they were working on implementing a schema change in a day rather than a month. That was the state of the art — a month to add a column, because the semantic contracts were so tightly welded to the physical structure that touching one meant touching everything.

Hadoop’s answer was brute force. Sheer machine power, schema-on-read, commodity hardware — throw everything in and figure it out later. It broke the operational rigidity overnight. And it also broke every semantic contract the relational era had built. We traded meaning for speed and went too far. The data lake became a data swamp because nobody could remember what anything meant — the constraints that encoded that meaning were gone.

The lakehouse tried to find a middle ground. Iceberg, Delta, Hudi — the flexibility of the lake with some structure of the warehouse. Better. But the semantic layer remained an afterthought.

catalogs, documentation, and governance overlays that nobody maintained because nobody’s career depended on them being right.

Even recent efforts like Snowflake’s Open Semantic Interchange initiative acknowledge the gap — the industry is only now trying to standardize how semantic meaning travels between tools.

Each swing of the pendulum traded one problem for another. Rigidity for meaninglessness. Meaninglessness for a partial structure. What none of them achieved was decoupling — semantic precision that doesn’t require physical rigidity. Context that travels alongside the data but isn’t welded to the table structure. Change the schema in seconds. The context updates through the Contextualize pipeline. The meaning stays current without the rigidity.

That decoupling is what ECL provides. It’s the first architecture that doesn’t force you to choose between knowing what your data means and being able to change it.

The Graveyard of Good Intentions

I know what the skeptics are thinking, because I’ve thought it myself: we’ve heard this before.

Bill Inmon literally wrote the book on this in 2007 — Business Metadata: Capturing Enterprise Knowledge — which covers semantics, ontologies, business rules, and the capture of tacit knowledge. He laid out a complete methodology for capturing it. The methodology was sound. The economics weren’t there yet.

Business glossaries in the 2000s promised to capture institutional knowledge. They became static documents that nobody updated. Semantic layers in the 2010s promised a unified layer of meaning. They became another piece of middleware to maintain. Data catalogs promised discoverability and governance, but soon proved to be useless. Many became expensive shelfware. Enterprise knowledge graphs promised connected meaning. Most never made it past the proof-of-concept stage.

Every generation of data practitioners has pointed at the same north star: capture business meaning as a first-class artifact. Every generation has underestimated the organizational gravity that pulls teams back to “just get the data there, and we’ll figure out what it means later.”

So what makes this time structurally different? One thing: the consumer changed from forgiving to unforgiving.

When the consumer was a human analyst, missing context was inconvenient. The analyst would Slack a colleague, read the dbt code, ask in standup, and check the wiki. Humans are remarkably good at filling semantic gaps through social channels. Bad metadata produced frustrated analysts, not system failures.

When the consumer is an AI agent, missing context produces systematic errors at scale. The agent doesn’t Slack anyone. It doesn’t read tribal knowledge. It sees a column called rev_adj, makes its best inference, and acts — confidently, consistently, and potentially wrong across every downstream decision. Bad context doesn’t produce frustration. It produces hallucination at an enterprise scale.

For the first time, the cost of missing context exceeds the cost of maintaining it. That economic inversion is what none of the previous attempts had. Business glossaries failed because humans bore the cost of maintaining them, while the benefit was diffuse. The Context Store succeeds or fails based on whether agents produce reliable results — and that feedback loop is immediate, measurable, and impossible to ignore.

The graveyard is real. But the economics changed.

What Replaces It

ETL asked: Did the data land? ECL asks: Can the data be trusted? I introduced the ECL framework in my earlier article on data engineering after AI.

Extract remains. Data still moves from source systems to analytical environments. That work still requires engineering judgment about reliability, latency, and failure modes. AI handles more of the mechanical construction. Humans make the architectural decisions.

Contextualize is the new center of gravity. A dedicated, agentic pipeline that builds and maintains a living store of semantic context. It isn’t documentation. It isn’t a catalog. It’s an engineering artifact with its own trigger model, validation layer, and storage — the Context Store.

The Context Store holds two types of objects. Context objects capture long-lived semantic definitions — what “revenue” means, who validated that definition, when, and at what confidence level. These compounds increase in value over time. Decision objects capture what agents produce when they act on context — which definitions they used, what they inferred, and what they recommended. These create the audit trail.

Link connects entities across the data landscape — and emerging standards like Model Context Protocol (MCP) are starting to standardize how agents access data without moving it. Not just table joins — semantic relationships between business entities across systems. A customer in CRM is linked to a user in your product, linked to a session in your support tool. Whether you implement that as a graph, a mapping table, or a markdown file matters less than whether the linkage is validated and the semantic relationship is explicit.

And because data is inherently social in nature, you don’t build this all at once. You start with one business flow. One critical table. Early bind where you control the data and can hold producers accountable for meaning. Late bind where data comes from outside your accountability boundary — third-party feeds, undocumented internal systems, legacy data where the person who knew what the fields meant left five years ago. Even one table, well contextualized, starts compounding as you connect it to the next one, and the next one.

Long Live the Context Architect

The physical warehouse workers who resisted robotics didn’t save their jobs. They delayed their own transition. Those who moved into robotics coordination, system design, and exception architecture found themselves more valued, more strategic, and more central to the operation than they were when driving forklifts.

Data engineers who built their identity around moving data from one bucket to another have felt that identity under pressure for a while now. That pressure isn’t going away. AI will write your Spark jobs. AI will generate your dbt models. AI will build more pipelines in a year than your team could build in a decade.

But AI cannot decide what “revenue” means for your organization. It cannot negotiate data contracts between producing and consuming teams. It cannot design the appropriate level of context for an agent addressing a specific business problem. It cannot build the organizational agreements that make semantic definitions stick. That work requires institutional knowledge, cross-functional coordination, and architectural judgment. That work is context architecture.

The data engineer’s value migrates from pipeline reliability to semantic reliability. From “the job ran” to “the meaning is right.” From operating the warehouse floor to designing the system that makes robotic operation trustworthy.

The frontier is genuinely open. Nobody has this figured out yet. The practitioners who invest in the architecture of meaning — not just the mechanics of movement — will define this discipline for the next decade.

ETL is dead. Long live the Context Architect.

Data Engineering Weekly #260

Ananth Packkildurai — Mon, 09 Mar 2026 04:31:18 GMT

Best practices for orchestrating Databricks at scale

Save your spot now

underCurrent: A one-day conference for data engineers and architects

Confluent is hosting a free one-day conference with a catch: there’s no catch. It’s a single-track event with no sponsors and no product pitches—just technical talks for data engineers and architects.

🎙️ Speakers include Joe Reis, Holden Karau, and Max Beauchemin
🚫 No vendors. No sales pitches
✨ 100% free to attend
📅 March 26
📍 San Francisco
🎟️ Limited to 100 seats — register for free here

Vinoth Govindarajan: OpenClaw Architecture

Production AI agents fail at scale because uncontrolled state mutations corrupt execution and create unpredictable behavior. In “The Agent Stack,” Vinoth Govindarajan outlines OpenClaw’s architecture, in which isolated execution contexts and strict invariants prevent state leakage, while sessions enable async pause-resume semantics. The pattern standardizes how teams decouple short-term context from persistent state, ensuring agents reliably rehydrate their mental model and enforce authorization boundaries that gate tool access to user privilege levels.

Part 1, Part 2, Part 3.1, Part 3.2, Part 4

Pinterest: Unified Context-Intent Embeddings for Scalable Text-to-SQL

Navigating sprawling data warehouses forces analysts to choose between slow manual exploration and unreliable keyword-based search. Pinterest Engineering built a production Analytics Agent that embeds historical SQL queries as semantic intent signatures, injecting business glossary terms and extracting structural patterns (join keys, filters, usage signals) to retrieve contextually relevant tables at scale. The system reached 40% internal adoption within two months by standardizing discovery through an asset-first pattern, converting years of institutional SQL knowledge into a searchable, governance-aware library.

https://medium.com/pinterest-engineering/unified-context-intent-embeddings-for-scalable-text-to-sql-793635e60aac

Francesca Lazzeri: AI evals platforms: A comparative guide for production AI systems

Production AI systems fail silently in ways demos never expose, forcing teams to replace manual testing with automated evaluation as the enterprise LLM market scales toward $71.1 billion by 2034. A comparative analysis of six leading eval platforms reveals a consolidation around open standards (OpenTelemetry, OpenInference) and specialized architectures—Microsoft AI Foundry embeds red-teaming agents into Azure workflows, while Galileo replaces expensive LLM judges with smaller consensus models (Luna) to reduce eval latency. The shift standardizes safety as a structural property of development, enabling teams to catch jailbreaks and data leaks early while choosing platform fit based on stack priorities: simulation-first, research rigor, or ecosystem depth.

https://medium.com/data-science-at-microsoft/how-do-you-know-your-ai-actually-works-b1a380a07825

Sponsored: The AI Modernization Guide

Download the free guide

Netflix: MediaFM - The Multimodal AI Foundation for Media Understanding at Netflix

Understanding content at scale requires machine-readable representations that capture narrative structure, not just visual features—a challenge intensified as streaming catalogs exceed tens of thousands of titles. Netflix built MediaFM, a tri-modal transformer that fuses video frames, audio (wav2vec2), and subtitles into shot-level embeddings using Masked Shot Modeling, with a [GLOBAL] token injecting title-level context (synopsis, genre) to ground each segment. The model powers ad placement, clip ranking, content tagging, and cold-start recommendations by contextualizing shots within narrative sequence, outperforming external benchmarks and enabling machine-readable understanding across Netflix's entire catalog.

https://netflixtechblog.com/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d

Nabin Debnath: Building a Least-Privilege AI Agent Gateway for Infrastructure Automation with MCP, OPA, and Ephemeral Runners

AI agents in infrastructure automation bypass traditional guardrails by making runtime decisions without human validation, risking silent resource destruction or credential exfiltration at scale. The author writes about the Agent Gateway to treat the agents as untrusted requesters, layering Model Context Protocol (MCP) for tool discovery, Open Policy Agent (OPA) for intent-based authorization, and ephemeral Kubernetes runners for isolated execution. The pattern enforces least privilege by mediating all API calls through policy code, validates plan integrity against immutable hashes, and surfaces decision reasoning via OpenTelemetry—standardizing agent governance with SLO targets (100ms policy decisions, 5s runner startup) that prevent silent bypasses.

https://www.infoq.com/articles/building-ai-agent-gateway-mcp/

Dropbox: Using LLMs to amplify human labeling and improve Dash search relevance

Enterprise search ranking requires massive labeled datasets, but traditional human annotation is prohibitively slow and cannot scale to sensitive content across billions of internal documents. Dropbox Dash uses LLMs as labeling force multipliers by calibrating a small human-labeled set to generate millions of relevance judgments offline, then training lightweight production models (XGBoost) on synthetic labels at scale. The pattern standardizes judgment consistency by pairing contextual research tools (for acronyms and ambiguous queries) with programmatic prompt optimization (DSPy), enabling continuous ranking improvements while keeping human oversight as the ground truth rather than replacing it.

https://dropbox.tech/machine-learning/llm-human-labeling-improving-search-relevance-dropbox-dash

Zalando: Why We Ditched Flink Table API Joins: Cutting State by 75% with DataStream Unions

Declarative SQL joins in Flink multiply state across operators, forcing teams to choose between snapshot overhead or operational instability—a scaling bottleneck for pipelines enriching millions of real-time product records. Zalando replaced chained Table API joins with a custom KeyedProcessFunction that unions all streams into a single keyed DataStream, storing each product’s enriched state once in RocksDB instead of redundantly across join operators. The shift cut state size by 75% (235GB to 56GB), reduced snapshot time by 77% (11 minutes to 2.5 minutes), and lowered AWS costs by 13%—demonstrating how imperative control over stream topology recovers efficiency when declarative abstractions misalign with physical execution.

https://engineering.zalando.com/posts/2026/03/why-we-ditched-flink-table-api-joins-cutting-state.html

Aihua Xu & Andrew Lamb: Variant Type in Apache Parquet for Semi-Structured Data

Semi-structured data in columnar formats forces a choice between slow JSON parsing or rigid schemas that block evolution, creating friction in pipelines handling heterogeneous records. Apache Parquet’s new Variant type uses binary-encoded metadata plus value fields, enabling direct nested field access without full-document parsing while preserving native types (timestamps, integers) that JSON loses. The type standardizes schema flexibility through “shredding”—extracting hot fields into strongly-typed columns for predicate pushdown and pruning—allowing heterogeneous records to coexist in one column, reducing migration overhead and accelerating adoption across DuckDB, Spark 4.0, and Snowflake.

https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/

Pranav Mehta: Silent Data Loss in ClickHouse: 3 Reasons Your Distributed Queue Keeps Growing

ClickHouse distributed inserts silently fail when coordination services downtime, execution timeouts, or concurrency limits block the async flush pipeline, leaving data trapped in on-disk queues while clients receive no error signals. The author identifies three failure modes: Keeper/ZooKeeper downtime forcing ReplicatedMergeTree read-only, oversized insert blocks exceeding max_execution_time that cork sequential queue processing, and exhausted user concurrency slots starving background INSERT workers. The pattern demands proactive monitoring of DistributedFilesToInsert (alert at 50+ files), debugging via system.distribution_queue.last_exception, and inode-aware filesystem choice (XFS over ext4) to prevent silent data loss and system crashes from queue explosion.

https://medium.com/@pranavmehta94/silent-data-loss-in-clickhouse-3-reasons-your-distributed-queue-keeps-growing-9bf6b8af88e5

Data Engineering Weekly #259

Ananth Packkildurai — Mon, 02 Mar 2026 03:57:15 GMT

AI is moving fast. Is your data platform ready?

Download the AI Modernization Guide

underCurrent: A one-day conference for data engineers and architects

Confluent is hosting a free one-day conference with a catch: there’s no catch. It’s a single-track event with no sponsors and no product pitches—just technical talks for data engineers and architects.

🎙️ Speakers include Joe Reis, Holden Karau, and Max Beauchemin
🚫 No vendors. No sales pitches
✨ 100% free to attend
📍 San Francisco 📅 March 26
🎟️ Limited to 100 seats — register for free here

Netflix: DataJunction as Netflix’s answer to the missing piece of the modern data stack

Metric inconsistency and definition sprawl across distributed teams create onboarding bottlenecks and fragment analytics workflows. Netflix built DataJunction, an open-source semantic layer that decouples metric definitions from compute through a graph-based metadata model and SQL generation engine. This standardizes metrics across the experimentation platform, reducing onboarding from weeks to hours, while enabling expansion across all business verticals and LLM integration for auditable metric lineage.

https://netflixtechblog.medium.com/datajunction-as-netflixs-answer-to-the-missing-piece-of-the-modern-data-stack-92af926b40a5

Benoit Pimpaud: Specs Should Be Equations, Not Essays

As AI automates code generation, the engineering bottleneck shifts from writing implementation to defining precise specifications. the author argues that natural language specifications create compounding ambiguity when parsed by LLMs and proposes layered specifications that combine text, diagrams, and mathematical notation as constraint definitions for AI iteration. Mathematical specs eliminate interpretation drift, enabling AI agents to generate correct programs by satisfying invariants rather than reconstructing intent from prose.

https://fromanengineersight.substack.com/p/specs-should-be-equations-not-essays

Notion: Balancing cost and reliability for Spark on Kubernetes

Spark clusters on Kubernetes face a fundamental tension between aggressive cost optimization through spot instances and job reliability during capacity interruptions. Notion reduced compute costs by 60–90% through EKS migration with Karpenter bin-packing, then open-sourced Spot Balancer—a Kubernetes webhook that enforces stable spot-to-on-demand ratios per job, preventing cascade failures during AWS termination windows. Spot Balancer abstracts infrastructure trade-offs into developer-friendly stability tiers, enabling teams to optimize costs without sacrificing job completion rates.

https://www.notion.com/blog/balancing-cost-and-reliability-for-spark-on-kubernetes

Apache Iceberg: Introducing the Apache Iceberg File Format API

It is indeed an exciting development in Iceberg to support a plugable file format API spec. As we increasingly handle unstructured data, this will significantly enhance data management practices through unified governance and compliance. Interestingly, Apache Hudi’s RFC-100 is, in fact, the feature request to support the Lance File Format.

https://iceberg.apache.org/blog/apache-iceberg-file-format-api/

Delta Lake: The next evolution of Delta - Catalog-Managed Tables

We went through the full cycle, from exposing the files directly through Hadoop to Snowflake-style cloud data warehouses, to Iceberg-style direct file access, back to catalog-managed tables.

Nonetheless, it will be interesting to watch DuckLake-style catalog-managed tables vs object-store-style managed tables.

https://delta.io/blog/2026-02-02-delta-catalog-managed-tables/

Microsoft Fabric: Under the hood: an introduction to the Native Execution Engine for Microsoft Fabric

The Apache Gluten project is continually making an impact on the Spark ecosystem, bringing unique optimization and efficiency. Microsoft Fabric writes an under-the-hood story of adopting Apache Gluten in its Fabric platform.

https://blog.fabric.microsoft.com/en-us/blog/under-the-hood-an-introduction-to-the-native-execution-engine-for-microsoft-fabric/

Pinterest: Piqama - Pinterest Quota Management Ecosystem

As companies scale, manual and static quota systems become bottlenecks, forcing engineers to choose between over-provisioning resources and managing brittle enforcement logic. Pinterest developed Piqama, a unified quota platform that dynamically right-sizes limits using historical data stored in Apache Iceberg, then applies custom enforcement strategies across batch schedulers and online services. Piqama centralizes resource governance across hardware and service metrics, enabling teams to optimize capacity allocation while linking consumption directly to financial costs.

https://medium.com/pinterest-engineering/piqama-pinterest-quota-management-ecosystem-dc7881433bf5

LinkedIn: Engineering LinkedIn’s job ingestion system at scale

Ingestion systems struggle to scale source onboarding—hard-coded extraction logic creates engineering bottlenecks that slow integration of new data partners. LinkedIn shifted extraction logic from code to configuration files called Sitemaps, enabling AI tools and browser plugins to onboard sources without engineering deployments. At the same time, a transactional state machine enforces precise failure boundaries across parallel mining tasks. The configuration-driven approach reduces onboarding time from weeks to hours, allowing LinkedIn to ingest 20TB daily across thousands of global sources.

https://www.linkedin.com/blog/engineering/infrastructure/engineering-linkedins-job-ingestion-system-at-scale

Shopify: The generative recommender behind Shopify’s commerce engine

Recommendation systems traditionally treat purchases as isolated events, missing the temporal and causal structure that shapes buyer journeys across millions of products. Shopify transitioned to an autoregressive sequence model that treats commerce journeys as token sequences, implementing RoPE-inspired rotary encoding combined with relative attention bias to capture temporal gaps and seasonality across its catalog. The time-aware attention mechanism drove +0.94% order growth and +0.71% conversion lift while achieving 7.3x training speedup through optimized CUDA kernels, enabling Shopify to integrate richer context into a unified generative framework.

https://shopify.engineering/generative-recommendations

Alibaba: PostgreSQL Blink-tree Implementation

As we increasingly use AI to code, understanding database internals is more critical than ever. Alibaba Cloud engineers break down how PostgreSQL utilizes the Blink-tree architecture to achieve massive concurrency. By adding link pointers to sibling nodes and high keys to mark boundaries, PostgreSQL allows searches to proceed without lock-coupling. This enables the system to gracefully handle concurrent page splits—following links when data exceeds old boundaries—and significantly outperforms the more rigid lock-subtree approach used in MySQL’s InnoDB.

https://www.alibabacloud.com/blog/postgresql-blink-tree-implementation_602913

Data Engineering After AI

Ananth Packkildurai — Tue, 24 Feb 2026 03:03:53 GMT

A few days back, I ran a LinkedIn poll asking what stays core to software engineering as AI increasingly writes the code. 53% said architecture and trade-offs. 20% said quality and ownership, and 25% said product and problem discovery.

The poll wasn’t specifically about data engineering, but the answer it yielded applies directly to us. When AI can generate a pipeline as fluently as a senior engineer, the question isn’t whether our toolbox is changing — it clearly is. The question is: what kind of thinking has always been too important to automate, and why we let it get buried under the more mechanical work in the first place.

My answer is that the irreducible work was never about moving data. It was always about meaning. And the framework we’ve been using — ETL — was never really designed to capture meaning.

The ETL Era and Why It’s Ending

Extract, Transform, Load made sense as a job description for a specific historical moment. Source systems were siloed, formats were inconsistent, and somebody had to write the code that moved data from where it lived to where it could be used. The data engineer was that somebody.

But if we’re honest, the transformation step was always the most brittle part. Teams encoded business rules as SQL logic or Python functions, buried them in pipeline code, version-controlled them alongside infrastructure, but rarely treated them with the same rigor as application code. When the definition of “active user” changed — and it always changed — someone had to find every place that definition lived and update it, hoping they caught them all.

AI is now competent at generating this kind of code. Not perfect, but competent enough that the mechanical work of pipeline construction is no longer a meaningful differentiator. If your professional identity is built around being good at writing transformation logic, that identity is under pressure.

But this isn’t a story about loss. It’s a story about clarity. The mechanical work was always obscuring the more important work underneath it. AI forcing that reckoning is, in a strange way, a gift.

Introducing ECL — Extract, Contextualize, Link

The framework emerging as a replacement isn’t a technical architecture so much as a reorientation of purpose. Instead of Extract, Transform, Load, think Extract, Contextualize, Link.

Extract remains. Data still needs to move from source systems to analytical environments, and that work still requires engineering judgment — about reliability, latency, volume, and failure modes. AI will increasingly handle the mechanical parts, but the architectural decisions about what to extract, when, and how belong to people who understand both the source systems and the downstream consequences.

Contextualize is where the real shift happens. This is the work of giving data semantic meaning — understanding that “revenue” is calculated differently by Finance and Sales, that a timestamp in a clickstream event means something different than a timestamp in a billing record, that a null value in one system represents the absence of information while in another it represents an explicit user choice. AI can draft this work at scale — inferring field definitions, classifying entities, and mapping relationships across a data landscape that no human team could manually annotate in full. What AI cannot do is be accountable for itself. The judgment of whether an inference is correct, the organizational authority to declare a definition, the decision to formalize a discovered pattern into an enforced contract — that belongs to humans. Contextualize is where AI inference and human judgment meet, structured by a pipeline built specifically for that purpose.

Link is about entity relationships across the data landscape — connecting a customer record in your CRM to a user record in your product database, linking an event in your analytics system to a session in your support tool. As AI generates more of the code that consumes data, the ability to reason about how entities relate across systems becomes more valuable, not less. Linkage is what makes context portable — what allows the meaning built in one part of the landscape to be grounded in its relationships to the rest.

The rest of this article discusses how ECL works at the architectural level, not as three abstract concepts, but as three concrete pipelines — and why you need all of them.

Early Binding — Contracts as Executable Constraints

The first technique is early binding: capturing semantic intent at the point of data production, before the data moves.

Data contracts are the practical implementation of this idea. At their core, contracts are agreements between data producers and their consumers — specifying schema, data quality expectations, ownership, and the semantic meaning of each field.

Data Engineering Weekly identified this gap precisely in their piece Data Contracts: A Missed Opportunity. While the data industry was debating what contracts were and drafting governance frameworks to describe them, software engineering had quietly converged on a different organizing principle: treating specifications as executable constraints with real failure semantics. The data industry treated contracts as documentation. Software engineers treated them as interfaces — things that could break, that had versioning implications, that enforced behavior rather than merely describing it.

A data contract that lives in a wiki and gets updated when someone remembers is the documentation. A data contract that is enforced at the point of production — that fails a pipeline when a schema changes without notice, that alerts a consumer when quality thresholds are violated, that an AI agent can reason about deterministically — that is architecture.

This matters more in an AI-heavy world, not less. When AI agents generate transformation code, bad contracts are amplified at scale. The agent will faithfully implement whatever logic it’s given; if the contract governing its inputs is ambiguous or unenforced, the errors it produces will be systematic rather than isolated. Early binding is the mechanism by which human intent gets formalized into something AI can actually work with.

But early binding alone has a fundamental limitation. And understanding that limitation is what makes the Contextualize pipeline necessary.

The Problem Early Binding Alone Can’t Solve

Consider what happens to a well-contracted dataset as it moves through a modern Medallion architecture.

At the Bronze layer, data lands close to its source — raw, minimally transformed, the contract’s guarantees still largely intact. Silver applies conformance rules: deduplication, type casting, and light standardization. By the time data reaches Gold, the pipeline has made a series of editorial decisions on the data’s behalf. Aggregations collapse granular events into metrics. Engineers bake business logic into the shape of the table. The Gold layer is an artifact optimized for a specific set of questions — the ones that seemed important when the pipeline was built.

Early binding contracts help at the source, but they can’t prevent this erosion at every subsequent hop — especially when those contracts are treated as descriptive rather than executable. If there’s no enforcement mechanism preventing meaning from drifting across transformations, the telephone game plays out silently in your pipeline. By the time a consumer queries the Gold layer, they’re working with an artifact whose original intent may be several editorial decisions removed from the contract.

This is the problem that early binding alone cannot solve. Each transformation layer progressively collapses the context captured at the source. You need a complementary approach—one that preserves the ability to recover context when it’s actually needed.

Late Binding — The Agentic Contextualized Pipeline

Traditional late binding deferred the application of business rules to query time. What it didn’t defer was the definition of those rules — domain experts still had to specify them upfront, just applied through a semantic layer rather than baked into a physical table. In complex domains, that knowledge engineering process was its own bottleneck.

The more forward-looking approach is to defer definition itself — and hand that work to a dedicated pipeline.

The Contextualize pipeline is a separate, agentic pipeline that runs alongside your data infrastructure. Its job is singular: build and maintain a living, validated store of semantic context. It isn’t part of the Extract pipeline. It isn’t a query-time process. It’s a first-class engineering artifact with its own triggering model, validation layer, and storage.

The trigger is event-driven, not scheduled. Every new dataset that lands automatically kicks off the pipeline. For existing datasets, continuous profiling monitors for meaningful changes — a new column appears, a column is dropped, a data distribution shifts in ways that suggest something changed upstream. Any of these events re-triggers the pipeline for the affected entities. Semantic context isn’t a one-time annotation exercise. It tracks the data as it evolves.

The pipeline itself is agentic. An AI agent analyzes the incoming data — schema, sample values, statistical profiles, lineage — and infers semantic meaning. What does this field represent? What business entity does it belong to? What relationships exist between it and other data in the landscape? It produces structured, versioned context artifacts: inferences about meaning that didn’t require a domain expert to pre-specify every scenario.

Those inferences don’t automatically commit. They route to a validation layer that works like a labeling workflow — because structurally, it is one. An LLM-as-Judge validates high-confidence inferences before any human review triggers. Medium-confidence ones surface to domain experts for labeling. The pipeline flags low-confidence or contested inferences for deeper investigation. The humans aren’t reviewing every artifact; they’re reviewing the uncertain ones. Every labeling automation technique that works in ML pipelines applies here.

Validated artifacts land in a Context Store — a dedicated, versioned, queryable store of semantic definitions, entity classifications, and relationship maps. This is the new infrastructure component that ECL requires. Downstream agents don’t query raw data and infer meaning on the fly. They query the Context Store first, ground their understanding in validated context, and then query the data. The context is stable, reusable, and auditable — the opposite of ephemeral query-time inference.

Early Binding vs Late Binding — When to Choose What

The decision criterion isn’t about semantic maturity or how well-understood a domain is. It’s about where the data comes from relative to your accountability boundary.

When a dataset originates within a controlled environment — produced by a team or system within your organization’s sphere of accountability — early binding is the right tool. The producer and consumer share an organizational context. Contracts can be negotiated, enforced, and held to. The producing team can be made accountable for the schema they declare and the semantics they commit to. Prescribed context is possible because the relationship that makes it enforceable exists.

When a dataset originates outside that boundary — third-party feeds, partner data, public datasets, marketplace sources — that relationship doesn’t exist. You cannot hold an external provider to a data contract. The schema can change without notice. The semantics are inferred, not declared. This is where the Contextualize pipeline earns its place. Discovered context is the only kind available.

But the boundary is not purely organizational. Poorly governed internal data — produced by a team with no accountability to its consumers, with undocumented schemas and inconsistent definitions — is effectively uncontrolled even if it sits within the same organization. The real test is not position on an org chart. It is accountability. Early bind where accountability exists. Let the Contextualize pipeline discover where it doesn’t.

The feedback loop holds in both directions. Discovered context built up through repeated profiling, inference, and validation can graduate into a prescribed context over time. An external dataset that your organization ingests consistently enough to profile, validate, and republish as an internal data product crosses the boundary from uncontrolled to controlled at that point. The Contextualize pipeline is what makes that transition possible — and makes the resulting contract trustworthy rather than assumed.

A data environment that treats all data as early-bindable is brittle. It can only contract what it already understands, and it has no mechanism for the uncontrolled data that makes up a growing share of the analytical landscape. A data environment that treats all data as requiring discovery never formalizes what it learns into enforceable guarantees. The architecture that works reads the accountability boundary correctly and applies the right technique on both sides.

Context Propagation — The Relay, Not the Pipeline

With three pipelines now in play, the question becomes: how does context actually travel through the architecture without getting lost?

The conventional mental model is wrong. Context doesn’t travel through the data pipeline—if it did, it would be lost at every transformation step, which is precisely the Medallion erosion problem. Context travels alongside the pipeline, as metadata, lineage records, and contract provenance. The transformations change the data; the metadata preserves the meaning.

The relay works like this. Early binding stamps prescribed context at the point of origin — schema, field-level semantics, producing team ownership, quality thresholds — as an executable contract living in metadata, not column values. Lineage tooling propagates this through Bronze, Silver, and Gold, maintaining a record of the transformations applied and the contract that governed the data at each stage. The Contextualize pipeline reads that lineage as part of its inference process — understanding not just what a field looks like today, but also the history of how it arrived and the commitments made about it at the source. Validated inferences land in the Context Store, which becomes the relay’s destination: a durable, queryable record of what the data means, grounded in both original contract and accumulated lineage.

The analogy that makes this concrete is git. A file can be modified heavily across dozens of commits — refactored, renamed, moved, rewritten — but the context of how it got there is never lost, because it lives in the commit history, not in the file itself. The Gold layer is the latest commit. The lineage graph is the git log. The Context Store is the understanding you build by reading that log systematically rather than hoping the current file tells the whole story.

This reframe — from pipeline to relay — changes what data engineers are actually responsible for building. The transformations are increasingly automatable. The metadata infrastructure, the lineage graph, the Contextualize pipeline that reads it, the Context Store that accumulates from it — that is the engineering surface that requires sustained human judgment.

The Context Store as the New Engineering Surface

Which brings us to where the most consequential engineering work has migrated.

The Context Store is where business definitions live — not as documentation in a wiki, not as logic engineers have baked into a Gold table, but as versioned, validated artifacts that downstream systems can query and trust. This is where the validation workflow resolves the competing interpretations of “revenue” from Finance and Sales — not organizational politics, but a confidence-based process that determines which inference earns formalization. Where AI consumers find the grounded, stable context they need to act reliably rather than reverting to ad hoc inference.

This surface distinguishes queryable data from trustworthy data. A table can be perfectly partitioned, indexed, and replicated while being semantically wrong — built on a definition that drifted from its source contract three transformations ago and never caught because no Contextualize pipeline was watching. The Context Store is where that failure mode gets closed.

As AI generates more transformation code and AI agents consume more data at scale, the stakes of this surface rise. An agent operating on a stale or conflicting context artifact produces systematic errors rather than recoverable ones. The engineering work that governs trustworthiness — designing the trigger model for the Contextualize pipeline, structuring the labeling workflow, deciding what validation confidence threshold earns formalization, and versioning context artifacts as definitions evolve — requires human judgment at every step.

Practitioners are still working out the patterns for doing this at scale. The tooling is maturing. How organizations govern ownership of the Context Store, adjudicate conflicts between teams, and manage the graduation from discovered to prescribed context are genuinely open questions. This is where the frontier actually is.

The New Data Engineer — Context Architect

Return to the poll. 53% said architecture and trade-offs are what remain irreducibly human. In the data engineering context, ECL is what that looks like in practice.

The data engineer of the next decade owns the architecture of meaning. They design the contractual foundations at the source—executable, enforceable, versioned. They build the lineage infrastructure that carries context through every transformation layer without losing it. They design and govern the Contextualize pipeline and the Context Store — the infrastructure where inferences get built, validated, and formalized into the definitions that everything downstream depends on. They understand when to prescribe context upfront and when to let it be discovered, and they build the systems that make both possible.

But this is not only a technical role. Context erosion is as much an organizational failure as a technical one. Teams don’t share semantic definitions because no ownership model incentivizes them to do so. Nobody enforces contracts because producing teams have no accountability to the consumers they serve. In this new frame, the data engineer is the person who builds both the technical system and the organizational agreement that holds it together. They sit at the intersection of architecture and coordination — the two things the poll respondents correctly identified as irreducibly human.

The title “Data Engineer” may need an update. What we are actually describing is a Context Architect — someone whose primary material is not data movement but data meaning, not pipelines but provenance, not transformation logic but the semantic infrastructure that makes transformation logic trustworthy.

An Open Frontier

I want to be honest about what ECL is and what it isn’t. It is a reorientation — a way of thinking about what the work actually is, now that AI is handling more of what the work used to look like. It is not a finished methodology. The tooling that links early binding contracts to the Contextualize pipeline and Context Store is still maturing. The organizational patterns for governing who owns the Context Store, how conflicts between teams get adjudicated, and how discovered context earns formalization don’t yet have established templates. Practitioners are working out the engineering patterns for building contextual pipelines that operate reliably at scale in production environments right now, figuring it out as they go.

That’s precisely what makes this moment worth paying close attention to. The frontier is genuinely open. The practitioners who invest in the architectural and organizational work of context — who treat contracts as executable infrastructure, who build lineage as a first-class engineering concern, who govern the Contextualize pipeline and Context Store as seriously as they once owned the ETL pipeline — will define the discipline for the decade ahead.

The 53% who said architecture and trade-offs are irreducibly human were right. We didn’t yet know which architecture, or which trade-offs.

Now we do.

Data Engineering Weekly #258

Ananth Packkildurai — Mon, 23 Feb 2026 01:00:43 GMT

AI is moving fast. Is your data platform ready?

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the AI Modernization Guide

Garry Tan: Half the AI Agent Market Is One Category. The Rest Is Wide Open

AI Agents thrive in RL environments with a verifiable target and quick feedback. Software manufacturing is a perfect model for such an environment, but the challenge persists in other categories. It will be an exciting decade for software engineering as we build new infrastructure that we never imagined.

https://garryslist.org/posts/half-the-ai-agent-market-is-one-category-the-rest-is-wide-open

LangChain: How to Use Memory in Agent Builder

Agents fail to improve over time when they treat every conversation as stateless and discard learned preferences or workflows. The article explains how LangChain’s Agent Builder implements short-term and long-term memory as a filesystem of Markdown files, enabling persistent instructions and reusable skills. Explicit memory updates, modular skill loading, and direct file editing enable agents to reliably evolve behavior without increasing core prompt complexity.

https://blog.langchain.com/how-to-use-memory-in-agent-builder/

LinkedIn: Scaling LLM-Based ranking systems with SGLang at LinkedIn

LLM-based ranking systems face strict latency and concurrency constraints because they score thousands of items per query without requiring text generation. The article explains how LinkedIn optimized SGLang for prefill-only ranking through batching improvements, scoring-only execution paths, prefix KV reuse, and Python runtime parallelization.

https://www.linkedin.com/blog/engineering/ai/scaling-llm-based-ranking-systems-with-sglang-at-linkedin

Spotify: Our Multi-Agent Architecture for Smarter Advertising

Fragmented decision logic across buying channels prevented Spotify from translating high-level campaign goals into unified execution plans. The article explains how Spotify built Ads AI, a multi-agent orchestration layer with intent routing, specialized resolution agents, and data-grounded media planning using real-time tool integration. The architecture reduced campaign setup time from minutes to seconds, simplified user inputs, and grounded recommendations in historical performance data.

https://engineering.atspotify.com/2026/2/our-multi-agent-architecture-for-smarter-advertising

Uber: Database Federation: Decentralized and ACL-Compliant Hive™ Databases

Monolithic Hive warehouses create shared-fate outages, resource contention, and weak governance when thousands of datasets share a single database. The article explains how Uber implemented database federation by reorganizing datasets into domain-specific units, updating Hive Metastore pointers to avoid data duplication, and deploying both real-time and batch synchronizers to maintain consistency. The decentralized architecture improves ACL compliance, strengthens resource isolation, and reclaims storage while enabling zero-downtime migration at the petabyte scale.

https://www.uber.com/en-IN/blog/database-federation/

Anton Borisov: AutoMQ: Shared Storage Architecture Deep Dive

Kafka’s shared-nothing architecture imposes high replication costs, slow failover, and tight coupling between storage and compute. The article explains how AutoMQ replaces local disk replication with S3-backed shared storage, using layered abstractions, WAL batching, metadata-driven ownership, and epoch fencing to enable stateless brokers and zero-copy failover. AutoMQ design eliminates the 3x replication tax and simplifies scaling to “add compute,” while accepting higher cold-read latency from object storage.

https://medium.com/fresha-data-engineering/automq-shared-storage-architecture-deep-dive-043c5226847e

Pinterest: Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest

OOM in Spark jobs is an infamous issue across data processing, creating operational overhead and inefficient cluster utilization. The article explains how Auto Memory Retries dynamically adjusts executor resources by retrying failed tasks with higher CPU allocation or larger executors through modified Spark resource profiles. The elastic strategy reduced OOM failures by 96%, lowered infrastructure costs by avoiding over-provisioning, and improved overall pipeline reliability.

https://medium.com/pinterest-engineering/drastically-reducing-out-of-memory-errors-in-apache-spark-at-pinterest-c55d7dac2257

StarTree: Consistent, Scalable Compaction for Real-Time Upserts in Apache Pinot

Near-Real-Time upsert is my favorite subject to study, and I've worked with many OLAP engines. Apache Pinot always stands out for its flexible indexing and fast upsert capabilities. The article explains how StarTree’s SegmentRefreshTask compacts segments in the background by merging only valid records and ensuring atomic visibility with bitmap-based consistency controls. The approach reduces storage costs, supports sustained high ingestion rates, and maintains predictable query latency at a billion-key scale.

https://startree.ai/resources/upserts-compaction-in-apache-pinot-startree/

Zepto: Debezium at Scale: An Open Source CDC Story from Zepto

High-velocity CDC pipelines can overwhelm downstream databases due to redundant updates and MVCC-induced write amplification. The article explains how Zepto optimized Debezium by introducing an in-memory reduction buffer to collapse duplicate updates and a Postgres UNNEST-based batching strategy to reduce parsing overhead. These improvements stabilized CPU and I/O usage, eliminated replication lag during peak traffic, and ensured the database processes only final record states.

https://blog.zeptonow.com/debezium-at-scale-an-open-source-cdc-story-from-zepto-aa4b12e32bf7

Data Engineering Weekly

Data Engineering Weekly #273

How to Build a Data Platform

Fei-Fei Li: A Functional Taxonomy of World Models

Gorgias: When Event Time Meets Reality: Lessons from Building Billing on Apache Flink

Cloudflare: Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse.

Sponsored: AI Modernization Guide

Helpshift: Migrating from a Monolithic Orchestrator to Apache Airflow

Indeed: Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search

Netflix: Dynamic Repartitioning for Time Series Workloads

Jack Vanlightly: Broker-Visible vs Client-Local Parallelism

Criteo: Introducing CLEPR, our model for semantic understanding

WMG: Why we shrank our TimescaleDB chunks from 30 days to 7

Data Engineering Weekly #272

How to Build a Data Platform

Netflix: High-Throughput Graph Abstraction at Netflix

Slack: Slack AI - The Path to Multi-Cloud

Sponsored: Agents for Data Engineering

Uber: Solving the Identity Crisis for AI Agents

Mehul Batra: A field journal on Ray Data and Daft for multimodal data lake

Sponsored: AI Modernization Guide

LakeOps: Routing Multiple Query Engines with Iceberg

Rion Williams: Prepare for Launch: Enrichment Strategies for Apache Flink

Agoda: How Agoda Simulates Booking Flows to Test Flight Integrations

Databricks: Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs

Giannis Polyzos: When Tables Became the Language of Time

Data Engineering Weekly #271

How to Build a Data Platform

Netflix: The Evolution of Cassandra Data Movement at Netflix

Grab: The Hugo evolution: Engineering Grab’s unified, one-click data ingestion platform with Apache Flink

Sponsored: Agents for Data Engineering

Meta: A Blueprint for Valuing Content When A/B Tests Are Not an Option

Uber: Scaling Real-Time Traffic Forecasting with a Graph-Aware Transformer

Sponsored: Free Course: AI-Driven Data Engineering

Airbnb: Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure

Pinterest: Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

Yelp: How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%

LinkedIn: Crosscheck: Benchmarking AI Models in the Real World

Jack Vanlightly: Introducing Dimster, a performance benchmarking tool for Apache Kafka

Data Engineering Weekly #270

The Data Platform Fundamentals Guide

Airbnb: Viaduct 1.0 and the future of Airbnb’s data mesh

Netflix: Data Projects: Managing Data Assets at Netflix Scale

Sponsored: Agents for Data Engineering

Meta: Migrating Data Ingestion Systems at Meta Scale

Databricks: The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available

Sponsored: Free Course - AI-Driven Data Engineering

Eric Sun: A Query Proxy for Analytical and Fast Data

The Craft: How we rebuilt search ranking at Faire with deep learning

Alibaba: When MySQL Meets the Columnar Storage Engine DuckDB in the AI Era

Marc Bowes: Aurora DSQL: Meet Coupler

Pete: Full-Text Search with DuckDB

Data Engineering Weekly #269

The Data Platform Fundamentals Guide

Meta: How We Built an AI Second Brain for 60K Knowledge Workers

Salesforce: How Informatica Built a Multi-Agent AI System to Reduce Data Workflows from Months to Days

Netflix: Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Sponsored: Free Course: AI-Driven Data Engineering

Confluent: Stream Processing vs. Real-Time OLAP: Flink, ClickHouse & Pinot Compared

Kirill Bobrov: The Power of Data Sketches: A Comprehensive Guide

Whatnot: The ML Feature Pipeline That Got Slower and No One Noticed

Pinterest: Enhancing Ad Relevance: Integrating Real-Time Context into Sequential Recommender Models

Grab: Enhancing Flink deployment with shadow testing.

Halodoc: Building Self-Healing Data Pipelines at Halodoc

Data Engineering Weekly #268

Free Course: AI-Driven Data Engineering

Event Highlight: Don't Miss AI Council

Grab: Data Mesh at Grab Part II: The Foundational Tools behind Certification

Doug Turnbull: Can agents replace the search stack?

Pinterest: Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer

Sponsored: The AI Modernization Guide

Pinterest: From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest

FiveTran: How we accelerated transpilation by compiling SQLGlot with mypyc

Robin Moffatt: Materialized Tables in Apache Flink

Alexey Makhotkin: 5NF and Database Design

ultrathink: SQLite in Production: Lessons from Running a Store on a Single File

Capital One: Spark tuning: executor optimization for performance

Data Engineering Weekly #267

Free Course: AI-Driven Data Engineering

Monzo: A “meshy” approach to Data: Enabling 100+ teams to build Data Models