Data Engineering Weekly

Data Engineering Weekly #280

Ananth Packkildurai — Mon, 27 Jul 2026 03:37:19 GMT

Editor’s Note: leetdata.ai & aidataengineer.io updates

Last week we were busy fixing some bugs and upgrading a few features.

aidataengineer.io now supports search over DEW articles & a seamless browsing experience on all the articles published so far.
We fixed bugs in subscriptions in leetdata.ai, and we're thrilled to see more paid subscribers there. Python code practices now tuned for pandas.

We are actively working on two things.

Cohorts for leetdata.ai: traditional cohort focus on a tool, but we are building an agent to build cohorts reflecting real-world design by mining the DEW articles.
Simulation Learning model, where a real-world outage/scenario is simulated, and you work to learn it. We are actively looking to partner with data tooling companies on this - more exciting development coming soon!!!

Joe Reis: To Every Agent Its Own Database

Are we prepared for the data infrastructure shake-up for the agentic workload? The author proposed a DuckDB-based solution with each agent having its own copy of the data to work with. Though I agree there are a few versioning solutions available, they are still built for humans to resolve conflicts. Data can’t afford to have multiple versions of the same facts locked by different agents. I believe this is the area where next-generation data tools will invest a lot.

https://joereis.substack.com/p/to-every-agent-its-own-database.

Datafold: The Modern Data Stack - Open-source edition

There has been a lot of debate about open source last week. I found this to be an excellent work on gathering all the open-source tools remaining in the modern data stack.

https://www.datafold.com/blog/the-modern-data-stack-open-source-edition/

OlegWock: Guide to data tools landscape for developers

Another existing playbook talking about the data tools landscape. If you’re a beginner to the data industry, this is an excellent overview to understand the landscape of data engineering.

https://sinja.io/blog/data-landscape-guide-for-developers

Housing: Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently)

I love the concept of a single source of truth. Too bad we have to wait until we are living in the singularity to actually see it work.

The author talks about the importance of having a certification process for standard metrics and the organizational alignment over it. The certification model is something that should be baked into every data product's deliverables.

https://medium.com/engineering-housing/why-most-single-source-of-truth-initiatives-fail-and-what-successful-teams-do-differently-7bf4846e4b82

Hex: The life and times of data teams in the AI era

The article reiterates the importance of data quality in the AI era and how it will play a vital role in data engineering. Getting data into people at the right time, with high quality, will still be a challenge. The challenge is multi-fold as traditional certified dashboard building is slowly fading, and headless chat focuses on intelligence emerging.

https://hex.tech/blog/data-teams-in-ai-era/

Bojan Ciric: The Knowledge Spine: Why Your Ontology Needs to Grow a Backbone

Ontologies fail when they remain documentation rather than production infrastructure. The author’s “knowledge spine” makes the useful shift: version concepts like code, map them virtually to governed sources, federate domain graphs, and make agents resolve meaning through the same queryable backbone. The architecture is persuasive, but central semantics cannot manufacture consensus; success still depends on domain ownership, tested mappings, and stewards empowered to resolve conflicting definitions.

https://www.linkedin.com/pulse/knowledge-spine-why-your-ontology-needs-grow-backbone-bojan-ciric-ykn9c/

JioHotstar: Journey of an Ad Request: The Hidden Engineering

Ad-tech is a compact lesson in real-time data systems: selecting an ad is not a ranking problem alone, but a constrained allocation problem under a 100 ms deadline. The author describes JioHotstar’s blend of eligibility filtering, frequency caps, pacing, waterfall priority, and uncertain programmatic fill. The durable data-engineering challenge sits after serving—event pipelines must reconcile every decision, impression, and conversion into trustworthy delivery and attribution metrics.

https://blog.hotstar.com/journey-of-an-ad-request-the-hidden-engineering-e7b7f9921e46

Netflix: In-House LLM Serving at Netflix

Netflix’s in-house LLM platform is compelling because it refuses to treat LLMs as a separate product category. The hard-won lesson is that model quality lives in serving details: guided decoding, batch-aware constraint logic, API compatibility, and observability determine whether a prototype can become dependable infrastructure.

https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c

Airbnb: Personalizing Airbnb search by learning from the guest journey

Airbnb writes about compressing seven years of high-signal history and 21 days of views into daily guest embeddings, then combines them with live queries at serving time. This is an exciting area of data pipelining, thinking in terms of embedding models instead of relational.

https://medium.com/airbnb-engineering/personalizing-airbnb-search-by-learning-from-the-guest-journey-bcefd1915624

Artem Krylysov: How MVCC and Transactions Work in RocksDB

RocksDB’s LSM design supplies the raw material for MVCC—immutable versions and sequence numbers—but snapshots, SuperVersions, and compaction rules make concurrent reads safe in practice. The crucial warning is that native transactions are a local, configurable layer, not a blanket serializability guarantee: optimistic commits favor low contention, pessimistic locks favor hot keys, and range phantoms still need explicit range locking.

https://artem.krylysov.com/blog/2026/07/23/how-mvcc-and-transactions-work-in-rocksdb/

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

Data Engineering Weekly #279

Ananth Packkildurai — Mon, 20 Jul 2026 04:07:20 GMT

How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating under a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your copy now

William Inmon: Data Management In The Age Of AI

The author reminds us that generative AI does not diminish data management; it shifts the control point from database schemas to the text and meaning fed into the model. The case for tying ontologies and taxonomies to the enterprise logical data model is sound, but filtering text alone is not governance—provenance, permissions, and feedback must travel with the context.

https://williaminmon.substack.com/p/data-management-in-the-age-of-ai?r=22julh

Snowflake: Apache Ossie (Incubating): The New Name for Open Semantic Interchange

Apache Ossie’s incubation is an interesting development. I’ve been part of a few sync-ups, and I’m thrilled to see how this is evolving. It is more than a rename: it gives the still-fragmented semantic-layer market a neutral home for portable metrics, dimensions, and business relationships.

https://www.snowflake.com/en/blog/apache-ossie-open-semantic-interchange-incubator/

Cerebras: How We Built Our Knowledge Base

I often make a point that building a knowledge base from the vast corpus of data is a typical data engineering problem, and no, data catalogs have nothing to do with knowledge engineering. Cerebras makes a compelling case for the same, ingesting every source into a single Postgres embeddings table, then fusing full-text, vector, IDF, and age-decay signals using reciprocal rank fusion.

https://www.cerebras.ai/blog/how-we-built-our-knowledge-base

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Honeycomb: Transforming How We Run Kafka at Honeycomb

Kafka migrations are constrained less by the target platform than by the semantics hidden in consumers. Honeycomb’s move from Confluent/ZooKeeper on EC2 to KRaft on EKS was shaped by internally checkpointed offsets, rehearsed rollbacks, and telemetry-defined pause points. The strongest lesson is sociotechnical: operational independence is real only when runbooks and expertise survive the original expert—and when customer communication is treated as part of the cutover.

https://www.honeycomb.io/blog/transforming-how-we-run-kafka-honeycomb

StarTree: Designing Apache Pinot for High Availability in the Cloud

StarTree writes about designing HA for Apache Pinot, highlighting that Pinot availability depends on Kubernetes placement, segment replica groups, broker health routing, and durable deep storage sharing the same failure-domain model. The key test is better than any replica count: can the serving path answer queries before Kubernetes rebuilds the failed zone?

https://startree.ai/resources/apache-pinot-for-high-availability/

Razorpay: How We Refresh Razorpay’s Data Warehouse 10x Faster with Graphs and Indexes

Razorpay’s 10x warehouse refresh gain came from rejecting two expensive defaults: rebuilding every denormalized Fact and streaming every join. The warehouse models each Fact as a dependency graph, uses compact secondary indexes to locate historical lake partitions, and leaves high-cardinality dimensions to bucketed Iceberg runtime joins. The deeper lesson is excellent: incremental batch is often the practical middle ground when streaming state and materialization amplify each other’s cost.

https://engineering.razorpay.com/how-we-refresh-razorpays-data-warehouse-10x-faster-with-graphs-and-indexes-538abc244703

Scribd: Stop Building Models. Start Building Systems.

Scribd explains how a fast model, a sampled judge pass, a focused SME review, and a versioned golden dataset enabled a three-person team to classify roughly 375 million items in two months. The design is intentionally replaceable: models and prompts can change, while disagreements, rationales, and regression data compound into durable operational knowledge.

https://tech.scribd.com/blog/2026/fast-llm-human-in-the-loop-classification.html

Matt Butrovich: Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet

The Iceberg community keeps Java planning as the oracle while DataFusion Comet executes reads through Iceberg Rust, turning nearly 10,000 Spark tests into cross-language differential tests. Native acceleration still falls back for writes and format-v3 tables, yet every exposed gap improves the shared Rust library rather than a proprietary fast path—an exciting development to adopt.

https://iceberg.apache.org/blog/accelerating-iceberg-rust-development-with-datafusion-comet/

Artur Borycki: Puffin-Backed Vector Indexes: Attaching Approximate Nearest Neighbor Indexes to Apache Iceberg Snapshots for Compute-Disaggregated Query Engines

The author proposes a more interesting approach that attaches sharded Vamana (Graph-based ANN) indexes to Iceberg snapshots via Puffin, inheriting atomic commits, time travel, and garbage collection while keeping executors disposable. The architecture is elegant for analytical retrieval, but still a design-stage bet: benchmarks remain projected, while large index storage, snapshot-bound freshness, and shard-dependent recall separate it from low-latency serving.

https://arxiv.org/html/2606.04196v1

Konstantinos Chantsaris: Designing a Spatial ETL Pipeline in PostGIS

The author shows how Spatial ETL fails quietly when geometry semantics are treated like ordinary columns: normalize coordinate systems, keep fragment-level physical measures separate from commune-level population measures, and index the exact geographic expression used in distance queries. The most valuable lesson is not PostGIS syntax; projections, spatial granularity, and analytical denominators are part of the data contract because each can produce believable nonsense.

https://medium.com/@kchantsaris_84495/designing-a-spatial-etl-pipeline-in-postgis-72565b53aeb8

Data Engineering Weekly #278

Ananth Packkildurai — Mon, 13 Jul 2026 02:44:42 GMT

Deep Dive: See how Dagster uses AI internally

What does it actually look like to use AI as part of your day-to-day engineering workflow? At Dagster, we’ve developed workflows that use AI to quickly validate concepts, identify dead ends, and focus our engineering time on what matters most.

Join our live session this week where we’ll share the prompts, processes, and lessons we’ve learned from using AI to accelerate product and engineering decisions.

Register now

Editor’s Note: Updates on leetdata.ai

Since last week, we have launched two new features in leetdata.ai

Experiences - for the data engineers to share their interview experiences anonymously to help their fellow data engineers.
Systems: An in-depth, interview-focused learning experience covering the various tools we use in data engineering. We started with Spark, Kafka, data pipeline, and data warehouse. We will expand this more in the coming releases.

Here is what the current leaderboard looks like; visit leetdata.ai

BAIR: Intelligence is Free, Now What? Data Systems for, of, and by Agents

Agents will soon become the dominant workload for data systems. An agent querying a database doesn’t behave like a person or a BI tool. These two properties open a new era of data engineering research for building data systems for, of, and by agents. A thought-provoking article and a must-read for system builders.

https://bair.berkeley.edu/blog/2026/07/07/intelligence-is-free-now-what/

Bharath Vadhoola: The language tax on AI

Another article that made me think a lot this week is the language tax on AI.

Years ago, I was fortunate to talk to someone from Guinea during an Uber ride. He talked about the invention of the Adlam script (written form) for the Fula language (Spoken form) and the social impact of having a written language for the Fulani people, and his own life.

To achieve global social impact, we must ensure equality across languages in the AI era, specifically by reducing the higher token costs across languages.

https://www.linkedin.com/pulse/language-tax-ai-bharath-vadhoola-zhghc/

Microsoft: Flint - A visualization language for the AI era

As AI starts to impact the BI & visualization layer, one of the challenges is building rich visualizations within existing BI engines. I started building an allergy to mermaid diagrams; d2 is somewhat tolerable. Flint seems like a welcome addition to try out.

https://www.microsoft.com/en-us/research/blog/flint-a-visualization-language-for-the-ai-era/

Netflix: Measuring the Impact of Personalized Recommendations

Both attribution and measurement are exciting challenges in product analytics. Netflix writes about building a low-rank choice model to simulate dynamic member preferences. It validates this using small-scale A/B nudge tests, demonstrating that personalization drives engagement primarily through precise targeting of mid-popularity titles rather than sheer visual exposure.

https://netflixtechblog.medium.com/measuring-the-impact-of-personalized-recommendations-4c26be3a4d96

Grab: Scaling Grab’s Data Lake: Our journey to Apache Iceberg adoption

Having a table management system on top of the object storage brings much-needed integrity and efficiency in data management at scale. Grab writes a case study on its adoption of Iceberg and compares the advantages it brings versus the data lake model for managing parquet files in object storage.

https://engineering.grab.com/our-journey-to-apache-iceberg-adoption

Lyft: From Day 1 to Production: Building Lyft’s Analytics & Rides Intelligence Assistant as Onboarding Project

Natural-language analytics only becomes useful when it survives the unglamorous production work around it. Lyft’s ARIA story shows that the gap between a LangGraph/SQL prototype and an internal data product is auth, observability, state management, and responsive streaming—not another model prompt. Shipping it as a three-week onboarding project is a smart test of whether a platform’s documentation and paved roads actually work.

https://eng.lyft.com/from-day-1-to-production-building-lyfts-analytics-rides-intelligence-assistant-as-onboarding-5c5643c192d9

AWS: MCP tool design: Practical approaches and tradeoffs

MCP servers fail less often due to protocol limits than due to treating existing APIs as agent-ready. AWS lays out the real tradeoff: richer descriptions and schemas improve tool selection, but they consume the same context that agents need to reason. The most useful design principle is progressive disclosure—defaults and constrained schemas for common paths, taxonomy or agentic control only when ambiguity justifies the latency and cost.

https://aws.amazon.com/blogs/machine-learning/mcp-tool-design-practical-approaches-and-tradeoffs/

Santosh Shinde: The Medallion Architecture Reconsidered: What It Solved, and Where It’s Cracking

Medallion’s real contribution was imposing a recoverable path from raw data to trusted analytics; the mistake was treating Bronze, Silver, and Gold as three mandatory physical destinations. The author argues convincingly that AI workloads expose their one-dimensional model: governed data products—not a longer chain of medals—must serve BI, features, vectors, and agents. Keep immutable replay and logical refinement, but let contracts and fitness-for-purpose define the boundary.

https://levelup.gitconnected.com/the-medallion-architecture-reconsidered-what-it-solved-and-where-its-cracking-d6073aa1b0fd

Praveen Krishnan: Direct Lake is Magic, Until It Isn’t: How We Handled 1.5 Billion Rows Without Crashing Power BI

The Direct Lake promise is compelling, but this is a familiar lakehouse lesson: query engines inherit every physical-design decision beneath their semantic layer. The author’s 1.5-billion-row case shows why capacity sizing, column discipline, partitioning, and file layout must be tested at production scale—and why silent DirectQuery fallback belongs in the release gate. Performance features are only magical when their operating mode is observable.

https://medium.com/@praveenkrishnan18k/direct-lake-is-magic-until-it-isnt-how-we-handled-1-5-billion-rows-without-crashing-power-bi-a265ebf8d097

Beyond Redaction: Anatomy of a Privacy-Safe Data Platform

Ananth Packkildurai — Fri, 10 Jul 2026 10:20:21 GMT

The hard problem is not making personal data disappear. It is preserving the minimum useful properties for an allowed purpose, limiting who can link the data, and producing evidence that the controls actually ran.

The most useful question to ask of a privacy platform is not “Can it hide this column?” Almost every platform can. Ask instead: what property must this recipient retain, for which permitted purpose, and what evidence will prove that the platform enforced the decision?

That framing changes the architecture. A data scientist may need stable equality to deduplicate records. A support agent may need a governed path to re-identify one customer. A partner may need aggregates but must not be able to correlate its copy with another partner’s. An engineer provisioning a test environment may need intact foreign key constraints but no production identities. Those are different jobs. A single control called “mask” cannot answer all of them safely.

Redaction, encryption, tokenization, governed views, aggregation, synthetic data, and clean-room-style access are not competitors in a feature checklist. They are controls with different privacy and utility properties. The mature design selects among them deliberately, then makes the decision inspectable.

One terminology point matters throughout this article: a deterministic token is usually pseudonymous, not automatically anonymous. It can lower exposure while retaining the ability to link records; depending on the recipient, auxiliary data, and access to the key or vault, it may remain personal data under privacy law. That is useful—not a failure—but it means “PII-free” is usually too strong a claim. The UK ICO makes the distinction explicit in its pseudonymization guidance.

This is a technical reference architecture, not legal advice. Legal and privacy teams must review each production policy against the organization’s jurisdictions, contracts, and processing purposes.

TL;DR: Match the Control to the Utility

Need to join approved internal tables?

Use purpose-bound deterministic tokenization. It preserves equality inside the approved domain while keeping raw identifiers out of the analytical copy.

An external partner needs insight?

Use aggregation, a protected query, or a clean-room workflow. These approaches deliver useful results while reducing direct linkability and release risk.

Authorized staff need to re-identify a person?

Use vault-backed reversible tokenization. It makes recovery permissioned, purpose-bound, and auditable.

Does an auditor need proof of a release?

Use a signed manifest plus a protected audit trail. Together, they connect the artifact to its policy, transformation, approvals, and evidence of enforcement.

An LLM or RAG workflow may expose data?

Use retrieval authorization plus output guardrails. Enforce access outside the model and check sensitive output before release.

The five-stage pipeline: discover, authorize, protect, use, prove

1. Discover and classify candidates—not certainty

Discovery begins with metadata: connectors, schemas, column names, lineage, ownership, and tightly controlled sampling. A practical classifier layers cheap signals over expensive ones:

Column-name and metadata heuristics quickly find well-labeled fields.
Pattern recognizers find structured identifiers, including region-specific forms.
NLP and contextual analysis identify names, addresses, and free text that do not match a regex.

The output should be a set of candidates: data type, sensitivity tier, confidence, evidence, recognizer version, and review state. A boolean is_pii loses the uncertainty that operators need. Low confidence may block a high-risk export, route a field to human review, or narrow the downstream purpose; it should not silently become “not sensitive.”

Recognizer count is a weak buying signal. Ask what runs by default, which models or language packs are optional, what data leaves the environment during analysis, and how the system measures false positives and false negatives on your schemas. Discovery is important, but it is not proof that an output is free of personal data.

There is a further operational trap: classification drifts. New fields arrive, owners rename columns, free-text use expands, and upstream sources change their formats.

Treat classification as a versioned, recurring control rather than a one-time migration.

A useful platform records the scan scope and timestamp, compares the latest classification with the last approved state, and sends material changes to a review queue. Without that loop, the policy engine makes precise decisions based on stale assumptions.

2. Authorize the use, not merely the field

Classification tells us what a value may be. Authorization determines whether a particular actor may use it at this time. Good policy evaluation considers at least:

data class and sensitivity;
principal, role, and tenant;
declared purpose and retention window;
recipient and destination;
jurisdiction, contract, and approved legal basis;
consent or opt-out state where applicable; and
The required control and approval path.

Consent is one input, not a universal switch. India’s DPDP Act permits processing based on consent or certain legitimate uses; HIPAA also permits defined treatment, payment, and health care operations uses without individual authorization. A policy template can be a useful accelerator, but it cannot turn a regulation name into executable compliance on its own. See the DPDP Act and HHS guidance on HIPAA uses and disclosures.

The practical pattern is policy-as-data with versioning, approval, simulation, and rollback.

A useful result is not just “allow” or “deny,” but: allow this role to use a pseudonymous join key for fraud analysis until this date; deny raw identifier export; record the decision and policy version.

Enforce at the point of use

The policy engine should express a decision; the platform should enforce it at a dependable point in the data path. For warehouse workloads, teams can use native row-access policies, column masking policies, tags, and governed views. For an API, a policy-enforcement point can filter records before serialization. For a file release, the export worker and destination access policy enforce the decision. The central service should not become an overly optimistic proxy that expects every downstream query to request permission.

This separation also makes change management safer. The lifecycle is propose → simulate → approve → enforce → verify → rollback. Simulation should answer concrete questions before a policy goes live: which tables, columns, users, and jobs would change; which scheduled workloads would fail; and whether a new rule would materially raise or lower exposure. A denial that breaks a critical batch without warning is not evidence of maturity. It is a reason teams later bypass the privacy control.

3. Select the smallest control that preserves the required utility

The distinction that matters is not “redaction versus tokenization.” It is which property of the original value the recipient truly needs.

Redaction, therefore, remains a valid privacy strategy. Use it when no downstream utility exists. The mistake is presenting it as sufficient for every purpose—or treating a join-preserving token as if it automatically makes sharing safe.

Control selection should also reflect the recipient. A data scientist working inside a controlled tenant may need a stable token. An external partner may instead need aggregates, a protected query interface, or a narrowly scoped clean-room workflow. The policy can therefore drop the same field in one export, tokenize it in another, and grant access through a governed view in a third. Privacy architecture is a decision system, not a permanent label attached to a column.

Deterministic tokenization: preserve equality, bound linkage

Deterministic tokenization is powerful because it can preserve equality: the same canonical input produces the same token inside an approved domain. Two tables can then join without carrying the direct identifier alongside the analytical copy.

An illustrative irreversible token reference might look like this:

def token_reference(value, tenant_id, entity_type, purpose_domain, key):
    canonical = canonicalize_by_type(value, entity_type)
    message = canonical_json({
        "version": 1,
        "tenant": tenant_id,
        "entity": entity_type,
        "purpose_domain": purpose_domain,
        "value": canonical,
    }).encode()
    mac = hmac.new(key, message, hashlib.sha256).digest()
    return "tok_v1_" + base64.urlsafe_b64encode(mac[:20]).decode().rstrip("=")

This is a design sketch, not a drop-in cryptographic service. Production code needs keys from a managed KMS or HSM, key identifiers and rotation, authenticated service boundaries, telemetry that does not log the input, explicit token-format contracts, and collision detection appropriate to the token length and scale. Truncating HMAC-SHA-256 to 64 bits, for example, creates an avoidable false-join risk in large datasets.

Tenant isolation belongs in the same design, not as a metadata afterthought. Derive or select keys by tenant and purpose, bind tenant and entity context into the token request, verify the caller’s tenant before the lookup, and scope every vault read and write. A tenant_id column is useful context, but it is not an isolation boundary by itself. The identity layer, service layer, database access pattern, and key hierarchy must agree on the same tenant boundary.

Canonicalization is a product decision, not just string cleanup. Teams may normalize phone numbers to a canonical international form, but they should not automatically discard email local-part case; customer IDs need a source-system contract. If two pipelines canonicalize differently, the desired join fails. If they canonicalize too aggressively, distinct people may collapse into one identity.

Three modes, three different promises

Irreversible pseudonymization uses a keyed deterministic reference with no recovery path. Use it when an approved use needs equality but excludes re-identification.

Reversible tokenization normally needs a vault. One sound pattern maps an opaque, stable token to an encrypted original value; the stored envelope includes a unique nonce, authentication tag, key ID, and associated data such as tenant, entity, and token version. Detokenization is a privileged workflow: strong authentication, a purpose check, perhaps dual approval, a rate limit, and an immutable audit event. A bare HMAC has no reverse operation.

Per-recipient tokens add a recipient or purpose domain. This prevents two recipients from directly correlating records by token equality while preserving joins inside each recipient’s domain. It does not preclude correlation: recipients may have auxiliary data, collude, or use other quasi-identifiers. Describe the property precisely: it prevents direct correlation through this token namespace.

The legal and operational consequences are the same across all modes: tokens reduce exposure but do not end the privacy analysis. NIST similarly advises organizations to evaluate disclosure risk and, where appropriate, conduct re-identification studies rather than equating a transformation with de-identification. NIST SP 800-188 is a useful baseline.

Encryption, format preservation, and the names that must stay honest

Hashes, encryption, masking, and tokenization are not interchangeable.

A hash or HMAC is one-way. It is not encryption.
AES-GCM provides authenticated encryption when implemented with sound key and nonce management; it is not format-preserving.
Format-preserving encryption (FPE) retains a constrained output format and can help retrofit legacy systems, but it is still encryption and may preserve properties that create re-identification risk.
SQL substring replacement or repeated characters are masking, not FPE.

Do not present FF3-1 as a current default. NIST’s current public draft revision of SP 800-38G removes FF3 and focuses on FF1 after cryptanalysis exposed weaknesses, prompting tighter domain-size requirements. Use a reviewed implementation appropriate to the data domain and current organizational policy; do not build a custom FPE scheme. NIST’s draft revision explains the change.

Safe exchange is a controlled release, not a file write

Teams often overclaim the export stage, even when they apply sensible controls. A safer release workflow is:

Authorize the recipient, purpose, destination, and retention period.
Transform only the approved fields and preserve only the properties the recipient needs.
Re-scan with versioned detectors and block releases on policy-defined findings.
Run a quality check: referential integrity, token-domain consistency, and expected row/column counts.
Produce a signed manifest with the file hash, transformation and policy versions, detector configuration, findings summary, approver, and expiry.
Deliver through a revocable access channel where possible, then record receipt and revocation events.

The manifest should say what it actually proves: this signed artifact had this hash and these checks under this policy. It should not claim that a classifier proved the data contains no PII. A scan has false negatives; a signature proves artifact integrity and provenance, not anonymity.

The recipient is part of the security model too. If a team places a package in an unrestricted object store, a signed manifest does not make it safe. Prefer time-bound delivery, recipient-specific access, encryption the recipient can use without exposing the sender’s master keys, revocation where the sharing model permits, and deletion/retention confirmation. For some high-risk use cases, a controlled query environment serves users better than a portable copy.

This distinction matters in non-production provisioning. Stable, purpose-bound tokens can preserve foreign-key relationships across users and orders, producing realistic test data without distributing direct identifiers. But the resulting data can still be linkable, so teams must treat it as pseudonymous data: tenant-scoped, time-limited, access-controlled, and deleted when the environment is retired.

Evidence first: a hash chain is a building block, not a guarantee

Evidence connects policy to reality: policy version → request → decision → transformation → enforcement result → delivered artifact. That lineage is more useful than a compliance dashboard because it answers a concrete audit question about a concrete event.

A hash-chained log detects changes to a retained sequence, but it does not defend itself if an attacker can rewrite the database and recompute every subsequent hash. A credible design adds controls such as:

write-once or independently held storage;
signatures and externally verifiable checkpoints;
separation between the audited service and the log store;
time synchronization, retention rules, access monitoring, and tested verification; and
Periodic evidence review by someone other than the system owner.

This is consistent with NIST’s audit guidance, which calls for protecting audit information and logging tools from unauthorized access, modification, and deletion. NIST SP 800-53, AU-9 provides the relevant control family.

Evidence is not the same as certification. A dashboard can provide valuable operational evidence, but it does not by itself establish SOC 2, HIPAA, GDPR, or any other compliance outcome. The strongest evidence is scoped and reproducible: a reviewer can take one delivery ID, recover the relevant policy and approvals, verify the signed artifact and log checkpoint, and determine which controls governed the delivery. That is a far more useful promise than “compliance reporting.”

The AI boundary: output scanning is necessary, but not sufficient

Large language model applications add another exposure path: retrieved material, user input, tool calls, cache state, and generated text. An output scanner can redact or block a detected identifier before display, and it deserves a place in the request path. It is not a permission system.

The safer design applies controls before, during, and after generation:

retrieve only documents the requesting principal may access;
enforce tenant and purpose filters outside the model;
Minimize sensitive content before it enters prompts where possible;
treat tool results and model output as untrusted until validated;
restrict tool egress and log the exact data supplied to the model; and
Continuously test prompt injection, cross-tenant retrieval, and sensitive-output failures.

This matters especially for agents. A Model Context Protocol tool such as classify_text can be useful, but an agent’s decision to call it is not an enforcement boundary; prompt injection or a flawed plan can bypass it. The policy enforcement point must sit outside the model and apply even when the model is wrong or hostile input influences it. OWASP’s Sensitive Information Disclosure guidance makes the same defense-in-depth case.

What to test in a technical evaluation

Marketing terms are cheap; executable tests are better. Ask a platform team to demonstrate these scenarios in a controlled environment:

Utility: Can two approved datasets join on a stable pseudonym while direct identifiers stay out of the analytical copy?
Bounded linkage: Does the same value produce different tokens for two recipients, and can the recipients still join only within their own domains?
Governed recovery: Who can detokenize, for what purpose, with what approval, rate limit, and evidence? Demonstrate both denial and approval.
Release assurance: Show the policy decision, transformation version, detector result, signed manifest, expiry, and revocation for one export.
Tenant isolation: Attempt a cross-tenant read, a tokenization request, and an evidence lookup. The platform must enforce isolation in its identity, service, store, and key boundaries—not infer it from a tenant_id field alone.
AI resistance: Demonstrate that unauthorized documents cannot enter retrieval and that prompt-injected tool output cannot exfiltrate data.

The evidence may be source code, configuration, an auditable demonstration, or an independent assessment. The important distinction is between a claimed feature and a testable control.

The bottom line

Redaction remains a valuable data-minimization control. Deterministic pseudonymization is valuable when an approved use genuinely needs stable equality. Encryption, governed views, protected query interfaces, and disclosure controls each solve other parts of the problem. None creates privacy by itself.

The architecture becomes credible when it makes three things explicit: the utility the policy allows a recipient to retain, the linkability that remains, and the evidence that proves the system applied the policy. That is the standard worth using when building—or buying—a privacy-safe data platform.

Data Engineering Weekly #277

Ananth Packkildurai — Mon, 06 Jul 2026 04:58:46 GMT

Deep Dive: See how Dagster uses AI internally

What does it actually look like to use AI as part of your day-to-day engineering workflow? At Dagster, we've developed workflows that use AI to quickly validate concepts, identify dead ends, and focus our engineering time on what matters most.

Join our live session this week where we'll share the prompts, processes, and lessons we've learned from using AI to accelerate product and engineering decisions.

Register now

Editor’s Note: aidataengineer.io & leetdata.ai are now live!!!

Over the last two weeks, we launched both aidataengineer.io [aide] and leetdata.ai. DEW has so far focused on delivering the handcrafted articles for the week to the readers. However, I found two gaps when I started conversing with many data engineers.

How do I get started in a career in data engineering? leetdata.ai is the answer for it.
I’m working on my job and on how to validate my daily design compared with my peers - aide is the answer for it.

People are already spending hours on both platforms, which I’m absolutely thrilled to see. Rest assured, we are just getting started on these two platforms. We have some exciting feature additions coming to these two platforms, making them a one-stop source for all things data engineering. If you have any suggestions or feature implementation requests, please feel free to reach out to me on both LinkedIn and Substack.

Databricks: From monolith to Lakebase to LTAP: rethinking the database from storage up

Databricks publishes an in-depth article on the design thinking and high-level architecture of the LTAP system. The system builds on top of the Postgres protocol, with the replication and page server over a network. Bringing OLTP-style guarantees into the Lakehouse format is certainly appealing, especially with the rise of agent development and data engineering, not only generating data marts but also moving beyond to handle structured, unstructured, embedded, and multi-dimensional data.

https://www.databricks.com/blog/lakebase-ltap-rethinking-database-storage

Boaz Palgi: Databricks LTAP and the Unfinished Problem of Unified Data

An interesting take on the Databricks LTAP announcement, RegattaDB compares the LTAP's functioning with its design. The article raises very valid concerns about the system's concurrency guarantee and the use of copy-on-write. Anyone who has worked closely with Iceberg knows that it is pretty bad at concurrent writes and copy-on-write for fast-changing records.

https://regatta.dev/blog/databricks-ltap-storage-unification-is-not-enough

Addy Osmani: Loop Engineering

Coding agents start every run cold and fill any gap in intent with a confident guess, so durable leverage lives in the system around them. The author reframes this as loop engineering — scheduled automations, isolated worktrees, SKILL.md context, and split maker-checker sub-agents feeding an on-disk state file that survives each run. The loop engineering pattern aligns closely with the data engineering workflow, and I look forward to seeing how this will impact the data engineering function.

https://addyosmani.com/blog/loop-engineering/

Microsoft: Introducing Durable Functions in PostgreSQL

Postgres as a queue, especially around the job queue, is something many companies adopted without using it as intended. The “pg_durable” attempts to streamline the pattern by introducing durable functions in PostgreSQL.

https://techcommunity.microsoft.com/blog/adforpostgresql/introducing-durable-functions-in-postgresql/4526821

Meta: Meta’s AI Storage Blueprint at Scale

Meta writes about overhauling its storage architecture to address bottlenecks that limit GPU utilization and slow down AI research. The primary bottleneck was caused by a legacy layered storage architecture that relied on stateful, slow metadata lookups and inefficient data proxies, stalling GPU training cycles. Meta’s solution was to rebuild the foundation with a unified metadata schema for faster lookups, eliminate data proxies in favor of a direct-streaming client SDK, and implement a tiered caching strategy that allows data to be hydrated on-demand across regional flash and memory.

https://engineering.fb.com/2026/07/01/data-infrastructure/metas-ai-storage-blueprint-at-scale/

Affirm: Re-architecting Affirm’s Upfunnel Platform: How We Cut Experiment Cycle Time from Months to Days

Affirm writes about migrating an upfunnel messaging service from a monolithic Python application to a standalone Kotlin microservice to improve reliability and reduce configuration bloat. By replacing a static database model with a dynamic, in-memory rule engine and implementing a robust shadow-validation pipeline, Affirm achieved independent deployability and significantly improved system performance.

https://medium.com/affirmengineering/re-architecting-affirms-upfunnel-platform-how-we-cut-experiment-cycle-time-from-months-to-days-20a047eb6599

Stripe: Scaling up your microservice testing with Apache Spark

Stripe writes about separating deterministic decision logic from the request path, then wrapping it in an Apache Spark harness that replays years of historical traffic through both current and candidate implementations. The harness turns production history into an executable regression asset, quantifying which records and rules change before a high-risk change ships, and anchoring it to a decision-engine boundary shared across online and offline paths.

Part 1: https://stripe.dev/blog/microservice-testing-with-apache-spark

Part 2: https://stripe.dev/blog/microservice-testing-with-apache-spark-part-2

Target: Scaling Marketing Campaign Forecasting with Generative AI

Rule-based matching and static segmentation decay as campaign types diversify, leading to false positives and manual overrides. Target writes about a RAG architecture that embeds historical campaigns into a grounding index, then has an LLM filter and rank the top three against an explicit matching hierarchy. The system lifts coverage from 75% at top-one to 100% at top-three, eliminating manual search while grounding the offer-propensity model in contextually relevant campaigns.

https://tech.target.com/blog/scaling-marketing-campaign-forecasting-ai

Expedia: Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs

Decoding the Spark SQL plan to identify the bottleneck is both an art and a science that requires considerable cognitive effort. I truly appreciate how LLMs commoditized this part of it. The author writes about feeding Spark plan metadata from an MCP server into an LLM. The grounded workflow cuts long-running job runtimes by 40–95% and compute costs by 50–95%, converting opaque plan internals into evidence-backed fixes that engineers can verify.

https://medium.com/expedia-group-tech/using-llms-to-analyze-spark-sql-plans-a-practical-approach-to-debugging-long-running-jobs-35eace7eeec4

Announcing leetdata.ai — A Practice Ground for Data Engineers

Ananth Packkildurai — Thu, 02 Jul 2026 08:27:41 GMT

A software engineer preparing for a career walks a well-worn path: grind LeetCode, study system design primers, rehearse mock interviews on platforms built for exactly that purpose. An entire practice infrastructure stands ready — thousands of problems ranked by difficulty, instant feedback, a community sweating through the same exercises.

Now try preparing as a data engineer. You grind... the same LeetCode. The same array puzzles and binary tree inversions. Then you walk into an interview, and someone asks you to design a dimensional model for an order management system, or to reason through late-arriving data in a pipeline. Nothing you practiced shows up in the room. Nothing in the room showed up in your practice.

We built practice infrastructure for every software discipline except the one that moves the world’s data. Today, I’m doing something about it.

leetdata.ai is live. It’s a practice platform built for data engineers — real problems, data modeling challenges, mock design interviews, and a leaderboard to keep you honest. It’s open to everyone, starting now.

Why is Data Engineering Weekly launching it

For years, I’ve curated Data Engineering Weekly by reading what the industry actually ships — the architecture posts, the postmortems, the migration stories, the debates about modeling and orchestration. And for years, the same question has landed in my inbox from early-career readers: “This is great, but where do I practice?”

I never had a good answer. I could point them to blog posts about dimensional modeling, but reading about modeling is like reading about swimming. At some point, you have to get in the water, and there was no pool.

leetdata.ai is the pool. It’s not a prep site bolted onto a trend — it’s the answer to a gap I’ve watched widen with every issue of this newsletter.

Fundamentals matter more now, not less

Here’s the uncomfortable timing: I’m launching a fundamentals platform in the middle of an AI boom, when models write more of our pipeline code every week. Doesn’t AI make this obsolete?

The opposite. AI collapses the value of syntax recall and raises the value of judgment. When a model can generate a working pipeline in seconds, the engineer’s job shifts entirely to the questions the model can’t answer for you: Is this the right grain for the fact table? Should this be incremental or a full refresh? What happens when the upstream schema changes? What does “correct” even mean for this dataset?

Engineers who understand systems from first principles direct AI. Everyone else takes direction from it. Engineers entering this field today will spend their careers reviewing, correcting, and architecting around machine-generated code — and you cannot review what you cannot reason about. Fundamentals are the leverage.

Practice is the job

Here’s what makes leetdata.ai different from generic interview prep: the practice and the work are the same thing. Partitioning strategies, dimensional modeling, pipeline design, data quality tradeoffs — these aren’t proxies for the job. They are the job.

When you solve a data modeling problem on leetdata.ai, you’re rehearsing the exact conversation you’ll have in a design review six months into your first role. Every problem you complete makes you better at the interview and better at the work waiting on the other side of it. That’s the whole point.

What you’ll find inside

Data engineering problems. SQL, transformations, pipeline logic — the day-to-day craft. Solve each problem in SQL, Python, or PySpark, filter by topic — joins, aggregation, window functions — and work through the same questions companies like Databricks, Netflix, and Stripe ask in their interview loops. Each problem answers the question every engineer quietly asks: can I actually do this work?

Data modeling problems. Modeling is the most under-trained, most interview-critical skill in data engineering. It’s the round that filters out most candidates, and until now there was nowhere to practice it deliberately. On leetdata, you build the schema yourself on an interactive canvas — add tables, draw relationships, define cardinality — against real scenarios like designing a denormalized search index for an e-commerce catalog. Then you validate your design and compare it against a reference. This is the heart of the platform.

Mock design interviews. Knowing the answer and holding a design conversation under pressure are different skills. The mock interviews let you rehearse the second one — a timed session, a full problem context, and an AI interviewer you can talk through your approach with, exactly like the real round. Tradeoffs, follow-up questions, thinking out loud — practice it before it counts.

A progress dashboard. Not streaks and vanity metrics. The dashboard shows which capabilities you’re building and where the gaps are, so your practice compounds instead of wandering.

A leaderboard. See where you stand among peers solving the same problems. A little competition sharpens practice — and the people beside you on that board are your future colleagues.

A daily problem. Every day, one problem. No decision fatigue about what to practice next — open the daily, solve it, and let the habit compound. You don’t build fundamentals in a weekend binge; you build them in twenty focused minutes a day.

Two asks

If you’re building your data engineering career: sign up and open your first problem. Start with the supply chain shipment tracking data model. It looks simple until the shipment updates its status out of order, and then you learn firsthand why event ordering shapes every modeling decision you’ll ever make. That moment — where the problem teaches you something the tutorial never mentioned — is what leetdata.ai delivers, problem after problem.

If you’re a senior engineer reading this: this platform is for you too. Fundamentals rust quietly — when did you last design a schema from a blank canvas instead of inheriting one? Work through a modeling problem and see if your instincts still hold, or run a mock interview before your next job move. And if the problems remind you of someone on your team who’d benefit, send it their way.

Back to first principles

Data Engineering Weekly has always been about understanding this field deeply rather than skimming it. leetdata.ai extends that mission from reading to doing. The industry left data engineers without a practice ground for too long, and the timing has never mattered more — the engineers who master fundamentals now will be the ones directing the AI-augmented data teams of the next decade.

The platform is live. Your first problem is waiting.

Start solving at leetdata.ai.

Data Engineering Weekly #276

Ananth Packkildurai — Mon, 29 Jun 2026 03:52:17 GMT

How to Build a Data Platform From Scratch

Download your copy now

Jack Vanlightly: Can We Agree on a Storage/Workload Architecture Taxonomy?

OLTP, OLAP, HTAP, and now LTAP? Yes, we need to agree on storage/ workload architecture. I wrote about the emerging LakeDB architecture in lakedb.cloud, describing its target characteristics. It is high time to rephrase this taxonomy and update the Lakedb characteristics.

https://jack-vanlightly.com/blog/2026/6/21/can-we-agree-on-a-storage/workload-architecture-taxonomy

Pinterest: Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

Maintaining backward compatibility for schema changes is a must-have for any data platform. The adoption of message protocols such as protobuf enables compile-time schema evolution. You would encounter an issue where protocols such as Protobuf are indexed-based systems, whereas catalogs are name-based. Pinterest writes about its data management system, which follows basic schema management principles to enable backward-compatible schema evolution.

https://medium.com/pinterest-engineering/automated-schema-evolution-in-pinterests-next-generation-db-ingestion-framework-36c5c07070de

Uber: Zone-Failure-Resilient OpenSearch® at Uber

Uber writes about how it achieved Zone-failure-Resilience by combining OpenSearch’s native shard-allocation awareness features with its in-house isolation-group infrastructure. Uber combines this with forced shard allocation awareness to prevent chaotic data rebalancing during an outage, prioritizing overall cluster stability over immediate replication.

https://www.uber.com/us/en/blog/zone-failure-resilient/

Sponsored: AI Modernization Guide

Download the free guide

Tejas Sridhar: Computational Synapses: Rethinking Notebooks as Stateful Reasoning Systems

The author makes an interesting argument that we need to stop treating data notebooks as simple, linear scripts and start architecting them as "stateful computational nervous systems. The author proposes three distinct layers of observability in Notebook.

The Dataflow Layer: Tracking what depends on what (standard lineage).
The Reasoning Layer: Tracking why the pathway exists (turning business logic and assumptions into explicit code objects).
The Memory Layer: Tracking what actually happened during execution (cache status, runtime logs, validation checks).

https://tejas-sridhar.medium.com/computational-synapses-rethinking-notebooks-as-stateful-reasoning-systems-7c47197b132d

Razorpay: Turning Scattered Data Into Queryable Segments at Scale: How Razorpay Built Its Customer Data Platform

Razorpay writes about its consent-native Customer Data Platform to unify billions of scattered user transactions into cohesive profiles. The architecture utilizes Spark for segment computation, DynamoDB for sub-30ms real-time serving, and Theta Sketches for rapid audience size estimation.

https://engineering.razorpay.com/turning-scattered-data-into-queryable-segments-at-scale-how-razorpay-built-its-customer-data-3937c4b012de

Grab: Architecting isolation, identity, and auditability for AI agents.

Grab writes about Palana, a Kubernetes-native substrate isolating each agent in its own namespace — handing agents placeholder tokens that an egress proxy swaps for real Vault credentials. The platform keeps real secrets out of agent runtime and routes every call through OPA-checked Envoy egress, anchored to an external control plane that terminates uncooperative agents.

Part 1: https://engineering.grab.com/palana-part-1-secure-platform-for-ai-agents

Part2: https://engineering.grab.com/part-2-palana-architecture

Fixel Smith: Six SQL patterns I use to catch transaction fraud

There is a lot more you can do with SQL alone. I believe we will be switching from spec-driven development to SQL-Driven development soon. SQL is something many have tried to break, with no success. The article is a classic example of why SQL is so powerful, yet simple.

https://analytics.fixelsmith.com/posts/sql-fraud-patterns/

Data Strata: How Ads Ranking Works: The Data System Behind Every Ad You See

I’ve been getting a full 360-degree view of ad tech and data engineering for the ads engine recently. It is indeed a fascinating industry in which the entire financial model relies on data engineering efficiency. The article is a pretty good read to headstart data systems for ad ranking.

https://medium.com/@datastrata/how-ads-ranking-works-the-data-system-behind-every-ad-you-see-67c9d0d654b8

KoliStat: the-stats-duck v0.6.0 — statistics that live in your SQL

Data profiling is a key part of the system for overall data management. A good profiling tool can help auto-generate data contracts, run data quality checks efficiently, and be used for synthetic data generation as an indicator of the data distribution, etc. The author highlights that the stat-ducked plugin provides out-of-the-box integration with DuckDB. I believe this should be the default for the Lakehouse formats.

https://kolistat.com/blog/the-stats-duck-v0-6-0/

You Don't Graduate From Data Engineering: Why We Built aide for Continuous Learning

Ananth Packkildurai — Tue, 23 Jun 2026 03:47:56 GMT

Two years ago, Maya finished a twelve-week data engineering bootcamp. She crushed it. She built a batch pipeline, wired up Airflow, wrote her first dbt models, and landed a job two months later. The bootcamp did exactly what it promised: it got her in the door.

Today, Maya leads a small data team. Streaming has eaten half her workload. Her company moved to a lakehouse. The ecosystem deprecated half the tools she learned, and the other half shipped three major versions while she was busy shipping features. She still knows how to learn — but the structured, guided, motivating environment that got her started ended the day she graduated.

Maya’s problem isn’t that her bootcamp was bad. Her bootcamp was great. Her problem is that learning that ends is learning that decays.

That’s the gap we built aide to close. Today, we’re launching aidataengineer.io — and the idea behind it is simple: in data engineering, you never actually graduate.

The bootcamp paradox

Let’s give bootcamps their due, because they earn it. A good bootcamp is structured, intense, and motivating. It compresses months of scattered self-study into a focused sprint with a curriculum, a cohort, and a deadline. For going from zero to employable, a few things work better. We are not here to bury the bootcamp.

Yet every bootcamp shares one trait that no amount of quality can fix: it ends. The format bounds itself by design. A fixed curriculum, a fixed window, and a graduation date are the whole point — they create the urgency that makes the format work. The trouble is that data engineering refuses to stay inside that window.

Think of it as the half-life of a data skill. The Spark API you mastered gets a new optimizer. The orchestration tool you trusted gets displaced by something leaner. Streaming, lakehouse architectures, data contracts, and a steady drip of new query engines keep moving the frontier. A skill that felt cutting-edge at graduation is merely current a year later and quietly dated the year after that.

The curriculum that taught you so well was a snapshot, and the field kept moving after the photo.

So the real skill — the one that actually compounds over a career — is continuous learning: the habit of staying current, deliberately and forever. That’s the layer bootcamps were never designed to provide. It isn’t a replacement for the bootcamp. It’s the thing that comes after the bootcamp and never stops.

To be fair, not everyone needs a formal system for this. Plenty of strong engineers stay sharp through sheer curiosity, a good RSS habit, and hard problems at work. If that’s you, keep doing it. But for most practitioners — and for every team that needs its skills to grow predictably rather than by luck — “just stay curious” isn’t a plan. It’s a wish.

A wish needs structure to become a habit, and that structure is exactly what’s been missing.

Why Data Engineering Weekly is the right foundation

Here’s the thing about continuous learning that nobody likes to admit: access to content was never the bottleneck. The internet overflows with data engineering material. The bottleneck is everything around the content — knowing what’s worth reading, in what order, and actually following through instead of bookmarking forty tabs you’ll never open.

Data Engineering Weekly has spent years solving the hardest part of that problem. Across 256+ editions, it has curated more than 3,000 articles spanning 100+ technologies, filtering the firehose of the modern data ecosystem down to the writing that genuinely matters. That editorial judgment — built one week at a time — is something no algorithm can fake and no new entrant can shortcut.

aide builds directly on that foundation. We took the curation that Data Engineering Weekly already nailed and added the two things it was never meant to provide on its own: structure and follow-through. The result turns a world-class reading list into a world-class learning system.

Introducing aide

aide is an AI-powered learning platform built on the Data Engineering Weekly archive. It lets you search the entire corpus in plain language, learn any article deeply with a Socratic AI tutor, track your skill growth over time, and bring that curated knowledge directly into the AI tools you already use — for yourself or for your whole team.

The easiest way to hold the whole product in your head is four words: Search, Learn, Track, Integrate. You find the right thing to study, actually learn it instead of skimming, watch your skills grow, and carry that knowledge into your daily workflow. The next four sections walk through each pillar in turn.

Search & Discover: find the right thing to learn

Every learning session starts with a question, and most learning tools answer questions badly. They make you guess the exact keyword and punish you when you guess wrong.

aide’s hybrid search works the way you actually think. Ask a real question — “how do I handle late-arriving data in a streaming join?” — and it combines semantic understanding with full-text matching to surface the right articles, even when your wording never appears in them. You describe the problem; aide finds the depth.

This is what makes learning just-in-time instead of someday. You hit a thorny Kafka rebalancing issue at work on a Tuesday afternoon, and the curated answer is one question away — not buried in a course you’ll start “when things calm down.” And when you’d rather explore than search, the discover feed surfaces the latest editions and editor picks, while a rich taxonomy of technologies, use cases, companies, and domains lets you browse with intent instead of drowning in an undifferentiated archive.

Learn with Agent DEW: your Socratic AI tutor

Finding the right article is the easy part. Actually, learning it is where almost everyone falls down — because reading is not the same as knowing. You skim a great article, nod along, feel smarter for ten minutes, and retain almost none of it a week later.

Passive reading is a leaky bucket.

Agent DEW is how aide plugs the leak, and it’s the feature we’re proudest of. It turns any article into an active lesson with a personal Socratic tutor. Here’s how a session feels.

Agent DEW takes an article and breaks it into digestible sections, so you’re never staring down a wall of text. For each section, it explains the core concept in plain language, then turns the tables and asks you a question to check whether the idea actually landed. You answer in your own words. Agent DEW evaluates your response and gives you a clear verdict — and, crucially, a real explanation of why, whether you got it right or wrong. There’s no hollow “Correct!” and no vague “Not quite.” You learn something from every answer. After you’ve worked through the sections, a final quiz pulls it all together, and because your progress is saved, you can step away and pick up exactly where you left off.

Why does this work when re-reading doesn’t? Because it’s built on how memory actually forms. Explaining a concept and then immediately retrieving it under a question is active recall — the single most reliable way to move knowledge from “I’ve seen this” to “I know this.” Agent DEW bakes that retrieval loop into the reading itself. A one-time bootcamp lecture gives you the concept once and moves on; Agent DEW makes you use it, which is the difference between learning that fades and learning that sticks.

Picture a session on Spark’s lazy evaluation. Agent DEW explains that Spark defers execution until an action is called, then asks why that design choice matters. You reply that it lets Spark optimize the whole plan before running anything. Agent DEW confirms you’ve got the core idea, sharpens it by adding how the optimizer fuses transformations to avoid redundant passes over the data, and moves you to the next section. Two minutes, one concept, genuinely understood — not just highlighted.

Track Your Growth: see yourself getting better

Motivation is the quiet reason bootcamps work. The cohort, the deadline, the visible progress bar — they manufacture momentum. The moment you graduate, all of that disappears, and self-directed learning stalls not because the material is too hard but because progress becomes invisible.

aide makes progress visible again. It surfaces your skill readiness across technologies and tracks your learning activity over time, so your effort adds up to a trajectory you can actually see. Instead of a vague sense that you “should learn more streaming,” you can watch your streaming readiness climb session by session.

That turns motivation from something a bootcamp imposes from the outside into something you generate from within. You’re not chasing a deadline anymore. You’re watching yourself get measurably better — and that’s a far more durable engine for a habit meant to last a whole career.

Bring aide into your workflow

There’s a failure mode that kills even well-designed learning tools: they live in yet another tab. If staying sharp requires you to stop what you’re doing, switch context, and visit a separate site, it quietly drops to the bottom of the list every single day.

So we made aide come to you. For paid subscribers, aide connects directly to the AI tools you already work in — your assistant of choice — so the entire curated Data Engineering Weekly knowledge base becomes ambient. It’s right there, in the place you already spend your day.

Picture debugging a gnarly consumer-lag problem inside your AI assistant. Instead of a generic, plausible-sounding answer assembled from the open internet, your assistant draws on curated Data Engineering Weekly material and answers with citations you can actually trust. That’s the whole difference: answers grounded in the best data engineering writing of the last several years, with provenance, exactly when you need them.

Learning stops being a destination you visit and becomes a layer underneath your normal work.

For teams, this quietly extends the reach of everything else. A team’s curated knowledge base is no longer trapped behind a login — it’s available inside every engineer’s daily tools, which means the learning surface grows far beyond the web app.

Built for data engineering teams

If you manage a data team, you live with two uncomfortable unknowns. You don’t really know where your team’s skills are strong and where they’re dangerously thin — until a project stumbles into the gap. And you don’t really know whether the money you spend on training is doing anything at all. Bootcamps and course budgets vanish into individual inboxes, and the results, if any, are invisible to you.

aide is built to turn both unknowns into dashboards.

The skill matrix shows you every team member’s proficiency across the technologies that matter — Spark, Kafka, SQL, Airflow, and the rest of the modern stack — at a glance. You can see, in one view, who your Kafka experts are and where you have exactly one person holding up an entire capability. You spot the gaps before they become project risks, rather than discovering them in a postmortem.

Learning activation and completion answers the question every manager eventually has to defend: Is this working? You can see who’s actively learning, who’s finishing sessions, and how engaged the team is overall. Training stops being an act of faith and becomes something you can measure and report on.

Invite and manage, keeping the administrative overhead near zero. Adding members takes a click, work-email validation keeps the team clean, role-based access keeps things appropriate, and team plans are built for real teams. Paid seats also include the workflow integration, so your team’s curated knowledge base reaches every engineer’s AI tools, not just the web app.

Step back, and the strategic case is clear. A bootcamp is a per-person expense with no shared visibility and no continuity — you pay, someone learns something, and the knowledge walks around in one person’s head. aide is a continuous, measurable team capability: skills that grow on purpose, gaps you can see, and a training story you can take upstairs with numbers attached. That’s the ROI conversation managers have always wanted to have and never had the data for.

Who aide is for

Aspiring data engineers. Breaking in is overwhelming because the resources are scattered and unstructured. aide gives you curated fundamentals and a tutor that guides you through real-world articles step by step. You don’t need prior experience — you need a path, and aide is one.

Working data engineers and data scientists. The stack never stops moving, and staying current is a job in itself. With 3,000+ articles across Spark, streaming, lakehouse, data quality, and more, you can search for anything the moment you need it and learn it deeply with Agent DEW instead of just bookmarking it.

Team leads and managers. You need your team’s skills to grow predictably, and you need to prove it. aide gives you skill readiness across the whole team, visibility into who’s actually learning, and the ROI story your training budget has always lacked.

The bigger picture

Step back from the features for a moment, because the features are not really the point. The point is a shift in how technical careers actually work.

The engineers and teams who win over the long run are not the ones who learned the most in a single intense burst. They’re the ones who keep learning systematically, week after week, long after the certificate is framed and forgotten. In a field that reinvents itself every few years, the durable advantage isn’t what you know today. It’s how reliably you keep knowing the right things tomorrow.

That habit is hard to sustain on willpower alone. It needs curation so you’re not lost, structure so you actually retain what you read, visible progress so you stay motivated, and a home inside your daily workflow so it never becomes one more thing you’ll get to later. aide is the infrastructure for that habit — built on the best curation in the field, and designed for the long game rather than the sprint.

Get started

You can try it all free for 7 days — no credit card required.

Individuals can jump straight into a free trial, search the full archive, and run a Socratic session on any article that catches their eye. Premium unlocks the full experience for the long haul, and our team plans (built for groups of four or more) add the skill matrix, the learning dashboards, and team-wide workflow integration.

Start your free trial → if you’re an individual ready to stay sharp.

See team plans → if you’re a manager ready to make your team’s growth visible.

Bootcamps have a graduation day. aide doesn’t — and neither does your career.

Data Engineering Weekly #275

Ananth Packkildurai — Mon, 22 Jun 2026 04:02:10 GMT

How to Build a Data Platform

Download your free copy now

MIT CISR: The Case for a Semantic Layer

Data decontextualized from its source loses the business meaning and rules that made it trustworthy, leaving GenAI agents to hallucinate without machine-readable context. MIT CISR frames the semantic layer as the answer — taxonomies, ontologies, and business rules engines encoding usage constraints, illustrated by Healthcare IQ’s Semantic ETL. The semantic layer automated 80% of Healthcare IQ’s hospital onboarding, anchored in shared terminology that lets organizations expand AI without reinventing governance for each use case.

https://cisr.mit.edu/publication/2026_0501_SemanticLayer_LefebvreWixomLegnerVandermeulenBeath

Lyft: Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions

Metric inconsistency and definition sprawl across distributed teams fragment analytics workflows, seeding decisions with conflicting values drawn from the same underlying data. Lyft builds a Metric Semantic Layer as a Python package, encoding Golden Metrics in YAML and Jinja SQL templates, with governance by paired Business and Operational Owners. The layer flows a single update through every downstream dashboard, ML model, and BI tool, anchored in clean YAML that doubles as an MCP knowledge base, reducing agent hallucination.

https://eng.lyft.com/metric-semantic-layer-how-lyft-governs-and-scales-key-data-definitions-56bee3643c29

Mohammed Benaissa: Idempotency in Data Engineering - The Quiet Property That Saves Your Pipelines

Functional data engineering treats every task as a pure function — same inputs producing the same output — yet most pipelines silently break that contract the moment a retry fires. The author reframes the fix as idempotency — pairing a deterministic key with MERGE or dynamic partition overwrite, proven by a run-twice hash test. The pattern collapses retried loads into harmless no-ops, anchored to writes designed to absorb duplicates instead of delivery guarantees that prevent them.

https://medium.com/@halanmyanime/idempotency-in-data-engineering-the-quiet-property-that-saves-your-pipelines-22ee357f37b2

Sponsored: AI Modernization Guide

Download the free guide

Netflix: Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning

Netflix writes about predicting media asset delivery with boosted-tree regression on daily-snapshotted production signals, building a phase-agnostic model that fills schedule gaps without look-ahead bias. The model reaches accuracy 11 weeks sooner than manual schedules and reduces Accumulated Error Days, anchored to serving logic that defers to manual dates where it underperforms.

https://medium.com/netflix-techblog/predicting-risk-in-content-launches-how-data-driven-insights-can-transform-launch-planning-587b1f2de928

Helpshift: Building a Centralized Alerting Framework for Data Quality Monitoring and Incident Management

Failure detection is rarely the bottleneck — fragmented monitoring and generic alerts mean issues get found but not acted on, because responders lack context overnight. The author builds a centralized alerting framework in Snowflake, separating detection, notification, and escalation, then pages on-call via Splunk with secure external access. The framework reduces MTTR by using validation-specific context rather than generic messages, anchored to loosely coupled layers that allow new checks and channels to evolve independently.

https://medium.com/helpshift-engineering/building-a-centralized-alerting-framework-for-data-quality-monitoring-and-incident-management-2f90d93a65b5

Instacart: Semantic IDs: Product Understanding at Scale

Recommendation models learn from engagement volume, so they skew toward popular staples and leave cold-start and tail products invisible behind a rigid category tree. Instacart compresses product embeddings into hierarchical semantic IDs via an RQ-VAE, regularized with a contrastive loss that uses catalog taxonomy as free supervision rather than engagement data. The shared-prefix codes drove a 34% add-to-cart lift and surfaced 2.7x more emerging brands, anchored to a vocabulary every downstream model reads alike.

https://tech.instacart.com/semantic-ids-product-understanding-at-scale-5283e0288f5a

Salesforce: How Data 360 Segmentation Processes a Quadrillion Records Across Arbitrary Customer Data Models

Distributed systems stay predictable by controlling schema, relationships, and workload shape — but handing those to the customer forces reliability from runtime adaptation, not design-time assumptions. Salesforce interprets arbitrary customer schemas and relationship graphs at runtime in Data 360, adding phased query planning that decomposes optimization when plan candidates reach the billions. The platform maintains 99.95% reliability across three million monthly Spark jobs, anchored in workload-size estimation that right-sizes each job rather than overprovisioning.

https://engineering.salesforce.com/how-data-360-segmentation-processes-a-quadrillion-records-across-arbitrary-customer-data-models/

Shopify: Teaching Sidekick to say no - automated data curation with LLM judge consensus

Production logs are a survivorship-biased corpus — they capture only successful queries, so a fine-tuned model never learns when to refuse an impossible request. Shopify teaches its Sidekick text-to-query skill to say no — seeding a four-judge LLM ensemble with a small annotated set and gating on unanimous consensus to label refusals. The consensus gate raised the skill’s pass rate from 0.762 to 0.798 at 86% refusal accuracy, anchored to a flywheel that recycles production traffic each cycle.

https://shopify.engineering/sidekick-curation

Trivago: How We Cut Kafka Consumer Deployment Costs by 83%

Architectural choices that fit the workload when made silently calcify into bottlenecks as throughput grows, while reactive scaling hides the root cause. Trivago traces a Kafka consumer’s chronic lag to three compounding bottlenecks that each moved nothing alone — coupled commits, a delay ceiling, and an undersized gRPC pool. Migrating to reactor-kafka, removing the delay, and resizing the pool cut pod count from 60 to 6, anchored to fixing compound bottlenecks together.

https://tech.trivago.com/post/2026-06-12-how-we-cut-kafka-consumer-deployment-costs-by-83

Data Engineering Weekly #274

Ananth Packkildurai — Mon, 15 Jun 2026 05:29:03 GMT

How to Build a Data Platform

Download your free copy now

Anthropic: How Anthropic enables self-service data analytics with Claude

Data != Software. The Anthropic blog about enabling the self-serve analytics essentially highlights the same. The key takeaway from the article,

The most important aspect of ensuring analytics agents are accurate is a strong data foundation, which includes the data models, transforms, tests, and tables in a data warehouse, along with the metadata that describes them. Standard data engineering and data quality practices such as dimensional modeling, shift-left testing, freshness, and completeness checks on critical pipelines all still apply (and we won't relitigate these).

https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude

Airbnb: Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world

Decentralized data ownership breaks down when teams build domain tables without shared conventions, leaving consumers to reconcile inconsistent identifiers and namespaces across the warehouse. Airbnb resolves the trade-off with three principles — no hybrid models, identifier naming tied to the modeling choice, and dedicated namespaces that separate product, global, and team-owned tables. The framework lets product-facing domains pick separate models while cross-cutting concerns like payments and messaging stay monolithic, anchored to centralized principles with decentralized modeling.

https://medium.com/airbnb-engineering/scaling-beyond-one-how-airbnb-evolved-its-data-architecture-for-a-multi-product-world-6125645d470c

Uber: Simplifying Data and Product Integrations with a Data Abstraction Layer

User-facing data products break when queries are tightly coupled to physical schemas, forcing weeks-long migrations every time a table is renamed, split, or moved. Uber builds a Data Abstraction Layer fronting OLAP, Docstore, and Hive — routing requests through schema-eligibility, availability-window, and column-continuity rules to pick physical candidates at query time.

https://www.uber.com/us/en/blog/data-abstraction-layer/

Sponsored: AI Modernization Guide

Download the free guide

LinkedIn: Semantic Search for AI Agents at Scale: Retrieval and Ranking for LinkedIn’s Hiring Assistant

LinkedIn writes about building MUSE, a dual-tower Matryoshka embedding model trained on LLM-Teacher relevance labels — serving 2048-dim truncations to ANN retrieval and 4096-dim vectors to the L2 ranker. The system lifts relevance by 2.7%, and InMail sends by 4.1% on fewer sourced candidates, anchored to embeddings that distill multi-step qualification reasoning into a single cosine score.

https://www.linkedin.com/blog/engineering/ai/semantic-search-for-ai-agents-at-scale-retrieval-and-ranking-for-linkedins-hiring-assistant

Capital One: DataAgents: How we turned 9 months of analysis into 10 days

AI agents amplify whatever data foundation they sit on — good data scales insight, bad data scales confusion across hundreds of entities at production speed. Capital One pairs an authoritative cloud-asset data product with a three-phase agent workflow — feasibility assessment, per-entity Spark SQL detection logic, and one-by-one MEDIUM-confidence validation. The pattern compresses a 350-resource-type dormancy analysis from nine months to ten days, anchored to confidence-rated outputs with per-entity-documented reasoning.

https://medium.com/capital-one-tech/dataagents-how-we-turned-9-months-of-analysis-into-10-days-8d6ed482f5d7

John Kutay: Building an AI Database for Agentic GTM Operations

Compute redundancy compounds at scale — re-running enrichment, re-extracting evidence, and re-embedding payloads on every query turns scaling into a cost spiral instead of a leverage curve. Rippling writes about pairing ML entity resolution with a blocking tree to compress the n² candidate space, then applies DRY(E) to precompute embeddings once and refresh only changed conversations.

https://medium.com/@john.kutay/building-an-ai-database-for-agentic-gtm-operations-6a0c86fb8cdc

Naidu Rongali: Why We Moved from Hive-Style Data Lakes to Apache Iceberg?

The author walks through Apache Iceberg’s metadata hierarchy — snapshots, manifest lists, and column-ID-tracked manifest entries — replacing directory-based discovery with statistics-driven file pruning. The architecture brings ACID transactions, schema evolution, and time travel into open object storage, anchored to a metadata layer that decouples logical table state from S3 file layout.

https://medium.com/@rongalinaidu/why-we-moved-from-hive-style-data-lakes-to-apache-iceberg-f9f23e7a64d3

Kiran Pothina: Ship Fast, Break Nothing: dbt CI/CD, Monitoring, and Model Health

Compute redundancy compounds at scale — CI pipelines that rebuild every model on every PR turn feedback loops into bottlenecks and push engineers to skip review entirely. The author wires four-stage slim CI across six dbt projects, using fresh manifest generation from main and zero-copy incremental clones before a state:modified+ build with --defer.

https://kiran-pothina.medium.com/ship-fast-break-nothing-dbt-ci-cd-monitoring-and-model-health-029d261a7a31

Andreas Andreakis: Why DBLog Is Snapshot-Equivalent

Streaming abstractions hide physical execution behind informal correctness arguments — “it works” leaves downstream consumers to discover edge cases in production rather than reason from stated guarantees. The author formalizes DBLog’s snapshot-equivalence — interleaving chunk reads with CDC events and applying latest-event-wins replay against a global frontier, machine-checked in Isabelle/HOL. The proof shows that replay output equals the source database at the frontier, restricted to backfill keys, anchored to a virtual cut that requires no lock, write pause, or single-moment read.

https://aandreakis.com/posts/why-dblog-is-snapshot-equivalent

Data Engineering Weekly #273

Ananth Packkildurai — Mon, 08 Jun 2026 02:44:02 GMT

How to Build a Data Platform

Download your free copy now

Fei-Fei Li: A Functional Taxonomy of World Models

The World Model is one of the most overloaded terms today. The author writes about the taxonomy of the world model into three functions,

Renderer: A renderer outputs observations in the form of pixels meant for human eyes, and the quality that matters most is visual fidelity.
Simulator: A simulator outputs a state: a geometrically, physically, or dynamically faithful representation of the world that humans and computer programs can both compute on and interact with.
Planner: A planner outputs actions.

From a data warehouse perspective, the metric tree is an attempt to build a simpler version of a business simulator. We need to think about extending the simulation of the entire business as it emerges.

https://drfeifei.substack.com/p/a-functional-taxonomy-of-world-models

Gorgias: When Event Time Meets Reality: Lessons from Building Billing on Apache Flink

In real-time pipelines, we assume that the watermark configuration alone preserves ordering, but reprocessing historical data with keyBy operators breaks source-level alignment downstream. The author aligns deduplication and consolidation using the same customer-user key, removes intermediate repartitioning, and extends consolidation by 1 day when event lag exceeds 15 minutes. The combined fix cuts overlapping consolidation windows by 10x without delaying real-time billing, anchored to lag-aware timer registration with explicit cleanup.

https://medium.com/gorgias-engineering/when-event-time-meets-reality-lessons-from-building-billing-on-apache-flink-581ff895c60d

Cloudflare: Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse.

Cloudflare writes that Multi-tenant ClickHouse partitioning hides query-planner coupling at scale — the part-list mutex and vector copy serialize across concurrent queries, even when each query reads few parts. Cloudflare patches ClickHouse’s MergeTreeData hotspot with three upstream fixes — replacing the exclusive lock with std::shared_lock, deferring the parts-vector copy, and binary-searching the sorted namespace prefix.

https://blog.cloudflare.com/clickhouse-query-plan-contention/

Sponsored: AI Modernization Guide

Download the free guide

Helpshift: Migrating from a Monolithic Orchestrator to Apache Airflow

Helpshift writes about migrating a monolithic Clojure-based scheduler system to Apache Airflow. The team writes about its journey and the overall impact on metrics such as reliability, availability, performance, data quality, & the developer productivity.

https://medium.com/helpshift-engineering/migrating-from-a-monolithic-orchestrator-to-apache-airflow-30fde94bcdc0

Indeed: Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search

We have long relied on distillation to compress complexity. But while humans reason within finite cognitive dimensions, foundation models now navigate data at an order-of-magnitude greater scale. We are moving from a paradigm of static, structural data modeling to a new era where the embedding model itself has become our primary, high-dimensional data model. - Ananth

This case study perfectly illustrates the thesis: Indeed’s shift from traditional feature engineering to a unified, transformer-based UBM architecture demonstrates the evolution from static, structural data design to the embedding model as the primary, high-dimensional data model for reasoning.

https://engineering.indeedblog.com/blog/2026/06/distilling-long-tail-user-behavior-into-scalable-embeddings-for-job-search/

Netflix: Dynamic Repartitioning for Time Series Workloads

Netflix discusses how static Cassandra partitioning breaks down under evolving time-series workloads when a small percentage of IDs accumulate disproportionate event volumes, driving wide partitions that cause tail-latency timeouts. Netflix layers two runtime adjustments onto its TimeSeries Abstraction — a background worker returning future Time Slice partitions, and per-ID dynamic splitting that routes reads via Bloom filters.

https://netflixtechblog.com/dynamically-splitting-wide-partitions-in-cassandra-for-time-series-workloads-0eded064f456

Jack Vanlightly: Broker-Visible vs Client-Local Parallelism

Messaging system parallelism has a hidden cost — where the unit of parallelism lives determines whether scaling stays cheap or explodes into broker-managed TCP connections and state. The author compares broker-visible parallelism with client-local parallelism using virtual threads, showing that the latter scales a 60K msg/sec workload from 60,000 consumers down to 60. Client-local parallelism shifts coordination cost off the broker but adds client complexity, leaving share groups in need of a parallel-processing library layer, anchored to per-message acknowledgment semantics.

https://jack-vanlightly.com/blog/2026/6/3/broker-visible-vs-client-local-parallelism

Criteo: Introducing CLEPR, our model for semantic understanding

Criteo writes about building CLEPR, a two-tower contrastive embedding model with separate keyword and product encoders — training on session-stitched pseudo-clicks and deduplicating repeated pairs to strip popularity bias. CLEPR scores 10 billion keyword-product pairs daily as a semantic guardrail before performance ranking, lifting CTR by 6%, anchored to embeddings that generalize to human-judged relevance.

https://medium.com/criteo-engineering/introducing-clepr-our-model-for-semantic-understanding-d3984eed84c8

WMG: Why we shrank our TimescaleDB chunks from 30 days to 7

Storage configuration choices made when a database is small calcify silently as ingest grows, turning yesterday’s reasonable defaults into compression failures and backfill storms. WMG Lab shrinks TimescaleDB chunk intervals from 30 days to 7 across its hot hypertables — using set_chunk_time_interval to retune future chunks only, without rewrites or locks. The smaller chunks let compression finish cleanly and reduce backfill to a single 7-day decompression instead of a month, anchored to the 25% active-chunk memory rule.

https://tech.wmg.com/why-we-shrank-our-timescaledb-chunks-from-30-days-to-7-07cab8afefc5

Data Engineering Weekly #272

Ananth Packkildurai — Mon, 01 Jun 2026 03:18:34 GMT

How to Build a Data Platform

Download your free copy now

Netflix: High-Throughput Graph Abstraction at Netflix

One emerging data model envisions the organization model as a property graph abstraction, though the underlying data may be stored in a relational model, key/value store, or document store. GraphQL is one abstraction, but it leads to many complications while integrating the api. Netflix writes about its high-throughput graph abstraction and explains how it handles read-aside cache and write-aside cache.

https://netflixtechblog.com/high-throughput-graph-abstraction-at-netflix-part-i-e88063e6f6d5

Slack: Slack AI - The Path to Multi-Cloud

Building AI Infrastructure for a multi-tenant stack with enterprise security requirements is challenging in itself. Slack writes about evolving from SageMaker-managed endpoints to Bedrock provisioned throughput, Bedrock on-demand spillover, and a multi-cloud Vertex AI architecture with normalized APIs, model hierarchies, circuit breakers, health-aware routing, and feature-level model selection. The platform focuses on reducing the infrastructure lock-in, enabling same-day model migration, improving the quality of complex reasoning by about 10%, cutting latency for low-token workloads by about 67%, and increasing resilience against regional and provider-wide failures.

https://slack.engineering/slack-ai-the-path-to-multi-cloud/

Uber: Solving the Identity Crisis for AI Agents

The article seems so timely, as I recently finished a similar design for AI Identity. The conventional way to start the design is to assign the user role to the agent to work on their behalf, but the model soon fails. The services-centric IAM role also fell apart, since the same service can potentially assume multiple roles. Uber provides an in-depth account of how it handles these challenges.

https://www.uber.com/us/en/blog/solving-the-agent-identity-crisis/

Mehul Batra: A field journal on Ray Data and Daft for multimodal data lake

Multimodal data lake pipelines on Kubernetes require engines that can handle mixed CPU, GPU, and async inference workloads, along with cataloging and checkpointing, across text, PDF, image, audio, video, and LLM metadata. The author compares Ray Data 2.55.1 and Daft 0.7.13 on the same KubeRay, DigitalOcean, Gravitino, Iceberg, Lance, and H100 setup, measuring wall time, GPU occupancy, memory, code size, native primitives, concurrency behavior, and completion reliability across eight use cases. Ray Data scored 56 versus Daft’s 47 out of 70 because both engines tied on most multimodal batch workloads, Daft delivered stronger native media ergonomics, and Ray Data won the decisive async LLM inference workload by completing the 50k-email job with enforced concurrency while Daft failed to finish.

https://mehulbatra.medium.com/a-field-journal-on-ray-data-and-daft-for-multimodal-data-lake-e0c26839f5b5

Sponsored: AI Modernization Guide

Download the free guide

LakeOps: Routing Multiple Query Engines with Iceberg

Multi-engine Iceberg deployments let Spark, Trino, Flink, DuckDB, Athena, Snowflake, and StarRocks share the same tables. Still, they leave query placement, cost control, dialect compatibility, and workload isolation to each client team. LakeOps writes about QueryFlux, a Rust-based SQL routing proxy that supports existing database protocols, selects engine groups via routing rules, translates SQL dialects with sqlglot, enforces concurrency limits, and dispatches queries based on cost, latency, throughput, or health.

https://lakeops.dev/blog/routing-multiple-query-engines-with-iceberg

Rion Williams: Prepare for Launch: Enrichment Strategies for Apache Flink

Flink enrichment jobs become fragile when they depend on referential data that is not already present in state, forcing teams to choose between slow per-record lookups, incomplete early results, or complex state preloading. The article compares external enrichment, CDC-backed gradual enrichment, State Processor API–based two-phase bootstrapping, and gated enrichment as ways to manage the launch-time pain of warming the state before live traffic depends on it.

https://rion.io/2026/01/27/prepare-for-launch-enrichment-strategies-for-apache-flink/

Agoda: How Agoda Simulates Booking Flows to Test Flight Integrations

Agents perform relatively well when there is a faster feedback loop to tune, but unfortunately, in many business cases, the transactions are sensitive. Simulation is the most critical aspect of engineering components, not only for testing but also as an RL environment for Agents. Digital twins are emerging as a critical component of simulation. Agoda describes a simulation system for booking flows to test flight integrations.

https://medium.com/agoda-engineering/how-agoda-simulates-booking-flows-to-test-flight-integrations-204ec4f2e128

Databricks: Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs

PySpark Pandas UDFs improved Python execution with Arrow serialization and batching, but still incur Pandas/Arrow conversion costs, add memory copies, and struggle with complex data types. Databricks writes about the introduction of native Arrow UDFs, Arrow aggregate functions, Arrow UDTFs, and DataFrame-level mapInArrow and applyInArrow APIs that operate directly on PyArrow arrays, record batches, and tables. Arrow UDFs preserve columnar execution end-to-end, run about 10% faster than Pandas UDFs, use about 40% less memory, and support complex datatype and table-in/table-out transformations with familiar Python and SQL interfaces.

https://www.databricks.com/blog/introducing-arrow-udfs-pyspark-faster-leaner-replacement-pandas-udfs

Giannis Polyzos: When Tables Became the Language of Time

Batch and streaming architectures create duplicated logic, fragmented corrections, and inconsistent state when systems treat logs as the source of truth instead of modeling how data changes over time. The author reframes streams and tables as two views of the same history, using Flink for streaming-first compute, lakehouse commits for durable memory, and Apache Fluss as table-first streaming storage with native updates, deletes, schemas, snapshots, and columnar access. Table-centric streaming collapses batch jobs, replays, late corrections, and real-time dashboards into different temporal views of evolving state, making unification an architectural consequence rather than a glue layer.

https://ipolyzos.substack.com/p/when-tables-became-the-language-of

Data Engineering Weekly #271

Ananth Packkildurai — Mon, 25 May 2026 16:05:00 GMT

How to Build a Data Platform

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating undering a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your free copy now

Netflix: The Evolution of Cassandra Data Movement at Netflix

The Change Data Capture (CDC) from the operational store is often expensive and involves multiple staging hops and an expensive merge operation in Iceberg. Netflix writes one such case study with its Cassandra, the challenges with capturing operational data into Iceberg tables, and its solution to avoid partition skew with a layered approach.

https://netflixtechblog.medium.com/the-evolution-of-cassandra-data-movement-at-netflix-6e13329c80a1

Grab: The Hugo evolution: Engineering Grab’s unified, one-click data ingestion platform with Apache Flink

Grab narrates a similar challenge with the CDC, highlighting the disintegration of data ingestion across multiple operational data stores with schema management and ingestion issues. The unified pipeline, with the Flink pipeline, auto-detects schema changes and ingests the data back into the Hive Tables.

https://engineering.grab.com/one-click-data-ingestion-platform-with-apache-flink

Meta: A Blueprint for Valuing Content When A/B Tests Are Not an Option

Content is a primary driver of the Quest ecosystem. With the recent announcement at Google I/O about seamless shopping integration with content, it is evident that content-driven commerce has reached the mainstream. How do you value the contents when there is no A/B testing option available? Meta writes about implementing the DoubleML method to tackle the challenge.

https://medium.com/@AnalyticsAtMeta/meta-a-blueprint-for-valuing-content-when-a-b-tests-are-not-an-option-7880bac721f1

Uber: Scaling Real-Time Traffic Forecasting with a Graph-Aware Transformer

Uber writes about rebuilding the traffic forecasting stack, DeepETT, a real-time traffic forecasting system. DeepETT approaches forecasting as a fixed-input graph-aware transformer that combines pre-aggregated segment, road-graph, regional, historical, real-time, and event features with continuous Flink-based calibration.

https://www.uber.com/us/en/blog/scaling-real-time-traffic/

Airbnb: Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure

Counting and Finding Unique Users are the two hard problems in Data Engineering.

One of the long-standing questions in data engineering is: since many real-world systems are fundamentally about connections, why can’t we model them using the graph data model? Airbnb highlighted the reasons for the scalability issues with Graph and its adoption of JanusGraph, using DynamoDB as a backend.

https://medium.com/airbnb-engineering/scaling-airbnbs-identity-graph-with-a-unified-knowledge-graph-infrastructure-ebac467b7836

Pinterest: Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

The user journey/user sequence of actions is one of the most important signals for analyzing user behavior. Pinterest publishes a comprehensive case study on how to approach user sequence data as a product and its architectural patterns.

https://medium.com/pinterest-engineering/making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use-2a56a928cae1

Yelp: How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%

Usage-driven data retention & storage class optimization is a must-have tool for your Lakehouse management, given the growing need to ingest more data. Yelp applies the art and science of table management by collecting usage metrics at the table-partition level to optimize storage.

https://engineeringblog.yelp.com/2026/05/partition-access-visualizations.html

LinkedIn: Crosscheck: Benchmarking AI Models in the Real World

Static AI benchmarks lose signal as models optimize toward them, collapsing role-, industry-, and task-specific performance into one number that answers no professional’s actual question. LinkedIn writes about Crosscheck, which extends the Bradley-Terry comparison model with time-decay weighting, low-data regularization, and confidence-aware ordinal tiering — surfacing only differences supported by 95% statistical evidence.

https://www.linkedin.com/blog/engineering/ai/crosscheck-benchmarking-ai-models-in-the-real-world

Jack Vanlightly: Introducing Dimster, a performance benchmarking tool for Apache Kafka

Kafka performance benchmarks rarely travel — results lack the configuration, hardware, and version metadata that another engineer needs to reproduce or trust them. The author builds Dimster, a Kafka benchmarking tool centered on dimensional testing — sweeping config axes like batch.size or consumer type while emitting self-contained result bundles. Dimster runs explore, drain-backlog, and correctness modes on Kubernetes as a portable runtime, making benchmark campaigns reproducible across any cloud or laptop, anchored to traceable result artifacts.

https://jack-vanlightly.com/blog/2026/5/20/introducing-dimster-a-performance-benchmarking-tool-for-apache-kafka

Data Engineering Weekly #270

Ananth Packkildurai — Mon, 18 May 2026 01:54:53 GMT

The Data Platform Fundamentals Guide

Download your free copy now

Airbnb: Viaduct 1.0 and the future of Airbnb’s data mesh

The term Data Mesh focuses on the platform & services, Viaduct aims to be the multi-tenant GraphQL system, unlike the data mesh as we know it in data engineering. However, the first principle of Viaduct is something the data teams should drive towards to bring a more connected experience across the business functions.

One global schema. Independent teams contribute schema and resolvers

https://medium.com/airbnb-engineering/viaduct-1-0-and-the-future-of-airbnbs-data-mesh-6bab4ec98b89

Netflix: Data Projects: Managing Data Assets at Netflix Scale

Netflix writes about the operational challenges of enforcing ACLs at the individual user level. When pipelines run under an On-Behalf-Of permission model, access frequently breaks when employees change teams or leave the organization. To address this, Netflix adopts the concept of Data Projects, replacing user-bound identities with durable, team-owned application identities.

https://netflixtechblog.medium.com/data-projects-managing-data-assets-at-netflix-scale-7ca25888591e

Meta: Migrating Data Ingestion Systems at Meta Scale

Meta writes about successfully migrating tens of thousands of mission-critical pipelines at the petabyte scale. The blog focused on how it ensures the shadow testing mode to enable end-to-end robustness of the migration process. The partition-level metadata flags are an interesting approach. If a partition was marked as “bad,” the system automatically halted new delta landings and forced merges with older, known-good partitions, stopping the bleeding instantly.

https://engineering.fb.com/2026/05/12/data-infrastructure/migrating-data-ingestion-systems-at-meta-scale/

Databricks: The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available

The data infrastructure went through a full lifecycle, from Hive metastore to S3 as the primary metastore, and back to catalogs. Databricks writes about the convergence of table formats and the catalogs, with how Unity Catalog and Delta Lake formats interplay.

https://www.databricks.com/blog/convergence-open-table-formats-and-open-catalogs-catalog-commits-generally-available

Eric Sun: A Query Proxy for Analytical and Fast Data

The author writes about the Query Proxy, which fronts Snowflake, Databricks, StarRocks, and Iceberg via gRPC, with service-credential-based RBAC, and returns async query IDs and presigned Parquet URLs to clients. The proxy collapses per-engine clients into a single interface, federates hot Postgres data with warm Iceberg partitions, and tracks query-shape telemetry, all anchored to a stateless gateway model.

https://eric-sun.medium.com/a-query-proxy-for-371907878996

The Craft: How we rebuilt search ranking at Faire with deep learning

Faire writes about replacing XGBoost with a Deep and Cross Network trained with session-normalized listwise cross-entropy, then accelerates new-ranker development by fine-tuning Brand Page models from Product Search. The rebuild increases Product Search order volume by 2.14% in North America and 1.54% in Europe while cutting new-surface launch cycles by half through a shared multi-task representation.

https://craft.faire.com/how-we-rebuilt-search-ranking-at-faire-with-deep-learning-14f080679c83

Alibaba: When MySQL Meets the Columnar Storage Engine DuckDB in the AI Era

AliSQL writes about integrating DuckDB as a pluggable storage engine, transforming MySQL into a lightweight Hybrid Transactional and Analytical Processing (HTAP) database. Motherduck recently released the client-server protocol DuckDB, as it is slowly moving to a warehouse-style engine on its own.

https://www.alibabacloud.com/blog/when-mysql-meets-the-columnar-storage-engine-duckdb-in-the-ai-era_603117?spm=a2c65.11461433.0.0.2e5d5355O7iepw

Marc Bowes: Aurora DSQL: Meet Coupler

The author writes about the design of Coupler, a highly scalable CDC solution for Aurora DSQL. Traditionally, CDC is hard on databases and degrades performance. Because DSQL’s architecture already decouples its read/write paths using a low-latency journal fan-out, Coupler acts as another subscriber, making it highly performant.

https://marc-bowes.com/dsql-coupler.html

Pete: Full-Text Search with DuckDB

The author writes about DuckDB’s Full-Text Search (FTS) extension for data workflows, such as exploratory text mining and document archive analysis. The development is worth watching, as I have always found cases that balance both free-text analytics and typical exploratory analysis. Apache Pinot & ParadeDB are the two systems I know of that try to handle these cases.

https://peterdohertys.website/blog-posts/full-text-search-w-duckdb.html

Data Engineering Weekly #269

Ananth Packkildurai — Mon, 11 May 2026 02:50:23 GMT

The Data Platform Fundamentals Guide

Download your free copy now

Meta: How We Built an AI Second Brain for 60K Knowledge Workers

Meta writes about building an AI Second Brain that combines a PARA-based workspace model, authenticated infrastructure access via MCPs and CLIs, agentic execution loops, and reusable markdown-defined skills that continuously organize, retrieve, and act on structured work context across tools and projects. The article is a classic case of data engineering evolving into context engineering, making a broader impact in the industry.

https://medium.com/@AnalyticsAtMeta/how-we-built-an-ai-second-brain-for-60k-knowledge-workers-78c507dd795b

Salesforce: How Informatica Built a Multi-Agent AI System to Reduce Data Workflows from Months to Days

Enterprise data workflows break down when discovery, governance, data quality, and orchestration span disconnected systems, requiring repeated human coordination across tools, teams, and execution stages. Informatica writes about the CLAIRE platform, which addresses fragmentation through a multi-agent architecture that combines orchestration agents, semantic context modeling, deterministic tool routing, and specialized execution agents to coordinate workflows involving 50–60 model calls with validation checkpoints and adaptive planning.

https://engineering.salesforce.com/how-informatica-built-a-multi-agent-ai-system-to-reduce-data-workflows-from-months-to-days/

Netflix: Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

ML platforms fragment across registries, orchestrators, feature stores, and experimentation systems, forcing practitioners to traverse multiple tools to trace lineage or reuse assets across domains. Netflix writes about its Metadata Service, which ingests events via a unified AIP URI scheme and materializes a Model Lifecycle Graph in Datomic, connecting models, features, pipelines, and experiments through asynchronous enrichment. The graph collapses multi-system investigations into a single GraphQL traversal from models to features to pipelines to A/B tests, standardizing cross-domain discovery anchored in a normalized entity model.

https://netflixtechblog.com/democratizing-machine-learning-at-netflix-building-the-model-lifecycle-graph-5cc6d5828bb1

Confluent: Stream Processing vs. Real-Time OLAP: Flink, ClickHouse & Pinot Compared

Real-time data platforms routinely collapse when teams treat stream processors and OLAP engines as substitutes, exposing dashboards to stateful pipelines or routing continuous ETL through scan engines. Confluent writes about drawing the boundary at computation timing — stream processors evaluate data in motion through event-time windows and watermarks. In contrast, OLAP engines evaluate data at rest through columnar scatter-gather scans.

https://www.confluent.io/blog/stream-processing-vs-real-time-olap-flink-clickhouse-and-pinot-compared/

Kirill Bobrov: The Power of Data Sketches: A Comprehensive Guide

Exact answers to cardinality and quantile queries become infeasible at the billion-event scale because COUNT DISTINCT shuffles every unique value across the cluster rather than aggregating partially. The author walks through Apache DataSketches — Theta, HyperLogLog, CPC, KLL, and Frequent Items — replacing full-data shuffles with kilobyte hash summaries that merge across partitions. Sketches shift analytics from query-time scans to precomputed dimensional cubes, trading bounded approximation for parallel ingestion and late-data merges, anchored in a fixed-memory streaming model.

https://luminousmen.com/post/the-power-of-data-sketches-a-comprehensive-guide/

Whatnot: The ML Feature Pipeline That Got Slower and No One Noticed

Hourly pipelines leave a shorter window to catch errors, compressing the failure window and exposing silent degradation — zero-row regressions, runtime drift, and cadence slippage that never trigger threshold alerts. Whatnot writes about handling such pipeline failures by splitting monitoring into two tiers — a zero-row signals page on-call, and distribution shifts routed to Slack — backed by a 2-day Redis TTL that absorbs missed updates.

https://medium.com/whatnot-engineering/the-ml-feature-pipeline-that-got-slower-and-no-one-noticed-8e90c224eae3

Pinterest: Enhancing Ad Relevance: Integrating Real-Time Context into Sequential Recommender Models

Sequential recommender models trained on offline behavioral history fail on contextual surfaces because the user embedding ignores what the viewer is currently browsing. Pinterest writes about injecting a context layer into the two-tower query tower — concatenating Transformer history with subject-Pin features and training with synthetic context from positive labels. The hybrid flow precomputes the Transformer offline and runs the context layer online, lifting Recall@K by 3-10x and ROAS by 0.7%, anchored to dynamic user embeddings.

https://medium.com/pinterest-engineering/enhancing-ad-relevance-integrating-real-time-context-into-sequential-recommender-models-bc3a2f9b682e

Grab: Enhancing Flink deployment with shadow testing.

Staging environments cannot reproduce production-specific failure modes in streaming applications — checkpoint incompatibility, traffic shape, and environment drift only surface under live traffic. Grab writes about adding a Shadow Testing stage to its Flink deployment pipeline — running a parallel shadow job in an isolated Kubernetes namespace with prefixed Kafka consumer groups and dedicated shadow sinks. The pre-deployment shadow run absorbs the failure window that caused 10-minute rollback outages, raising Deployment Frequency and cutting Change Failure Rate, anchored to connector-level isolation.

https://engineering.grab.com/enchancing-flink-shadow-testing

Halodoc: Building Self-Healing Data Pipelines at Halodoc

Generic retry-everything pipeline recovery breaks down at scale because CDC restarts, OOM failures, orphaned warehouse queries, and cascading backfills each demand distinct checkpoint logic. Halodoc layers six targeted recovery mechanisms — CDC restart with three-gate eligibility checks, file-size-aware mini-batching, OOM-classified retry scaling, and watermark-based warehouse lock cancellation. The platform cuts CDC recovery to under 5 minutes, drops on-call alerts by 80%, and collapses backfill setup to 15 minutes, anchored to per-failure-mode contracts.

https://blogs.halodoc.io/building-self-healing-data-pipelines-at-halodoc/

Data Engineering Weekly #268

Ananth Packkildurai — Mon, 04 May 2026 01:09:52 GMT

Free Course: AI-Driven Data Engineering

Get started now

Event Highlight: Don't Miss AI Council

- The technical conference for humans who ship

Join the people actually building AI & data infrastructure - and hear them share what’s working, what broke in prod, and what’s coming next. May 12–14 in San Francisco.

Speakers include: The co-inventor of ChatGPT. Creator of DuckDB. Creator of Codex. Engineers from ClickHouse, Databricks, Datadog, and LangChain.

→ Save 20% on your ticket with code DATAEW20 through 5/5

Grab: Data Mesh at Grab Part II: The Foundational Tools behind Certification

Scaling a data mesh across hundreds of thousands of assets demands enforceable trust, consistent governance, and reliable quality without centralized bottlenecks. Grab operationalizes certification through three integrated platforms — Hubble for metadata and event-driven certification, Genchi for continuous quality validation, and a Data Contract Registry enforcing producer-consumer agreements. The system converts data mesh principles into a metadata-driven workflow, reducing dataset sprawl and driving convergence toward certified, reusable assets anchored to analytics and AI workloads.

https://engineering.grab.com/data-mesh-2

Doug Turnbull: Can agents replace the search stack?

Search APIs depend on layered pipelines for query understanding, retrieval, and reranking, creating complexity that struggles to adapt across heterogeneous user intents. The author writes about testing GPT-5 and GPT-5-mini agents with basic BM25 and e5 embedding tools on Amazon ESCI, lifting NDCG from 0.289 to 0.453 — exploration prompts and duplicate-query rejection further closed the gap on smaller models. The findings reframe retrieval for product-style “finding things” workloads as an agent-driven loop. However, knowledge-gap tasks like MSMarco still favor traditional embedding stacks anchored in dense-retrieval quality.

https://softwaredoug.com/blog/2026/04/28/search-apis-replaced-by-agents

Pinterest: Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer

Root-leaf ML serving architectures unlock GPU specialization but bottleneck on network bandwidth when fanning out feature payloads to partitioned model inference. Pinterest writes about building Feature Trimmer, a “Send What You Use” system that treats the model signature as the source of truth — version-aware allowlists sync with bundle deployments through the same staged rollout, fallback, and concurrency safeguards as model configs.

https://medium.com/pinterest-engineering/optimizing-ml-workload-network-efficiency-part-i-feature-trimmer-ae20beb08d69

Sponsored: The AI Modernization Guide

Will your data platform accelerate your AI initiatives or become their biggest bottleneck? Learn how to build a data platform that's ready for AI:

- Transform from Big Complexity to AI-ready architecture
- Real metrics from organizations achieving 50% cost reductions
- Introduction to Components: YAML-first pipelines that AI can build

Get the guide now

Pinterest: From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest

Optimizing ad retrieval for offsite conversions requires modeling sparse, noisy, delayed signals that engagement-based candidate generators were never designed to surface. Pinterest writes about built a two-tower shopping conversion retrieval model that unifies conversions and click-duration-weighted engagement under a single multi-task head, paired with a parallel DCN v2 and MLP cross-layer architecture, and an advertiser-level loss to stabilize sparse Pin-level supervision.

https://medium.com/pinterest-engineering/from-clicks-to-conversions-architecting-shopping-conversion-candidate-generation-at-pinterest-04cae5e1455b

FiveTran: How we accelerated transpilation by compiling SQLGlot with mypyc

Fivetran writes about compiled SQLGlot with mypyc, transpiling type-annotated Python into C extensions, contributing five upstream string primitives, and inlining hot paths like sentinel tokens, native i64 integers, and pre-built dispatch dictionaries — all while preserving the pure Python install path. The compiled sqlglot[c] package accelerates parsing by 5x, SQL generation by 2.5x, and optimizer passes by 2-2.5x, anchored to a dual-distribution model that keeps SQL transpilation portable across multi-engine data lake architectures.

https://www.fivetran.com/blog/how-we-accelerated-transpilation-by-compiling-sqlglot-with-mypyc

Robin Moffatt: Materialized Tables in Apache Flink

Streaming SQL frameworks split table definitions from the population logic, leaving INSERT jobs orphaned across restarts and forcing operators to manage schema evolution and lifecycle as separate concerns. The author walks through Flink 2.2’s Materialized Tables, which bind the refresh query to the table definition and support CONTINUOUS or FULL refresh modes, partition-scoped reloads, suspend/resume via savepoints, and unified batch-streaming semantics through a single FRESHNESS parameter. The construct collapses three legacy patterns — CREATE/INSERT, CTAS, and external schedulers — into a single durable object. However, catalog support beyond Paimon and the embedded scheduler remains anchored in early-stage maturity gaps.

https://rmoff.net/2026/04/28/materialized-tables-in-apache-flink/

Alexey Makhotkin: 5NF and Database Design

Traditional database normalization tutorials present 5NF through contrived table-splitting exercises that obscure the underlying business logic and leave practitioners unable to apply it in practice. The author reframes 5NF design around two logical patterns — the AB-BC-AC triangle for independent M: N relationships across three anchors, and the ABC+D star pattern, where a fourth entity binds three 1:N links — thereby driving table construction directly from business requirements rather than from post hoc decomposition. The approach replaces 5NF reasoning with a deterministic logical-to-physical workflow, anchored to anchor-link modeling that produces normalized schemas without invoking decomposition theorems.

https://kb.databasedesignbook.com/posts/5nf/

ultrathink: SQLite in Production: Lessons from Running a Store on a Single File

Single-file embedded databases promise operational simplicity, but their filesystem-level locking model breaks down when modern container orchestration introduces concurrent writers across overlapping deploys. Ultrathink writes about running a production e-commerce store on Rails 8 with four SQLite databases on a shared Docker volume, diagnosing lost orders through sqlite_sequence after eleven rapid Kamal blue-green deploys caused overlapping containers to corrupt WAL writes despite successful Stripe charges.

https://ultrathink.art/blog/sqlite-in-production-lessons

Capital One: Spark tuning: executor optimization for performance

Spark applications often underperform when executors are configured by default, leaving CPU cores and memory underutilized while introducing fault-tolerance risks and network overhead across worker nodes. Capital One Tech walks through executor sizing trade-offs — fat executors maximize data locality but concentrate failure risk, thin executors improve parallelism but flood the network — and codifies an optimal configuration recipe reserving cores and memory for OS overhead, capping executors at 3-5 cores, and accounting for the 384 MB or 10% memory overhead. The framework converts executor tuning from guesswork into a deterministic sizing exercise, anchored to balanced parallelism, fault tolerance, and resource utilization across distributed Spark clusters.

https://medium.com/capital-one-tech/spark-tuning-executor-optimization-for-performance-c757b39f0efe

Data Engineering Weekly #267

Ananth Packkildurai — Mon, 27 Apr 2026 00:52:38 GMT

Free Course: AI-Driven Data Engineering

Enroll today

Monzo: A “meshy” approach to Data: Enabling 100+ teams to build Data Models

Decentralized data ownership breaks down when cross-team dependencies remain implicit and upstream schema changes silently cascade through downstream models. Monzo introduces Interfaces—explicitly declared, tested dbt models that serve as governed data contracts—stabilizing cross-domain consumption across its 12,000-model warehouse. The migration has already reduced processing costs by 40% and accelerated data landing times by 25%, proving that formalized contracts scale distributed data modeling.

https://monzo.com/blog/a-meshy-approach-to-data

Aparna Dhinakaran: Context Management in Agent Harnesses

Long-running AI agents degrade as context windows fill with unbounded tool outputs, stale conversation history, and redundant file reads. The author analyzes five agent frameworks—Pi, OpenClaw, Claude Code, Letta, and Arize's Alyx—revealing convergence on hard file caps, token-triggered compaction, and isolated sub-agents. These patterns mirror the classical memory hierarchy—registers, cache, and swap—suggesting that context management is maturing into an invisible system-level discipline.

https://x.com/aparnadhinak/status/2048492731929149929

Spotify: Flow generation through natural language: An agentic modeling approach

LLMs struggle to reason over deeply nested domain-specific schemas that lack representation in pretraining data. Shopify builds a bidirectional transpiler that converts its Flow automation JSON into Python—improving syntactic correctness by 22% and semantic correctness by 13% for its fine-tuned Qwen3-32B model. The approach delivers a Sidekick assistant that runs 2.2x faster and 68% cheaper than the closed-source frontier model it replaces.

https://shopify.engineering/fine-tuning-agent-shopify-flow

Sponsored: The AI Modernization Guide

Download the free guide

Pratish Yadava: Data agents - When enterprise analytics learns to reason

Traditional dashboards answer predefined questions but struggle to diagnose root causes or recommend actions within live business workflows. The author outlines continuous data agents that interpret intent and make bounded decisions—anchored in governed semantic layers and modular, domain-specific orchestrators. This operating model moves analytics from passive reporting into decision-adjacent automation with explicit guardrails and escalation paths.

https://medium.com/data-science-at-microsoft/data-agents-when-enterprise-analytics-learns-to-reason-13345ec8998e

Pinterest: Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest

Content platforms waste significant compute re-fetching identical pages, disguised by URL variations introduced by tracking tags, session tokens, and click identifiers. Pinterest engineers MIQPS—a data-driven algorithm that renders pages with and without each query parameter to empirically classify content-changing signals from noise. The system strips redundant parameters at runtime via precomputed offline maps, reducing duplicate fetches and improving catalog deduplication at scale.

https://medium.com/pinterest-engineering/smarter-url-normalization-at-scale-how-miqps-powers-content-deduplication-at-pinterest-4aa42e807d7d

Meltwater: Doing More With Less: Rethinking Entity-Level Sentiment at Scale

Entity-level sentiment analysis scales linearly when systems re-encode the same document once per entity, multiplying inference costs without improving accuracy. Meltwater extracts per-entity embeddings from a single shared Transformer forward pass, proving that local mention context carries sufficient sentiment signal. The approach reduces inference costs by 45.5% and improves accuracy by 3.02%, converting linear per-entity scaling into near constant-time processing.

https://underthehood.meltwater.com/blog/2026/04/23/rethinking-entity-level-sentiment-at-scale/

Halodoc: Implementing Apache Yunikorn on EMR on EKS at Halodoc

Kubernetes-native Spark workloads trigger aggressive node scaling when the default scheduler evaluates pods independently—causing cost whiplash from rapid scale-outs followed by immediate underutilization. Halodoc adopts Apache YuniKorn's bin-packing strategy to fill existing nodes before provisioning new ones, paired with hierarchical queues that govern cross-team resource boundaries. Node utilization reaches 96%, with a 10% reduction in EC2 costs and increased Spot instance adoption due to improved scheduling predictability.

https://blogs.halodoc.io/implementing-apache-yunikorn-on-emr-on-eks/amp/

Netflix: Scaling Camera File Processing at Netflix

Media production pipelines struggle to manage massive daily camera footage when raw metadata remains unconformed and unsearchable across downstream workflows. Netflix integrates FilmLight's API into its Media Production Suite to parse and normalize camera metadata at ingest—conforming it to a standardized schema that enables automated retrieval and pipeline validation. The system deploys as stateless serverless functions on CPU-only instances, scaling elastically to handle spiky VFX plate generation without dedicated GPU infrastructure.

https://netflixtechblog.com/scaling-camera-file-processing-at-netflix-6dab2b1e80be

Z1: Airflow DAG Bundles: Managing DAGs Across Teams Without Helm Upgrades

Onboarding new data pipelines in Airflow typically requires Helm upgrades, pod restarts, and infrastructure tickets—turning every DAG addition into a deployment bottleneck. The author leverages Airflow 3.x DAG bundles with an S3-backed sidecar sync pattern that hot-reloads pipeline configurations without downtime or centralized repository dependencies. New DAGs appear in the Airflow UI within 30 seconds of a commit, decentralizing the entire pipeline lifecycle to self-service.

https://blog.platform.zerotoone.ai/blog/airflow-dag-bundles-without-helm-upgrades/

Data Engineering Weekly #266

Ananth Packkildurai — Mon, 20 Apr 2026 03:09:39 GMT

Free Course: AI-Driven Data Engineering

Enroll today

Animesh Kumar: AI-Ready Data vs. Analytics-Ready Data

Analytics and AI systems fail when teams treat data readiness as a single maturity path instead of two distinct axes optimized for different consumers. The author distinguishes analytics-ready data—designed for human interpretation through aggregation, stability, and explainability—from AI-ready data, which requires contextual completeness, timeliness, and semantic richness often lost in aggregation pipelines.

https://medium.com/@community_md101/ai-ready-data-vs-analytics-ready-data-f67ef0804341

Whatnot: The model is the easy part: Building the LLM Platform at Whatnot

Production LLM systems fail primarily in the surrounding infrastructure rather than the model layer, where non-deterministic outputs, unconstrained inputs, and missing feedback loops undermine reliability and trust. Whatnot writes about structuring its LLM platform around three pillars—velocity, reliability, and trust. It uses the post-exposure A/B logging to isolate divergent outputs, a reusable tool registry, and LLM-as-a-judge calibration workflows to detect production drift early.

https://medium.com/whatnot-engineering/the-model-is-the-easy-part-building-the-llm-platform-at-whatnot-ec8730fa9bdf

Slack: Managing context in long-run agentic applications

Long-running multi-agent systems lose investigative coherence as stateless LLM APIs force context accumulation, degrading reasoning quality and amplifying hallucinations across rounds. Slack writes about addressing this with three context channels—a Director’s Journal for working memory, a Critic’s Review that scores findings on a five-level credibility rubric, and a Critic’s Timeline that prunes incoherent findings by enforcing narrative consistency.

https://slack.engineering/managing-context-in-long-run-agentic-applications/

Sponsored: The AI Modernization Guide

Download the free guide

Atlassian: Engineering the Forge Billing Platform for Reliability and Scale

The billing pipeline, especially the usage-based billing pipeline, is often very complex, given the system's sensitivity to correctness. Atlassian describes Forge Billing as a deterministic pipeline—routing 300M daily usage events through StreamHub and UTS for deduplication and schema validation, then splitting them into cold-tier raw storage and StarRocks hot-tier aggregations for fast developer queries. The architecture handles counter and gauge metrics through idempotency keys and last-write-wins windowing, enabling full charge traceability from Developer Console back to raw events while scaling to 1B daily events within a year.

https://www.atlassian.com/blog/atlassian-engineering/engineering-the-forge-billing-platform-for-reliability-and-scale

Giannis Polyzos: From Events To Real-Time Profiles On Apache Fluss

Real-time entity decisions fail when profile state lives in a separate OLAP layer, introducing ingestion lag between event arrival and visibility. The approach builds risk profiles directly in Apache Fluss by mapping identifiers to integers, encoding group membership as Roaring Bitmaps, and merging updates at write time using the Aggregation Merge Engine without stateful Flink jobs. This design removes the need for a separate profile store, reduces state latency from hours to seconds, and ensures correct recovery through replay-safe inverse operations in the UndoRecoveryOperator.

https://ipolyzos.substack.com/p/from-events-to-real-time-profiles

Thiago Baldim: The journey to Agentic BI

Agentic BI tools amplify data quality issues—ambiguous schemas, undocumented columns, and fragmented sources of truth—rather than resolving them at query time. SafetyCulture writes about rebuilding its data platform on Kimball architecture with SCD Type 2 dimensions, over 90% dbt test and documentation coverage, and column-level ownership aligned to business stakeholders, reducing pipeline execution time from 14 hours to 1.5 hours.

https://medium.com/@thiagobaldim/the-journey-to-agentic-bi-617975c106b7

Pinterest: Scaling Recommendation Systems with Request-Level Deduplication

Scaling recommendation models creates infrastructure pressure because request-level data—especially long user action sequences—is redundantly stored and processed once per candidate item across storage, training, and serving. Pinterest writes about applying request-level deduplication across the ML lifecycle using request-sorted Iceberg datasets, SyncBatchNorm, and user-level masking to preserve training correctness, and a Deduplicated Cross-Attention Transformer that caches user context for reuse across ranked items.

https://medium.com/pinterest-engineering/scaling-recommendation-systems-with-request-level-deduplication-93bd514142d9

Just Eat: Daedalus and the Data Labyrinth

Data governance fails when organizations share data without sharing definitions, ownership, lineage, and reusable business logic across the systems humans and AI agents rely on. Just Eat frames modern governance as a layered navigation system—combining a business glossary, data catalog, metadata, data quality signals, lineage, and a semantic layer—to connect business language with trusted data assets and machine-usable definitions. These governance layers turn complex data platforms from opaque labyrinths into navigable systems, enabling consistent analytics, more reliable AI-generated queries, and a lower risk of conflicting metrics.

https://medium.com/justeattakeaway-tech/daedalus-and-the-data-labyrinth-2c166b1d9866

Teads: We Let AI Agents Orchestrate Our ML Experiments

Teads writes about extending its Datakinator platform with agentic orchestration by exposing APIs through MCP, enriching them with dataset probing and error-retrieval tools, and adding cost guardrails that estimate and gate expensive runs. This approach scales experiment throughput from hundreds to thousands, enables autonomous retry and correction of failed runs, and delivers 5–10% model improvements that translate to nearly $1M in margin gains despite increased cloud usage.

https://medium.com/teads-engineering/we-let-ai-agents-orchestrate-our-ml-experiments-fc8606816fde

A Response to Our Reader Survey

Ananth Packkildurai — Wed, 15 Apr 2026 16:26:18 GMT

Thank you to every reader who took a few minutes to fill out the survey. Your feedback is the clearest signal we have — and this round gave us a lot to act on.

One piece of feedback came through clearly enough that we’d be wrong to gloss over it: several of you feel we’ve leaned too heavily into AI coverage. That’s a fair critique. So we did what data engineers do: we audited ourselves.

We reviewed 233 articles published across 25 issues, from July 2025 through April 2026. The audit made one thing clear — while the majority of our content has remained relevant to data engineers, we have not been as precise as we should be. That precision gap showed up in 44 articles, or 18.9% of the total, which we now classify as outside the newsletter’s editorial scope: prompting tutorials, generic agent framework walkthroughs, model-release commentary, and LLM inference economics that follow the AI news cycle more than the data engineer’s day-to-day reality.

The survey also returned encouraging numbers. Data Engineering Weekly earned a Net Promoter Score of +17.3, and 79.6% of respondents rated their experience as Satisfied or Very Satisfied. We don’t take those numbers for granted. They reflect real signals from practitioners who are selective about where they invest their attention. But the critique is what drove this post, and the critique is what we are acting on.

What the Data Says

Across 233 articles published in 25 issues: Category Articles Share of the total 233 articles

Core DE 99 42.5%
Adjacent but Relevant 60 25.8%
Not DE 44 18.9%
Context Engineering (DE extension) 30 12.9%

The 44 out-of-scope articles are the problem we are fixing. The 189 in-scope articles are the standard we are holding ourselves to.

The survey revealed two truths at once: readers were right to call out drift, and we still believe that some of the work at the intersection of data infrastructure and AI belongs squarely within the future of data engineering. The rest of this post explains exactly where we draw that line — and what changes now.

You can read the complete analysis here
https://docs.google.com/spreadsheets/d/1kLq_touW4Z0SXDjCWEg4zt1DZzBoZFyTM2d2KJAR_tQ/edit?usp=sharing

Our Editorial Categories

Our test is straightforward: if a working data engineer cannot reasonably connect a piece to the systems they build, operate, govern, or evolve, it does not belong in the newsletter.

We use four categories to apply that test. They are not new instincts — they formalize the editorial judgment we have always tried to apply, and they give us, and you, a shared framework for accountability.

Core DE

Classical data engineering topics: ingestion, storage, orchestration, batch/streaming, table formats, query engines, data modeling, quality, governance, and platform reliability.

This is the foundation. Articles in Core DE cover the systems data engineers’ own every day — pipeline architecture, storage layer decisions, table format trade-offs, query engine internals, data quality frameworks, and the operational realities of running data infrastructure at scale.

Context Engineering (DE extension)

Context/semantic layers, ontologies, knowledge graphs, NL-to-SQL, data agents, and other systems that turn enterprise data into governed machine-usable context.

One category deserves a more explicit explanation, because it sits at the center of both the feedback we received and the direction this field is heading.

The reader survey did not change my view that Context Engineering belongs in Data Engineering Weekly. It forced a more disciplined articulation of that view — and a much stricter standard for what qualifies.

If you have read ETL is Dead or Data Engineering After AI, you know where I stand. AI coding agents are reshaping how we work with data, and that shift is not temporary. The data engineer’s value is migrating from pipeline reliability to semantic reliability — from “the job ran” to “the meaning is right.” ETL is dead, the way landlines are dead: the pipelines still run, but nobody builds their data strategy around them anymore. The new leverage lies in designing systems that make AI-generated code trustworthy and grounded in a governed, machine-usable context. Extract, Contextualize, Link rather than Extract, Transform, Load.

I would not be doing justice to our Data Engineering Weekly community if I excluded the engineering questions that come with that shift. The issue was never whether DEW should cover AI-adjacent work. The issue was whether each piece remained grounded in the systems, semantics, governance, and infrastructure that data engineers actually own.

That is the line. In scope: systems that make enterprise data structured, governed, semantic, and machine-readable. Out of scope: prompting tactics, agent framework tutorials, model release commentary, and generic application-layer AI workflows.

Adjacent but Relevant

AI/ML platform, feature store, search/retrieval infrastructure, eval/observability, or AI governance topics that materially touch DE skills or infrastructure without being pure DE.

Data engineers do not work in isolation. The systems they build feed ML platforms, power search infrastructure, and underpin AI governance frameworks. We include this category selectively — only when the infrastructure implications are concrete, the engineering depth is substantial, and the connection to a working data engineer’s responsibilities is immediate rather than abstract. An article on feature store architecture belongs here. A think-piece on the AI market does not.

Not DE

Prompting, generic agent frameworks, model-news / benchmark/market commentary, app-layer AI orchestration, pure ML modeling, or other pieces with no substantial data-infrastructure center of gravity.

This is the content we are eliminating. It may be technically credible and widely read, but it is not written for data engineers and does not serve your work.

What Changes Now

Here is what is different, starting with the next issue:

Zero Not DE articles per issue. Every article will be evaluated against the four categories above before publication. If it does not clear the bar, it does not go in.

A stricter standard for Adjacent but Relevant. The infrastructure angle must be concrete, the engineering substance must be real, and the relevance to a working data engineer must be direct — not abstract or aspirational.

Strict vendor neutrality. Data Engineering Weekly will remain strictly vendor-neutral. We generally exclude content published primarily to promote a product, platform, or company, with rare exceptions for work of exceptional technical depth and broad practitioner value. The content we select is written by practitioners for practitioners and published on personal blogs, company engineering blogs, or open platforms, where the goal is to share real experience, not to drive adoption.

Periodic transparency reports. We will share our category breakdown across recent issues so you can hold us to this standard.

Help Us Build the Signal

Data Engineering Weekly is a community effort. The best issues are shaped by readers who surface work worth sharing.

If you have read something — a post, an engineering deep-dive, a company blog — that fits clearly within Core DE, Context Engineering, or Adjacent but Relevant, submit it. Contributions are reviewed weekly, and we read everything that comes in. The process is lightweight: open a pull request with the article title, link, author details, and topic tags using our Contributing Guide. The same editorial categories and vendor-neutrality standard described here apply to all submissions.

As we advance, each issue should feel sharper, more grounded, and more consistently useful to working data engineers. That is the standard we are setting for ourselves, and the standard we invite you to hold us to.

If you have feedback on this policy or on individual articles, reply directly to any issue. We read every response.

Data Engineering Weekly #265

Ananth Packkildurai — Mon, 13 Apr 2026 03:24:30 GMT

This week: Multi-Tenancy for Modern Data Platforms

Join Brooklyn Data Co. and Dagster Labs for a live deep dive on multi-tenancy for modern data platforms. We’ll cover:

- Code location isolation and project structure patterns
- Managing dependencies across tenants (including AI models)
- Operational strategies that scale with your organization
- Lessons learned from real production implementations

Save your spot for practical guidance that you can apply immediately.

Reserve your spot now

dbt: Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

AI will write all the code generated in data engineering. It is the fundamental shift we all have to prepare for. dbt published its benchmark update, claiming GPT-5.3-Codex with Semantic Layer acheives 100.0% accuracy.

https://docs.getdbt.com/blog/semantic-layer-vs-text-to-sql-2026?version=1.10

Rill: Introducing Metrics SQL: A SQL-based semantic layer for humans and agents

Staying on the metrics and the semantic layer, Rill introduces Metrics SQL to define logic once in YAML and expose it through standard SQL, automating aggregations, enforcing row-level security, and serving governed definitions to AI agents via an MCP server without exposing raw schemas. Deterministic metric resolution eliminates inconsistencies across consumers, while a semantic pushdown roadmap targets native MEASURE support in OLAP engines like ClickHouse and Snowflake.

https://www.rilldata.com/blog/introducing-metrics-sql-a-sql-based-semantic-layer-for-humans-and-agents

Meta: How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines

AI coding assistants fail in proprietary codebases because tribal knowledge—implicit design decisions and legacy constraints—remains absent from training data and documentation. Meta Platforms deploys a swarm of 50 specialized agents to map a 4,100-file pipeline into concise context artifacts, using tiered explorer, analyst, critic, and fixer roles with automated decay detection. This system increases context coverage from 5% to 100%, captures over 50 non-obvious patterns, reduces tool-call volume by 40%, and cuts codebase research time from two days to 30 minutes.

https://engineering.fb.com/2026/04/06/developer-tools/how-meta-used-ai-to-map-tribal-knowledge-in-large-scale-data-pipelines/

Sponsored: The AI Modernization Guide

Download the free guide

Netflix: Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Real-time dashboards with rolling windows invalidate traditional caches because shifting intervals cause repeated misses on otherwise unchanged historical data. Netflix builds an interval-aware caching proxy that decomposes Druid queries into one-minute buckets, serves historical segments from Cassandra, and fetches only the uncached tail from Druid using exponential TTLs ranging from 5 seconds to 1 hour. The system achieves 82% partial cache hit rates, reduces Druid query volume by 33%, improves P90 latency by 66%, and shifts the bottleneck from compute-heavy Druid to low-cost Cassandra storage.

https://netflixtechblog.com/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-22fadc9b840e

Booking.com: Scaling Experimentation Quality at Booking.com

Underpowered experiments, premature peeking, and inconsistent reporting degrade decision quality as experimentation scales without statistical rigor. Booking.com embeds experimental quality across design, execution, and decision-making through data science ambassadors, peer-review practices, and tooling such as a Quality Tab that enforces power calculations and pre-registered hypotheses in real time. These changes increase the share of high-quality experiments, with the largest gains in design, where proper power improves the reliability of results and decision confidence.

https://booking.ai/scaling-experimentation-quality-at-booking-com-726152ee4ef0

Andros Fenollosa: From zero to a RAG system: successes and failures

Legacy engineering knowledge locked in unstructured simulation files and technical documents remains inaccessible when confidentiality requirements prohibit sending proprietary data to external LLM APIs. The author writes about building a local RAG system using Ollama, LlamaIndex, and ChromaDB—filtering out non-text files to reduce indexable load by 54% and serving source documents from Azure Blob Storage while staying within a 100GB disk constraint. The architecture delivers confidential retrieval over 1TB of legacy engineering data while establishing batch checkpointing and error-tolerant ingestion as the critical patterns for production RAG deployments at scale.

https://en.andros.dev/blog/aa31d744/from-zero-to-a-rag-system-successes-and-failures/

All Things Distributed: S3 Files and the changing face of S3

The introduction of S3 Files certainly generated a lot of interest on my reading list last week. I’m still studying the impact of S3 files on data pipeline engineering. One thing to note, S3 Files indeed breaks the read-on-write (Write in S3 Files, but read in S3) consistency model. I wonder if the data infrastructure really wants to go back to that world; nonetheless, this is an exciting blog to read to understand the thought process behind S3 Files.

https://www.allthingsdistributed.com/2026/04/s3-files-and-the-changing-face-of-s3.html

Apache Kafka: KIP-848: The Next Generation of the Consumer Rebalance Protocol

Whether you like or dislike Apache Kafka, its KIPs are among the best learning materials for distributed systems. Consumer rebalancing is one of the hottest debated topics in the Kafka world. KIP-848 moves rebalance logic from consumer clients to the Group Coordinator—introducing a ConsumerGroupHeartbeat API, three-layered epochs for group, assignment, and member state, and server-side Range and Uniform assignors that drive incremental partition reconciliation without global synchronization barriers.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol

Data Engineering Weekly

Data Engineering Weekly #280

Editor’s Note: leetdata.ai & aidataengineer.io updates

Joe Reis: To Every Agent Its Own Database

Datafold: The Modern Data Stack - Open-source edition

OlegWock: Guide to data tools landscape for developers

Housing: Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently)

Hex: The life and times of data teams in the AI era

Bojan Ciric: The Knowledge Spine: Why Your Ontology Needs to Grow a Backbone

JioHotstar: Journey of an Ad Request: The Hidden Engineering

Netflix: In-House LLM Serving at Netflix

Airbnb: Personalizing Airbnb search by learning from the guest journey

Artem Krylysov: How MVCC and Transactions Work in RocksDB

Data Engineering Weekly #279

How to Build a Data Platform From Scratch

William Inmon: Data Management In The Age Of AI

Snowflake: Apache Ossie (Incubating): The New Name for Open Semantic Interchange

Cerebras: How We Built Our Knowledge Base

Sponsored: AI Modernization Guide

Honeycomb: Transforming How We Run Kafka at Honeycomb

StarTree: Designing Apache Pinot for High Availability in the Cloud

Razorpay: How We Refresh Razorpay’s Data Warehouse 10x Faster with Graphs and Indexes

Scribd: Stop Building Models. Start Building Systems.

Matt Butrovich: Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet

Artur Borycki: Puffin-Backed Vector Indexes: Attaching Approximate Nearest Neighbor Indexes to Apache Iceberg Snapshots for Compute-Disaggregated Query Engines

Konstantinos Chantsaris: Designing a Spatial ETL Pipeline in PostGIS

Data Engineering Weekly #278

Deep Dive: See how Dagster uses AI internally

Editor’s Note: Updates on leetdata.ai

BAIR: Intelligence is Free, Now What? Data Systems for, of, and by Agents

Bharath Vadhoola: The language tax on AI

Microsoft: Flint - A visualization language for the AI era

Sponsored: The Data Platform Fundamentals Guide

Netflix: Measuring the Impact of Personalized Recommendations

Grab: Scaling Grab’s Data Lake: Our journey to Apache Iceberg adoption

Lyft: From Day 1 to Production: Building Lyft’s Analytics & Rides Intelligence Assistant as Onboarding Project

AWS: MCP tool design: Practical approaches and tradeoffs

Santosh Shinde: The Medallion Architecture Reconsidered: What It Solved, and Where It’s Cracking

Praveen Krishnan: Direct Lake is Magic, Until It Isn’t: How We Handled 1.5 Billion Rows Without Crashing Power BI

Beyond Redaction: Anatomy of a Privacy-Safe Data Platform

TL;DR: Match the Control to the Utility

The five-stage pipeline: discover, authorize, protect, use, prove

1. Discover and classify candidates—not certainty

2. Authorize the use, not merely the field

Enforce at the point of use

3. Select the smallest control that preserves the required utility

Deterministic tokenization: preserve equality, bound linkage

Three modes, three different promises

Encryption, format preservation, and the names that must stay honest

Safe exchange is a controlled release, not a file write

Evidence first: a hash chain is a building block, not a guarantee

The AI boundary: output scanning is necessary, but not sufficient

What to test in a technical evaluation

The bottom line

Data Engineering Weekly #277

Deep Dive: See how Dagster uses AI internally

Editor’s Note: aidataengineer.io & leetdata.ai are now live!!!

Databricks: From monolith to Lakebase to LTAP: rethinking the database from storage up

Boaz Palgi: Databricks LTAP and the Unfinished Problem of Unified Data

Addy Osmani: Loop Engineering

Sponsored: The Data Platform Fundamentals Guide

Microsoft: Introducing Durable Functions in PostgreSQL

Meta: Meta’s AI Storage Blueprint at Scale

Affirm: Re-architecting Affirm’s Upfunnel Platform: How We Cut Experiment Cycle Time from Months to Days

Stripe: Scaling up your microservice testing with Apache Spark

Target: Scaling Marketing Campaign Forecasting with Generative AI

Expedia: Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs

Announcing leetdata.ai — A Practice Ground for Data Engineers

Why is Data Engineering Weekly launching it

Fundamentals matter more now, not less

Practice is the job

What you’ll find inside

Two asks

Back to first principles

Data Engineering Weekly #276

How to Build a Data Platform From Scratch

Jack Vanlightly: Can We Agree on a Storage/Workload Architecture Taxonomy?

Pinterest: Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

Uber: Zone-Failure-Resilient OpenSearch® at Uber

Sponsored: AI Modernization Guide