Data Engineering Weekly #279
The Weekly Data Engineering Newsletter
How to Build a Data Platform From Scratch
We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams, operating under a single platform.
In this book, you’ll learn:
- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly
William Inmon: Data Management In The Age Of AI
The author reminds us that generative AI does not diminish data management; it shifts the control point from database schemas to the text and meaning fed into the model. The case for tying ontologies and taxonomies to the enterprise logical data model is sound, but filtering text alone is not governance—provenance, permissions, and feedback must travel with the context.
https://williaminmon.substack.com/p/data-management-in-the-age-of-ai?r=22julh
Snowflake: Apache Ossie (Incubating): The New Name for Open Semantic Interchange
Apache Ossie’s incubation is an interesting development. I’ve been part of a few sync-ups, and I’m thrilled to see how this is evolving. It is more than a rename: it gives the still-fragmented semantic-layer market a neutral home for portable metrics, dimensions, and business relationships.
https://www.snowflake.com/en/blog/apache-ossie-open-semantic-interchange-incubator/
Cerebras: How We Built Our Knowledge Base
I often make a point that building a knowledge base from the vast corpus of data is a typical data engineering problem, and no, data catalogs have nothing to do with knowledge engineering. Cerebras makes a compelling case for the same, ingesting every source into a single Postgres embeddings table, then fusing full-text, vector, IDF, and age-decay signals using reciprocal rank fusion.
https://www.cerebras.ai/blog/how-we-built-our-knowledge-base
Sponsored: AI Modernization Guide
AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.
Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.
Honeycomb: Transforming How We Run Kafka at Honeycomb
Kafka migrations are constrained less by the target platform than by the semantics hidden in consumers. Honeycomb’s move from Confluent/ZooKeeper on EC2 to KRaft on EKS was shaped by internally checkpointed offsets, rehearsed rollbacks, and telemetry-defined pause points. The strongest lesson is sociotechnical: operational independence is real only when runbooks and expertise survive the original expert—and when customer communication is treated as part of the cutover.
https://www.honeycomb.io/blog/transforming-how-we-run-kafka-honeycomb
StarTree: Designing Apache Pinot for High Availability in the Cloud
StarTree writes about designing HA for Apache Pinot, highlighting that Pinot availability depends on Kubernetes placement, segment replica groups, broker health routing, and durable deep storage sharing the same failure-domain model. The key test is better than any replica count: can the serving path answer queries before Kubernetes rebuilds the failed zone?
https://startree.ai/resources/apache-pinot-for-high-availability/
Razorpay: How We Refresh Razorpay’s Data Warehouse 10x Faster with Graphs and Indexes
Razorpay’s 10x warehouse refresh gain came from rejecting two expensive defaults: rebuilding every denormalized Fact and streaming every join. The warehouse models each Fact as a dependency graph, uses compact secondary indexes to locate historical lake partitions, and leaves high-cardinality dimensions to bucketed Iceberg runtime joins. The deeper lesson is excellent: incremental batch is often the practical middle ground when streaming state and materialization amplify each other’s cost.
Scribd: Stop Building Models. Start Building Systems.
Scribd explains how a fast model, a sampled judge pass, a focused SME review, and a versioned golden dataset enabled a three-person team to classify roughly 375 million items in two months. The design is intentionally replaceable: models and prompts can change, while disagreements, rationales, and regression data compound into durable operational knowledge.
https://tech.scribd.com/blog/2026/fast-llm-human-in-the-loop-classification.html
Matt Butrovich: Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet
The Iceberg community keeps Java planning as the oracle while DataFusion Comet executes reads through Iceberg Rust, turning nearly 10,000 Spark tests into cross-language differential tests. Native acceleration still falls back for writes and format-v3 tables, yet every exposed gap improves the shared Rust library rather than a proprietary fast path—an exciting development to adopt.
https://iceberg.apache.org/blog/accelerating-iceberg-rust-development-with-datafusion-comet/
Artur Borycki: Puffin-Backed Vector Indexes: Attaching Approximate Nearest Neighbor Indexes to Apache Iceberg Snapshots for Compute-Disaggregated Query Engines
The author proposes a more interesting approach that attaches sharded Vamana (Graph-based ANN) indexes to Iceberg snapshots via Puffin, inheriting atomic commits, time travel, and garbage collection while keeping executors disposable. The architecture is elegant for analytical retrieval, but still a design-stage bet: benchmarks remain projected, while large index storage, snapshot-bound freshness, and shard-dependent recall separate it from low-latency serving.
https://arxiv.org/html/2606.04196v1
Konstantinos Chantsaris: Designing a Spatial ETL Pipeline in PostGIS
The author shows how Spatial ETL fails quietly when geometry semantics are treated like ordinary columns: normalize coordinate systems, keep fragment-level physical measures separate from commune-level population measures, and index the exact geographic expression used in distance queries. The most valuable lesson is not PostGIS syntax; projections, spatial granularity, and analytical denominators are part of the data contract because each can produce believable nonsense.
https://medium.com/@kchantsaris_84495/designing-a-spatial-etl-pipeline-in-postgis-72565b53aeb8
All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.