Data Engineering Weekly #47

Weekly Data Engineering Newsletter

Welcome to the 47th edition of the data engineering newsletter. This week's release is a new set of articles that focus on Erik Bernhardsson's building a data team, Jamie Brandon's Against SQL, Benn Stancil's self-serve is a feeling, Saxo Bank's enabling data discovery, OpenLineage's backfilling Airflow DAGs using Marquez, Shopify's tuning Trino queries, DoorDash's modularize the recommendation engine, Uber's tuning ML models, FinancialTimes' learning from rapid experiments, and OLAP != OLAP Cubes.


Erik Bernhardsson: Building a data team at a mid-stage startup - a short story

The blog is possibly one of the best narrations highlighting the real-world complexity of data engineering. The blog is walking through various stages of data team as an organization grows and can be a catalyst to bring data-driven culture.

https://erikbern.com/2021/07/07/the-data-team-a-short-story.html


Jamie Brandon: Against SQL

SQL is the lingua franca for databases, but how does this stand with other general-purpose languages? The author takes a fresh perspective on SQL, discussing some of the shortcomings and discuss the possibility of what is after SQL looks.

https://scattered-thoughts.net/writing/against-sql/


Benn Stancil: Self-serve is a feeling

Every organization loves to talk about self-serve analytics, but the definition of self-serve after vague. Is it a bot answering all business questions that is self-serving? The author narrates all aspects of self-serving and emphasizes the importance of chase the self-serve experience that makes the data team and the data consumers feel most at home.

https://benn.substack.com/p/self-serve-is-a-feeling


OpenLineage: Backfilling Airflow DAGs using Marquez

Backfilling is a vital aspect of the data pipeline to fix the computing or produce a newer version. In a typical functional data engineering, backfilling can have a cascading downstream effect. Though systems like Airflow does provide backfilling capabilities out-of-the-box, the scope is limited to DAG definition. Marquez writes an exciting blog that narrates how to use the Marquez lineage API to trigger end-to-end backfilling.

https://openlineage.io/blog/backfilling-airflow-dags-using-marquez/


DataHub/ Saxo Bank: Enabling Data Discovery in a Data Mesh - The Saxo Journey

Saxo Bank writes about its data infrastructure with an in-house central data management application, "Data Workbench." powered by Data Hub and Great Expectations. The blog narrates the data inconsistency issues resulting from inconsistent naming and the Saxo Bank's approach with the data glossary feature.

https://medium.com/datahub-project/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f


Shopify: Shopify's Path to a Faster Trino Query Execution - Infrastructure

Shopify writes about its experience in tuning the Trino query infrastructure. The workload-specific Trino clusters, analysis on the coordinator node congestion, limit the number of drivers per query to preventing the compute starvation are some of the exciting reads.

https://shopifyengineering.myshopify.com/blogs/engineering/faster-trino-query-execution-infrastructure


DoorDash: Leveraging the Pipeline Design Pattern to Modularize Recommendation Services

Doordash writes about its experience applying pipeline design patterns to the explore page to improve the modularization. The blog is an exciting read on the pipeline approach to decoupling retrieval and ranking to efficiently solve the information retrieval problem.

https://doordash.engineering/2021/07/07/pipeline-design-pattern-recommendation/


Uber: Tuning Model Performance

Creating and maintaining a high-performing model is an iterative process. Uber writes about its ML platform Michelangelo and the support for iterative tuning and one-off comprehensive tuning of ML models.

https://eng.uber.com/tuning-model-performance/


Fiancial Times: 6 Lessons from rapid experimentation at the Financial Times

The fast-changing content platform brings challenges to run through A/B testing. The Financial Times writes about its lesson learned from adopting the rapid experiments strategy.

https://medium.com/ft-product-technology/6-lessons-from-rapid-experimentation-at-the-financial-times-19524ea36040


Holistic Blog: OLAP != OLAP Cube

OLAP and OLAP Cubes are often confused, where OLAP specifies the access pattern, and the OLAP Cubes specify the data structure. The blog is walking through the distinction of the two terms.

https://www.holistics.io/blog/olap-is-not-olap-cube/


Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers' opinions.