Data Engineering Weekly #66
Weekly Data Engineering Newsletter
Data Engineering Weekly - Brought to You by RudderStack - the Customer Data Platform for Developers
RudderStack Provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools.
There is a lot of exciting stuff to catch up on in DataEngineeringWeekly this week, especially since we took off last week.
dbt labs: Coalesce - the analytics engineering conference 2021
Coalesce - the analytics engineering conference is a delight to watch this week. dbt as a metric layer is an exciting evolution and is the key highlight of the conference. If you missed the conference, you can watch all the replays here.
James Le: What I Learned From the Open Source Data Stack Conference 2021
Open-source Data Stack conference is another exciting data conference focused on the modern data stack in the open-source world. The author writes an excellent summary of the conference.
You can catch up on all the replays of the open source data stack conference 2021 here.
AWS: Top Announcements of AWS re:Invent 2021
AWS published the top announcements from the AWS re: Invent 2021 conference. The top announcements from the data engineering perspective are,
SageMaker Studio Lab, a free service to learn and experiment with ML
The availability of new storage-optimized EC2 instances (Im4gn & Is4gen)
LinkedIn: Evolving LinkedIn’s analytics tech stack - Lessons from a large-scale data platform migration
LinkedIn shares its analytical stack transition story from Teradata data warehouse systems to open source big data technologies. The analytical stack includes 1400+ datasets, 900+ data flows, and 2100+ users. The
migration strategy with improving the data model is an exciting read.
Tableau: Top Data books of 2021
Though the title says the top data books, the shortlisted books focus on data visualization or Tableau platform. Nonetheless, it is great to read data visualization books.
Sponsored: The Data Stack Show Live - What is the Modern Data Stack?
Join The Data Stack Show Live for a special panel with experts from Databricks, dbt, Fivetran, Essence VC, and Hinge. The panel will look at the modern stack from all angles and discuss the future of data tooling.
Erik Bernhardsson: Storm in the stratosphere - how the cloud will be reshuffled
Erik Bernhardsson writes an exciting prediction on cloud vendors' trends builds a case on top of the success of Snowflake over Redshift. It is undoubtedly true in the analytical world where AWS solutions always package and sell open-source tools but never go beyond simplifying the developer workflow. A couple of interesting predictions to highlight,
Kubernetes will be some weird thing people loved for five years, just like Hadoop was from 2009-2013, but the world will move on.
YAML will be something old jaded developers bring up after a few drinks. You know it's time to wrap up at the party at that point.
Shreya Shankar: The Modern ML Monitoring Mess: Rethinking Streaming Evaluation
A streaming sliding window with a finite interval is a go-to ML metrics monitoring strategy. The threshold, window size, and alerts are still defined manually for each metric. The author argues why this procedure to evaluate ML on streams of data is broken, highlighting representation differences, varying sample size & delayed feedback on the sliding window.
Microsoft: SynapseML - A simple, multilingual, and massively parallel machine learning library
Microsoft announces the release of SynapseML (previously MMLSpark), an open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines. With SynapseML, developers can build scalable and intelligent systems for solving challenges in domains such as Anomaly Detection, Computer Vision, Deep Learning, Text analytics, etc.
Data@Monzo: Mapping our data journey with column lineage
Monzo writes about its journey to bring column-level lineage to track and understand scope changes across the data warehouse and automatically detect unused columns. TIL about
ZetaSQL, which helps to parse & analyze BigQuery Sql, and looking forward to playing around with it.
PayPal: Building Data Quality into the Enterprise Data Lake
PayPal writes about Rule Execution Framework to manage a centralized rule configuration system to manage data quality rules & rulesets. The adoption of SQL to write complex data validation rules and the workflow focused on the domain owners to define the data quality rules are exciting.
Links are provided for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employers’ opinions.