From the course: Introduction to Modern Data Engineering with Snowflake

Unlock this course with a free trial

Join today to access over 24,400 courses taught by industry experts.

Efficient transformations with streams

Efficient transformations with streams - Snowflake Tutorial

From the course: Introduction to Modern Data Engineering with Snowflake

Start my 1-month free trial Buy for my team

Efficient transformations with streams

“

So far, you've learned how to perform transformations against your data using some of Snowflake's core features: SQL, Snowpark, and UDFs. You'll run into these concepts and features over and over again, and it's good to have a baseline knowledge of them as you learn more about building data pipelines. In this video, we'll level things up a bit and start focusing on efficient transformations. If you've noticed in all of the exercises that we've completed on transformation so far, we've performed our transformations and calculations against entire sets of data. Every time we wanted to modify our views, like daily_city_metrics_v, for example, we reprocessed every single row in that view to do so. But what if we could focus on only processing rows with changes rather than all of the rows within a table or view? That would be far more efficient. For example, let's say you have a table with 20 million rows. You need to calculate an aggregate value, like a sum, using a column in that table…

Contents