In this lesson we will:
- Discuss the approach that business are taking to transforming their data as part of Modern Data Platform solutions.
What Are Data Transformations?
Once we have data loaded into the repository, we will then likely need to cleanse and transform it in order to make it useful for our business.
As an example, imagine we ingest details about all of our thousands of daily e-commerce orders into the database
A typical transformation might be to add these totals into a.
By doing this, the job of our data analysts.
From ETL to ELT
Historically, businesses used a process referred to as ETL to populate data warehouses. This involved Extracting data from source applications and databases, Transforming it into the required formats, and then Loading it into the warehouse for consumption.
In the modern data platform, we flip this process around by executing the transformations directly within the database or data warehouse, after it's been loaded. The process therefore changes to Extract, Load and Transform or the acronym ELT.
dbt
dbt is the leading Modern Data Platform for transormations.
dbt is increasingly being deployed within modern data stacks to carry out data transformation.
An example transformation requirement might be to take all of the incoming customer orders, clean up the data for consistency, and aggregate it into a “sales by region” summary table for our business users.