From the course: Data Management Essential Training
Unlock this course with a free trial
Join today to access over 24,700 courses taught by industry experts.
Challenges in data processing and integration
From the course: Data Management Essential Training
Challenges in data processing and integration
- [Narrator] This data processing stage of our lifecycle is vital for getting high quality, complete, consistent, and useful data to our decision makers. However, none of this is easy. I want to highlight a few of the challenges you might face in this stage and provide some ideas for conquering them. Data quality issues are the biggest challenge here. As we integrate data from multiple sources in many different formats, it's likely there will be inconsistencies or missing data. Making sure to prioritize the data cleaning I've talked about in this chapter will help to mitigate these challenges and leave you on a strong foot for the next stage. Probably the second biggest challenge is being able to scale as more data sources are integrated, or as the business grows, the amount of data that is going to flow through the system is going to increase. Designing a system that has the ability to scale and manage these increases and still performs is vital to our project being a success…
Contents
-
-
-
-
-
(Locked)
Introduction to data processing1m 28s
-
(Locked)
Data cleaning and preprocessing2m 37s
-
(Locked)
Data transformation techniques2m 39s
-
(Locked)
Extract, transform, load (ETL) processes3m 19s
-
(Locked)
Data integration approaches3m 7s
-
(Locked)
Real-time data processing1m 45s
-
(Locked)
Big data processing2m 3s
-
(Locked)
Challenges in data processing and integration2m 2s
-
(Locked)
Practical look at pipelines4m 47s
-
(Locked)
-
-
-
-
-
-