From the course: Data Management Essential Training

Unlock this course with a free trial

Join today to access over 24,700 courses taught by industry experts.

Challenges in data processing and integration

Challenges in data processing and integration

From the course: Data Management Essential Training

Challenges in data processing and integration

- [Narrator] This data processing stage of our lifecycle is vital for getting high quality, complete, consistent, and useful data to our decision makers. However, none of this is easy. I want to highlight a few of the challenges you might face in this stage and provide some ideas for conquering them. Data quality issues are the biggest challenge here. As we integrate data from multiple sources in many different formats, it's likely there will be inconsistencies or missing data. Making sure to prioritize the data cleaning I've talked about in this chapter will help to mitigate these challenges and leave you on a strong foot for the next stage. Probably the second biggest challenge is being able to scale as more data sources are integrated, or as the business grows, the amount of data that is going to flow through the system is going to increase. Designing a system that has the ability to scale and manage these increases and still performs is vital to our project being a success…

Contents