REAL-TIME CHANGE DATA CAPTURE USING STAGING TABLES AND DELTA VIEW GENERATION FOR INCREMENTAL LOADING OF LARGE DIMENSIONS IN A DATA WAREHOUSE - Miss. Paurnima Ghugarkar; Mr. Yogesh Borude; Prof. Prabhudev Irabashetti (International Journal of Innovations in Engineering Research and Technology, 2021)
In the big data era, data become more important for Business Intelligence and Enterprise Data Analytics system operation. The load cycle of traditional data warehouse is fixed and longer, which can’t timely response the rapid and real time data change. Real-time data warehouse technology as an extension of traditional data warehouse can capture the rapid data change and process the real-time data analysis to meet the requirement of Business Intelligence and Enterprise Data Analytics system. The real-time data access without processing delay is a challenging task to the real-time data warehouse. In this paper we discusses current CDC technologies and presents the theory about why they are unable to deliver changes in real-time. This paper also explain the approaches of dimension delta view generation of incremental loading of real-time data and staging table ETL framework to process the historical data and real-time data separately. Incremental load is the preferred approach in efficient ETL processes.