Tag: Incremental Load

SSIS Incremental Load with Datetime Columns

Posted on by Reza Rad

Previously I’ve mentioned how to implement Incremental Load within SSIS using CDC (Change Data Capture) feature. Unfortunately CDC is not always supported by the source database, so you have to implement an incremental load solution without CDC. In this post I’ll explain how to implement incremental load if there is a modified datetime column in Read more about SSIS Incremental Load with Datetime Columns[…]

Incremental Load: Change Data Capture in SSIS

Posted on by Reza Rad

15

Incremental Load is always a big challenge in Data Warehouse and ETL implementation. In enterprise world you face millions, billions and even more of records in fact tables. It won’t be a practical practice to load those records every night, as it would have many downsides such as;

  • ETL process will slow down significantly, and can’t be scheduled to run on small periods.
  • Performance of the source, and destination server will be affected badly, downtime of these systems would be longer.
  • More resources will be required to maintain the process. such as better processors, more RAMs… and adding these won’t help so much at the end, because the amount of data is increasing as times passes.
  • and many other issues.

So what would be the solution? Solution is Incremental Load approach. In this approach data will be loaded partially, preferably only part of the data that has been changed. A change set will be much smaller than the total amount of data. As an example in a 200 million records fact table which stored data for 10 years, only 10% percent of that data might be related to the current year and changes frequently, so you won’t usually required to re-load the rest 180 million records.

[…]