SSIS Demos in Action: Tutorial Videos

Facebooktwittergoogle_plusredditpinterestlinkedintumblrmailFacebooktwittergoogle_plusredditpinterestlinkedintumblrmail

podcast

SQL Server Integration Services is not a new technology, this technology is a mature data transfer and data consolidation tool which has been in the market since 2005. Prior than that SSIS had another name; DTS. However SSIS nowadays is capable of doing far more than what DTS had to offer.

You can transfer data from any source to any destination, you can upload or download files, you can set priority on execution of tasks, you can call third party applications or command lines, you can get part of the data from Web Service, you can zip or unzip result set, change structure of the result set and load it wherever you want.

[…]

Facebooktwittergoogle_plusredditpinterestlinkedintumblrmailFacebooktwittergoogle_plusredditpinterestlinkedintumblrmail
rssyoutuberssyoutube

Incremental Load: Change Data Capture in SSIS

Facebooktwittergoogle_plusredditpinterestlinkedintumblrmailFacebooktwittergoogle_plusredditpinterestlinkedintumblrmail

15

Incremental Load is always a big challenge in Data Warehouse and ETL implementation. In enterprise world you face millions, billions and even more of records in fact tables. It won’t be a practical practice to load those records every night, as it would have many downsides such as;

  • ETL process will slow down significantly, and can’t be scheduled to run on small periods.
  • Performance of the source, and destination server will be affected badly, downtime of these systems would be longer.
  • More resources will be required to maintain the process. such as better processors, more RAMs… and adding these won’t help so much at the end, because the amount of data is increasing as times passes.
  • and many other issues.

So what would be the solution? Solution is Incremental Load approach. In this approach data will be loaded partially, preferably only part of the data that has been changed. A change set will be much smaller than the total amount of data. As an example in a 200 million records fact table which stored data for 10 years, only 10% percent of that data might be related to the current year and changes frequently, so you won’t usually required to re-load the rest 180 million records.

[…]

Facebooktwittergoogle_plusredditpinterestlinkedintumblrmailFacebooktwittergoogle_plusredditpinterestlinkedintumblrmail
rssyoutuberssyoutube

Speaking at SQL Saturday 352 Sydney: ETL Architecture for Solving Data Warehouse Challenges

Hi, Thanks for all attendees to my session at SQL Satuday 352 Sydney, It was great to meet Grant, and Iman and also some Persian folks in this event. I was talking about my favorite subject: ETL Architecture for Solving Data Warehouse Challenges Session Description: Data Warehouses are heavily in use nowadays in most of Read more about Speaking at SQL Saturday 352 Sydney: ETL Architecture for Solving Data Warehouse Challenges[…]

Speaking at TechEd New Zealand 2014

It is my pleasure to speak in TechEd New Zealand 2014.  TechEd is happening between 9th to 12th of September, and it will be held in SkyCity, Auckland, New Zealand. the topic that I will talk about is:   ETL Architecture for Solving Data Warehouse Challenges Description: Data Warehouses are heavily in use nowadays in Read more about Speaking at TechEd New Zealand 2014[…]

SQL Satuarday 316: Christchurch: ETL Architecture for Solving Data Warehouse Challenges

I am honored that I will speak in SQL Saturday 316 Christchurch on 14th of June 2014. Presentation topic is : ETL Architecture for Solving Data Warehouse Challenges presentation Description: Data Warehouses are heavily in use nowadays in most of businesses around the world. Data Warehousing brings higher performance, faster and more insightful data analysis Read more about SQL Satuarday 316: Christchurch: ETL Architecture for Solving Data Warehouse Challenges[…]

ComCamp 2014; Presentation Files: Solution Architecture for Inferred Dimension Members with SSIS and MDS

Thanks to all attendees of my session in ComCamp 2014 in Auckland about Solution Architecture for Inferred Dimension Members with SSIS and MDS. Here is session Title, Description, and presentation file; Title: Solution Architecture for Inferred Dimension Members with SSIS and MDS Description: Data Warehouses are heavily in use nowadays in most of businesses around Read more about ComCamp 2014; Presentation Files: Solution Architecture for Inferred Dimension Members with SSIS and MDS[…]

Presentation Materials: TechDays Hong Kong 2014, ETL Architecture for Solving Data Warehouse Challenges

Thanks to all attendees of my session in TechDays 2014 Hong Kong about ETL Architecture for Solving Data Warehouse Challenges. Here is session Title, Description, and presentation file; Title: ETL Architecture for Solving Data Warehouse Challenges Description: Data Warehouses are heavily in use nowadays in most of businesses around the world. Data Warehousing brings higher Read more about Presentation Materials: TechDays Hong Kong 2014, ETL Architecture for Solving Data Warehouse Challenges[…]

Script to Generate and Populate Date Dimension Version 2: Adding Multiple Financial Years

Almost a year ago I published first version of T-SQL script to generate and populate a general purpose date dimension here. Today I want to publish the second version of that date dimension which has some corrections (fiscal calendar calculation fixed), and additional attributes (such as current year, current month, and current day) Multiple countries Read more about Script to Generate and Populate Date Dimension Version 2: Adding Multiple Financial Years[…]

Insert, Update, and Delete Destination table with SSIS

Previously I’ve wrote about design and implementation an UPSERT with SSIS. UPSERT is about Update existing records, and Insert new records. Today I want to extend this to cover DELETED records as well. So method used in this post can be used to find INSERTED / UPDATED / DELETED records from the source table and Read more about Insert, Update, and Delete Destination table with SSIS[…]

Speaking about Inferred Dimension Members with SSIS and MDS: Charlotte SQL Saturday

Hi folks, I proud to announce that I will speak in the SQL Saturday #237 Charlotte BI Edition Which will run in 19th of October 2013 in Charlotte, North Carolina  In this session I will speak on one of my favorite topics which is Design and implementation architecture for Inferred Dimension Members with SSIS and Read more about Speaking about Inferred Dimension Members with SSIS and MDS: Charlotte SQL Saturday[…]