Combine CSV Files without Duplicating Headings using C#

By:
On:
In: Data Integration
With: 3 Comments

The simple script below shows how to combine csv files without duplicating headings using C#. This technique assumes that all of the files have the same structure, and that all of them contain a header row. The timings in this post came from combining 8 csv files with 13 columns and a combined total of 9.2 million rows. I first tried combining the files with the PowerShell technique described here. It was painfully slow and took an hour and a half! This is likely because it is deserializing and then serializing every bit of data in the files, which adds a lot of unnecessary overhead. NextRead More →

Select columns from multiple CSV files

By:
On:
In: Data Integration, PowerShell
With: 0 Comments

I recently encountered a scenario where I needed to use PowerShell to select columns from CSV files and output the results to a new set of files. This was necessary because an additional column was accidentally introduced to CSV files that were being loaded hourly with SSIS. When the additional column appeared, SSIS began to choke on the files, resulting in package failures. The hourly process actually continued working, but a couple dozen CSV source files needed to be modified before they could be loaded into the data warehouse. The diagram above shows the required modification. PowerShell to the Rescue PowerShell is my favorite Business Intelligence tool that isRead More →