Fetch Files and/or Folders with Filtering and Masking: Power Query

Posted by on Aug 3, 2015 in Power BI, Power Query | 2 Comments
Facebooktwittergoogle_plusredditpinterestlinkedintumblrmailFacebooktwittergoogle_plusredditpinterestlinkedintumblrmail

4

Fetching list of Files in a folder is easy with Power Query, it is one of the built-in source types. However fetching list of folders is not a built-in function. In this post I’ll show you how to fetch list of files, and also fetch only list of folders. The method explained in a way that you can customize the code and apply any conditions as you want later on. Conditions such as File or folder name masking to fetch only names that contains special character strings.

Fetch All Files in a Folder

For fetching all files in a folder you can simply use the GUI Get Data, and under File, choose Folder

1

Browse for Folder

2

and then simply you will see list of all files.

3

As you see in the above table extracted, there is a column named Content, which has the content of the file. you can click on that to see content of the file if you want.

What is the M Code Written Behind the Scenes

The code behind the scenes for this transformation used Folder.Files function

Fetch All Files and Folders

There is another M function to fetch all files and folders listed under a folder, named Folder.Contents. This function returns the Content column with data type of the record and content of it. for Files data type usually is Binary, and for Folders it is table. So it can be easily distinguished and separated. Here is list of all files and folders fetched:

Result set contains both files and folders

4

As you see the Content column shows the data type of the value. To check if a record is Folder or not, we have to compare its data type to Table. We can add a column to the table to check data type of the content column in each record. data type can be checked with Value.Is function.

And the result set shows if the record is folder or not

5

To fetch only folders, we can filter the data set with Table.SelectRows function. I’ve sorted the result set descending by creating date of the folder. Here is the code:

And the result:

6

 

Fetch Files and Folders with Masking

Now that you’ve got through the M code, it would be really easy to apply any masking option to this. Here are couple of examples:

1- Fetch Only Folders Created after Specific Date

Result

7

2- Fetch Only Files with .txt extension and name similar to “amp”

Result:

8

 

Facebooktwittergoogle_plusredditpinterestlinkedintumblrmailFacebooktwittergoogle_plusredditpinterestlinkedintumblrmail
rssyoutuberssyoutube
Reza Rad
Reza Rad is an Author, Trainer, Speaker and DW/BI Consultant. He has a BSc in Computer engineering; he has more than 15 years’ experience in databases, programming and development mostly on Microsoft technologies. He is a Microsoft MVP in Data Platform for seven continues years (from 2011 till now) for his dedication in Microsoft BI. He is author of some SQL Server and BI books, and also Power BI online book; from Rookie to Rock Star.

2 Comments

  • Awesome Post ! Power Query is an amazing tool !

    I am looking to fetch workbooks from a folder within another folder. So say I have folder X which contains 100 other folders. In each folder is another folder called Y where my work book is. Do you know of a way to have power query reference a main folder (X) then for each folder within (X) find the folder that contains my workbook to query?

    Cheers!

    • Hi Stefan,
      Thanks for your kind feedback.
      It is easy to do such thing in Power Query. start with Get Data from the main folder. Choose Folder as the source. and all files under any sub folders will be showed as a table output. then filter name of the file you want.

      Cheers,
      Reza

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">