Integrate Power BI Into Your Application: Part 6 – Real-time Streaming and Push Data

It’s been a while since the last post about Power BI REST API. I have written previously about how to get the REST API, and use it in your application, how the authentication works, and things such as embedding content, and data source management. In this post, I’m going to explain one of the functionalities of REST API which is pushing data to Streaming data set. I’ve previously explained how to do real-time streaming using Azure Streaming Analytics and Power BI. However, this post explains real-time functionality just with the REST API. You won’t need azure services besides it. If you want to learn more about Power BI, read Power BI book, from Rookie to Rock Star. If you want to learn more about REST API, follow this series:

Part 1: Register your Application

Part 2: Authenticate

Part 3: Embed Content

Part 4: Refresh Data Set

Part 5: Data Source and Gateway Management

Prerequisite

For running the sample in this post, you need to have an application that is registered and followed the authentication process. If you want to see how it works, here are the two steps you have to have prepared as the prerequisite:

Step 1: Register your Application

Step 2: Authenticate

You also need a Power BI account to create a data set in the service.

Streaming Dataset

In Power BI, you can create different types of dataset. The normal data set that everyone knows about it is a data set that automatically generates when you create a Power BI Desktop solution and publish it to the website. However, There is another very important type of data set, called Streaming data set. This dataset is used for pushing data into Power BI in a real-time scenario.

How to create a streaming data set

Go to Power BI Service (http://powerbi.microsoft.com) website. Log in to the website with your Power BI Account. When logged in create a new streaming dataset;

There are three ways to create a streaming dataset. I have explained the Azure Stream Analytics previously. For this example, choose API;

In the next part, is where you specify the name of your dataset, and add fields to it. Your fields can be of type DateTime, Text, or Number. with adding every new field, you will see a JSON format will be created underneath, which is the format that the data needs to be sent through REST API.

Push, Hybrid or Streaming?

One very important option to choose is the Historic Data Analysis. By default, this option is off. It means the streaming data will not be stored anywhere. Service keeps up to 200,000 rows of data. when row # 200,001 comes in, the first row will be deleted. If you want the history to be saved, you have to switch this option to On. When you turn this option your dataset from a streaming dataset, changes to be a Hybrid dataset.

Streaming dataset

Only streams the data rows. Keeps up to 200,000 rows. First In, First Out. When row #200,001 comes in, row #1 will be removed. Cannot be used easily in reports, because there is no historical data. You can create a report with that, however, because it only focuses on streaming data. not all options are available.

Hybrid dataset

Streams the data, and also keep the data rows. You will have the rich experience of Power BI Report. Because data is stored, you can interact with that easier from a report.

Push dataset

Created when you create a solution from Power BI Desktop with importing data into it. This dataset is not a streaming dataset. it just stores the data. the data should be then scheduled to refresh. we are not talking about this type of dataset in this post.

In this example, doesn’t matter which option to choose.

Create a Dashboard and Report

After creating the dataset, you need to create a report with a simple line chart with value and timestamp on it, and then save it, and pin it to a dashboard. The dashboard’s chart at the moment will be blank because there are no data entries in it for now. In the next step, we will be adding rows into this data set from an application.

Connecting to the dataset from REST API

As mentioned before, you need to prepare your application for this step as a prerequisite. In the application ready for this (after registration, and authentication), you can add few lines of code to pass this data through REST API. Here is a sample code;

// Create a Power BI Client object. It will be used to call Power BI APIs.
            using (var client = new PowerBIClient(new Uri(ApiUrl), tokenCredentials))
            {
                // Get a list of dashboards.
                var datasets = await client.Datasets.GetDatasetsInGroupAsync(GroupId2);

                // Get the first report in the group.
                var dataset = datasets.Value.FirstOrDefault();

                if (dataset == null)
                {
                    return View(new TileEmbedConfig()
                    {
                        ErrorMessage = "Group has no datasets."
                    });
                }

                Random random = new Random();                

                for (int i = 0; i <= 5000; i++)
                { 

                var requestBody = new RowsRequestBody()
                {
                    rows = new List<OnlineData>()
                    {
                       new OnlineData() { ts = DateTime.Now, value = random.Next(0, 100) }
                    }
                };

                await client.Datasets.PostRowsInGroupAsync(GroupId2, dataset.Id, "RealTimeData", requestBody);
                    System.Threading.Thread.Sleep(1000);
            }

Majority of the code above is about finding the dataset (which I explained it in details in data source management and refreshing dataset posts). After finding the dataset, I used a loop (highlighted lines) to create a random number from 0 to 100, and pass it to the service and dataset with a function named PostRowsInGroupAsync. This process is in a loop and happens every second for 5 minutes. as a result, now you can see the dashboard data changes;

Summary

In this post, you’ve learned how easy is to create a streaming dataset and interact with that with REST API. You’ve learned that there are two types of datasets that you can work with for streaming; Hybrid and Stream only. One is storing data as well as streaming it. the other one is just streaming. If you are interested to learn more about REST API, read blog series below;

Part 1: Register your Application

Part 2: Authenticate

Part 3: Embed Content

Part 4: Refresh Data Set

Part 5: Data Source and Gateway Management

Reza Rad

Trainer, Consultant, Mentor

Reza Rad is a Microsoft Regional Director, an Author, Trainer, Speaker and Consultant. He has a BSc in Computer engineering; he has more than 20 years’ experience in data analysis, BI, databases, programming, and development mostly on Microsoft technologies. He is a Microsoft Data Platform MVP for 12 continuous years (from 2011 till now) for his dedication in Microsoft BI. Reza is an active blogger and co-founder of RADACAD. Reza is also co-founder and co-organizer of Difinity conference in New Zealand, Power BI Summit, and Data Insight Summit.
Reza is author of more than 14 books on Microsoft Business Intelligence, most of these books are published under Power BI category. Among these are books such as Power BI DAX Simplified, Pro Power BI Architecture, Power BI from Rookie to Rock Star, Power Query books series, Row-Level Security in Power BI and etc.
He is an International Speaker in Microsoft Ignite, Microsoft Business Applications Summit, Data Insight Summit, PASS Summit, SQL Saturday and SQL user groups. And He is a Microsoft Certified Trainer.
Reza’s passion is to help you find the best data solution, he is Data enthusiast.
His articles on different aspects of technologies, especially on MS BI, can be found on his blog: https://radacad.com/blog.