I’ve hear this a lot in my presentations that “Does Power BI support real-time data?”, And I answer yes. In fact this feature was available long time ago with Azure Stream Analytics and Power BI, however recently some new changes happened in this area which allows Power BI dashboards to monitor real-time data changes on the dashboard from a data coming from API, and also PubNub. In this post I’ll show you how easy is to create a dashboard that shows real-time data coming from API. If you want to learn more about Power BI, read Power BI online book; from Rookie to Rock Star.
Real-Time Data, Instant Monitoring
One of the key features of monitoring tools is the ability to monitor the real-time data instantly when the data refreshes. That means as a user you don’t need to click on refresh button to see the latest data. This single key feature brings heaps of value to Power BI as a monitoring tool. This ability isn’t new, in fact Amir NetZ showed an awesome demo of an IoT device working with Azure Stream Analytics and Power BI as a real-time monitoring. So what is new is that we can now have this feature in the new generation of Power BI with a dashboard tile connected to streaming data. Streaming data sets can be now either API, Azure Stream Analytics (coming soon), and PubNub. More streaming services will come very soon I believe.
In this post I want to show you only streaming data through API, and showing that in a dashboard tile in Power BI. This single streaming option by itself opens many gates to Power BI as a real-time data monitoring tool. So let’s start.
Prerequisite
For this example I will be using Microsoft sample .NET application named PBIRealTimeStreaming. You can download it from here.
You need Visual Studio (Not SSDT) for running this example solution.
Real-time Data Set
Start with logging in to Power BI service, create a new dashboard (or open an existing dashboard). And create a New Tile.
In the New Tile pane you now can see an option to add a tile for real-time data.
Select this option and click Next. if this is the first time you are using this service, then you probably don’t have any streaming data set existing there. So go to Manage Data to create one;
Now you are in streaming data sets window. you can see list of existing data sets, or create a new one with click on Add Streaming Dataset option on top right hand side.
Create Sample Streaming Data Set
As I’ve mentioned before at the time of writing this post only these three options for streaming data is available: API, Azure Stream Analytics, and PubNub (which is a data streaming service).
Because for this example we are using API as a streaming option, choose API and continue. Now create a new data set with a name such as mySampleRTDS or any other names you like. You have to create two columns as below;
- ts column with DateTime data type
- value column with Number data type
These are columns that will be created and passed through a .NET application through Power BI API to this data set.
Now click on create. This will create the data set with a Push URL that can be used in the application through API. copy this URL. We will be using that in the application next step.
Sample Streaming Data API Application
As I mentioned before, download the sample project. and then select the project under this path:
samples/consoleapp/getting-started-for-dotnet-real-time/PBIRealTimeStreaming
And open the PBIRealTimeStreaming solution in Visual Studio. This project creates a data set including a datetime value and a numeric value and will pass that through Power BI API (which will be discussed later in another post) to Power BI service.
Open Program.cs file and in line 33 change the value of realTimePushURL to the URL which we’ve copied from above step.
Run the application with pressing F5, or right click on project, Debug, Start New instance. The application will generate a data point in each second and send it through API.
Create Dashboard Tile
Now, let’s create the dashboard tile for this streaming data set. Follow the steps and choose the mySampleRTDS as the data streaming source for the tile.
There are number of charts/visualizations that you can use with streaming data set. Choose Card for the first example. and set the value of that to “value” data field
Add another dashboard tile for the same data set, and set this to be a Line chart with ts field in Axis, and value field in Values.
After adding each chart you will immediately see the real-time dashboard working and changes every second (because streaming data from application updates every second);
API Data Can Be Anything
The sample used here had only one datetime field and a value field. However you can create your own data stream from any event, notifications, or data set that you want to be monitored real time. With a simple application the stream of data can be generated. Or it can come from Azure Stream Analytics or PubNub. Don’t limit your options, think about where you can use this feature to solve a real-time monitoring challenge.
Reza is author of more than 14 books on Microsoft Business Intelligence, most of these books are published under Power BI category. Among these are books such as Power BI DAX Simplified, Pro Power BI Architecture, Power BI from Rookie to Rock Star, Power Query books series, Row-Level Security in Power BI and etc.
He is an International Speaker in Microsoft Ignite, Microsoft Business Applications Summit, Data Insight Summit, PASS Summit, SQL Saturday and SQL user groups. And He is a Microsoft Certified Trainer.
Reza’s passion is to help you find the best data solution, he is Data enthusiast.
His articles on different aspects of technologies, especially on MS BI, can be found on his blog: https://radacad.com/blog.
thanks Reza, will this work for connecting to a SQL DB through Odata?
Hi Brad,
this works with push mechanism, SQL DB or Odata doesn’t support push by itself. So you need an application to read data from those and push it. which can be similar to the .NET application used here and pass the data to REST API.
With SQL DB you can have DirectQuery which is different story from pushing real-time data.
Cheers,
Reza
I find your blogs excellent Reza, keep them coming.
I saw this one a while ago and have been waiting for the mysterious “Azure Stream Analytics” streaming dataset option to pop up on my PowerBI desktop. Most people are seeing API and Pubnub only. Any idea when the Streaming Analytics option will be available – how do you see it and I don’t?
It is tempting to migrate my Streaming Analytics source back to Pubnub just to get this feature.
Thanks Leigh for your kind words and feedback. Glad you liked it.
For stream Analytics the process is different. Instead of creating Data set from Power BI and connect it to Stream Analaytics (like what you do for REST API and PubNub), you have to start process from Stream Analytics itself. there in Stream Analytics you can define the output to be Power BI Data set (you can define name of the data set, the work group of it, and name of table etc). Here is a blog post with a full example of Stream Analytics and Power BI I have written:
https://radacad.com/stream-analytics-and-power-bi-join-forces-to-real-time-dashboard
Cheers
Reza
Which version of Visual Studio was this developed on? I’m using VS 2010 and when I try to open the SOLN or PROJ file it says:
This project is incompatible with the current version of Visual Studio.
Hi Ryan,
It is with VS 2015 I believe
Cheers
Reza
Hi
Tried your sample but can’t make it work. I see that the timestamp ha a different format from the application compared to your screen dumps in the article.
I get :
“[{ \”ts\”: \”13.05.2017 06.14.57\”, \”value\”:39 }]”
An the tile do not show anything with this format.
Any ideas what is going on ?
Regards
Hi Rolf,
You can try and use Format String to something with YYYY MM DD HH mm SS. maybe this is because of your computer locale configuration.
Cheers
Reza
HI I am unable to see azure stream analytics data set option when we are creating a new data set how to to get that one to my account. why I am unable to find that one any idea on it ?
You won’t see this in Power BI.
You need to set this up from Azure portal. Create azure stream analytics. and then an output for Power BI
Cheers
Reza
what is difference between live data and streaming data??
Hi,
Streaming data: helps you to build a dashboard which changes any time a new data entry is available.
Live query: means your dashboard data will be always up-to-date from the data source. but live query is not aware of changes in the data source. so dashboard refreshes every 15 minutes to give you up-to-date information.
Cheers
Reza
Hi i want to be create one dashboard to show all server DB size,DB growth for weekly and server space etc.how can i create like this kind of dashboard.please suggest.
Regards,
Saran
Hi Sara,
Do you want such dashboard to be real-time?
If so, then you need to create an application that calculate these information and send it through REST API to Power BI.
otherwise if it is not necessary to be real-time. then create the report in Power BI Desktop with scheduled refresh.
Cheers
Reza
Hi there. I am trying to link Power BI to Project Online custom fields. Are you able to please help me do this? Cheers
Project Online seems to have OData feed. so you can use that in Power Query as a data source.
https://msdn.microsoft.com/en-us/library/office/jj163015.aspx
Cheers
Reza
Hi,
I’d like to ask one thing. Is it possible to use Python? Can we write such an application in Python and push the data.
Regards,
Kamal
Hi Kamal
I think (not sure) you should be able to do it from there too. for sending real-time data, you need a REST API URL, authentication configuration, and a mechanism to send JSON scripted data. if these are possible to do from Python, then you are good to go.
Cheers
Reza