Object Detection Application with AI Builder and Power Apps

In the last posts, I have explained how we able to set up the AI Builder for Power Apps and Microsoft Flow. Also how we able to do classification in Power Apps has been explained.

Now in this post, I am planning to show how to create an Object detection app using AI builder in Power Apps.

Prerequisite :

1- First you need to set up your AI builder, so it did not please follow the  link: See more

2- you need to have an excel file that contains the label for prediction 

3- A list of pictures, regarding specific objects

we are going to create an application, to detect the Green Teabox in the picture.

Need to create an Excel file for labelling the file.


In the Power Apps, click on the data and entity, then click on the Get Data.

In the choose data source page, click on the Excel connector.


you need to first upload the file into Onedrive. then choose the file.

Browse the link and then click on the Next to load the data.

Select the Green Tea, then click on the Next.


The first row, need to be as header


After transforming the data, click on the Next.


In the Map Entities, for Load setting, choose the “Load to New Entity”, then choose the option “Delete rows that no longer exists”.  For the Primary name field, choose the “Lable” and for the filed type, change the Multiline text, to the Text.

Now click on the Next.



the data will be available.

After loading the data, check the Data section, to see the Green Tea.


Then, in Power Apps, click on the Build option, to create an object detector.


choose Object Detection, from the list.


In the new page, choose the name of the Model Name, then click on the Create model.


The first step is to “Select the Object Name” that is the Label of data.


Choose the file Lable Name “Green Tea”.


You need to choose the Label field.



For all objects, choose all labels to select all “Green Tea Rose”, “Green Tea Rose”, and “Green Tea Cinnamon”.


In the next steps upload the images, at least 15 images need to be selected for each label.

Now after loading the images, need to tag it.


For each picture, you need to tag it, again for each label you need to assign a picture.

You need to assign a tag for each picture. As you can see in the picture, you able to add tags for it. select the picture and select the boundary for it then add the related tag for it.


In this example, I already uploaded 30 pictures and tagged all of them.

Now you need to train the model, that is the fourth steps.




The model will be trained and this process takes a couple of minutes.



After a couple of minutes, the Model will be trained and become available to be published.

After the model train, click on it to see the accuracy. Also, you can now publish to be available for application.


Also, you can test it before creating an application to see how it works, click on the “Click Test” option, then import a test image.

Now you can see the classification and object detection.

After publishing the model, back to Home tab and choose a “Blank Canvas” to create an object detection application.



Now, you can create a smart application in mobile format, click on the insert, then select the AI Builder. from AI Builder menu, choose the Object detector.


Now just drag and drop the component into the white area

Just tes the application and import a new image to see how it works ( to run the application click on the Run button at the right side)



Leila Etaati on LinkedinLeila Etaati on TwitterLeila Etaati on Youtube
Leila Etaati
Trainer, Consultant, Mentor
Leila is the first Microsoft AI MVP in New Zealand and Australia, She has Ph.D. in Information System from the University Of Auckland. She is the Co-director and data scientist in RADACAD Company with more than 100 clients in around the world. She is the co-organizer of Microsoft Business Intelligence and Power BI Use group (meetup) in Auckland with more than 1200 members, She is the co-organizer of three main conferences in Auckland: SQL Saturday Auckland (2015 till now) with more than 400 registrations, Difinity (2017 till now) with more than 200 registrations and Global AI Bootcamp 2018. She is a Data Scientist, BI Consultant, Trainer, and Speaker. She is a well-known International Speakers to many conferences such as Microsoft ignite, SQL pass, Data Platform Summit, SQL Saturday, Power BI world Tour and so forth in Europe, USA, Asia, Australia, and New Zealand. She has over ten years’ experience working with databases and software systems. She was involved in many large-scale projects for big-sized companies. She also AI and Data Platform Microsoft MVP. Leila is an active Technical Microsoft AI blogger for RADACAD.

Leave a Reply