You might know that I am an enormous enthusiast of Machine Learning and Artificial Intelligence. I dive into various tools, frameworks, and solutions in my spare time. Some time ago, I wrote here about ML.NET Model Builder. This tool is still developing and one one the last features is detecting objects from images. Today I would like to share with you my knowledge of how to create such Object Detection Models within ML.NET Model Builder.
Let’s start with a definition of Object Detection and a brief explanation of what it can be used for. Object Detection is the detection of given objects in a presented image with the same features that can be assigned to a given category. It is closely related to the concept of Computer Vision. Provided that there are several objects in the presented image, object detection aims to capture images that belong to the same collection. In the photo below you can see the successful detection of a lovely dog. 🙂
We will use drone inspection images of wind turbines from Mendeley Data. These images include a variety of photographs of wind turbines that were taken from multiple perspectives. This will give our model a better chance of being effective.
Now we need to prepare our data which means tagging all the photos. For that purpose, we will work with a tool called VoTT. You can use a desktop or web version. In this article, we will focus on the first one.
Preparing data using VoTT
After downloading VoTT, open it and select ‘New Project’:
After that, you should see the form, so add Display Name and change Security Token to Generate New Security Token. Next, go to Source Connection and click the bottom ‘Add Connection’:
Here you need to provide the Display Name again and choose Provider. In our case, it will be ‘Local File System’ but there are additional options such as ‘Azure Blob Storage’ or ‘Bing Image Search’.
Click on Save Connection and change Target Connection and Source Connection to the connection you have just created.
Another step is to save your project and add a tag. How to do that? Select the first icon in the Tags toolbar to attach a new tag, put the name of the tag “WindTurbine” and click Enter on your keyboard:
Finally, you are able to tag objects by clicking and dragging to draw a rectangle around each wind turbine in the picture:
When you have labelled all of your images, export the file that will be used by Model Builder for training then. Go to Export Settings by clicking on the fourth icon in the left toolbar and leave the Provider as VoTT JSON, cause we need this format.
Creating Object Detection Model
Ok, so it is time to open Visual Studio 2022. After creating a console application, you can proceed to build a machine learning algorithm. In the first step, open the context menu of your project and select the ‘Add’ tab there and then ‘Machine Learning Model’. You should now see something like this:
So choose the ‘Object detection’ field and then prepare the Azure Machine Learning workspace by clicking ‘Set up workspace’. You should fill out here some inputs (create a new subscription, workspace and compute if you haven’t done it earlier):
Subsequently, load the previously prepared JSON file with data:
The next step is about training the model. In order to do so, select the ‘Start training’ button. At this moment, your data is uploaded to the Azure Storage service and the learning process begins in the Azure ML service. In general, the training process takes some time, and the amount of time may vary depending on the size of the computation you choose and the amount of data.
You have the option to track the progress of your runs in the Azure Machine Learning portal by selecting the ‘Monitor current run’ in the Azure portal link in Visual Studio.
In the Evaluate phase, you get an overview of the results of the training process. In this case, the accuracy is 1.0, meaning that the model is more than likely over-fitting due to too few images in the dataset.
Additionally, you have the possibility to use the ‘Try your model’ experience to check if your model is performing as expected.
The score displayed on each detected bounding box indicates the confidence of the detected object. For example, in the image above, the score on the bounding box around the wind turbine indicates that the model is 99% confident that the detected object is a wind turbine.
After all, there is an option for project templates that can be used to leverage the model. The choice is between Console App and Web API. What does it look like and how to take advantage of that? To be continued in another article. 🙂
As I expected and mentioned in the last post about ML.NET Model Builder, this tool is being developed constantly. One of the added features is Object Detection and that’s what I wanted to introduce you to and demonstrate how easy it is to play with.