Azure Custom Vision in action

  • Article

Azure platform offers many ready-to-use cognitive services, allows you easily enrich your applications with innovative features.

In this article we will see how to use the Custom Vision service. We will train a model to learn how to recognize some types of coats. We will then test this model and finally we will see how to use it within an application.

 

Prerequisite

The part about the notebook can be done, among others, with:

In this article, we will use the "Data Science" virtual machine

Creating "Data Science" virtual machine

  • Connect to the Azure Portal: https://portal.azure.com
  • Click on " Create a resource », and then search for the virtual machine "Data Science"

 

Select " Data Science Virtual Machine for Linux (Ubuntu)  ». Then Click on the button "Create"

 

 

Fill in the virtual machine creation form

At the end of the process click on the "Create" button

 

The virtual machine is now created. We will connect on its Jupyter server. The connection is an URL type:

https://< Your_VM_IPAddress >: 8000

In our case, this address will be:

https://40.86.210.115:8000

 

The virtual machine IP address can be found on the Azure portal:

 

The following window may appear. Click on " Details, then Go on the Web page »

 

In the login window, enter the name and password you have used when creating the virtual machine:

After a few seconds you must land on the Jupyter server home page.

We will create a folder to facilitate our work. Click on "New" then "Folder"

 

Check the box to the left of the newly created folder (the folder is named « Untitled Folder »), then click the button « Rename » on top of the page.

Rename the folder and then click on the "Rename" button

 

 

Creating the Notebook

Enter the previously created folder. Click on the button "New"   then "Python 3"

 

Once the Notebook created, click "Untitled" and give it a name. Click on "Rename"

 

Download pictures

In the first cell of the Notebook, copy the code below to recover the training and tests images

! curl-O https://canadastore.blob.core.windows.net/franmerpictures/Pictures.zip

Then click « run » on the toolbar (or use the combination key "Shift Enter"  or "Ctrl Enter" ). For the rest of the article, after copying the code in the cells, think well to execute the code.

If everything goes well, the Pictures.zip file must appear in your folder

We will now unzip the file. Use the command below in a new cell:

! unzip Pictures.zip

 

 

Attention: star between the hooks, means the code is still running. Wait until the star disappears.

 

Three new files had to appear:

displaying an image

We will display an image and see some basic functions to deal with pictures. We will use « Matplotlib » library. « Matplotlib » documentation can be found here: https://matplotlib.org/.

Copy the code below to a new cell:

#Afficher une image / Display a picture

# Importation de librairie matplotlib / Import the matplotlib libraries

import matplotlib.pyplot as plt

import matplotlib.image as im

 

img = im.imread('hardshell_jackets/10116634x1038116_zm.jpeg')

imgplot = plt.imshow(img)

 

In the first run, the following message may appear. This will not affect the outcome.

Retrieve picture information

An image is intrinsically an array of values of pixels (Picture Element) and It is important to be able to recover picture information, in order to use them with object recognition algorithms. Below is a sample code for retrieving certain information. Copy and paste it in a new cell. Don't forget to run the code.

#Information sur l'image / Picture's information

print ("Picture Shape: ")

print(img.shape)

 

#Total number of pixels is accessed by img.size:

print ("Number of pixels: ")

print (img.size)

 

#Image datatype is obtained by img.dtype:

print("Picture data type:")

print(img.dtype)

 

 

Integration with the "Custom Vision" cognitive service

Microsoft offers many Cognitive Services. The complete list is available through the following link: https://azure.microsoft.com/en-us/services/cognitive-services/directory/

In addition, these services may be called as a web service to return the prediction results.

In this example, we will use the service "Custom Vision" . This service is available through the following link: https://customvision.ai/.

But before, we still have small preparations to do in our Notebook.

Installing the Custom Vision SDK

As a first step, we will check the presence of "Pip" and its update.  "Pip" is a tool for installing Python packages. In a new cell of the Notebook, copy the following code:

#installation de pip / pip installation

import sys

! {sys.executable} -m pip freeze

! {sys.executable} -m pip --version

! {sys.executable} -m pip install --upgrade pip

 

 

From your notebook, enter the following command: (So we use " Pip "To Install the Package Of the "Custom Vision" cognitive service) :

#Installation du SDK Custom Vision / Install the Custom Vision SDK

! {sys.executable} -m pip install azure-cognitiveservices-vision-customvision

 

 

Configuring the "Custom Vision" cognitive service

We will now create a "Custom Vision" service via the following link:

Https://customvision.ai

Click on " Sign In »

 

Can choose to click on "Continue with trial" or click on "Sign Up for Azure". For our example, we will choose " Continue with trial » :

Once the service is created, retrieve the information from your service by clicking the gear in the top right of the window

Then recover the values of your keys

Return to your Jupyter notebook to assign the key values to the variables:

# Assignez les clefs de votre service "Custom Vision" aux variables / Assign your Custom service keys to variables

training_key = 'c5926d0852854f459c85facff519c759'

prediction_key = '32446a7de71b496798d38fded67fc2da'

 

 

Creating a Custom Vision project

From the notebook, we will create a project in the "Custom Vision" cognitive service. In a new cell, copy the code below and run it:

from azure.cognitiveservices.vision.customvision.training import training_api

from azure.cognitiveservices.vision.customvision.training.models import ImageUrlCreateEntry

 

trainer = training_api.TrainingApi(training_key)

 

# Create a new project

print ("Creating project...")

project = trainer.create_project("MyProject")

 

 

If you return to the "Custom Vison" service web page, you will find a new project:

 

Labels creation

Before adding images to the service, we will create our first labels to identify the images.

Add the following code to a new cell and run the code:

# Création des labels les images / Add tags for pictures

 

print("Creating tags")

hardshell_tag = trainer.create_tag(project.id,"Hardshell")

insulated_tag = trainer.create_tag(project.id,"Insulated")

print("Tags created")

 

 

Loading Images

Now let's add the images in our custom vision project and add the labels at the same time. Copy and run the code below in a new cell:

# Téléchargez les images / Upload pictures

import os

 

print("Adding hardshell images...")

hardshell_folder = "jacket_images/hardshell_jackets"

for file in os.listdir(os.fsencode("jacket_images/hardshell_jackets")):

with open(hardshell_folder + "/" + os.fsdecode(file),mode="rb") as Hardshell_img_data:

trainer.create_images_from_data(project.id,Hardshell_img_data,[hardshell_tag.id])

 

print("Adding insulated images...")

insulated_folder = "jacket_images/insulated_jackets"

for file2 in os.listdir(os.fsencode("jacket_images/insulated_jackets")):

with open(insulated_folder + "/" + os.fsdecode(file2),mode="rb") as insulated_img_data:

trainer.create_images_from_data(project.id,insulated_img_data,[insulated_tag.id])

 

print("Done !")

 

 

 

Check the presence of the images in the web interface of the cognitive service, and check that the images have the right label.

 

 

 

 

 

 

Training of the project

After adding images and set the different labels, we can now train a model for identifying our images.

In a new cell, copy and run the code below:

#Entrainement du projet / Train the project

import time

 

print("Training....")

iteration = trainer.train_project(project.id)

print (iteration.status)

while (iteration.status != "Completed"):

iteration = trainer.get_iteration(project.id,iteration.id)

print("Training status: " + iteration.status)

time.sleep(1)

 

# Set this iteration of the project as the default.

trainer.update_iteration(project.id,iteration.id,is_default=True)

print("Done!")

 

 

 

If we check in the web interface of the cognitive service, a first iteration of our model was created.

 

 

Test the model

Now that the model has been trained, we will test it with another set of images

As a first step, we will create a function to convert images and send a data value to the cognitive service endpoint.

To do this conversion, we will create a function. Copy the code below into a new cell:

def read_image_data(img_path):

import numpy as np

from io import BytesIO

import matplotlib.pyplot as plt

 

# Load the numpy array

#imgArray = np.load(img_path)

imgArray = im.imread(img_path)

# Write the array data to a stream

imageStream = BytesIO()

plt.imsave(imageStream, imgArray)

# Read the stream from the beginning

imageStream.seek(0)

return imageStream.read()

 

print ('read_image_data function created!')

 

 

 

We will use the model we have trained to make predictions on test images. We will also use the function created previously to convert the images before sending them to the model.

In a new cell in your Notebook, copy the code below:

import os

import time

import numpy as np

 

#Importation du point de terminaison de prédiction / Import the prediction endpoint

from azure.cognitiveservices.vision.customvision.prediction import prediction_endpoint

from azure.cognitiveservices.vision.customvision.prediction.prediction_endpoint import models

 

#Création d'une instance du point de terminaison de prédiction

#Create an instance of the prediction endpoint

predictor = prediction_endpoint.PredictionEndpoint(prediction_key)

 

#Boucle sur l'ensemble des images / Loop through the test images

folder = 'Test'

files = os.listdir(folder)

i=1

x = np.linspace(0,10)

y = np.sin(x)

 

for file in sorted(files):

 

#Les images sont traitées dans l'ordre / process files in order

#Récupération du chemin de l'image et lecture des données / Get the file path and read the image data

 

img_path = os.path.join(folder,file) #Test pour savoir si c'est un fichier ou un dossier

if os.path.isfile(img_path):

print("File name: " + file)

img = im.imread(img_path)

plt.imshow(img)

plt.show()

#plt.plot(x,y)

#plt.figure(i+1)

 

img_data = read_image_data(img_path)

#Récupération des prédictions pour les images / Get the predicted tags for the image

results = predictor.predict_image(project.id, img_data, iteration.id)

fig = plt.figure()

#Affichage des labels et des probabilités / Display the tag name and probability for each prediction

for prediction in results.predictions:

print ("\t" + prediction.tag_name + ": {0:.2f}%".format(prediction.probability * 100))

 

print('\n')

#Pause technique :) / Technical Delay

time.sleep(0.25)

i=i+1

 

 

Our model returns predictions to try to identify whether the scanned images are coats or not, and if so, what kind of coat is it. Below a sample of prediction results

     

 

Using the cognitive service with Visual Studio

 

Now that our model is ready, It could be interesting to use the service API to Integrate predictions in an application. Below is a quick illustration of integrating the cognitive service into an application.

 

Start Visual Studio 2017 and create a "Console App"project.

 

Once the project created, go to « Tools / NuGet Package Manager / Manage NuGet Packages for Solution... " and add the Custom Vision Packages as illustrated below:

 

 

 

Add the 2 following packages:

 

 

Then in the file "Program.cs" add the code below. Replace the keys by those of your cognitive service project

 

using Microsoft.Cognitive.CustomVision.Prediction;

using Microsoft.Cognitive.CustomVision.Training;

using Microsoft.Cognitive.CustomVision.Training.Models;

using System;

using System.Collections.Generic;

using System.IO;

using System.Linq;

using System.Threading;

 

 

namespace myTutorialCustomVision

{

class
Program

{

//private static MemoryStream testImage;

static
void Main(string[] args)

{

var projectID = "<YourProjectID>";

var trainingKey = "<YourTrainingKey>";

var predictionKey = "<YourPredictionKey>";

string imagesPath = @"..\..\..\myTutorialCustomVision\Images";

string[] imageFiles = Directory.GetFiles(imagesPath, "*.jpg");

 

TrainingApi trainingApi = new TrainingApi() { ApiKey = trainingKey };

var project = trainingApi.GetProjects();

Guid myGuid = project[0].Id;

 

var iterations = trainingApi.GetIterations(Guid.Parse(projectID));

var validIterations = iterations.Where(iter => iter.Status == "Completed");

var lastIterationTrained = validIterations.OrderBy(iter => iter.LastModified).Last();

 

 

Iteration lastIteration = trainingApi.GetIteration(myGuid, lastIterationTrained.Id);

lastIteration.IsDefault = true;

trainingApi.UpdateIteration(myGuid, lastIteration.Id, lastIteration);

 

 

foreach (var image in imageFiles)

{

FileStream fileStream = new FileStream(image, FileMode.Open, FileAccess.Read);

PredictionEndpoint endpoint = new PredictionEndpoint() { ApiKey = predictionKey };

Console.WriteLine("Making a prediction for picture: " + Path.GetFileName(image));

var result = endpoint.PredictImage(myGuid, fileStream);

foreach (var c in result.Predictions)

{

Console.WriteLine($"\t{c.Tag}: {c.Probability:P1}");

}

 

}

Console.ReadKey();

 

}

}

}

 

 

Test of the application

To test the application and the predictions, I added a folder images in the "Solution Explorer" and I have copied 2 images in this folder

 

 

 

Result

The application will return, for each image, a prediction for each Labels.

 

You may get the error " Operation returned an invalid status code '429' » If you are testing with a large number of images. Often this error comes from the fact that your service is in trial version or undersized. To correct the problem, just change the "Pricing tier" on the Custom Vision Web page, in the "Account" section.