classification

Object detection and image classification with Google Coral USB Accelerator

A couple of weeks in the past I revealed a tutorial on how you can get started with the Google Coral USB Accelerator. That tutorial was meant that will help you configure your system and run your first demo script.

Immediately we’re going to take it a step further and discover ways to utilize the Google Coral in your personal customized Python scripts!

Inside at the moment’s tutorial you will study:

  • Image classification with the Coral USB Accelerator
  • Image classification in video with the Google Coral Accelerator
  • Object detection with the Google Coral
  • Object detection in video with the Coral USB Accelerator

After reading this guide, you’ll have a robust understanding of the way to make the most of the Google Coral for image classification and object detection in your personal purposes.

To discover ways to perform image classification and object detection with the Google Coral USB Accelerator, simply hold studying!

Object detection and image classification with Google Coral USB Accelerator

For this guide I will probably be making the next assumptions:

  1. You already own a Google Coral USB Accelerator.
  2. You might have followed my earlier tutorial on the right way to set up and configure Google Coral.

For those who haven’t adopted by install guide, please discuss with it before persevering with. Finally, I’ll word that I’m connecting my Google Coral USB Accelerator to my Raspberry Pi to collect results — I’m doing this for 2 reasons:

  1. I’m at present writing a ebook on using the Raspberry Pi for Pc Vision which may even cover the Google Coral.
  2. I cover the Raspberry Pi very often on the PyImageSearch weblog and I know many readers are fascinated with how they will leverage it for pc imaginative and prescient.

In the event you don’t have a Raspberry Pi but nonetheless need to use your Google Coral USB Accelerator, that’s okay, however be sure to are operating a Debian-based OS.

Again, check with my previous Google Coral getting began information for extra info.

Challenge structure

Let’s evaluate the undertaking included in at this time’s “Downloads”:

As we speak we shall be reviewing 4 Python scripts:

  1. classify_image.py  – Classifies a single image with the Google Coral.

  2. classify_video.py  – Actual-time classification of every frame from a webcam video stream using the Coral.

  3. detect_image.py  – Performs object detection using Google’s Coral deep learning coprocessor.

  4. detect_video.py  – Actual-time object detection utilizing Google Coral and a webcam.

We’ve got three pre-trained TensorFlow Lite fashions + labels obtainable within the “Downloads”:

  • Classification (educated on ImageNet):
    • inception_v4/  – The Inception V4 classifier.

    • mobilenet_v2/  – MobileNet V2 classifier.

  • Object detection (educated on COCO):
    • mobilenet_ssd_v2/  – MobileNet V2 Single Shot Detector (SSD).

In case you are interested in tips on how to practice your personal classification and object detection models, be sure you seek advice from Deep Learning for Pc Imaginative and prescient with Python.

For both
classify_image.py  and
detect_image.py , I’ve offered two testing pictures in the “Downloads”:

  • janie.jpg  – My lovable beagle.

  • thanos.jpg  – Character from Avengers: Finish Recreation.

For the
classify_video.py  and
detect_video.py  scripts, we’ll be capturing frames immediately from a digital camera related to the Raspberry Pi. You need to use one of the following with at the moment’s example scripts:

  • PiCamera V2 – The official Raspberry Pi Foundation digital camera.
  • USB Webcam – Any USB digital camera that supports V4L will work, reminiscent of a Logitech branded webcam.

Image classification with the Coral USB Accelerator

Figure 1: Image classification utilizing Python with the Google Coral TPU USB Accelerator and the Raspberry Pi.

Let’s get started with image classification on the Google Coral!

Open up the
classify_image.py  file and insert the next code:

We begin of by importing packages. Most notably, we are importing
ClassificationEngine  from
edgetpu  on Line 2.

From there we’ll parse three command line arguments by way of Strains 10-17:

  • –model : The path to our TensorFlow Lite classifier.

  • –labels : Class labels file path related with our model.

  • –image : Our input image path.

Utilizing these three command line arguments, our script will have the ability to deal with suitable pre-trained models and any image you throw at all of it from the command line. Command line arguments are one of the number one issues individuals e-mail me about, so make sure to assessment my tutorial on argparse and command line arguments in case you need a refresher.

Let’s go forward and load the
labels :

Strains 21-27 facilitate loading class
labels  from a textual content file right into a Python dictionary. Afterward, the Coral API will return the anticipated
classID  (an integer). We will then take that integer class label and lookup the related 
label  worth in this dictionary.

Shifting on, now let’s load our classification
model  with the
edgetpu  API:

Our pre-trained TensorFlow Lite classification
mannequin  is instantiated by way of the
ClassificationEngine  class (Line 31) where we move in the path to our mannequin by way of command line argument.

Let’s go ahead and load + preprocess our
image :

Our
image  is loaded (Line 34) and then preprocessed (Strains 35-42).

Take word that we made an unique
copy  of the image — we’ll be annotating this copy of the image with the output predictions later within the script.

How straightforward is it to perform classification inference on an image with the Google Coral Python API?

Let’s discover out now:

On Line 47, we make classification predictions on the input
image utilizing the
ClassifyWithImage  perform (an excellent straightforward one-liner perform name). I actually like how the
edgetpu  API allows us to specify that we solely want the highest results with the
top_k  parameter.

Timestamps are sandwiched round this classification line and the elapse time is then printed by way of Strains 49 and 50.

From here we’ll course of the
outcomes :

Looping over the
outcomes  (Line 53) we first discover the highest end result and annotate the image with the label and proportion score (Strains 56-60).

For good measure, we’ll additionally print the other outcomes and scores (however solely in our terminal) by way of Strains 63 and 64.

Lastly, the annotated unique (OpenCV format) image is displayed to the display (Strains 67 and 68).


That was simple. Let’s put our classification script to the check!

To see image classification in action with the Google Coral, be sure to use the “Downloads” part of this guide to download the code + pre-trained fashions — from there, execute the following command:

The output of the image classification script may be seen in Determine 1 on the prime of this part.

Here you’ll be able to see that Janie, my dog, is appropriately categorised as “beagle”.

Image classification in video with the Google Coral Accelerator

Figure 2: Actual-time classification with the Google Coral TPU USB Accelerator and Raspberry Pi using Python. OpenCV was used for preprocessing, annotation, and display.

In the earlier section, we discovered find out how to perform image classification to a single image — but what if we needed to perform image classification to a video stream?

I’ll be displaying you the best way to accomplish precisely that.

Open up a brand new file, identify it
classify_video.py  and insert the next code:

There are two differences in our first code block for real-time classification in comparison with our earlier single image classification script:

  1. On Line 2 we’ve added the
    VideoStream  import for working with our webcam.
  2. We not have a
    –image  argument since by default we shall be utilizing our webcam.

Just as earlier than, let’s load the
labels  and
mannequin , but now we additionally have to instantiate our
VideoStream :

Strains 19-31 are similar to our previous script the place we load our class labels and store them in a dictionary.

On Line 35 we instantiate our
VideoStream  object in order that we will read frames in our webcam (coated within the next code block). A
2.0  second sleep is added so our digital camera has time to warm up (Line 37).

Word: By default, this script will use a USB webcam. If you need to make use of a Raspberry Pi digital camera module, merely comment out Line 35 and uncomment Line 36.

Let’s begin our loop:

We start looping on Line 40.

Line 43 grabs a
body  from the threaded video stream.

We go ahead and preprocess it precisely as we did within the earlier script (Strains 44-51).

With the
frame  in the right PIL format, now we will make predictions and draw our annotations:

Just as before, Line 55 performs inference.

From there, the highest result’s extracted and the classification label +
score  is annotated on the
orig  body (Strains 59-66).

The frame is displayed on the display (Line 69).

If the
“q”  key’s pressed, we’ll break from the loop and clean up (Strains 70-78).


Let’s give image classification in video streams with the Google Coral a attempt!

Be sure to use the “Downloads” part of this guide to download the code + pre-trained fashions — from there, execute the following command:

An example of real-time image classification could be seen above in Figure 2.

Using the Google Coral USB Accelerator, the MobileNet classifier (educated on ImageNet) is absolutely capable of operating in real-time on the Raspberry Pi.

Object detection with the Google Coral

Determine 3: Deep learning-based object detection of an image using Python, Google Coral, and the Raspberry Pi.

We’ve already discovered learn how to apply image classification with the Google Coral — but what if we not solely needed to classify an object in an image but in addition detect where within the image the item is?

Such a activity is known as object detection, a way I’ve coated quite a couple of occasions on the PyImageSearch blog (check with this deep learning-based object detection information in case you are new to the idea).

Open up the
detect_image.py  file and let’s get coding:

Our packages are imported on Strains 2-7. For Google Coral object detection with Python, we use the
DetectionEngine  from the
edgetpu  API.

Our command line arguments are just like the
classify_image.py  script with one exception — we’re additionally going to provide a
–confidence  argument representing the minimum chance to filter out weak detections (Strains 17 and 18).

Now we’ll load the labels in the identical method as in our classification scripts:

And from there we’ll load our object detection 
mannequin :

We will now load our input image and carry out preprocessing:

After preprocessing, it is time to carry out object detection inference:

Strains 49 and 50 use Google Coral’s object detection API to make predictions.

With the ability to cross our confidence threshold (by way of the
threshold  parameter), is extremely convenient on this API. Truthfully, I wish OpenCV’s DNN API would comply with go well with. It saves an if-statement afterward as you’ll be able to imagine.

Let’s course of our
results :

Looping over the
results  (Line 56), we first extract the bounding
box  coordinates (Strains 58 and 59). Conveniently, the
field  is already scaled relative to our enter image dimensions (from any behind the scenes resizing the API does to fit the image into the CNN).

From there we will easily extract the category
label  by way of Line 60.

Next, we draw the bounding box rectangle (Strains 63 and 64) and draw the anticipated object 
text  on the image (Strains 65-68).

Our
orig  image (with object detection annotations) is then displayed by way of Strains 71 and 72.


Let’s put object detection with the Google Coral USB Accelerator to the check!

Use the “Downloads” section of this tutorial to obtain the source code + pre-trained fashions.

From there, open up a terminal and execute the next command:

Just for enjoyable, I decided to use object detection to a display capture of Avengers: Endgame film (don’t worry, there aren’t any spoilers!)

Here we will see that Thanos, a character from the film, is detected (Figure three)…though I’m unsure he’s an precise “person” if you recognize what I imply.

Object detection in video with the Coral USB Accelerator

Figure 4: Actual-time object detection with Google’s Coral USB deep learning coprocessor, the right companion for the Raspberry Pi.

Our remaining script will cover tips on how to carry out object detection in real-time video with the Google Coral.

Open up a new file, identify it
detect_video.py , and insert the following code:

To start out, we import our required packages and parse our command line arguments (Strains 2-Eight) Once more, we’re using
VideoStream  so we will entry our webcam (since we’re performing object detection on webcam frames, we don’t have a
–image  command line argument).

Next, we’ll load our
labels  and instantiate both our
model  and video stream:

From there, we’ll loop over frames from the video stream:

Our body processing loop begins on Line 41. We proceed to:

  • Seize and preprocess our frame (Strains 44-52).
  • Carry out object detection inference with the Google Coral (Strains 56 and 57).

From there we’ll process the results and display our output:

Right here we loop over every of the detected objects, grab the bounding field + class label, and annotate the body (Strains 61-73).

The frame (with object detection annotations) is displayed by way of Line 76.

We’ll continue to process extra frames until the
“q”  (give up) key’s pressed at which level we break and clean up (Strains 77-85).


Let’s put this Python + Coral object detection script to work!

To carry out video object detection with the Google Coral, ensure you use the “Downloads” section of the guide to obtain the code + pre-trained models.

From there you’ll be able to execute the next command to start out the item detection script:

For our remaining example of making use of real-time object detection with the Google Coral, I decided to let Janie in my office for a bit as I recorded a demo (and even determined to sing her a bit of music) — you possibly can see the end in Figure 4 above.

The issue with the Raspberry Pi 3B+ and Google Coral USB Accelerator

Determine 5: USB three.zero is far quicker than USB 2.0. To take full benefit of Google Coral’s deep learning capabilities a USB 3.0 port is required, nevertheless, the Raspberry Pi 3B+ does not embrace USB 3.zero capability. (image source)

You may need observed that our inference outcomes are pretty just like what we get hold of with the Movidius NCS — doesn’t Google advertise the Coral USB Accelerator as being quicker than the NCS?

What’s the issue here?

Is it the Google Coral?

Is it our code?

Is our system configured incorrectly?

Truly, it’s not one of the above.

The problem right here is the Raspberry Pi 3B+ only helps USB 2.0.

The bottleneck is the I/O happening from the CPU, to USB, to the Coral USB Accelerator, and back.

Inference velocity will dramatically enhance once the Raspberry Pi 4 is released (which will definitely help USB three, giving us the quickest potential inference speeds with the Coral USB Accelerator).

What about customized fashions?

This tutorial has targeted on state-of-the-art deep studying fashions which were pre-trained on common image datasets, together with ImageNet (for classification) and COCO (for object detection).

But what in the event you needed to run your personal pre-trained models on the Google Coral?

Is that this potential?

And in that case, how can we do it?

I’ll be answering that actual question inside my upcoming e-book, Raspberry Pi for Pc Vision.

The e-book shall be released in Autumn 2019, however when you pre-order your copy now, you’ll be getting a discount (the worth of the ebook will improve once it officially releases later this yr).

When you’re all in favour of pc imaginative and prescient + deep studying on embedded units such as the:

  • Raspberry Pi
  • Movidius NCS
  • Google Coral
  • Jetson Nano

…then it is best to undoubtedly pre-order your copy now.

Abstract

On this tutorial, you discovered learn how to make the most of your Google Coral USB Accelerator for:

  1. Image classification
  2. Image classification in video
  3. Object detection
  4. Object detection in video

Specifically, we used pre-trained deep learning models, together with:

  • Inception V4 (educated on ImageNet)
  • MobileNet V4 (educated on ImageNet)
  • MobileNet SSD V2 (educated on COCO)

Our outcomes have been far, much better than making an attempt to make use of the Raspberry Pi CPU alone for deep learning inference.

General, I was very impressed with how straightforward it’s to make use of the Google Coral and the
edgetpu  library in my own custom Python scripts.

I’m wanting ahead to seeing how the package deal develops (and hope they make it this straightforward to transform and run customized deep studying models on the Coral).

To obtain the source code and pre-trained to this publish (and be notified when future tutorials are revealed here on PyImageSearch), simply enter your e mail tackle in the type under!

Downloads: