My Cart 0

Face Tracking OpenCV, Python, & Arduino

Face Tracking OpenCV

In a previous article, I showed you how you can establish communication between Arduino and Python.  In this tutorial, I will show you how you can use OpenCV, Python, and Arduino to detect and track faces. Face tracking can be used in a variety of robotics projects and applications. Once you learn the basics from this face tracking OpenCV project, you can use your imagination to put these skills to work! So without wasting any more time, let’s get right into it.

Face Tracking OpenCV Project Materials

For this project, you’ll need an Arduino Uno, servos, pan-tilt kit, breadboard kit, and webcam. (Laptop built-in camera also works.) Along with the hardware components, you will also need the following software:

Set Up the Python Environment

First we need Python 2.7 up and running. To do this first download and Install python 2.7.14.
To check if it is installed correctly Goto : Windows Search >> Type “IDLE” >> Hit Enter.
You should see a Python shell that looks something like the photo below.

python opencv

You can also use the Command Prompt and script from the terminal. In search bar type ‘CMD,’ and hit enter to open Command Prompt. In CMD type >> python and hit enter. The Python interface will look similar to the photo below.

face tracking opencv

If you see an error in CMD, do not worry. You probably need to set the environment variable. You can follow this tutorial here to set up the environment variable.

Install ‘pyserial’ , ‘OpenCV” and “numpy” in Python

To install these modules we will use use pip install. First open CMD and type the following codes:

>pip install serial
>pip install opencv-python
>pip install numpy

Install each of the modules one by one. These modules will help us detect and recognize objects (face tracking in this case). And, they’ll help us connect our Arduino board to Python.

Face Tracking with Python

Before starting to write code first thing to do is make a new folder as all of the code needs to be stored at same location. So create a new folder, name it anything you want. Now download the ‘Haarcascade‘ paste it in the folder.

Now open Notepad (I recommend NotePad++), and write the script given below. Save it as ‘face.py’ in the same folder as haarcascade.

Arduino Face Tracking Code

After the Python script is ready, we need to create an Arduino sketch to control the servos. Refer the code below. Copy and paste it in a new Arduino sketch. Save the sketch as ‘servo.ino‘ in the same folder as face.py and haarcascade. Upload the code and move onto the next step to make the connections.

#include<Servo.h>

Servo servoVer; //Vertical Servo
Servo servoHor; //Horizontal Servo

int x;
int y;

int prevX;
int prevY;

void setup()
{
  Serial.begin(9600);
  servoVer.attach(5); //Attach Vertical Servo to Pin 5
  servoHor.attach(6); //Attach Horizontal Servo to Pin 6
  servoVer.write(90);
  servoHor.write(90);
}

void Pos()
{
  if(prevX != x || prevY != y)
  {
    int servoX = map(x, 600, 0, 70, 179); //tune this range to generate map
    int servoY = map(y, 450, 0, 179, 95); //tune this range to generate map

    servoX = min(servoX, 179);
    servoX = max(servoX, 70);
    servoY = min(servoY, 179);
    servoY = max(servoY, 95);
    
    servoHor.write(servoX);
    servoVer.write(servoY);
  }
}

void loop()
{
  if(Serial.available() > 0)
  {
    if(Serial.read() == 'X')
    {
      x = Serial.parseInt();
      if(Serial.read() == 'Y')
      {
        y = Serial.parseInt();
       Pos();
      }
    }
    while(Serial.available() > 0)
    {
      Serial.read();
    }
  }
}

Downloadable Code : servo.ino

Pan-Tilt Mechanism

I have used a readily available kit for the Pan-Tilt. If you want you can make one yourself using wood or plastic, or you can even 3D print one. I used a kit you can get on Amazon, Gearbest, or Banggood.

pan-tilt servos

You can use our guide to assemble the Pan-Tilt mechanism properly.  Once you finish that, it’s time to move on to the electronics.

Wiring Diagram for Face Tracking OpenCV

The circuit is pretty simple. Just attach two servos to Arduino as shown in the wiring diagram below.

wiring diagram face tracking opencv

The vertical servo signal pin goes to Digital pin 5. The horizontal servo signal pin goes to Digital pin 6. Power goes to +5V and Ground goes to Ground.

Note: To reduce any jitters, connect an electrolytic capacitor in parallel to the power supply.

Here’s what the final assembly looks like.

final setup face tracking arduino opencv

Test the Face Tracking OpenCV project

After everything is done, the last thing to do is to test to see if it works. First, make sure that the servos are properly connected to the Arduino and sketch has been uploaded.

After the sketch is uploaded, make sure to close the Arduino IDE so that the Serial port is free to connect to Python.

Next, open face.py with Python IDLE and press ‘F5’ to run the code. You can also navigate to the project directory in terminal and run face.py code directly from the command line.

It will take a few seconds to connect to Arduino. Then you should be able to see a window streaming the webcam. Now the code will detect your face, and the servos will track it.You may have to tune the pan and tilt servo range to best suit the coordinates given by Python & OpenCV. Check out the video to see how you can tune the readings for more dynamic control.

If everything is setup correctly, the servos should move as you move. As an added feature, you can mount the camera directly on the servos. Then the mechanism will mimic your movements while you’re looking directly at it.

That’s all for this tutorial. If you have any questions, be sure to find us on Facebook. You can also check out our online Arduino Course. It’s designed to help students learn the fundamentals of Arduino to be successful with more advanced projects like this one. Click here to check it out!

Did you like this post?

Electronic hobbyist, Technology enthusiast, I like to hack, make and create. You can find my tutorials also at https://www.instructables.com/member/WolfxPac/

Comments

  • Bonzadog
    05/20/2019

    Why on earth use an old version of python Python 2.7? Wh have Python 3.7 these days.

    • Liz Miller
      05/23/2019

      Try it with Python 3.7 and see how it goes 🙂

      • cheezusmaria
        06/21/2019

        Hello Liz, i tryed this with phyton 3.7 and it worked after i change couple easy thing. But somehow my servos are not working, so its just tracking my face,finds center and follows it etc.
        i changed arduino=serial.Serial(‘COM6’,9600), my servos are working , i tryed with some easy arduino code. So i am just stuck, whats your idea what can be wrong you think ?

        • cheezusmaria
          06/21/2019

          a little more information i changed these parts for fix some errors in my computer ;
          data=”X{0:f}Y{1:f}Z”.format(xx,yy)
          arduino.write(data.encode())

          • Liz Miller
            06/22/2019

            Yes, so the write method takes in datatype byte. Presumably, if you provide data of the type bytes(b'(xxxx)) you’ll be able to write it to the Arduino via the Serial port.

        • Liz Miller
          06/22/2019

          Verify, that your Arduino is on COM6. Are you able to connect to it via Serial when you run the face.py code? You’ll also need to verify that your Arduino contains the Facetracking servo.ino sketch prior to launching the python code. Try these things, & let me know how it turns out!

          • cheezusmaria
            07/09/2019

            Hello again Liz, thank you for your answer.
            i checked everything but somehow i cant send data to ardunio. i tryed too many way, i verifyed everything.Ardunino is on COM6, i have the connection with ardunio when i run face.py, and ardunio has servo.ino, i am getting outputs on pycharm like;
            X:268
            y:190
            x+w:449
            y+h:371
            385.5
            280.5
            center of rectangle is :(385.5,280.5)
            output=’X358.500000Y280.500000z’ but i just cant see any movement on servos. i cant send the data to servo. i checked connections also they all correct. i am using pycharm with anaconda. i have no idea why it doesnt work please help 🙂

          • Liz Miller
            07/10/2019

            I’m not really sure what the issue is. I recommend setting up a new python program, connect to the Arduino, and send some raw data to it. I’m not familiar enough with pycharm and anaconda to know if they support direct serial communication. Another option is to configure the python code and run it directly through Command Line or Terminal rather than through an IDE. Keep me posted on how it goes!

  • EoinD
    05/29/2019

    Hey so I’m trying to do this build and have everything set up as said but keep getting an error on the formatting.
    data = “X{0:d}Y{1:d}Z”.format(xx, yy)
    Value Error: Unknown format code ‘d’ for object of type ‘float’

    If you have any information or advice on where I could be going wrong I would really appriecate it!

  • EoinD
    05/29/2019

    Hey I’m having trouble with this build, I have followed all the steps using the code given but am getting the error;
    data = “X{0:d}Y{1:d}Z”.format(xx, yy)
    Value Error:Unknown Format code ‘d’ for object type ‘float’
    If you have any kind advice on where I am going wrong I would be very thankful.

    • Liz Miller
      05/29/2019

      Yes, it’s possible that you’re getting a float value for your data.

      Try changing the ‘d’ to an ‘f’ so the new line would read:
      data = “X{0:f}Y{1:f}Z”.format(xx, yy)

      Let me know if that helps!

  • pat
    06/05/2019

    Hi Liz
    Thanks for the tutorial.
    Just a question: Are you using the macbook webcam in the tutorial?
    Did you get to try this with a USB2 cam mounted on the pan and tilt brackets?
    Thanks

    • Liz Miller
      06/05/2019

      Pat, you’re welcome. Glad you’re enjoying this project. I used the built-in camera with the MacBook on this.

      For an additional webcam, try changing the VideoCapture() input parameter to 1. I believe this method detects cameras starting with 0 and incrementing by device number.

      Here’s the code snippet that you’ll want to modify:

      #To capture the video stream from webcam.
      cap = cv2.VideoCapture(1)

      Hope this helps! Be sure to check back in and let us know how it goes! 🙂

      • pat
        06/06/2019

        Thanks so much for the speedy response.
        I have a question regarding below section of the arduino code( POS function section):

        int servoX = map(x, 600, 0, 70, 179);
        int servoY = map(y, 450, 0, 179, 95);

        servoX = min(servoX, 179);
        servoX = max(servoX, 70);
        servoY = min(servoY, 179);
        servoY = max(servoY, 95);

        -how do you determine 179/70 and 179/95 for servo range for the two motors?
        -and how does having min max one after another work? Lets say if x is 500 wouldn’t the outcome be 500 after going through min max? min (500, 179) = 179 and then max (500, 70) = 500 (since 500 is the later outcome it’s what’s gonna be allocated to memory location for servoX)
        Thanks again

        • Liz Miller
          06/12/2019

          Pat, great questions.

          To address your first question, this section of code is to map the “detected face” as found by a green box in OpenCV to the position of the servos. If you’re using 9g servos (SG90) like the tutorial, you’re limited to 180 degrees of rotation. The x-direction is for the pan servo, and the y-direction is for tilt servo.The way you determine these numbers is a lot of trial and error. It’s a lot of “what looks good.” For my numbers, I have the pan servo rotating from 70-179 and the tilt servo rotating from 95-179. You can see the outcome in the results video.

          As far as your second question goes, the purpose of having min and max calculations back-to-back is to constrain the servo mapping between the defined angle ranges. For your example, if servoX=500, then here’s what the code would result in:

          servoX = min(servoX, 179); //results in min(500, 179) --> 179 is smaller than 500
          servoX = max(servoX, 70); //results in max(179, 70) --> 179 is larger than 70
          

          so the servoX would be set to 179 degrees. By taking the minimum of servoX in the first calculation, you’re reassigning servoX to the result of that calculation. Then, in the second calculation, you’re taking the larger of the two to constrain the rotational range.

          Hopefully that makes sense. Be sure to keep me posted on how your project turns out!

  • Ebubekir
    07/02/2019

    Hi Liz. What do sync prevX and prevY . I am rpi and arduino serial communication.

    • Liz Miller
      07/08/2019

      Ebubekir, I’m not sure what you’re referring to. Do you have a specific line you’re looking at?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

error:

Pin It on Pinterest