Thursday, February 2, 2012

Hand gesture using opencv


Hi ! In this post I will be describing the code for hand gesture recognition using OpenCV.The code is written in C on Dev C++.For installing the necessary libraries on Dev C++ you can check my previous post. So basically to start with I had to extract the hand region .Which can be done by many ways for eg                                                                                                                                                                                  1) you can segment the hand region using RGB values i.e.R G B values of hand will be different from background                                                                                                                                                         
OR
2) you can use edge detection
 OR
3) background subtraction.

     I have used background subtraction model. OpenCV provides us with different back ground subtraction models I choose codebook ( no specific reason).What it does is it calibrates for some time to be exact for some frames.In which for all the images it acquires; it calculates the average and deviation of each pixel and accordingly designates boxes. For more information please refer a book.

     So at this stage we have removed the background and in the foreground we only have our hand. For those who are new to CV it is like a black and white image with only the hand as white.

  
   In the next part what we intend to do is recognise the gesture. Here we use Convex Hull to find the finger tips.Convex hull is basically the convex set enclosing the hand region.


     The red line bounding the hand is convex hull .Basically it’s a convex set ; means if we take any two points inside the red region and join them to form a line then the line entirely lies inside the set.



     The yellow dot is the defect point and there will be many such defect points i,e every valley has a defect point. Now depending upon the number of defect points we can calculate the number of fingers unfolded.



summary :-
  • The hand region extraction has been done using background substraction using codebook method.
  • For Tip points i have used convex hull 2 and for depth points convexity defects.
The main code for extracting the contour and detecting the convexity points is in the function
void detect(IplImage* img_8uc1,IplImage* img_8uc3);

Place the camera in front of a steady background ; run the code ,wait for some time .Once the calibration has been done . U see the connected component image showing some disturbance.Bring your hand in cameras view . Enjoy .

VIDEOS:-




CODES:-

Link 1 : Convex Hull2 usage

Link 2 : Hand gesture recognition

                    FOR OPENCV 2.4


 Background subtraction has been done using codebook.
My code has been written over the basic example available in the opencv examples for codebook.So all that i have written has been included in a new function named detect() .

void detect(IplImage* img_8uc1,IplImage* img_8uc3) {

//8uc1 is BW image with hand as white And 8uc3 is the original image


CvMemStorage* storage = cvCreateMemStorage();
CvSeq* first_contour = NULL;
CvSeq* maxitem=NULL;
double area=0,areamax=0;
int maxn=0;


//function to find the white objects in the image and return the object boundaries

int Nc = cvFindContours(
img_8uc1,
storage,
&first_contour,
sizeof(CvContour),
CV_RETR_LIST // Try all four values and see what happens
);


int n=0;
//printf( "Total Contours Detected: %d\n", Nc );


//Here we find the contour with maximum area

if(Nc>0)
{
for( CvSeq* c=first_contour; c!=NULL; c=c->h_next )
{
//cvCvtColor( img_8uc1, img_8uc3, CV_GRAY2BGR );
area=cvContourArea(c,CV_WHOLE_SEQ );
if(area>areamax)
{areamax=area;
maxitem=c;
maxn=n;
}

n++;
}



CvMemStorage* storage3 = cvCreateMemStorage(0);
//if (maxitem) maxitem = cvApproxPoly( maxitem, sizeof(maxitem), storage3, CV_POLY_APPROX_DP, 3, 1 );


if(areamax>5000) //
check for area greater than certain value and find convex hull
{
maxitem = cvApproxPoly( maxitem, sizeof(CvContour), storage3, CV_POLY_APPROX_DP, 10, 1 );
CvPoint pt0;
CvMemStorage* storage1 = cvCreateMemStorage(0);
CvMemStorage* storage2 = cvCreateMemStorage(0);
CvSeq* ptseq = cvCreateSeq( CV_SEQ_KIND_GENERIC|CV_32SC2, sizeof(CvContour),
sizeof(CvPoint), storage1 );
CvSeq* hull;
CvSeq* defects;
for(int i = 0; i < maxitem->total; i++ )
{ CvPoint* p = CV_GET_SEQ_ELEM( CvPoint, maxitem, i );
pt0.x = p->x;
pt0.y = p->y;
cvSeqPush( ptseq, &pt0 );
}
hull = cvConvexHull2( ptseq, 0, CV_CLOCKWISE, 0 );
int hullcount = hull->total;
defects= cvConvexityDefects(ptseq,hull,storage2 );
//printf(" defect no %d \n",defects->total);

CvConvexityDefect* defectArray;
int j=0;
//int m_nomdef=0;
// This cycle marks all defects of convexity of current contours.
for(;defects;defects = defects->h_next)
{
int nomdef = defects->total; // defect amount
//outlet_float( m_nomdef, nomdef );
//printf(" defect no %d \n",nomdef);
if(nomdef == 0)
continue;
// Alloc memory for defect set.
//fprintf(stderr,"malloc\n");
defectArray = (CvConvexityDefect*)malloc(sizeof(CvConvexityDefect)*nomdef);
// Get defect set.
//fprintf(stderr,"cvCvtSeqToArray\n");
cvCvtSeqToArray(defects,defectArray, CV_WHOLE_SEQ);
// Draw marks for all defects.
for(int i=0; i
{ printf(" defect depth for defect %d %f \n",i,defectArray[i].depth);
cvLine(img_8uc3, *(defectArray[i].start), *(defectArray[i].depth_point),CV_RGB(255,255,0),1, CV_AA, 0 );
cvCircle( img_8uc3, *(defectArray[i].depth_point), 5, CV_RGB(0,0,164), 2, 8,0);
cvCircle( img_8uc3, *(defectArray[i].start), 5, CV_RGB(0,0,164), 2, 8,0);
cvLine(img_8uc3, *(defectArray[i].depth_point), *(defectArray[i].end),CV_RGB(255,255,0),1, CV_AA, 0 );
}
char txt[]="0";
txt[0]='0'+nomdef-1;
CvFont font;
cvInitFont(&font, CV_FONT_HERSHEY_SIMPLEX, 1.0, 1.0, 0, 5, CV_AA);
cvPutText(img_8uc3, txt, cvPoint(50, 50), &font, cvScalar(0, 0, 255, 0));
j++;
// Free memory.
free(defectArray);
}

cvReleaseMemStorage( &storage );
cvReleaseMemStorage( &storage1 );
cvReleaseMemStorage( &storage2 );
cvReleaseMemStorage( &storage3 );
//return 0;
}
}
}


thank you!! :)

47 comments:

  1. Could you possibly give a step by step explanation of what the code is doing in each line/segment as is done in the opencv tutorials handbook.

    If you have the time I would greatly appreciate it, as I am becoming familiar with C++ as I am becoming familiar with OpenCV

    ReplyDelete
    Replies
    1. I would surely do it in the vacations ! You would have to wait till 20th may !

      Delete
  2. Hi Aniket, Nice job and good notes. I am looking for a simple hand tracking OpenCV 2.4x C++ code. Would really appreciate it if you can point me to a website. I have googled but could not find anything. I am sure I must have missed it. Thanks again for this tutorial.

    ReplyDelete
    Replies
    1. hey mate, did you progress in this area ahead? Me working on the same grounds. Would be great if you can share your work experience.

      Delete
  3. i find the error.
    this comments , i don't understands.
    help me please...

    image2 @ 0x3bdb0]Could not find codec parameters (Video: 0x0000, yuv420p)

    this error , what happens..?
    i think that argc,argv is error
    so, these erase and for (n = 1; n < argc; n++
    this code erase , so, appear console window
    but, not appear two image; one image is source image

    help me please

    ReplyDelete
    Replies
    1. heyy !! the part of code which uses argc is been directly taken from the samples of opencv for background subtraction using codebook. So before testing this code do test the sample for background subtraction ! And all the errors you are getting majorly deal with the data acquisition part ! so do check the background subtraction sample !
      cheers !

      Delete
  4. Hey man, amazing work on hand tracking...... my background subtraction technique is giving me a little problem and I can't seem to find a solution online..... is it possible for u to gimme yr background subtraction code...

    ReplyDelete
    Replies
    1. my code has that too . and i am using code book for backgroung subtraction.

      Delete
  5. Hi Aniket,

    Thanks for the code sharing.
    I am new to opencv and need a little advise from you.
    Can you tell me how to find the coordinate of the index finger from the convex hull method to implement the virtual keyboard?
    Any opencv function can be used in this case?
    Hope you can help me.
    Thanks in advance.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. the CvSeq* hull stores the points you are looking for.since the orientation of convex hull is CLOCKWISE. the second point should be the one you are looking for.

      Delete
  6. hi i am using open cv2.2 and visual c++. I a getting an error fatal error LNK1104: cannot open file 'opencv_calib3d220d.obj'. can you please resolve the issue

    ReplyDelete
    Replies
    1. you might need to disable incremental linking:
      go to Project Properties -> Configuration Properties -> Linker (General) -> Enable Incremental Linking -> "No (/INCREMENTAL:NO)"

      Delete
  7. Hey which code are we supposed to implement? Link1 or Link2?
    Thanks.

    ReplyDelete
  8. Hi, Aniket Tatipamula thanks for sharing, now I understand a little bit more. But I've a question, why in your covnvex video you're in the frame and your hand can be detected without you? I'm trying to do the same but the I'm always detected and my hand isn't. Thanks

    ReplyDelete
    Replies
    1. ohh ! basically you will have to be in the video while calibrating (except for your hand keep it out ) . So now ,you will be treated as background when the code is running .

      Delete
    2. Thanks for answer, I thought you'll never answer T_T. But I make it work, But now I'm Trying to Detect Only the shape of the hand, even if I move or some other object cross the area, Do you know some Hand-hape detection or something like that?, but allways detect the hand by the Shape no by the Skin color.

      Thanks.

      Delete
  9. would be great if no google account would be required to see the code. wow, now i got a blogger account connected to my new google account. i never knew i wanted this :)

    anyway, thx for sharing.

    ReplyDelete
  10. Hi, Aniket Tatipamula. first, thanks you for your sharing. Do you have any site or link talking about the codebook method. i hardly understand the codebook method based on the code and the comments in the code and i cant search any site explaining the codebook method on google. if you have those sites or links explaining about the codebook method, kindly send the links to my email--> lhkeong.10@gmail.com or lhkeong-10@hotmail.com

    i very appreciate your help and your sharing. thanks you.

    ReplyDelete
    Replies
    1. you can check out " Learning OpenCV - O'Reilly " .

      Delete
  11. Hi aniket. Is there any possible way I can use the same image processing code with the image taken by an IP camera?? If so, how? Can I really integrate an IP camera directly to a devc++ program..Thank you so much..

    ReplyDelete
  12. check for CvCapture object and do necessary changes as stated in this link
    http://stackoverflow.com/questions/712998/opencv-with-network-cameras

    ReplyDelete
  13. Hi Aniket hope u doing well, I have this problem(Warning...........Source not compiled) can u help me plzzzzzzzzz .......... thanks

    ReplyDelete
    Replies
    1. Compile it. chk this - http://anikettatipamula.blogspot.in/2012/02/setting-up-opencv-on-dev-c.html

      Delete
  14. This comment has been removed by the author.

    ReplyDelete
  15. Hi aniket, we are doing mouse control using hand gesture as our summer project. We are facing difficulty in subtracting background. As we are doing the work in python, We couldn't implement any good background subtraction algorithm. We are using global hsv filter to detect the skin. But, It's not giving the desired output. can u plz elaborate how to use codebook method in python.

    ReplyDelete
    Replies
    1. am also wrkin on sme thing let me knw if you hav any prgrs

      Delete
  16. This comment has been removed by the author.

    ReplyDelete
  17. Buenas noches, me ha parecido muy interesante ésta página relacinado con el comvex hull, ya ensayé el algoritmo en figuras geometricas elaboradas en paint, como por ejemplo un triangulo, una estrella, una flecha, pero solo me funciona para la estrella, ¿Cómo podría realizarlo para las demás figuras?

    ReplyDelete
    Replies
    1. i tried google translate ! but i am not able to understand your question.

      Delete
  18. This comment has been removed by the author.

    ReplyDelete
  19. Hi Aniket, thank you for sharing.
    I need to use the background subtraction to detect hand, so your code is really useful, but i have a problem. When i try to run the program, it terminates without do anything. I have no errors in compiling, so the environment is well-configured. Any suggestions to understand my mistake?
    Thanks a lot!

    ReplyDelete
    Replies
    1. In particular i notice that the instruction 'model=cvCreateBGCodeBookModel()' probably causes the crash of the application. The program terminates just during the processing of that instruction.

      Delete
  20. Hi Aniket I tried to run your code first it ran well but now it has really become slow,
    it immediately draw lines where ever it detects skin color and then stops.

    ReplyDelete
  21. Hai Aniket,
    When I try to run the code I am not getting the right output. I'm mailing you the link to the output I am getting can you help me.

    https://www.youtube.com/watch?v=pOqe-2DHNJQ

    Thank You,
    Murali

    ReplyDelete
  22. hye..do you know how to integrate both the the coding from the videos of convex hull and hand gesture detection??

    -Azri-

    ReplyDelete
  23. Hi, it seems from what I see in this code :

    https://docs.google.com/file/d/0B7lDtwez94H3MWZlMGQwOTUtODk2Ni00NmNmLWI3NTUtNjA4MTY2NTc5NDk0/edit

    There's no explanation on how to get number of fingers??

    Could you provide the link that shows how to calculate finger? Thanks

    ReplyDelete
  24. M also from vjti.cn u plz explain me the code for gesture recognition?can we meet?

    ReplyDelete
  25. How to calibrate the thresholds???
    Are you working on any particular opencv projects??

    ReplyDelete
  26. Hi Aniket... i had just started working on hand gestures could you tell me which book i should refer to start from the basic and if you have some material or sample programs you would like to share, can you mail me at ami_301618@yahoo.co.in.

    ReplyDelete
  27. I need this project with matlab can you help me

    ReplyDelete
  28. Hi,Aniket can u post complete source code in c# ? Do you have any site from where I can get it? I need it plz share it as soon as possible at tayba.farooqui@gmail.com.Thanks in advance.

    ReplyDelete
  29. Hey!ForegroundCodeBook window and CodeBook_ConnectComp window are not displaying anything. They have grey and white screens respectively. What possibly could be the error. Need help. Thankyou.

    ReplyDelete