Too hard to say Goodbye

 All these troubles
               will give us stories to tell each other later in life                                                                                     ----- Shakespeare

Time flies.
Today, is the last day in cse155 this quarter.
Thank you all for leaving me such a wonderful memory of my first quarter in America.

Hope to see you in the next 6 months in UCSD ~

Thank you!

Final Report

Want 2 Find U

---

Too Hard To Say Goodbye

0. Description

 I. Milestones

II. Method 

III. Tools

0. Description

    Sketch-Based Face Matching

   1. the User draw a picture of a front face using mouse, and choose the color of the face and eyes and hair.

    /*the User note the key Points of the face, by pointing*/

  2. the application gives back the most closely matched one.

I. Milestones

II. Methods

     1. Using Haar Classifer to locate the Features on the Faces.

      1.1. Constrain the Located Features by changing the space located by Haar classifier.

     2. Using the Harris Corner Detector to get the Key Points.

      2.1. Use the Harris Corner Detector in Features that are located by Haar classifier.

      2.2. Get the uppermost, lowermost, leftmost, and rightmost Key Points within the Feature.

     3. Getting the Color of Face, Eyes mathematic  average.

      3.1. Use the average Color of the nose to be the representative of the Face.

      3.2. Use the Haar Classifier to locate the Eye, get the average color of the iris.

      

     4. Compare the Faces by checking the length and distances as well as shapes.

III. tool

     OpenCV 2.3 + Python 2.7

IV. Database

     Front Face Pictures grabbed from google.

V. Future Plan

   1. add more restrain. Such as the Moles, the eye-wrinkles and so on.

   2. train better classifier for the feature detection & Face Key Point detection.

   3. work out a better algorithm for the searching process.

   4. Try to link this to social network, and use the Location and other information to help in finding the person. 

VI. User Interface

VII. Result

(the most closely one, when i didn't put the original photo in my database)

Using Harris Corner Detector to get the KeyPoints

Using Harris Corner Detector to get the Key Points.(within the located features)
   The key points I got by using Harris Corner Detector are much better than using SURF of SIFT.
   Though, I have to admit that the picture becomes quite scary at the beginning.

But we can easily find that all the Features we need. the eyebrow, eyes, nose and mouth are Covered with a lot of red pots.

SO i decided to use the Harris Corner Detector and Haar Cascade Classifier together to get the key points I want.
When located in to different features, I can get the uppermost, lowermost, leftmost and right most. That is. it is enough to represent the width and length of that feature.

Draw by Mouse

Sketch the face by Mouse

The  user draw a sketch of a face by Mouse.
My code of this part is written in Python 2.7 using OpenCV lib.
 1. Track the movement of the mouse.
 2. Record the move and print it to another page.

it is very hard for the ordinary people without painting experience to draw a face, especially draw by mouse.
So I provide a sketch picture, as the background, to make it easier for the user to draw.

When done, the picture is saved as a new .jpg

How r u, Want 2 find U?

How r u, Want 2 find U?

     after 2 weeks of changing my mind again and again and again. Want 2 find u finally can move forward.

      1. Use Key Points instead of the Contour/ outlines

           😄 1. Key Points, in some degree, is better than the Contours to be representative of a picture

           😄 2. Key Points are clear and can speed up the Process of Comparing

           😢 1. Key Points may not contain as much as useful informations as the contours,    

                       However, I don't think the user who sketch a face can definitely confirm that ---- what he draw is what he want. 😄

      2. Use Key Point inside the Feature templates

      3. Use the Proportion instead of the length.

!!! The papers I follow..

       1.     Face Recognition: Features versus Templates

       2.     Feature Points Extraction from Faces

!!! What I will do this and next week:

      1. get the informations such as the LIP THICKNESS (necessary when i saw this..)

          the thickness of the Eyebrow if possible.

         

      2. check the Eye-lid --- if it is double fold eyelid. (necessary)

      3. check the shape of the chin.

      4. get the shape/outline of the eyes.

      5. Get the SKETCH, do the preprocess of it in the same way.

      6. Add the color of the Eye and Skin (already done)

      7. compare.

   

Using KeyPoints

1. Locate the Features

        use Haar Cascade Classifier to locate the Eyebrows, Eyes, Ears (not useful), Nose and Mouth.

2. Get Key Points

        use SURF algorithm to get the key points.

        2.1 about the Key Points, (x,y)

            1. Eyebrow: eyebrow_l1, eyebrow_l2, eyebrow_r1, eyebrow_r2 

            2. Eye:         eye_l1, eye_l2, eye_r1, eye_r2

            3. Iris:          iris_l, iris_r

            4. Nose:       nose_l, nose_m, nose_r

            5. Mouth:    mouth_l, mouth_m, mouth_r

            6. Jaw:         jaw

            7. Face:       face_l1, face_l2, face_r1, face_r2

x=111 y=104 laplacian=1 size=16 dir=90.000000 hessian=15466.430664

x=182 y=106 laplacian=1 size=15 dir=90.000000 hessian=12413.648438

x=192 y=110 laplacian=1 size=16 dir=90.000000 hessian=13282.147461

x=79 y=132 laplacian=1 size=15 dir=90.000000 hessian=13704.891602

x=92 y=137 laplacian=-1 size=16 dir=90.000000 hessian=11122.696289

x=105 y=175 laplacian=1 size=16 dir=90.000000 hessian=11498.058594

x=76 y=216 laplacian=1 size=16 dir=90.000000 hessian=14645.329102

x=220 y=139 laplacian=-1 size=32 dir=90.000000 hessian=15401.563477

x=93 y=146 laplacian=1 size=34 dir=90.000000 hessian=16285.532227

x=215 y=153 laplacian=1 size=31 dir=90.000000 hessian=13717.638672

x=228 y=211 laplacian=-1 size=31 dir=90.000000 hessian=22596.167969

x=53 y=227 laplacian=1 size=30 dir=90.000000 hessian=13204.163086

x=270 y=240 laplacian=1 size=30 dir=90.000000 hessian=18954.589844

x=89 y=153 laplacian=1 size=57 dir=90.000000 hessian=14741.933594

x=213 y=191 laplacian=1 size=62 dir=90.000000 hessian=15877.199219

x=230 y=221 laplacian=-1 size=58 dir=90.000000 hessian=32176.843750

3. Get the Length of the features, and the Distance among the features

      (to make it easier to present, just show it like below)

      (actually works like [(y2-y1)^2 + (x2-x1)^2]^(1/2))

        iris_dist           = iris_r - iris_l ; (red points)

eye_length      = eye_l2 - eye_l1 ; ( orange point, yellow point)

nose_width     = nose_r - nose_l ; ( green points)

mouth_length  = mouth_r - mouth_l ; (purple points)

        eyebrow_length

                               = eyebrow_l2 - eyebrow_l1; (blue points)

        eye_dist          = eye_r2 - eye_l1; (orange points)

        nose_mouth    = nose_m - mouth_m; (green & purple)

face_length     = face_r1- face_l1 ; (blue-green color points)

4. The Shape of the Face ----- Face Angle

       i didn't make it to get all the key points i want on the outline of the face.

       so i change it to be like 

               face_l1(x,y) -> find point face_l1(x,y) on FaceOutLine where x = Iris_l(x)

       and also face_r1(x,y)

       then find face_l2(x,y) -> find point face_l2(x,y) on FaceOutLine where x = eye_l1(x)

              also face_r2(x,y)

       face_angle1 = ( jaw(y) - face_l1(y) ) / ( jaw(x) - face_l1(x) )

       face_angle2 = ( face_l1(y) - face_l2(y) ) / ( face_l1(x) - face_l2(x) )

       face_angle = face_angle1 / face_angle2

after these four steps, I get the information of the face:

     1. the length of nose, mouth, eyes, eyebrow

     2. the distance of the Iris_Iris, Eye_Eye, nose_mouth...

use Proportion to compare

after getting the key points, i come across a problem
 😭  Problem

   Because the size of the head in different pictures are not the same, Using the Length and Distance to compare the pictures is far from accurate.

💡 Solution!

   Proportion is used to help me get rid of this problem.

     How?

        Consider the iris_dist to be the base.

        So..... I get:

     eye_iris                = eye_length / iris_dist ;
         eyebrow_iris        = eyebrow_length / iris_dist;
 nose_iris              = nose_width / iris_dist ;
 mouth_iris            = mouth_length / iris_dist ;
   
         eye_dist_iris         = eye_dist / iris_dist ;
         nose_mouth_iris   = nose_mouth / iris_dist ;

TrackBar & Color

TrackBar & Color
Last week, I disappeared from class. I got 2 midterms, and get sick after cse130-----OCAML : (.....

I didn't use the main color of the faces. And I didn't use HSV.

instead.

I tried to locate the nose, and get the average RGB value of the nose.

About Want 2 Find U
I added the find_by_face_color function, to my project.

1. TrackBar is used in this part to represent the COLOR.

(because the color of faces are usually white, yellow, red, brown, black. So I make 5 block in the track bar. (From low to high, That is from white to black))(you can see the track bar above the photo of Sophie Marceau)


(when you choose the first mark, you will receive all the photos have the color between white and yellow.
 The second, you will receive photos of skin color from yellow to red.....)

NOW you can see only one pic of each mark, because I only have 5 pics in my database to show my idea in class, but Want 2 find U will have more photos. 

NEXT: GET THE OUTLINE OF THE FACES, AND DO THE MATCH.

Color Histogram

Color histogram（hsv(hue,saturation,value)）

now I can locate the face in a photo, and make a HSV Color Histogram for it.

this is Sir Alex Ferguson, His face is always red.

i was imaging this two sportsman should seems similar in the diagram.

However, Nani seems a little brighter.

Two 'Vampires', Their Histograms are similar. (But actually Bella is not vampire in this pic, so the blue line is lower.)

What I am going to do this week, is 

1. Calculate the MAIN COLOR of the face and eyes.

2. Start the Work on Outline comparison.

3.Start the work on Color comparison

edge detection

Edge Detection 

Just do the edge Detection is far from enough.

What I should do is try to divide the Face by features.

And then do the Comparison Between the shapes of the features.

That is eyes to eyes, nose to nose.

In this way, Features seems like some small tags for a face.

(50,150)

(100,150)

OpenCV & Python2.7 on Mac OS X

Finally wake up  from the NightMare....

but.....another nightmare...in app store...made me change the name of my blog...

1.Setting up OpenCV on Mac OS X (binding Python2.7) 

        http://opencv.willowgarage.com/wiki/Mac_OS_X_OpenCV_Port


      1. download and install macports( http://www.macports.org/ )
            1.1 Mac OS X has already come with Python


      2. sudo port -v install opencv +python27
(before that remember to install command line tools in Xcode. or it will 
result in the conflicks in the installing process )
(Xcode -> preference -> components -> command line tools, then install it)

2. Start with OpenCV and Python2.7

    After spend the whole weekends dealing with setting up OpenCV with Python on Mac OS X, I finally start my project.


    Since OpenCV and Python are both open source, what I can refer to is very narrow. So I start with the The OpenCV Reference Manual

3. Track the Face and Eyes

       I write some small program in python 2.7, using openCV and haarcascades classifer.
       What I am able to do now is to locate the face, sometimes eyes in a photo...
       And also track the movement of the eyes and face from the video captured by webcam.
       

followings from webcam video

(seems this is a good way to find my eyes...)

 with glasses? it's ok, but.. we have to cut the influence...

s

4.What to do next?

   i.  Locate the Pupil (I mean.. the colorful part of the eyes.).
    ( 1. try to recognize circles in the given photo)
    ( 2. try to locate the circle inside the eyes ---- that is, pupil!)
  ii. Record the color of the eyes
 iii. Record the color of the face 

OpenCV & Python

There is an app using the name of FaceMatch in appStore, so I decided to change my project name to Want2FindU.

I found this video... And....finally have some confidence in my project
http://www.youtube.com/watch?v=4zDCs5l3Sj4

Works on Want2FindU, Starts

Works on Want2FindU,starts
.....
.......with my Nightmare on App Store



Nightmare on App Store

I've just bought a Macintosh to do my project with it. However I cannot login with my Chinese Account. 

What is worse, I cannot pay for the apps with my Chinese Credit Card. So I have to wait for activating my BOA card to continue setting up my new Apple ID.

This blocked my process on FaceMatch, and cannot follow my milestones in proposal.

So I did some reading instead and re-organized my thoughts.

1. Want2FindU will be a web application

2. Want2FindU will be written in python    /* to speed up, and keep in pace with the milestones. */
      - OpenCV for Python

3.Database of Want2FindU
     As I mentioned before, I want to use the portraits in Facebook or other Social Networks. But there is a PROBLEM.

     The users of the Social Networks just don't like to use the FRONTAL FACIAL IMAGE.
     However , if I ask the user to draw a picture of someone. I possibly receive a frontal facial image.

     I have some SOLUTIONS

      1. forget this problem, because there are billions of users in Facebook. There must be millions of people who uses the frontal facial image.

      2. change the 2D input image into 3D image
            -- 3D Face Reconstruction Using A Single 2D Frontal Facial Image
             http://jccit.ccit.ndu.edu.tw/ezfiles/7/1007/img/18/40-1-17_No-415.pdf

   
      3. change the 2D side facial image in database into  2D frontal facial image
            -- I didn't know if someone had done this before. But I think it is possible, because we are so familiar with the feature of a face and most of the faces are symmetrical.
            ( I will try solution 3 this week. and if I can find out the solution, I will post it on my blog)

      4. forget the Social Network. Set up my own web application, to ask the user take a photo from the laptop camera following the rules.

      5. forget the Social networks and my own Web application, just use some existing front facial image dataset.

      solution 1&5 seems easier for me.
      solution 2 is a little bit difficult.
      solution 3 | 4 is the most possible solution that I will use.

4. My Work & existing sketch-based image searching


       I did some comparasion between Want2FindU and other software(/image searching engine) , and did some analyze.
       http://want2findu.blogspot.com/2012/04/about-mindfinder-and-facematch.html


     This Week's goal;
        1. handle the problem with Apple ID.
        2. try to work out a way to change the side facial image into front facial image.
        3. start my work on Want2FindU
     

About the MindFinder AND Want2FindU

I found a short video of MindFinder in youtube.com
http://www.youtube.com/watch?v=UAFNtG0TTTo
1) Shapes
2) Complicated pictures
3) Sketch + tag
4) Sketch + tag + Color( main colors)

So, what I am going to do is a little bit similar  to their work.
However there is something different:

1) POSITION SENSITIVE?
    in the  MindFinder, the seaching is position sensitive.
    That is "the matched object should be at a similar position" as the input sketch, for example, when you draw a triangle on the upper left, then the pictures presented will be have some triangles in the same position.

    However in the Want2FindU, I am going to use the API from face.com to locate the face in the photo. That is to say, Want2FindU doesn't have to be POSITION SENSITIVE. I will just do the comparison between the sketching and the face, located by the face.com API.

this is one example in face.com. I think you are very familiar with this kind of App

 

2) SHAPE SENSITIVE?
   Want2FindU will be shape sensitive.


3) TAGS?
     If I want to finish this project following my original idea ---- link this with Social Networks, some Tag showing the information of the person, such as Nationalities,  Address, Age will give me great help in searching.


4) COLOR?
     In MindFinder, the user can choose a main color of his searching to get the photo with similar color.
     I will going to use this technique as well, however, only the main color is not enough, because the color of eyes is also necessary when recognize a person.

5) what I will ADD?
    As I mentioned in my blog, I will ADD SOME FUNCTIONS to help the user in drawing the picture closer to what is in his mind.
    ..........

6) Method.
   Edgel Index, indexing strategy for large-scale sketch-based image search.
   ..... this may be my be my job on this weekends... @_@....

7）Difficulty 
    The differences in faces are much slighter than the differences in other kind of photos.
 
     /* in this picture, you can see how the MindFinder works.
         The outline of the pictures is very clear. That's we, the users can check if the result is closed to our imagination very easily, because other photos is totally different.
         However, when it comes to the faces......
         The shape of the eyes, nose, mouth, ears..... are similiar .
         So the Want2FindU requires high accuracy....
*/

   

About the MindFinder

About the MindFinder
When I read the paper named, 'Edgel Index for Large Scale Sketch-based Imange Search', I found this software --- MindFinder.
However, I can only find a video of it.
http://www.youtube.com/watch?v=UAFNtG0TTTo

http://www.youtube.com/watch?v=UAFNtG0TTTo

About the FaceMatch

About the Want2FindU
I've got this idea many years ago, but only by now, with the wide spreading of Social Networks,  I can finally see the  chance to turn it into reality.

What is Want2FindU?
In my imagination, it will be an application that allow the users to draw a face of someone, and then searching for the most closely matches.

Problems:
0.people are not all artists.
1.Color Photo May Bring a Lot of Obstacles.
10.The Photos in Social Network is not Regular




PROBLEMS I: people are not all artists.


However, I have to admit that the faces is not very easy to draw for the ordinary people, who are not experts in painting. Therefore, I have to offer some functions to help in the process of organizing the pictures.
( I asked my friend to try to copy my photos. Even when I knew that it might be terrible, didn't I realize that it could be horrible)
This is the original photo of ME:

This is my friend's painting:
 ( we find that the first raw of the painting, seems like Picasso's work)
 ( The following one is interesting, I thinks it seems like Michael Jackson)

POSSIBLE SOLUTIONI:
   Provide some different kinds of features for users to drag to the canvas.
  ( I don't know if you once played this game, I think in some degree, I got this idea from it. )

PROBLEMS II: Color Photo May Bring a Lot of Obstacles.


  Color photo contains more content, and rocket up the time consumption. However, I think the color concentrate more on the detail than the outline. When we see a picture, even a gray-scale picture or a black-white picture, we can figure out the person in that picture. In some degree, Color in image influence both the user and the searching process.

POSSIBLE SOLUTION II:
 Change the photo into gray-scale image: (R,G,B) -> int gray = R＊0.3 ＋ G＊0.59 ＋ B＊0.11;
or Change it into black&white image by setting the THRESHOLD 128. 




PROBLEMS III:  The Photos in Social Network is not Regular

This is one of the portrait I used
 in the "Chinese Facebook" ---
 renren.com


As you can see...
This can't be a valuable picture 
in my application









POSSIBLE SOLUTION III:
 Add a function to ask the user capture their photo with the camera in their laptop or smartphone.
 or Just add some requirements.

 this is a picture I used in the in20year.com
It's a website, that can show what you will look like in 20 years.
 However when I put my photos( right, small one), The feedback really scared me.

But I learn a lesson from that, we should ask the user to put right, regular photo.

A Romantic Story, A Sweet Reason

A Romantic Story, A Sweet Reason

Have you heard about the  http://www.nygirlofmydreams.com/ ? Do you know about the Romantic Story in the  New York subway?

from: http://gawker.com/319937/patrick-moberg-has-found-his-new-york-dream-girl

from: nygirlofmydreams.com

Patrick Moberg, the website designer in NY, used 48 hours to find his Dream girl. It is a wonderful romantic story. Maybe from the day I saw this story, I have been thinking about how to seek for a person without knowing the name. Is it possible to just draw a picture, and then find out the one you want?

Now we have the Social Networks, like facebook.com, and have millions of photos of different faces. The romantic story in the NY Subway, gives me a sweet reason to do my project, Want2FindU --- Sketch-based Face Recognition application. If my Want2FindU is born successfully, it will be possible that the next Patrick may use no more than 10 seconds to find his dream girl.