Author: odelen

This week I did a little benchmarking between our existing MVP and our MVP once Neeraj’s dlib and face-cropping code were added in.  We learned from Professor Savviddes that including anything but the face itself could have a significant impact on the performance of fisherfaces. So once the dlib and face cropping code was added I was hoping for a significant boost in performance. The automated tests showed that not much of a performance increase was achieved. This was not a significant issue as our automated tests already perform very well (~80%) since they are only testing for changes in facial expressions.

Our live testing MVP script, however, did have a modest improvement in delineating between Me, Neeraj, and Kevan. One reason we suspected it did not do as well was because we used a 64×64 image training set for that live test. We need higher resolution testing data. I gathered 256×256 testing set, which included side-face views, of myself. My partners will also be doing the same.

I also ordered a 1080p webcam, mount, and data cable for us to use in case our existing camera (provided by the Professor) did not meet our needs.

My work, I believe, is ahead of schedule thanks to contributions made over the break.

Next week, we will be working on fusing Kevan’s detection into our own work as well as improving recognition.

One significant risk is not being able to increase our recognition accuracy to the required MVP levels. Fortunately, thanks to a boosting algorithm using fisherfaces we have improved our testing accuracy to about MVP levels. However, this testing accuracy is for a series of images where the face does not move and only the facial expressions move. Real-world scenarios will have many more confounding factors to deal with.

Another risk was the high false-positive rate with facial detection. However, we believe that the face alignment api we use will help filter these out.

Another risk is not being able to delineate between a stranger and non-stranger. Fortunately, thanks to a bug discovered in our code we now have a better stranger detection. But it is still lacking. 

It has also become apparent that the dlib code has performance issues when it comes to training as it takes about 1 second to process each image.  When added to the pca class to preprocess each image this makes the program quite slow.

No major changes have been made to the existing design, and the schedule does not need to be updated from last week.

I ended up spending a majority of Spring Break working on Capstone. It was not exactly expected but I enjoyed the programming and couldn’t get myself to stop.

Anyways, as a product of that work, I was able to write a working MVP program that accomplishes many of our MVP objectives by fusing together my work with facial recognition and hand detection.

That MVP script, using my webcam, does the following:

• detect faces (OpenCV). Hopefully, we can fuse Kevan’s work with detection to replace OpenCV’s here soon.
• recognize faces (my PCA + LDA = fisherfaces implementation from scratch) with < 1 second delay. Thanks to a novel boosting algorithm I implemented, the accuracy on our training set increased to 70-80% on average. However, accuracy in real-life webcam usage is lower because the faces are seen in new enviroments (tilt, brightness, etc). Hopefully, Neeraj’s work with dlib and face regularizatoin can help with this.
• Detects hands raised at the height of the persons head or higher with 1 second delay. Fairly high accuracy (~75% or greater). Built all from scratch (except k-means algo). Re-learns each individuals face pigments every 10 seconds (takes about ~2 seconds)
• Records all recognition and hand detection and displays it in charts.
• Runs a suite of automatic tests on the recognition portion of the script. Displays results in useful charts.

 

I can’t summarize all of the things I did over break. But I will touch on the important ones.

Novel Boosting Algo for Fisherfaces

Our fisherfaces implementation was not generating the accuracy on out testing set that we wanted (about ~55% with 4 classes). I came up with a boosting algo where several iterations of unique fisherfaces were run and their results fused. Each iteration differed on mainly two factors: Firstly, they learned on somewhat different training sets. Thus, each learned in a unique way, yet still saw most of the training data so that each did better than random guessing. Secondly, each weak learner used a random image format when doing PCA, LDA (options include gray, red, blue, green; hue was deemed ineffective). Each weak learner voted on classification.

 

I am currently far ahead of schedule.

Our risks are very much the same as the risks we dealt with last week.

One significant risk is not being able to increase our recognition accuracy to the required MVP levels. As we said before, Local PCA was not successful, so we switched to Global PCA. That still was not good enough, so we implemented LDA, used more data, and more eigenvectors. Still now, we do not have enough accuracy. We are hoping and hypothesizing that we made a mistake with the fisherfaces implementation, which when fixed will increase our accuracy. We then tried some other techniques (randomly change brightness and contrast in training images, include higher resolution images, include more people in training) and they still have not yielded much success.

Another risk that remains is that we need to come up with an algorithm to delineate between a stranger and non-stranger. Two approaches have already been fielded with limited success.

No major changes have been made to the existing design, and the schedule does not need to be updated from last week.

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).
  • This week I spent a significant amount of time trying to optimize our Fisherfaces implementation to achieve higher accuracy. Currently, we are at around 50-60% accuracy. Some things I tried include the following:
    • Randomly alter the contrast and brightness for all training images. This was an attempt to prevent poorly collected training data from causing the training to overfit on the training data and give poor results in actual testing. More specifically, if we had training data taken in only one or two light settings then testing in another light setting may lead to poor results. The Fisherfaces (eigenvectors) may really just be capturing the brightness features. Indeed, we often see the first few eigenvalues having very large values which might be because of their capturing pixel brightness.
    • We also tried to collect better training data. We focused on two aspects. One we collected data at a higher resolution (140 x 140) pixels while before we were just using 64 x 64. Using even higher resolution is expensive because of the large matrix multiplications involved which are a function of the image input dimension. We also collected data where the lighting in the room was varied across several settings.
    • I tried running the algorithm on more people. Usually, we tested only 3 people but I tried as many as 5 or 6 now.
  • I began researching how to extract the skin HSV thresholding given a bounding box of a persons face so that we could then detect the skin of that same person’s hand (therefore able to detect a hand-raise).

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?
  • • On schedule.
• What deliverables do you hope to complete in the next week?
  • Next week is Spring Break. If I find free time I may volunteer to do some work but I may also end up doing nothing, which our schedule (gant-chart) allows for us to do.

 

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

This week I went over-time in the amount of work I did. I was hoping to put sufficient work to cover not only the expectations this week but also for the next two weeks. My work is summarized below.

• • • Finished implementation Local PCA. Resulted were poor. This led to the developments in the next points.
    • Completed implementation of Global PCA. Results were better but not good enough. Used library version of K-means to cluster. Implemented classes to recognize arbitrary image using the clustering.
    • Implemented a script that uses identifies faces in real-time (combines my work related to Global PCA, LDA, Fisherfaces).

 

• • • Completed LDA and    Fisherfaces implementation. Our implementation improved our training and testing accuracy by about 10% (to about 45% per class across 3 classes). However, we are still not getting the performance we need. We believe we are not correctly implementing LDA as our clustering is not exhibiting the traits that LDA is characteristic of (see below clustering with 3 classes projected into a C – 1 = 2-dimensional space).
    • 

• • • Developed algorithms to delineate between a stranger and non-stranger. I devised two methods for detecting strangers.

1. Use an images E2 K-means distance. This distance is its distance to the nearest centroid. If this distance is larger than some threshold we consider the image as that of a stranger. We decide the threshold value by taking it as the E2 K-means distance of the training data (e.g. 75% of training data images had E2 distance <= 500 so the threshold is 500)
2. Similar threshold to before except we use the reconstruction mean-squared error as a metric.

This method seemed promising but it did not work out. Below is a chart summarizing the source of inaccuracies we detected in our training and testing. The False Positive part of the stacked bar represent errors where our k-means correctly predicted the identity but because of our thresholding we designated that image as a stranger.

In order to understand why, I visualized how the E2 and MSE metrics for training data and stranger data compared (below). As you can see, my implicit hypothesis was that stranger images would have larger E2 distances and larger MSE’s than the training data. This was not the case, unfortunately. I am still working on a solution to the problem of detecting strangers.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?
  • • On schedule.
• What deliverables do you hope to complete in the next week?
  • Since I accomplished much more than what I had to this week, I may take it easy for the next two weeks. The work I have been doing has been exciting so I may make progress anyways.

 

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).

I finished implementing Local PCA. The program takes in an arbitrary number of images, classes, and allows variable settings for eigenvectors chosen. It shows the reconstructions of all input images and also shows the k-means clustering results in a 3D plot. The clustering was not very good. After speaking with the professor, we decided we needed either more data, more eigenvectors, or to likely use LDA. We also started working on using dlib library to understand where the eyes in an image are so we can transform the image to get a frontal view.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?
  • • On schedule.
• What deliverables do you hope to complete in the next week?
  • Improve our PCA performance. Implement or use library K-mean to cluster points and take a step closer to performing the actual recognition.