Month: March 2019

This week I worked on training the classifier on AWS. I set up my AWS, including setting up the environment to be able to run my code. I am training with a dataset of 10k negative and 8k positive images, selecting 400 classifiers. I also incorporated hard negative mining. Specifically, I take any false positives in the test set and add those negative example to my training set. The training is still going on, and should be complete by beginning of the week. Additionally, I began writing code for the cascading classifiers in order to increase detection speeds. Previously, it took around 4seconds to detect all the faces in an image. I optimize this by parallelizing the sliding window search, and hope to further increase performance by using cascading classifiers.

Once the classifier has been trained, I will incorporate my facial detector into the mvp. This should be done by early next week. After this, I will work on improving accuracy for hand detection. Overall, I am slightly behind, but this is mostly because the classifier takes extremely long to train (a couple days). This has hindered my progress the past couple weeks.

This week I worked on face preprocessing code, modifying the alignment and cropping code to improve the preprocessing of the face and get only the face and with no background. I added more functions to the general preprocessing class and we have integrated it with the facial recognition. There was also a performance error last week that was causing the integrated face alignment code to run very slowly. I fixed the performance error and now the preprocessing is nearly instantaneous. I also changed the code so that all training images are preprocessed at the start as opposed to one by one. For testing, we will test images as they are sent for recognition, i.e. one-by-one. This code has been commited to the repository,

I worked with the camera and we deemed that it is not ideal for what we want as it requires making http requests and we wanted a direct stream to opencv. Omar has ordered a 1080p USB camera that we will be using. We feel that it should be suitable for our project purposes since our test conditions involve a small number of people sitting in one row.

I am behind schedule but since work in other classes has relatively freed up now, I will be dedicating much more time to capstone work. In the coming week I will be working to improve the facial recognition performance following our demo. Currently my code uses dlib face detection and that will be replaced with Kevan’s face detection which will be modified in the coming week. I will also be starting work on hand-raise detection next week.

Our facial detection is nearing completion and will soon be fused into our MVP. Hopefully, this fusion will be simple as the OpenCV facial detection code we are using is fairly straightforward (provide frame, get back a list of face bounding boxes).

We are still working on improving facial recognition. We hope that better face cropping, dlib correction, and zeroing out background noise will all lead to improvements.

There are no major changes to design/schedule.

This week I did a little benchmarking between our existing MVP and our MVP once Neeraj’s dlib and face-cropping code were added in.  We learned from Professor Savviddes that including anything but the face itself could have a significant impact on the performance of fisherfaces. So once the dlib and face cropping code was added I was hoping for a significant boost in performance. The automated tests showed that not much of a performance increase was achieved. This was not a significant issue as our automated tests already perform very well (~80%) since they are only testing for changes in facial expressions.

Our live testing MVP script, however, did have a modest improvement in delineating between Me, Neeraj, and Kevan. One reason we suspected it did not do as well was because we used a 64×64 image training set for that live test. We need higher resolution testing data. I gathered 256×256 testing set, which included side-face views, of myself. My partners will also be doing the same.

I also ordered a 1080p webcam, mount, and data cable for us to use in case our existing camera (provided by the Professor) did not meet our needs.

My work, I believe, is ahead of schedule thanks to contributions made over the break.

Next week, we will be working on fusing Kevan’s detection into our own work as well as improving recognition.

• What did you personally accomplish this week on the project?

Since I got back from spring break I have been working on optimizing my facial detection.  In order to increase the accuracy, I changed the dataset used for training and increased the the total number of training examples (1500 positive and 2000 negative). I also made some alterations to the adaboost algorithm. After making these changes, I am getting 88% accuracy for Faces, and 90% accuracy for Non-faces. This is only when  detecting a single face in a 24×24 image. I am using a test set of 500 positive and 500 negative images.

Next, I was working on the sliding window code to detect multiple faces in an image. This is the sample output:

There are multiple overlapping boxes, which is expected. I used Non-maximum suppression to merge overlapping boxes into one. The final output is as follows:

As you can see, there is a high false-positve rate. Additionally, not all the faces were detected. I am currently trying to improve the detection. I will try improve my training set, and use more examples. In addition, I will use more features (currently, I only select 100). The issue with these changes however, is that this will make training much slower.

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?

I think I am slightly behind schedule, as I was hoping to be completely done with detection by this week. This is mostly because I didn’t manage to get much done during spring break. However, I think I can catch up this following week.

• What deliverables do you hope to complete in the next week?

I will continue to optimize to get better accuracy and reduce the number of false-positives. I will also integrate my code into the MVP script.

One significant risk is not being able to increase our recognition accuracy to the required MVP levels. Fortunately, thanks to a boosting algorithm using fisherfaces we have improved our testing accuracy to about MVP levels. However, this testing accuracy is for a series of images where the face does not move and only the facial expressions move. Real-world scenarios will have many more confounding factors to deal with.

Another risk was the high false-positive rate with facial detection. However, we believe that the face alignment api we use will help filter these out.

Another risk is not being able to delineate between a stranger and non-stranger. Fortunately, thanks to a bug discovered in our code we now have a better stranger detection. But it is still lacking. 

It has also become apparent that the dlib code has performance issues when it comes to training as it takes about 1 second to process each image.  When added to the pca class to preprocess each image this makes the program quite slow.

No major changes have been made to the existing design, and the schedule does not need to be updated from last week.

I ended up spending a majority of Spring Break working on Capstone. It was not exactly expected but I enjoyed the programming and couldn’t get myself to stop.

Anyways, as a product of that work, I was able to write a working MVP program that accomplishes many of our MVP objectives by fusing together my work with facial recognition and hand detection.

That MVP script, using my webcam, does the following:

• detect faces (OpenCV). Hopefully, we can fuse Kevan’s work with detection to replace OpenCV’s here soon.
• recognize faces (my PCA + LDA = fisherfaces implementation from scratch) with < 1 second delay. Thanks to a novel boosting algorithm I implemented, the accuracy on our training set increased to 70-80% on average. However, accuracy in real-life webcam usage is lower because the faces are seen in new enviroments (tilt, brightness, etc). Hopefully, Neeraj’s work with dlib and face regularizatoin can help with this.
• Detects hands raised at the height of the persons head or higher with 1 second delay. Fairly high accuracy (~75% or greater). Built all from scratch (except k-means algo). Re-learns each individuals face pigments every 10 seconds (takes about ~2 seconds)
• Records all recognition and hand detection and displays it in charts.
• Runs a suite of automatic tests on the recognition portion of the script. Displays results in useful charts.

 

I can’t summarize all of the things I did over break. But I will touch on the important ones.

Novel Boosting Algo for Fisherfaces

Our fisherfaces implementation was not generating the accuracy on out testing set that we wanted (about ~55% with 4 classes). I came up with a boosting algo where several iterations of unique fisherfaces were run and their results fused. Each iteration differed on mainly two factors: Firstly, they learned on somewhat different training sets. Thus, each learned in a unique way, yet still saw most of the training data so that each did better than random guessing. Secondly, each weak learner used a random image format when doing PCA, LDA (options include gray, red, blue, green; hue was deemed ineffective). Each weak learner voted on classification.

 

I am currently far ahead of schedule.

18500-Design-Review-Report

Our risks are very much the same as the risks we dealt with last week.

One significant risk is not being able to increase our recognition accuracy to the required MVP levels. As we said before, Local PCA was not successful, so we switched to Global PCA. That still was not good enough, so we implemented LDA, used more data, and more eigenvectors. Still now, we do not have enough accuracy. We are hoping and hypothesizing that we made a mistake with the fisherfaces implementation, which when fixed will increase our accuracy. We then tried some other techniques (randomly change brightness and contrast in training images, include higher resolution images, include more people in training) and they still have not yielded much success.

Another risk that remains is that we need to come up with an algorithm to delineate between a stranger and non-stranger. Two approaches have already been fielded with limited success.

No major changes have been made to the existing design, and the schedule does not need to be updated from last week.

• What did you personally accomplish this week on the project? Give files or photos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week (12+ hours).
  • This week I spent a significant amount of time trying to optimize our Fisherfaces implementation to achieve higher accuracy. Currently, we are at around 50-60% accuracy. Some things I tried include the following:
    • Randomly alter the contrast and brightness for all training images. This was an attempt to prevent poorly collected training data from causing the training to overfit on the training data and give poor results in actual testing. More specifically, if we had training data taken in only one or two light settings then testing in another light setting may lead to poor results. The Fisherfaces (eigenvectors) may really just be capturing the brightness features. Indeed, we often see the first few eigenvalues having very large values which might be because of their capturing pixel brightness.
    • We also tried to collect better training data. We focused on two aspects. One we collected data at a higher resolution (140 x 140) pixels while before we were just using 64 x 64. Using even higher resolution is expensive because of the large matrix multiplications involved which are a function of the image input dimension. We also collected data where the lighting in the room was varied across several settings.
    • I tried running the algorithm on more people. Usually, we tested only 3 people but I tried as many as 5 or 6 now.
  • I began researching how to extract the skin HSV thresholding given a bounding box of a persons face so that we could then detect the skin of that same person’s hand (therefore able to detect a hand-raise).

• Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?
  • • On schedule.
• What deliverables do you hope to complete in the next week?
  • Next week is Spring Break. If I find free time I may volunteer to do some work but I may also end up doing nothing, which our schedule (gant-chart) allows for us to do.