Team F3: No Face

What we have done:

Claudia has finished implementing the dlib modification script. Given a face, it is able to change the color of a chosen facial landmark, eg. nose, mouth or left eyebrow. This will serve as an alternative to Avi’s adversarial network when it comes to making modifications.

 

What we are planning to do:

What we have done:

Dylan worked with Avi to get the pipeline of seeing the face identification, after it projected something onto the face. We still need to add the modification to test how our system tricks facial detection after being projected. Our recognition didn’t seem to work as well with the current setup, misclassifying a lot, so we will retrain it with some extra pictures of Dylan at the setup location since that is the only person who has access to the projector and the camera. Dylan also made the projector project in the right area a lot more accurately by dividing the camera picture into regions, and finding where to project by linearly interpolating within the region that we associate camera pixels and projector pixels. There was an error the previous homography matrix based approach that was also in the new approach where we first cropped the image to find where the face is, so part of the inconsistency with the homography matrix may have been due to that bug.

 

What we are planning to do:

Dylan is planning on adding Avi and Claudia’s projection onto what he is projecting and testing out the accuracy. Dylan is also considering creating a system of projecting onto a pattern and moving projected pattern until the two patterns are correlated to fix some of the offsets. Dylan is also considering using a tracking algorithm to make the calculation for where to project quicker, since it currently takes about 1.5 seconds per frame. However, there will probably not be enough time to get the tracking working.

What we have done:

Avi finished implementing and training two adversarial neural networks that change an image to fool the classifier.  One network had an unexpectedly high level of success, reducing the rate of classification to .07 from a test set that had an accuracy rate of .87.  The other network reduced the classification rate to .34.  The successful network outputs the same projection regardless of input, which was not intentional.  The impact of this needs to be explored further.

Dylan worked on setting up the projection onto the face. He received the projector in the mail so was able to start testing. He made a system to find the homography matrix to transform from an intended image to what is actually projected. Then he inverted the homography matrix and using a face image tried to project onto the right location on the face. The face projection seemed to project toward the correct part of the face in the center of the screen, but did not seem to move proportionally with the face. So he will need to look at why the homography transformation is not scaled aggressively enough.

Claudia continued on using Photoshop to make modifications manually, since her dlib implementation did not work as well. Trying it out manually actually helped, as it was easy to see what kinds of modifications would possibly work best for misclassification. Some results of this were that lip and skin color changes successfully misclassified some subjects, and facial hair additions changed individual probabilities to a significant extent.

What we are planning to do:

Avi will set up his code to work with the projector system in real time.  This includes blacking out the eyes in the output projection.  Once the system is set up, he will help to characterize the success of the projection in fooling our classifier.

Dylan will find how to make the homography transformation adjust the location of the projected image into the correct place. This may require utilizing techniques like using an affine transformation instead or just dividing the image into grids that I pre-calibrate so that our error is never more than a single grid. By the end of this week, we should be able to project properly sized images onto a person’s face.

Claudia will work on fixing her dlib implementation, and use it to make the modifications that Photoshop found to be successful.

What we did this week:

Dylan completed the image dewarping and mapping the transformed face from the image processing to the projected image. Dylan also began transferring our processing to AWS.

Avi has almost completed the training loop to train the adversarial network.  He has gotten to the point where gradients can be passed backwards through the networks to train and reduce loss consistently in a small test case.

Claudia began overlaying shapes onto the chosen facial landmark coordinates that are obtained from dlib. Some of them did not have much effect, as they did not change the classification probabilities significantly.

 

What we are planning to do:

Dylan is planning to finish transferring our scripts to AWS. Dylan also plans on verifying that the projection works as expected if the projector comes in.

Avi will finish implementing the training loop so that the Adversarial network will be able to show significant learning by the end of the week.  He will also implement other adversarial network architectures to train.

Claudia will try out more combinations of modifications and facial landmarks to find the ones that work best across all test individuals.

What we have done:

Dylan has changed some of the face detection functions so that they are easier for the other group members to use. Previously things like blacking out the eyes was solely done in a script, but now other members just need to call the function. Dylan has also worked on dewarping the face, after it is warped for the facial recognition. There seems to be a problem with the translation that takes place with the affine function. Though even without the translation, there is a lot of blurriness in the dewarped face. Dylan has also prepared for projecting images for once the projector arrives to him.

Claudia has worked closely with Dylan and helped with many of the tasks listed above.

Avi is ready to start implementing different neural network architectures.  He recently got facial classification to work with 94% accuracy for 6 different people.  This means that it will be possible to train the adversarial network.  Avi has also finished setting up the classifier to be easy to use for Dylan and Claudia.

What we are planning to do:

Dylan is currently blocked on using the projector. It is being shipped to him, but he can’t really test any of the projections until the projector arrives. Dylan plans on finishing the warping and hopefully projecting images accurately onto faces. The projecting onto faces is already partially complete, but need to find how well every part of face is projected onto to see if a more intense homography estimation is necessary.

Avi will implement one of the planned adversarial network structures in the next week, and set up the ability to train and characterize it.  This will make it easy to spend the next week trying multiple different architectures using the existing setup.  Right now, all testing will be with digitally produced changes to photographs.  Due to the separation of our group, Avi will be testing with only these digital changes (not the projector) for the next few weeks.

Claudia is going to start editing photos in Photoshop so that she can observe the effect on the classifier output.  She is going to start building an intuition for what changes are effective at reducing classification rates.  She will then try to directly program changes that are effective.  This is a new plan by our group, which will allow us to have more independent roles which give us two approaches at our main problem (which reduces risk).  We will also be able to meaningfully compare the successes of the machine learning approach versus the human learned approach.

What we have done:

Avi has a new strategy from Marios about how to classify faces that should train much faster than the current strategy, which we did not have enough data to make work.  This week, apart from presentations, Avi is implementing that strategy (and will have done so by midnight Sunday).

Claudia is in the process of collecting photos of individuals to be used as training data. This involves collecting around 100 photos each of about 20 people.

Dylan has tested the lens distortion calibration and the homography estimation between the camera and the projector. The lens distortion actually becomes much worse after the calibration, so Dylan is looking into reasons for why that is the case. However, since we are using the center for the image to get the face, and using the center of the projector to project onto the face, both of which are the places with the least lens distortion, there should be no need to get the lens distortion working.

 

What we are planning to do:

In the next week we will have collected a significant amount of training data and can start to implement and train our adversarial neural networks.

We can start projecting objects onto a person using the homography estimation between the projector and the camera.

What we have done:

Claudia has written a script for taking photos using the webcam, and is in the midst of collecting photos to be used as training data. This involves taking ~100 photos of about 20 individuals each.

Avi has delved into neural network architecture design to figure out how to build the adversarial network.  The current plan is to use part of the architecture from the OpenFace network so that the adversarial network can learn to identify the same important features, and then use an architecture similar to a GAN network to reconstruct a 96×96 image to input to the OpenFace network.  This is still in the process of being figured out.  We hope to have three promising architectures to try set up by next week.

Dylan has been working on the calibration between the camera and the projector. For camera and projector lens distortion, we will correct for the lens distortion using the OpenCV fixing of lens distortion. Instead of correcting for distortion on the camera and then the projector, we are just going to calibrate them both since that seems to minimize error. We may not actually need to do much lens distortion fixing since we are only using the center of the photo and center of the projector where lens distortion is least. For finding the relationship between the camera and the projector, we will project chessboard corners with the projector and then use the solve for the homography matrix with ransac from OpenCV. Will test the camera projector calibration, but need to find a better way to project than mirroring displays.

 

What we are planning to do:

• Finish up design review presentation for Monday
• Research GAN network architectures

What we have found out:

• Information about projector we are using:
  • Minimal distance with focus for projected image is 1.4 meters away from projector
  • At that 1.4 meters, a face is about 200 pixels wide, which is more than the 80 pixels wide we wanted
  • When we project a black square just on the eyes, we were comfortable looking into the projector
    • Though we will want to increase current size of black square

 

What we have done:

1. We selected and received a projector and 2 cameras from the 18500 inventory. All of them exceed the requirements for a projector and camera that we had, though the projector is much bigger than ideal. We do not have a requirement for projector size so that is fine.
2. Avi set up OpenFace to convert images into simple 128 dimensional embeddings.  Then he pulled data from a database of celebrity images and ran them through the OpenFace network.  The output was used to train a simple classification network that is able to distinguish between 20 different celebrities in the network.  The data he pulled had about 100 images of each celebrity, and about 60 were used for training.  The remaining 40 were used for validation, and the network was about 40% accurate.  He concluded that it is probably a working model that does not have enough data.
3. Claudia used dlib to extract facial landmarks, so that projections can be scaled to these landmarks. This can also be used to black out the eye area, making the experience more comfortable.
4. Dylan used Claudia’s facial landmark extractions to create a function that finds the pixels that need to be blacked out so that the projector does not project anything on someone’s eyes.
5. Dylan used Claudia’s facial landmarks to also create a height map across a person’s face. He did not finish the calculation for how far away each part of the face is from the camera that he and Claudia were working on because we found out that the depth map may not actually be important
6. We tested out putting the black-square over each member of the group’s eyes, and everyone was comfortable(no immediate eye strain) with the projection on their face.

 

What we are planning to do:

1. Avi is going to spend the next week researching neural network
   architectures to change an image without reducing its dimensionality
   and decide on appropriate loss functions for our future adversarial
   training.  If this goes well, he will also try to implement an image
   modifying network.
2. Claudia will create the program to help collect the images. She will also start collecting images. Claudia will also work on translating the image that we want to project to what we project.
3. Dylan will expand the black eye rectangle and make it cover each of the eyes. He will also complete the camera calibration for finding the geometric relationship between the camera and the projector.

Updates on design:

After talking with Marios and Emily, we found out that we may not even need depth information, since we can just correlate where we project with the facial features that dlib finds. We may decide to use the depth information in the future but for now, we will see how well projecting without depth information is and our backup will be using depth information. So for now we will only use 1 camera.