Month: April 2020

What we have done:

Claudia has finished implementing the dlib modification script. Given a face, it is able to change the color of a chosen facial landmark, eg. nose, mouth or left eyebrow. This will serve as an alternative to Avi’s adversarial network when it comes to making modifications.

 

What we are planning to do:

What we have done:

Dylan worked with Avi to get the pipeline of seeing the face identification, after it projected something onto the face. We still need to add the modification to test how our system tricks facial detection after being projected. Our recognition didn’t seem to work as well with the current setup, misclassifying a lot, so we will retrain it with some extra pictures of Dylan at the setup location since that is the only person who has access to the projector and the camera. Dylan also made the projector project in the right area a lot more accurately by dividing the camera picture into regions, and finding where to project by linearly interpolating within the region that we associate camera pixels and projector pixels. There was an error the previous homography matrix based approach that was also in the new approach where we first cropped the image to find where the face is, so part of the inconsistency with the homography matrix may have been due to that bug.

 

What we are planning to do:

Dylan is planning on adding Avi and Claudia’s projection onto what he is projecting and testing out the accuracy. Dylan is also considering creating a system of projecting onto a pattern and moving projected pattern until the two patterns are correlated to fix some of the offsets. Dylan is also considering using a tracking algorithm to make the calculation for where to project quicker, since it currently takes about 1.5 seconds per frame. However, there will probably not be enough time to get the tracking working.

What we have done:

Avi finished implementing and training two adversarial neural networks that change an image to fool the classifier.  One network had an unexpectedly high level of success, reducing the rate of classification to .07 from a test set that had an accuracy rate of .87.  The other network reduced the classification rate to .34.  The successful network outputs the same projection regardless of input, which was not intentional.  The impact of this needs to be explored further.

Dylan worked on setting up the projection onto the face. He received the projector in the mail so was able to start testing. He made a system to find the homography matrix to transform from an intended image to what is actually projected. Then he inverted the homography matrix and using a face image tried to project onto the right location on the face. The face projection seemed to project toward the correct part of the face in the center of the screen, but did not seem to move proportionally with the face. So he will need to look at why the homography transformation is not scaled aggressively enough.

Claudia continued on using Photoshop to make modifications manually, since her dlib implementation did not work as well. Trying it out manually actually helped, as it was easy to see what kinds of modifications would possibly work best for misclassification. Some results of this were that lip and skin color changes successfully misclassified some subjects, and facial hair additions changed individual probabilities to a significant extent.

What we are planning to do:

Avi will set up his code to work with the projector system in real time.  This includes blacking out the eyes in the output projection.  Once the system is set up, he will help to characterize the success of the projection in fooling our classifier.

Dylan will find how to make the homography transformation adjust the location of the projected image into the correct place. This may require utilizing techniques like using an affine transformation instead or just dividing the image into grids that I pre-calibrate so that our error is never more than a single grid. By the end of this week, we should be able to project properly sized images onto a person’s face.

Claudia will work on fixing her dlib implementation, and use it to make the modifications that Photoshop found to be successful.