Week 10
What we have done:
Dylan worked with Avi to get the pipeline of seeing the face identification, after it projected something onto the face. We still need to add the modification to test how our system tricks facial detection after being projected. Our recognition didn’t seem to work as well with the current setup, misclassifying a lot, so we will retrain it with some extra pictures of Dylan at the setup location since that is the only person who has access to the projector and the camera. Dylan also made the projector project in the right area a lot more accurately by dividing the camera picture into regions, and finding where to project by linearly interpolating within the region that we associate camera pixels and projector pixels. There was an error the previous homography matrix based approach that was also in the new approach where we first cropped the image to find where the face is, so part of the inconsistency with the homography matrix may have been due to that bug.
What we are planning to do:
Dylan is planning on adding Avi and Claudia’s projection onto what he is projecting and testing out the accuracy. Dylan is also considering creating a system of projecting onto a pattern and moving projected pattern until the two patterns are correlated to fix some of the offsets. Dylan is also considering using a tracking algorithm to make the calculation for where to project quicker, since it currently takes about 1.5 seconds per frame. However, there will probably not be enough time to get the tracking working.
Week 7
What we did this week:
Dylan completed the image dewarping and mapping the transformed face from the image processing to the projected image. Dylan also began transferring our processing to AWS.
Avi has almost completed the training loop to train the adversarial network. He has gotten to the point where gradients can be passed backwards through the networks to train and reduce loss consistently in a small test case.
Claudia began overlaying shapes onto the chosen facial landmark coordinates that are obtained from dlib. Some of them did not have much effect, as they did not change the classification probabilities significantly.
What we are planning to do:
Dylan is planning to finish transferring our scripts to AWS. Dylan also plans on verifying that the projection works as expected if the projector comes in.
Avi will finish implementing the training loop so that the Adversarial network will be able to show significant learning by the end of the week. He will also implement other adversarial network architectures to train.
Claudia will try out more combinations of modifications and facial landmarks to find the ones that work best across all test individuals.
Week 6
What we have done:
Dylan has changed some of the face detection functions so that they are easier for the other group members to use. Previously things like blacking out the eyes was solely done in a script, but now other members just need to call the function. Dylan has also worked on dewarping the face, after it is warped for the facial recognition. There seems to be a problem with the translation that takes place with the affine function. Though even without the translation, there is a lot of blurriness in the dewarped face. Dylan has also prepared for projecting images for once the projector arrives to him.
Claudia has worked closely with Dylan and helped with many of the tasks listed above.
Avi is ready to start implementing different neural network architectures. He recently got facial classification to work with 94% accuracy for 6 different people. This means that it will be possible to train the adversarial network. Avi has also finished setting up the classifier to be easy to use for Dylan and Claudia.
What we are planning to do:
Dylan is currently blocked on using the projector. It is being shipped to him, but he can’t really test any of the projections until the projector arrives. Dylan plans on finishing the warping and hopefully projecting images accurately onto faces. The projecting onto faces is already partially complete, but need to find how well every part of face is projected onto to see if a more intense homography estimation is necessary.
Avi will implement one of the planned adversarial network structures in the next week, and set up the ability to train and characterize it. This will make it easy to spend the next week trying multiple different architectures using the existing setup. Right now, all testing will be with digitally produced changes to photographs. Due to the separation of our group, Avi will be testing with only these digital changes (not the projector) for the next few weeks.
Claudia is going to start editing photos in Photoshop so that she can observe the effect on the classifier output. She is going to start building an intuition for what changes are effective at reducing classification rates. She will then try to directly program changes that are effective. This is a new plan by our group, which will allow us to have more independent roles which give us two approaches at our main problem (which reduces risk). We will also be able to meaningfully compare the successes of the machine learning approach versus the human learned approach.
Week 1
What we have found out:
- Information about projector we are using:
- Minimal distance with focus for projected image is 1.4 meters away from projector
- At that 1.4 meters, a face is about 200 pixels wide, which is more than the 80 pixels wide we wanted
- When we project a black square just on the eyes, we were comfortable looking into the projector
- Though we will want to increase current size of black square
What we have done:
- We selected and received a projector and 2 cameras from the 18500 inventory. All of them exceed the requirements for a projector and camera that we had, though the projector is much bigger than ideal. We do not have a requirement for projector size so that is fine.
- Avi set up OpenFace to convert images into simple 128 dimensional embeddings. Then he pulled data from a database of celebrity images and ran them through the OpenFace network. The output was used to train a simple classification network that is able to distinguish between 20 different celebrities in the network. The data he pulled had about 100 images of each celebrity, and about 60 were used for training. The remaining 40 were used for validation, and the network was about 40% accurate. He concluded that it is probably a working model that does not have enough data.
- Claudia used dlib to extract facial landmarks, so that projections can be scaled to these landmarks. This can also be used to black out the eye area, making the experience more comfortable.
- Dylan used Claudia’s facial landmark extractions to create a function that finds the pixels that need to be blacked out so that the projector does not project anything on someone’s eyes.
- Dylan used Claudia’s facial landmarks to also create a height map across a person’s face. He did not finish the calculation for how far away each part of the face is from the camera that he and Claudia were working on because we found out that the depth map may not actually be important
- We tested out putting the black-square over each member of the group’s eyes, and everyone was comfortable(no immediate eye strain) with the projection on their face.
What we are planning to do:
- Avi is going to spend the next week researching neural network
architectures to change an image without reducing its dimensionality
and decide on appropriate loss functions for our future adversarial
training. If this goes well, he will also try to implement an image
modifying network. - Claudia will create the program to help collect the images. She will also start collecting images. Claudia will also work on translating the image that we want to project to what we project.
- Dylan will expand the black eye rectangle and make it cover each of the eyes. He will also complete the camera calibration for finding the geometric relationship between the camera and the projector.
Updates on design:
After talking with Marios and Emily, we found out that we may not even need depth information, since we can just correlate where we project with the facial features that dlib finds. We may decide to use the depth information in the future but for now, we will see how well projecting without depth information is and our backup will be using depth information. So for now we will only use 1 camera.