Farreltin Fan

1. Work on Design Proposal.

2. Implemented the basic user interface using Tkinter. Then I realized the refresh rate of Tkinter is too long to keep our clock update each second. So Xinhe and I started to implement using Pygame.

3. Write a user interface to display temperature using Yahoo Weather API.

Jing Mao

1. Work on Design Proposal.

2. Started to implement the minimum average correlation energy filter for algorithm tests.

3. Got data from Facedatabase for training purpose.

Xinhe Zhang

1. Work on Design Proposal.

2. Started to Implement User interface using pygame. The new interface can refresh every second and can display the clock.

Farreltin Fan

1. I continued to implement the user interface: I wrote a basic pygame structure for interface. The refresh rate is 1/20 s-1. Then I reimplemented the weather interface.

2. After we put the Kinect on the top of the screen, we realized that Kinect won’t be able to film gesture on the edge of the screen, so decide to restrict our gestures in the center area of the screen. And we decide to put four applications by the four sides of the screen to make process of recognizing which application people are interacting with easier.

Jing Mao

1. I worked on MACE filter implementation on Matlab. I used 10 face images of the same person but with different angles and expressions for training. Then I input an image of the same person and another image of a different person. The correlation output showed a peak if the person was the same and didn’t have the peak if it was not the same.

2. The result could be visualized but I need a formal rule mathematically to determine whether the person is authorized or not. I am currently developing this rule for the system to achieve a high verification rate.

Xinhe Zhang

1. Collected gesture footages. Downloaded files from the VIVA challenge and manually classified into desired gesture groups.

2. Wrote video preprocessing pipeline. The videos are sampled by frames and resized to numpy array as the inputs to the classification network.

3. Updated website template from Start Bootstrap.

4. Helped Farreltin with interface implementation and kinect installation.

Farreltin Fan

1. I keep working on UI of the mirror. Without input from Kinect, I use key event to change between different functions

2. I finished implementing the weather UI and start to implement calendar UI

Jing Mao

1. Installed OpenCV and imported FaceRecognizer module for implementation.

2. Learned how to use the FaceRecognizer API and the Eigenfaces algorithm.

3. Currently, I am implementing Eigenfaces algorithm and use it for face recognition. I am also using the ORL database for training and testing.

Xinhe Zhang

1. Connect the pipeline with Kinect. Specifically, the kinect should be sampling frames in real-time and send back images sets for the classifier to generate tags.

2. Design Pygame event to send and receive signals indicating the gesture recognition classification results. The Kinect will be sampling frames constantly, and the classifier will be checking for gestures peiodically and update the UI part with corresponding signals should new gestures occur.

Farreltin Fan

1.This week I finished implementing every functions. The most difficult I encountered is when implementing using API, many modules could not be installed correctly due to differences between python version and Mac & Windows. Also since large amount of information was returned by API, it’s a challenge to select the right information and show them in the right place. At the first, I updated weather, calendar and gmail every time the functions are called, but the respond time is too long. Considering we need to combine each function with gesture recognition, shortening the respond time is very important. So we decide to update every five minutes( too frequent updates will make the whole program run slower)

2. Next step I will start to set up Kinect and combine the function with Kinect.

Jing Mao

1. This week I used OpenCV FaceRecognizer to implement the face recognition algorithms. I mainly studied how to use the functions in the module and developed the program in python.

2. I developed the program in mainly three stages: preparing training data, training facial recognizer and testing recognizer on test images.

3. I first collected images from the internet for training (details on how I chose the images are included the midway report for face recognition I sent previously by email). I also collected images separate from the training dataset for testing.

4. When training face recognizers, I found that for some pre-written recognizers, I need to feed in images of the same size (for example, for eigenface recognizer). I tried to resize the images in different ways but the result was not ideal. After comparing different algorithms, I finally used Local Binary Patterns Histograms (LBPH) Face Recognizer for implementation and achieved the accuracy of 80%. (a more detailed overview could be found in the midway report for face recognition)

Xinhe Zhang

1. This week I polished the video preprocessing pipeline to sample the frames smartly to capture the image with ideal features.

2. I wrote the baseline classification network and the current accuracy is around 22%.

3. I wrote my own classification network and improved the accuracy to 65%.

4. The accuracy is defined as the classification will give the correct tag of gesture from 5 tags 65 times per 100 trials. The current challenge is to connect the network to work with Kinect and fine-tune the model.

Farreltin Fan

This week I am trying to connect the Kinect to the computer stick. However, this process is kind of difficult. The computer can detect the Kinect but cannot run the Kinect Driver for some reason. I downloaded numerous Office Kinect tool but in Tuesday night, our system somehow was wiped out. So I downloaded our previous work and everything else. So this week I am a little short in progress. I will try to connect Kinect by Monday, if that doesn't work, then we probably have to use our own computer for presentation.

Jing Mao

1. This week I was mainly doing research on how to do face recognition in real time with web camera and Kinect. I have read several methods about integrating webcam or Kinect to face recognition and evaluated their pros and cons for our project.

2. I have read through some tutorials on how to integrate webcam into the OpenCV FaceRecognizer. It took me some time to learn the tutorial and I have implemented the basic code for the program. However, I will collect photos of myself and my teammates so that we could try recognizing our faces on the webcam.

3. During the research, I have also found that Kinect has its own integrated object detection. I will also implement that next week to evaluate whether we could detect faces by using Kinect more effectively.

Xinhe Zhang

1. This week I kept on working on classification network with pre-defined static dataset and increased the accuracy from 65% to 70%. Originally, the classification network starts with the inceptionV3, followed by max-pooling and dropout. Now I built the basic structure of the dueling network, and started the training process.

2. I am wokring on integrating kinect with the network. Specifically, I'm checking the function calls on Kinect to pass on real-time static images to my classification pipeline.

Farreltin Fan

Since our compute stick cannot connect with the Kinect, so Xinhe and I decided to wipe out the whole system and reinstall everything to see if the Kinect can be reconnected. After we successfully connect with Kinect, we start to integrate the features with the original UI. Xinhe and I installed open CV module to load the image information from the Kinect.

Jing Mao

1. This week I am mainly working on real-time face recognition.

2. I first made the face detection work with the camera on laptop.

3. Then I integrated the openCV face recognizer with the working face detection so that it could recognize faces.

4. I tried the model on my laptop and adjusted the confidence level so that the model could successfully recognize the authorized person.

5. However, the recognition time is a little longer and I will work on it next week to let it achieve better performance.

Xinhe Zhang

1. This week I kept working on the gesture recognition classification network. However, the difficulty is that it seems the current network has reached its maximum accuracy, so I researched on other effective architectures hoping to improve the results. I'm working on implementing the dueling network to improve the recognition result.

2. Over the carnival, I'm working with Farreltin for the real-time integration

Farreltin Fan

This week I was working on on integrating function with control. I've tried two version of facial recognition. The biggest problem of our project is recognition time is too long. We are trying different method to see which works the fastest.

Jing Mao

1. This week I finished implementing OpenCV face recognizer and trained the model by using our own face images.

2. We tested the trained model on our own web cam and let different people try to unlock the system. It turns out that the model could successfully detect whether the person is authorized or not.

3. However, as we use Kinect, the face recognition processing is really slow. And it also takes several seconds for the authorized person to unlock the system. So I am currently working on using another face recognition algorithm to achiever faster face recognition processing.

Xinhe Zhang

1. This week I worked on the integration of the real-time gesture recognition and the testing afterwards.

2. This week there's water damage in the lab room. Worth reporting.

3. This week though there are real-time feedbacks but they are not good as expected. I am currently working on fine-tuning the model.

Farreltin Fan

This week is basically test and debug. Since there is a recognition delay after integrating facial recognition with our main program. I worked with Jing to change the loop number in order to minimize the delay. I also worked with Xinhe on Gesture Recognition, since the kinect recognitize gesture in real time so we decide to add a delay after start position. I was trying to make the frame but the time is limited so I would just put the see through mirror in front of the monitor.

Jing Mao

1. This week I learned and implemented the deep learning face recognition algorithm for our facial authorization system.

2. I tested the algorithm on static test images and it works pretty well.

3. Then I integrated the algorithm with Kinect and test on real-time facial recognition but the processing time is a little slow.

4. We adjusted the code to achieve a faster recognition time of less than 3 seconds and the system is never unlocked for unauthorized person.

Xinhe Zhang

1. This week I make the gesture thing work! Hurray!!!

2. I reflected on what I have done throughout the semester.

3. I prepared and presented for the final presenation.

4. I thought briefly about how to make money out of this project. JK.