Overall Team Project Update
This week, all three of us worked on completing the design document. We incorporated the feedback we receive for our design presentation and adjusted our plan for the project accordingly. Aside from the design documents, we also made some progresses on the project individually.
Individual Team Member Responsibilities
Yue Zhang: Set up Mysql database to store user information. Connected to Facebook Graph API. Wrote script that requests data through the Graph API, and populate the database with the data received.
Ji Jin:Set up the Unity and the Android Visual Studio on computer. Found all Unity APIs which would be used in the future. Tried to transfer 2D locations on imgeas to 3D location in real world.
Zach (Zhengyuan) Pan: Set up OpenFace on Docker. Ran OpenFace demo on classifier training and feature extraction.
Spring Break
Individual Team Member Responsibilities
Yue Zhang: Set up Team Blog. Set up ARCore and HoloKit SDK on Google Pixel. Tried to build sample AR app in unity and exported to smartphone. Worked with Ji Jin to find solution to capture and save camera views. Tried to use ARCore camera, but existing ARCore functions only expose the projection matrix, which we cannot use.
Ji Jin:In this week, I have tried several ways to capture and save the camera view image in PC, such as using unity WebCam API and writing an additional script to access the phone camera, but none of these methods worked because ARCore was using the phone camera while the app was running.Therefore, Yue and I decided to use an additional camera to stream the video to the PC in real time through an app called IVCam. In this way, there would be no latency for image transfer between mobile and PC if the WIFI signal is strong and stable enough.
And, I have written several scripts to locate the objects at desired poison in real-world coordinate.
Zach (Zhengyuan) Pan: Finished facial detection, face alignment and image preprocessing. Collected image data of detecting one person.
Individual Team Member Responsibilities
Yue Zhang: Wrote scripts that communicate image processing result between PC and smartphone. We are using an online file to store the facial recognition result. Wrote python script that update online file with facial recognition results; rote another c# script that read info from the online file and display the results in the AR view.
Ji Jin:Wrote scripts that places the objects on required location in the real world;Wrote another script that uses focus point to allow users to interact with the objects seen in the view
Zach (Zhengyuan) Pan: Debug the current web interface for facial recognition. Reinstall and link the necessary modules due to update of Homebrew.
Individual Team Member Responsibilities
Yue Zhang: Wrote python script to retrieve user data from Facebook Graph API, store to online files. These files will be updated every hour. Wrote C# script to read user data from online files. Explore facial recognition code base to find solution to receive camera view.
Ji Jin:Wrote a C# script to build the user interface for AR, such as button and help messages. Found out another facial recognition model that could be used in Unity. Yue is helping to integrate that model in Unity. Currently, I am writing another C# script to allow users to view all their friends’ profile in our app.
Zach (Zhengyuan) Pan: Find out how to get image/video feed from camera on smartphone to the facial recognition algorithm and web interface, and output the recognition result to Unity object redering part.
Individual Team Member Responsibilities
Yue Zhang: Explored a new face recognition library called emgucv in c# and unity. Wrote script that detect faces and output face locations. Wrote script that get camera view from webcam on smartphone.
Ji Jin:Made a new AR view that displays user's all contacts. Built unity animation of AR contacts page. Explored other C# and python face recognition library.
Zach (Zhengyuan) Pan:
Individual Team Member Responsibilities
Yue Zhang: Build an android app that user can add faces to our database by taking pictures or select photo from android photo gallery. Adapted the face recognition script to take in video stream instead of image. Worked with Ji to adjusting position in AR and real world.
Ji Jin: wrote a c# script that could place an object in the real world to pin the position of face in the view. Yue and I tried to calibrate the position of the pin object when users have the holokit on. I am still working on finalizing the UI for unity.
Zach (Zhengyuan) Pan:
Individual Team Member Responsibilities
Yue Zhang: Try to explore google drive api to store user pictures. Modified face recognition code to improve speed
Ji Jin: Exploded the Openface library; modified all Unity UI; added the hands gesture control to ARCore
Zach (Zhengyuan) Pan: