This week was finishing integration and some timing testing. The wrinkle model was integrated into the mirror code, and times were taken with all of the models together ~2.1 seconds for an image. In order to make this number smaller, I change the architecture of the wrinkle model to match that of the acne detection model, causing its individual inference time to go down to ~0.12 seconds. Beyond that, face detection was integrated with the mirror, although we had to switch from blazeface to DeepFace, due to RPi compatibility issues.
Team Status Report for 11/22
Risks and Mitigations
Our team has fully integrated all the software side of the system. The biggest risk for now is the physical connections not having stable connections for long term use. Since we changed around our physical design pretty late and put in the orders late as well, we have to speed up the process of CADing the new design and connecting the camera, display, lights, and the SBC in a stable manner.
Another risk is to have the user sessions be intuitive (touch screen display being comfortable to use), since we changed our design from using buttons to a touch screen. We designed the UI to have menu items + ordered a larger display, but we plan to conduct user study pretty soon to gather more information on how comfortable it is to use our product.
Our latency risk has been mostly fixed this week by parallelizing our models with threads and shortening the preview. The blaze face also does a great job quickly capturing our face for each session. We will continue testing the latency to ensure that we consistently hit the target latency.
Design Changes
We are using a real mirror and a 7 inch display so that the user can more clearly see their face and the analysis results. These were the physical changes mentioned last week (we have decided to confirm on the changes). Our overall system has the same functionality.
Schedule
Our team is pretty much on track! We plan to focus on testing next week.
Corin’s Status Report for 11/22
Accomplishments
This week, I worked with Siena to fully integrate the models we got from Isaiah with the app and finalized our UI. The wrinkle model got added to our app, and we confirmed that all four conditions were detected during a user session. We also tested the inference latency of our models and succeeded in parallelizing the 4 different models to reduce our latency. We achieved a latency of about 1.5 seconds, which was bottlenecked by our slowest model (the burn model). I created a trend graph with the last 5 sessions for the 4 different conditions. When the user clicks on the dots on the trend graph, it displays the photo result of that session. I also continued working on the new CAD sketch for our physical design. We changed the physical design from last week, and focused on seamlessly integrating a larger display so that the user can more comfortable navigate around the touch screen display.
Schedule
I am pretty much on track. I just need to speed up the CAD design, but I think it would be done by early next week.
Next Week
We hope to get a larger display, build the full system once again, then start collecting data from our tests.
Siena’s Status Report for 11/22
Accomplishments
This week, I worked on integrating the face detection model. After Isaiah has laid out the baseline, I fully integrated it fully so that it is compatible with the rest of the code. Our models yield more accurate results. I also worked on integrated the wrinkle model and adding our recommendations based on the top two most severe analysis.
Finally, we worked on threading. Our 4 ML models now don’t run sequentially, but take advantage of the 4 cores on our RPi. The following is the average result:
oil: 0.3083 s
acne: 0.4713 s
burn: 1.5153 s
wrinkle: 0.3870 s
—————————
total: 1.5167 s
It seems like the bottleneck is our burn model.
Schedule
Right now, we’re behind in terms of the physical build because we changed our design. We plan on finishing this early next week so we can start our user study the week after Thanksgiving.
Next Week
I hope we can put together the physical model before we leave for break. 
Corin’s Status Report for 11/15
Accomplishments
This week, I worked with Siena to modify our app. We added features to view the history of the user sessions (we added a view trend page that shows the past 5 analyses). We also added a mock recommendation system (based on the classification results, we display mock recommendations below each session results). We haven’t finalized the recommendation system yet because we’re still working on categorizing the different skin types based on the classification & confidence. We also added lighting to our product – this did make the input image better. Through testing, we also realized the distance to the camera also played significant role (the results were best when we were pretty close to the camera, less than 1ft).
I also planned out the new physical design we would implement if we were to change directions to a real mirror and a display underneath. A new CAD sketch is needed since the design would become slightly larger and a different frame is needed to accommodate the real mirror dimensions.
Schedule
I am on track!
Next Week
We hope to finalize our physical design and have the full physical product built. Siena and I will work on turning our mock recommendation system into a real one, and hopefully we will integrate all 4 models for skin analysis.
Team Status Report for 11/15
Risks and Mitigations
This week, after the interim demo, we got some feedback on improving the physical design. It was harder to see through the mirror film for our screen display than we imagined. One suggestion made to our group was to change our design so that we have an actual mirror and the display right beneath the actual mirror instead of a see through mirror film. This would improve the quality of the reflection as well as the screen display. Although we are still contemplating on what design to go with, we have placed and received the parts to change our design as suggested.
We also got feedback on our latency for our inference models. We didn’t have the time measured for the inference models, and we got questions on whether our product will meet the timing we aimed for. After the interim demo, Isaiah measured the time for each inference models, which averaged 2.01 seconds. We are planning to figure out a way to run the models in parallel to achieve the timing for our MVP.
Besides the two concerns mentioned above, we are still working on improving the accuracy of our models. Isaiah is still working with the models to improve the accuracy, while Siena and Corin are working with the lighting that is optimal for the input image.
Design Changes
As mentioned in the first part of the section above, we are contemplating on getting rid of the mirror film and replacing it with an actual mirror. The display will be beneath the mirror instead of behind the mirror film, if the design changes. None of the software side will change, and our product will still deliver the same functionality (just a slight modification in the physical side of the product).
Schedule
We are pretty much on track!
Isaiah’s Status Report for 11/15
What did you personally accomplish this week on the project? Give files or
photos that demonstrate your progress. Prove to the reader that you put sufficient
effort into the project over the course of the week (12+ hours)
This week was spent on both finishing integration, and trying to make the current models work more effectively. 3/4 of the models are integrated and benchmarked in time, taking 2.018 seconds on average to analyze one image with the models. Also, work has been done to improve the model’s capabilities to detect more reliably. In particular, a patching system has been implemented to analyze multiple segments of an image.
What’s left is the just adapt the wrinkle model, integrate the face detection, and to finally perform testing and validation
Siena’s Status Report for 11/15
Accomplishments
This week, Corin and I worked on further developing our system app. First, we clearly defined what we want to output to the users for each of the four categories we have. This analysis output is now stored, and up to 5 of the most recent results are showed on our new “View History” page.
Corin and I also added LEDs to our interim setup. After testing it, we found that it gave more accurate results like expected.
Last but not least, I was involved in discussions about altering our physical setup after the feedback our team got after the demo.
Schedule
Mostly on track, a little bit behind because we may need to do another iterations of our physical design.
Next Week
I hope our team can finish our iteration after feedback and make a refined physical model. I also want to work on actually adding the recommendation system.
Corin’s Status Report for 11/8
Accomplishments
This week, our team started integrating all of our subsystems together. My job mainly was to assist Siena and Isaiah integrate the code they had for the camera + ml model to the mock app that I have previously built. I also quickly did a little bit of CAD work to laser cut the mirror frame. The wood we got was a bit smaller than we expected it to be, so I made a smaller version of the CAD sketches to make a prototype for our interim demo. I also added extra features to the app that we had before – Isaiah added code to outline the conditions on the image, and I added the feature on the app so that the user can view their picture with analysis on it.
Schedule
I am on schedule now. The next steps for me would be to modify the CAD so that we have a product that is well put together. I would also like to improve on the user experience (refining the app and the buttons).
Deliverables
Currently, we have a prototype with acne, sunburn, and oiliness analysis. Our next step is to add wrinkles as well and to improve the overall quality of our product with a better CAD design and lighting.
Team Status Report for 11/8
Risks and Mitigations
This week, we worked on integration and put together a mock mirror for our interim demo. Through this process and by testing the integrated model, we noticed a few challenges.
- Reflection – The quality of our “mirror” functionality is low. We plan to address this by adding a black background directly behind the two way mirror (in areas excluding our LCD, of course).
- Lighting – Lighting heavily impacts our analysis results. We have yet to install the LED lights discussion in our design report, so adding that will help us get more accurate results.
- Rounded cornet design – The plywoods we currently have don’t bend well, making it difficult for us to fill the edges of our casing. We plan to look into a different material for this. However, it would also be fine to have 90 degree corners instead of the rounded ones as it will just affect the appearance of the mirror, not the functionality.
Design Changes
After testing the mirror, we decided to add a preview feature before a picture is taken so the user gets more feedback from the system. It reduces confusion on the user’s end as they don’t have to guess about whether they are in frame or not. However, this won’t affect other aspects of our design.
Schedule
No change to schedule! We are on track 🙂