Posted on

Corin’s Status Report for 9/27

What did you personally accomplish this week on the project?

This week, I mainly worked with the team to complete our overarching design. In the beginning of the week, all of us decided on using a RPI 5 as our main SBC board and realized that we wouldn’t need additional microcontrollers or accelerators for now. We decided that the RPI camera module 3 and the RPI 5 inch touch screen display would be best compatible with our RPI while satisfying the necessary requirements (size/resolution). Both Siena and I worked on planning out how all the different hardware components will be laid out. I also created a CAD model of our physical design, planning out the dimensions of the actual mirror (12inch * 8inch) and placing the camera, touch screen display, and RPI in positions such that all necessary connections can be made.

Is your progress on schedule or behind?

My progress is on schedule. Last week, I mentioned that I want to focus on the LCD display because I thought we might be using an LCD display connected via GPIO + a communication protocol like i2c. Since our group decided on a touch screen display connected via HDMI (micro HDMI), we don’t need to worry about the pinout or a separate communication protocol. However, we will have to think about launching an app on our RPI that will be displayed on our screen.

What deliverables do you hope to complete in the next week?

For next week, I would like to modify the CAD design to a more detailed level after receiving all of our parts. I would also like to connect the camera and LCD display to the RPI and test the input/output of the images to the processor and processor data to LCD display (although the latter I think might take more than a week to set up the app).


Posted on

Siena’s Status Report for 9/27

Accomplishments

This week, our group held extensive discussions to solidify our implementation plans, ranging from component selection to defining high-level project goals. I was primarily responsible for the hardware design, focusing on how our chosen single-board computer would serve as the central hub of the magic mirror. This included planning hardware interfaces and ensuring it could host both our system application and the ML component.

Working closely with Isaiah, who is leading the software side, I finalized a condensed design plan after several iterations. We revisited our design goals multiple times, aligning on the user experience we wanted to deliver. While creating a detailed diagram, I also researched relevant resources and libraries that could support our implementation. After several drafts, we arrived at a finalized version of the design plan:

Schedule

I am currently on track. This week’s goal was to finalize our parts list and research how the board would interface with components such as the control buttons and camera, which has been completed.

Next Week

Next week, we expect to receive some of the parts we ordered. Since we will at least have access to the Raspberry Pi and camera from the capstone inventory, I plan to begin testing simple connections with the camera. This will help me gauge if my interface plans were sufficient in regards to the camera and decide on next steps. 



Posted on

Team Status Report for 9/27

  • What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

This week, we worked mostly on coming up with detailed hardware and software designs. As we were integrating the two and discussing with our mentors, we improved the design so that it is more fit for the purpose and target audience of our product. For example, we decided to leave out the accelerator and see if we meet our use case requirements first. There is always a possibility that we might not reach that and have us adjust the design to accommodate its usage. 

Additionally, we all lack experience in actually building physical products. We expect difficulties during this process, so we made a CAD for general guidelines and solidified the design and dimensions. 

  • Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We decided on the board for our product: RPi 5 8GB. Although we thought about including an additional accelerator, we realized that our product does not require a heavy ML workload for us to need an accelerator. To keep our design cheap and simple, we decided to attempt our ML processing on just the RPi. 

Our group also decided to include an app (we’re thinking about an on system app for now). This app is displayed on our LCD display to show the current session’s skin analysis as well as data stored from past user sessions. 

Training on the whole image datasets proved difficult, with poor generalization. This causes a shift to training with close up skin-patched based methods. These have seen more success in published papers. This requires a higher resolution camera, however that is very much within specs. The newer models of Raspberry Pi-compatible cameras all have at least 12MP resolution, which is plenty for accurate skin patches. This does have the side effect of requiring the user to be closer to the mirror than previously projected, ~40cm. However this is still within standard desktop mirror distances. Fine-tuning was not in schedule for this week, so we’re still on track in that regards.

  • Provide an updated schedule if changes have occurred.

We’re on track as we finalized our parts list and compiled a detailed implementation plan with block diagrams. We have also planned out our communication protocols for our device interface. 



Posted on

Isaiah Weekes’s Weekly Status Report for 9/27

What did you personally accomplish this week on the project? Give files orphotos that demonstrate your progress. Prove to the reader that you put sufficient effort into the project over the course of the week

Then, I got started training the model for the oily/normal/dry condition. This task proved challenging however. First, I trained it  on the datasets I collected last week, however, the loss was bad, and the model was overfitting with only ~38% validation accuracy. This indicates its only barely better than random. I attempted a couple remedies.

First, I attempted to modify the classification head’s architecture. Namely, reducing the hidden layer sizes, and reducing the number of layers. This only had a marginal effect on the accuracy.

Next, I attempted to test out other models. specifically MobileNetV2, and ResNet, and I got similar results. This indicates that the issue is likely with the data I have. From there, I attempted data augmentation. Each image was color jittered, randomly flipped, and rotated. This would in theory improve regularization. The result was similar however. This indicated that I either needed much more data, or to change the approach.

After looking into more approaches, I found this paper:

Sirawit Saiwaeo, Sujitra Arwatchananukul, Lapatrada Mungmai, Weeraya Preedalikit, Nattapol Aunsri, Human skin type classification using image processing and deep learning approaches, Heliyon, Volume 9, Issue 11, 2023, e21176, ISSN 2405-8440, https://doi.org/10.1016/j.heliyon.2023.e21176 (https://www.sciencedirect.com/science/article/pii/S2405844023083846)

In the paper, the team uses images from close up patches of skin to classify oily/dry/normal. Furthermore, they use an architecture similar to MobileNet, with much, much less data to achieve 92% accuracy. This approach would be doable without changing anything in our physical design, so it was my target. This did result in the camera gaining a new requirement: The resolution of the camera had to be high enough to capture rich details in a 224×224 patch (standard vision model input size). The lowest standard resolution that would fit this (assuming a face takes up  ~60% of the image frame) would be about 12MP.

I then checked over the other datasets to make sure they were compatible with patching/appropriate for it, and the acne and wrinkle datasets looked similar in zoom/image size to the dataset the paper used, so no modifications were needed. This also meant that I could re-introduce the burn dataset from before, as that dataset has burn patches that could be used, alongside the scraped images.

Is your progress on schedule or behind? If you are behind, what actions will be taken to catch up to the project schedule?What deliverables do you hope to complete in the next week?

This week, the schedule was just to pretrain the models, and finalize camera + processor decisions. This change in my approach doesn’t take me off of schedule, but it does mean that this next week will be pivotal in making sure this part of the project stays on course. It’s also a possibility that some individual metrics take different amounts of time/effort to get well-performing models of. Acne and sunburns are very distinct visual features, and the wrinkles dataset is quite large and comprehensive, so it is likely that oily/dry skin is the hardest task.

(Link to training graph)

https://drive.google.com/file/d/1QYQX7UeJfLUXG3gD1P8TnrqGQmz24VPU/view?usp=sharing

Posted on

Team Status Report for 9/20

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The most significant risk as of now is to ensure that we are able to deploy our model on the board we choose. For a raspberry pi + AI kit, using an external accelerator and connecting that to our main processor will be a tedious process, and from previous experience there is a lot of debugging involved. As for the Nvidia Jetson Nano, it will be much more convenient since the GPUs are included in the kit, but none of us have experience with using the board, so it’ll be a challenge to ensure that we can get the board up and running and have all the software stack set up. Another challenge is to ensure that all of our input/output data processing can be done. Once we decide on the camera and the LCD displays, writing the scripts for image processing (to feed into our model) and producing output data (analysis on each of the classes) can be challenging, since we do not have embedded experience with these boards. We are currently looking into 2 different boards with compatible cameras/lcd displays so that we have an alternative plan. However, because the parts are expensive, we want our first choice to work and are doing extensive research in open source CV projects that work with the boards we researched.

Another potential risk is the classification accuracy on the oiliness detection task. Unlike the other tracked factors, oiliness is much less visually clear. This could potentially result in a sub-85% classification error when trained only on the initial datasets. To counteract this, if model performance is below our expectations, we will gather new data points through web-scraping, and potentially new images we collect ourselves. Models to detect oiliness have been implemented prior, so likely gathering more data will drive down the error rates.

The datasets available for skin burns are either locked behind stock image distributors paywalls, or contain many images of much greater severity, and also don’t solely focus on the face. Potentially, this could cause a bias in our sunburn detection model to only classify significant burns. To counteract this, we will augment the existing dataset we were using with images of sunburns scraped from google images. Google image results for sunburns are much better aligned with the goals of our system, and match typical levels of sunburn people face in their day-to-day life.

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

We changed our computer vision model backbone from MobileNetV2 to MobileNetV3-small. MobileNetV3 is a more modern iteration, and has better performance on edge devices, while retaining similar classification accuracy (defined on imagenet top-1 and top-5 classification rate).

Upon further research, our team is now considering removing the two MCU units in our hardware block diagram. The goal of the project can be achieved with just a single computer board directly interfacing with our camera inputs and LCD outputs. This adjustment simplifies the architecture itself, but the complexity of the project remains as we are still interfacing with the same hardware components from the plan.

This design change, however, requires corresponding updates to our communication protocol and power supply to ensure compatibility with the connected devices. In addition, if we choose to work with a Raspberry Pi 5, we will have an external AI accelerator to meet our use-case requirement of displaying results on the LCD within the 7 seconds from the user being positioned correctly.

Provide an updated schedule if changes have occurred.

This week, we originally intended to finalize selecting the different hardware components for our project. However, upon research, we realized that a discussion with our advisors will be needed in order to make a decision and compile a parts list. This influenced our search for libraries as they will depend on which single board computer we select for our project. We will be catching up to our schedule until Wednesday of Week 5.

Posted on

Siena’s Status Report for 9/20

Accomplishments

This week, I researched the hardware components for our project such as the single-board computer (SBC), camera, and LCD. My main responsibility was evaluating the Raspberry Pi as a potential system controller. During this process, I realized that the two microcontrollers (MCUs) in our original design were unnecessary, since I found no trouble in connecting both the camera and LCD directly to the SBC. I also identified an AI accelerator plug-in to better custom the Raspberry Pi with our needs. Meanwhile, one of my teammates researched the NVIDIA Jetson, and we plan to discuss both options with our advisors next week.

In addition, I researched treatment approaches for the four skin conditions we will analyze (acne, oiliness, wrinkles, and sunburn) by reviewing online resources and recommendations.

My work is here Week 4

Schedule

We are slightly behind schedule because our group wanted guidance on selecting the SBC and needed approval for our revised hardware plan (which removes the two MCUs). To ensure we can catch up quickly, my groupmate and I have researched both SBC options under consideration, along with the corresponding camera and LCD for each. With this groundwork completed, we expect to be back on track by early next week.

Next Week

Next week, we aim to finalize our parts list, get the updated diagram approved, and begin working on the communication protocols for the selected components. I plan to compile a list of relevant SDKs and draft an initial plan for how to implement them.



Posted on

Corin’s Status Report for 9/20

What did you personally accomplish this week on the project? 

This week, I wanted to find a board that best suited our project. While Siena looked into raspberry pi boards, I looked into the Nvidia Jetson Orin Nano Super Developer Kit.

Below are the components of the board.

Price ($250)

I researched the basic setup that needs to be done when we get out board.

Hardware board setup: https://developer.nvidia.com/embedded/learn/get-started-jetson-orin-nano-devkit#intro

Software setup : Nvidia JetPack (software stack) – including Tesnsor RT, pytorch, etc

Choose to build project from source:

https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md

After the steps above, pytorch will be installed, and either python or c++ can be used.

Below is a link to start coding our own image recognition program

https://github.com/dusty-nv/jetson-inference/blob/master/docs/imagenet-example-python-2.md

 

For a live camera demo, 

An MIPI CSI Camera can be used (port 9 of the board image above).

Raspberry Pi Camera Module 2 ($12)

AP-IMX290-MIPIYUVx1 – harder to buy

AP-IMX334-MIPIx1 – harder to buy

We can also use USB webcams  (port 6 of our board image above).

Logitec C270($22)

Logitec C920($60) 

 

For LCD, we can use 

7 Inch IPS LCD Touch Screen Display ($46) 

5 Inch IPS LCD Touch Screen Display ($40)

The one concern is that there aren’t many resources out there on LCD displays for Nvidia Jetson orin nano boards. I’m pretty sure the two displays above will work with the cable from DP-HDMI (jetson orin nano does not have an HDMI port, only a DP).

Another option is to use TFT LCDs and use the gpio pins on the board. This will be a cheap option for us, since the costs will be below $30, but the display will be small.

More research has to be done on how to communicate the output data to our LCD. Communication methods will vary based on the LCD we decide on.

 

Is your progress on schedule or behind?

My progress is pretty much on schedule, but I want to focus more on the LCD aspect next week. I want to decide which display our project will use (will it be connected via DP or GPIOs), and exactly how the software side should progress to deliver our output data to the display.

 

What deliverables do you hope to complete in the next week?

I hope to have chosen our board, and to have a plan on how the output data will be delivered to our LCD. I don’t think we’ll have our board by next week, but based on our board decision, I plan to look at the datasheets for hardware configuration of the LCD and research how the communication will work from the board to the display.