This week I was able to make a large break though with the Jetson and Camera module and setting up the torso detection with Open Pose. Initially, Jeremy and I had set up a Plan B which utilized trt-pose and a different camera (IMX 219) but it was not going smoothly so we had ordered a different camera that would hopefully work with the software and was going to test it. However, working at night on Monday, I was able to get Open Pose working with the Jetson and the original Arducam we had a found a way to extract the data from the output.
Open Pose is run on the individual screenshots we take of the person and I was able to return a output as a JSON file that where we would be able to identify the key body point in terms of coordinates. Open Pose is able to identify a total of 25 key points across the body and we just need to use 1-8 to detect the upper torso. What I plan on doing in the upcoming week is to use these key points to automatically crop the image we are using and connect it with the Color Thief algorithm to detect the main piece of clothing the user is wearing.
OpenPose body key pointsRunning OpenPose on my bodyOutput JSON file from OpenPose
Last week I worked on categoritizing outfits and coming up with relevant tags. In order to do so, I did research through multiple popular clothing brands such as Zara and H&M that many students use and that sell various ranges of clothes. With the research, I came up with a few tags that can categorize clothings and tags that are relevant for outfit recommendations. The following Figure 1 demonstrates an abstract idea for tags and categorizations. Beside, I did more research on outfit recommendation algorithm, and I decided to hardcode weather based and also color based recommendation based on the color and weather coordination chart below (Figure 2, 3).
I think we are a week behind the very original schedule; however, because the rest are integration of modular tasks we have already accomplished and because we have done ramping-up, if we spend a few intensive days in the project, we will soon be on track.
This week, I will be working on wrapping up the recommendation systems and test it out to see if the result align with our expectations.
This week I worked on finishing up the important features of our app. As of now, I am able to open the app through Expo Go, show the splash screen with our logo, and choose and upload photos which the server responds to. The server is on localhost:3000 and the user is also able to cancel choosing/uploading photos which the server will also respond with a “cancelled = true” k-v pair in the json object that is returned. I also helped demonstrate all this during the interim demo that happened on Wednesday.
For the coming week, as long as I stay on schedule, then I will have a working version of the app up and running with the ability to add tags and labels when uploading the photo which will also be sent to the server in a json object. I also want to be able to work with Yun to use her database that she has been setting up instead of localhost:3000 just responding with an OK. After that, I plan to integrate everything into the smart mirror and possibly add functionality to turning on/off the LED lights once we order them and get delivered.
Splash screenChoosing a photoCrop once selectedImage previewServer response body
This week was all about getting the torso recognition software to work. I decided to go with trt-pose due to its compatibility with Nvidia Jetsons as well as its simplicity compared to OpenPose. We ordered a new camera that was compatible with this as well to work with the software. However, the camera ended up not being compatible with trt-pose. The SAINSMART imx219 camera does not work with trt-pose and we have another camera ordered to be compatible with the software. In the beginning of the week, Jeremy and I fully assembled the mirror and frame as well as testing it so the UI is visible through the mirror. This upcoming week, when the camera arrives, we will be testing it and making sure that trt-pose works no matter what. Also, as a backup plan we will be reaching out to groups that are using OpenPose to figure out what is wrong with our system and get the torso recognition up and running for the interim demo. This obstacle has set us back a week in terms of schedule but I am confident that we will be able to get it up in running before we go on our break for carnival.
This week, I touched a bit more upon the weather api as the one that I worked on last week was based on http request. It may cause unnecessary delay or complication in the future, so I worked on running everything completely locally, and was able to complete the task. On top of that I did more research on some basic recommendation algorithm, and also acquired some dataset regarding which outfits are appropriate for which weather.
I think we are bit behind of schedule, especially due to the unexpected delay in the integration of camera, jetson, and openpose. Two people are working on resolving that issue which delays the rest of the project. However, I am very optimistic about the rest of modular tasks as well, because I am almost done with the recommendation which means I can hop on the mobile app development and general integration as well.
Next week, I will work on interim demo presentation, and will work on integrating all modular tasks that my team has worked on so that we can have a demo. If I have some time left, then will work further on the basic recommendation system I have and improve it to be more sophisticated.
This week Wonho and I attached the mirror to our mirror frame. We ended up simply taping the four corners of the mirror onto the four corners of our mirror frame because the mirror bends a lot as the mirror is made of acrylic. In a well-lit environment, it is possible to see through the front of the mirror which is not what we want, so we have decided (as a temporary solution) to use the cardboard box that the mirror came in to cover the back of our smart mirror and block any light from coming in through the back. I tried putting my phone behind the mirror and setting it at a full brightness level to test if the smart mirror would still be able to show the monitor display and it is visible as expected.
We are running into some trouble with the Xavier not being able to recognize that we have a camera input. A week ago, we had no issues with this but they appeared out of nowhere. We tried switching out the CSI cables, reading through troubleshooting threads online and it still would not work. This is the only obstacle that is preventing us from testing out trtpose and I am planning to go in on either Sunday/Monday to get this working again.
I would say that we are slightly behind schedule, but as long as we get trtpose working before the interim demo, then we should be in a very good spot to finish up during the last two weeks of this semester.
This week we decided to move on to our plan B of using trt-pose after struggling for a week to get OpenPose working on our Jetson Xavier. We spent endless hours on it trying to debug and reinstall drivers but it still did not seem to work. I’ve wiped our SD card and have installed trt-pose on the Xavier. I just need to go in the lab and test it with the camera to see if it works. Other than working on this crucial part of our project, we decided to get the hardware components (the frame of the mirror) out of the way by setting an internal goal to finish the mirror this week
As a team, we all went to Home Depot to purchase the wood necessary for the project and were able to finish building the frame for the mirror. In terms of the hardware, we seem to be on track but lacking slightly in the software progress. This will be the week where I will really start to make progress with our CV code as trt-pose should be up and running soon.
Since I worked on modular tasks related to color detection last week, I moved on and started to work on weather API. Using weatherapi.com, I was able to successfully run weather API locally using node js which is the same programming language used for color detection. On top of that, I also visited TechSpark, got some advice from Ryan, who is in charge of woodshop, and also arranged our wood work schedules with him. As a team, we all worked together at wood shop, building the frame holding mirror and monitor.
I think we are on schedule as the wood frame is done, and the rest of modular tasks are almost done as well. Next week, I am hoping to connect weather API and color detection API together to build a small system that recommends an outfit based on those information.
The most significant risks that pose a threat to the overall completion of our project is the issues that we have been facing with OpenPose. There are a bunch of Github threads that have been made by others who have had problems with setting up OpenPose on their NVIDIA Jetsons yet the documentation is either outdated or incorrect depending on each person who had the issue. Our team faced the same challenges and therefore created a contingency plan where we will be using trtpose now. We plan to wipe the SD card that was in our NVIDIA Xavier and start from scratch by installing trtpose instead of OpenPose.
In terms of the mirror frame that we built, we made a few changes to our initial blueprint of what the frame would look like. Our most recent diagram shows the exact dimensions of each side of our frame and this is what we referred to when constructing the frame in the woodshop downstairs.
Diagram of mirror frame
None of the scrap wood in the woodshop matched the dimensions of our mirror frame so we had to go to Home Depot to buy wood. This incurred an extra cost of around $50, but it is still well under our budget limit. Another extra component to our mirror design that we thought of after constructing the frame was how we would hold the acrylic two-way mirror in place. The mirror itself is quite bendy and malleable for some reason, so we decided that we would attach triangular pieces to each corner of the frame which the mirror could stick to without falling forwards. A picture is shown below of our finalized frame.
Completed mirror frame
During the construction of the mirror, we got assistance from an instructor in Techspark who was generous enough to give us a few tips about woodworking as well. One obstacle was the fact that Home Depot claimed they sold this wood plank with dimensions of 2in x 10in x 8ft, but the height of the wood was actually shorter than 2 in which resulted in an unforeseen consequence of having to cut the wood again to match the design we wanted to in our original blueprint.
A generous soul helping out engineers who don’t know anything about woodworking.
We moved the frame back to our bench, so now that the frame is done, all we need to do now is get the software working and running. We are on schedule in terms of what we have built so far, but we are behind schedule with testing out the software because it hasn’t been set up properly yet. That will be our group’s main focus for the coming week.
This week I worked on figuring out what was wrong with setting up OpenPose with our Xavier. After hours of endless debugging and reinstalling, Wonho and I decided that it may be easier to switch to our plan B of using trt-pose. I’ve been working on the app a little more, but the past few days, I have been working with my team to get to the woodshop and create the frame for our mirror. We went to Home Depot to get two 2 x 10 x 8ft wood planks which we cut at the woodshop down in techspark with the assistance of an instructor there. As a result, I don’t have any photos to post in my individual status report, but will post the mirror frame that we built as a team in the team status report. For the next week, I hope to finish the app and get trtpose working so that we have something to show for the interim demo in the first week of April. The app dev process should be a little easier now that I learned more about frontend UI and GUI development in a class I am taking right now which also uses React and Node.js to create a backend and frontend that communicate via APIs. This is essentially the route that we are taking to have the frontend use a button that will send an API request to a backend which stores the user’s wardrobe.
