Author: cjye

Claire’s Status Report for 03/07

This week has been very unfortunate with all the midterms and projects I had due, so I wasn’t able to work on Capstone meaningfully. Aside from things that I got done last week coming to fruition this past week (e.g. requesting full access to Google Cloud platform within andrew.cmu.edu), I didn’t really start anything new. I will be traveling during spring break, but here are some deliverable tasks that I can achieve remotely:

  • Getting Google Assistant SDK sample to run on my ASUS computer
  • Altering sample code to take in text queries and pipe text outputs (and if not, at least determine how to make them into text)
  • Explore the need for other smart home SDK’s (Alexa?) if Google Assistant SDK is too difficult
  • Re-map some gestures to be more recognizable (working with Sung who will run it through OpenPose)

 

Team Status Report for 03/07

This week, we started going deeper into the machine learning aspect of things. After some experimentation with OpenPose on the Nano, it became abundantly clear that if we want to meet our speed requirements we definitely should not run it locally. It’s good to know this early on – now we know that AWS EC2 is definitely the only way forward if we want to keep our current design of utilizing both OpenPose and OpenCV. 

We also found out that OpenPose doesn’t recognize the back of hands, especially gestures where the fingers are not visible (like a closed fist with the back of the hand facing the camera). We are going to re-map some of our gestures so that we know that each gesture is, at minimum, recognized by OpenPose. This would greatly reduce the risk of a gesture that is never recognized later on, or the need for additional machine learning algorithms incorporated into the existing infrastructure. 

(OpenPose can detect the hand backwards, but cannot do the same with a fist backwards)

We are quickly realizing the limitations of the Nano and seriously reconsidering changing to the Xavier. We are in contact with our advisor about this situation, and he is ready to order an Xavier for us if need be. Within the next two weeks, we can probably make a firmer decision on how to proceed. So far, only the CPU has shown serious limitations (overheating while running basic install commands, running OpenPose, etc.). Once OpenCV is installed and running, we can make a more accurate judgement.

Claire’s Status Report for 2/29

This week, I did the Design Review presentation and worked on the report. I also spent a long time exploring the Google Assistant SDK and gRPC basics.

For the Google Assistant SDK, I got to the point where I was almost able to run the sample code on the Nano. I bumped into a lot of unforeseen permissions issues on the Nano, which took a few hours to resolve.

Now, I am stuck at a point where I need to register the device with Google Assistant, but despite a few hours of probing around I cannot get a good answer on why this is happening. It seems like there is, again, a permissions issue. There are not too many online resources for debugging this because it is a little niche and Google’s tutorial for it is quite incomplete.

I have also contacted the school’s IT desk so I can create a project under my school Gmail account rather than my personal one. I want to be able to create an account under the school’s Gmail to make my project “internal” within the CMU organization and skip some authentication steps later on in the process (i.e. having to provide proof of owning a website for terms and agreements) . The IT desk and I are preparing for additional permissions for my account so I can create actions on my Andrew email (CMU emails are normally denied that privilege).

For the gRPC, I was able to run some code based from the samples. I think there is potential for it to be very useful for communicating to either of the AWS servers we have. For the WebApp, it can give it information from the results from the command and to display it on screen.

For the deliverables next week, I will be completing the introduction, system specification, and project management sections of the design report. I will also continue working on the Google Assistant SDK samples on the Nano and try to get the issues resolved as soon as possible. I should also have a new project created on my school email instead by next week. Aside from that, I will be installing the WiFi card on to the Nano.

Claire’s Status Report for 2/22

This week, a lot of my work was focused on developing our design review decks, as I am presenting. I think the most important thing I did was fully flesh out our set of static gesture for the MVP.

We derived these gestures from ASL fingerspelling. We had to make sure that the gestures were unique from each other (the original set had some overlapping gestures) and if unique, distinct enough for the camera. Some examples of similar gestures are K and V. While they look distinct from each other in the image, we felt that they would not be too different from one person to another given differences in finger length and hands.

Aside from the decks, I also worked on getting the Nvidia Jetson running. I successfully booted the disk, but because it lacks WiFi abilities, I wasn’t able to get it to run anything too useful. I started a demo, and tried my hands at some basic machine learning setup to prep the Nano for image recognition. I am now learning how to train networks on my personal computer using its GPU.

This was surprisingly difficult on my machine, due to some secure boot and missing dependencies. After a few hours of installing Python libraries, I got to a point where I was not confident in how to fix the error messages.

Aside from that, because our designated camera hasn’t arrived yet I tried to borrow some webcams from the ECE inventory. Both didn’t work. One was connected through GPIO, and another was through the camera connector. Both were not detected by the Nano despite a few hours of tinkering and searching online. This could be troublesome – especially if the camera connector is broken. However, for now, it is most likely a compatibility issue with the Nano, as neither of the webcams were meant for this particular device. For now, we just have to wait for the camera to see.

The progress is still behind, but I feel fairly confident that it will work out. I can start looking into Google Assistant SDK while waiting for the parts to arrive, as those two tasks do not depend on each other.

As I am looking into other tasks to do while waiting for hardware, I think the best use of my time right now would be to start thinking about the Python scripts for automated testing, and to start testing out the Google Assistant SDK and possibly make my first query.

Thus, my deliverable next week are the design review report and a small Google Assistant SDK Python program. I am thinking something that even just takes in a line of input from the command line and outputs in text form would be a good enough proof of concept.

Team Status Update for 02/15

Hello from Team *wave* Google! Right now, the most significant risk is in the performance of the Jetson Nano. We are not sure if the Nano has the amount of computational power we need to complete both the feature extraction and the gesture recognition. If the Nano proves to be insufficient, we will need to quickly pivot to another piece of hardware, likely the Jetson TX1 or Jetson Xavier. We will try to get the Jetson Nano demo-ing as soon as possible in order to test if it has what it takes. We can do hardware testing and network training in parallel, as those two tasks don’t depend on each other. The gesture recognition demo on the Nano we saw online used OpenCV, but we want to also use OpenPose, which we are not sure if we can run on the Nano yet. This could greatly complicate our project, and the only way to mitigate is to start only. 

We had to change how we implement the gesture recognition aspect of this project. We originally thought that we would’ve been able to implement our own skeletal tracking, but upon talking to the professors and reading up papers on skeletal tracking, we realized that implementing skeletal tracking from scratch would be way too hard. Thus we have two alternative approaches. The first approach is to use OpenPose and train a model with pre-processed datasets, and use skeletal tracking provided by openpose to classify gestures. The other approach is to use OpenCV and have our users possibly wear a special glove. This glove would have joints labeled, and we would use OpenCV to imitate skeletal tracking and classify gestures. 

Finally, our Gantt chart and overall schedule has changed as a result of the longer than expected time for parts to arrive. This results in us not being able to completely setup the Jetson Nano and run OpenCV and OpenPose on the Nano in our initial time frame. Instead, we are forced to wait until our parts arrive, and instead run it first on the laptop. Also using OpenPose on the laptop proved more difficult than expected and would carry onto the next week.

 

Claire’s Status Report for 02/15

This week, I worked on getting parts for our Jetson Nano. The most important piece of hardware for meeting our requirements is the camera to go with our board. After some research, I decided to go with a SoC board by e-Con Systems specifically made for the Jetson Nano. I researched and compared factors such as the connector (and thus communication protocol, which affects latency), the size (is it appropriate for an embedded system? does it look intrusive?), the resolution (how much resolution do we need per image for accurate feature extraction?), and finally, the frames per second (how much information do we need to make dynamic gestures?). Unfortunately, the camera won’t be arriving for another two weeks at least, so some parts of the testing may be delayed for now. I hope to continue on with trying out some Jetson demos by borrowing a webcam from the inventory and working with that for now. Luckily, familiarizing myself with the Nano is not a super pressing task – the next task that is dependent on it isn’t due for a few weeks.

Aside from learning camera lingo, I also made a rough draft of our block diagram for the hardware specifications. We have shopped and submitted purchase forms for most of the hardware listed on this image. It took some time for us to find hardware that specifically works with an embedded system and looks sleek. In terms of purchasing parts, we are on time. We started our Gantt chart a little earlier than we should have (we didn’t realize purchase forms didn’t open until this week), but otherwise we are on schedule.

I also worked on collecting some data for each of the hardware and putting them into slides for our design review in a week.

Another factor is the missing microSD card. We just placed the order for it and we can’t set up the Nano without it.

By next week, I hope to have the microSD card and start setting up the Jetson. I will also talk to a TA about possibly borrowing a webcam for now to start setting up some demos on the Nano. I will also be working on the design review slides and presentation next week, and that will be another deliverable.