Posted on

Valeria’s Status Report for 2/19/22

This week I worked on some Figma frames to help represent how our web app is going to look and have a clear idea of what we want to put in the HTML templates. I also created another GitHub repository to put all of our images and created the folders for each sign. Since we needed to get started this week in building our testing database, I started taking pictures of signs for letters N to Z and numbers 5 to 9.  I took 5 pictures for each sign and you can see a sample of what the pictures looked like here.  The main idea was to have different angles for the photo to build the neural network in recognizing a sign at any angle. Apart from that, I looked into the possibility of building a neural network inside an EC2 instance since we found through research that building this network with our computers can potentially make them crash. I did find that it is possible but that we might need a GPU instance, which is something to consider. Lastly, I’ve spent the majority of this week, along with Aishwarya and Hinna, working on the design presentation.

Currently, our progress is mostly on schedule. However, we are currently slightly behind on the web app since we have taken priority to machine learning this week. Because we are currently behind on the web app, I’m planning on working on it this next week and not focusing as much on machine learning so that we do have the HTML templates set up. Luckily the Figma frames do help immensely on what elements to add to the pages so it shouldn’t take me more than a week to finish this up. For next week, I hope to accomplish finishing up the HTML templates and have all the pages set up with minimum actions like moving from the home page to a course module, etc. I also hope to continue building the testing database for my assigned signs (N-Z, 5-9) with at least 10 pictures per sign.  Apart from that, I would also be helping finish up our design paper.

Posted on

Team Status Report for 2/12/22

This week, our team gave the proposal presentation. We met twice before the presentation to practice what we were going to say. As part of our preparation for the proposal presentation, we also created a solution block diagram to visualize the main components needed for our project. Furthermore, we created a visualization of the different modes for our web application (training vs testing). After our proposal presentation, we met on Friday to discuss how we were going to design our machine learning model. We were researching what were the best types of neural networks to use for both images and videos to label them with a correct prediction. We discussed the limitations of convolutional networks and looked more into recurrent neural networks. We also discussed how we might want to approach feature extraction (modifying the coordinate points from the hands into a more useful set of distance data). Distance data may allow us to have greater prediction accuracy than raw image inputs, which can have interference from background pixels. Currently, our most significant risk is incorrectly choosing the neural network, as well as having our models not be accurate enough for users. Another potential risk is incorrectly processing our images during feature extraction leading to latency and incorrect predictions. Our current risk mitigation is that we are researching the best neural network. But we have decided that worst-case scenario we would choose convolutional neural networks, which would allow us to simply feed the images themselves as inputs with the consequences, however, of lower accuracy and more latency. Lastly, a potential worry is that we need to start training soon but our design is still in progress, so we have firm time constraints to keep in mind.

Posted on

Hinna’s Status Report for 2/12/22

Over this past week, my main focus was working on and practicing the project proposal presentation as I was our group’s speaker. In making the proposal presentation, I worked with the other members of my group to refine our use-case requirements and testing metrics, create a solution approach diagram, and create basic web app screens to explain the different learning and testing modes we plan to implement in our learning platform.

I also did research on ASL signs to help determine which ones to include in our web app as well as factors such as similarity between chosen signs, signs that deal with left/right hand dominance, signs that involve motion, signs that involve both motion and facial features, etc.

While our project is currently on schedule, there is definitely some time pressure associated with training our machine learning model, which we have yet to begin, despite not yet finalizing an implementation design for our project. Over the next week, I plan to do more research into types of neural networks that would best fit our project (as feedback from our initial proposal has us rethinking a CNN), and figure out tradeoffs between how we process input data (raw images, videos, data points from computer vision models, etc). With this research, I hope to work with my team to finalize a design and start making testing/training data for our ML model.

Posted on

Valeria’s Status Report for 2/12/22

This week I made the base for our web application. I set up a Github repository for the team to access our application where all the Django files are located, as well as a separate folder for our HTML templates.  I did research on what is the best camera to use for our computer vision and which one would help with our machine learning model. From what I found, the best one that is within our budget is the Logitech C920 camera since it has 30 fps. The 30 fps is going to help us when we are creating our neural network for our moving signs in our platform. Apart from that, I have also been researching neural networks, like the rest of our team, and trying to decide which one would be best for our project. From what I am currently finding, it seems that using two different neural networks, one for moving and one for static signs, can help us in the long run.

Our project is currently on schedule. For next week, I hope to finish my research on neural networks and finalize our design for the machine learning part of our project. Furthermore, I hope to get the HTML templates set, with a very basic layout of what the app is going to look like, and also start listing the functionality that we might need from AJAX. Apart from that, I would also be helping finish up our design presentation and paper so I do want to finish the design presentation slides by next week.

Posted on

Aishwarya’s status report for 2/12/22

I put together the initial starter code for hand detection and created a Git repo for it so that our team could experiment with what kind of data could be retrieved, its format, and limits on factors such as distance from the camera, hand movement speed, etc. This helped us verify that mediapipe is an appropriate tool for our project. I also collected research for our design documents related to different types of neural networks (convolutional, recurrent, etc.), and how we could go about formatting the input data (through feature extraction) to our NN. This helped drive our design meetings following completion of the project proposal presentations on Wednesday. In addition, I researched the various tools AWS offers to help streamline the development process, and our team is considering using Amazon Amplify and Amazon Sagemaker.

Next week, I hope that we can split up the different signs among the three of us, so that we can create ASL video data. I also hope to finalize the type and structure of our neural network(s) after completing our research on what would be the best approach with the greatest potential of maximizing prediction accuracy and minimizing model execution time. This way we can make more progress on designing an implementation for feature extraction and input data formatting in order to be compatible with the requirements of the neural network(s).