This week, since I was ahead of my personal schedule and with deadlines for the Design Review Report coming up, most of my time was focused on contributing to writing parts of the Design Review Report. I learned a lot from all of the different groups’ presentations and from the instructor feedback and have been working on incorporating suggestions and clarifying points about our project in the Design Review Report. In the next week, I will finish writing my portion of the Design Review Report and then start working with Andrew or Brian on getting their components of the project up and running before continuing with my own OS interface portion.
Andrew’s Status Report 10/3
I spent most of this week researching more into the mediapipe library. I tested more with my web cam, and have a better understanding on how the actual hand landmark is parsed and implemented within the cv pipeline. I explored more on how you could use the hand landmark information as well, detecting distance between fingers, distance between hands if multiple are detected on screen, etc. While I may not use this as a feature enhancement dataset for our gesture detection algorithm as we’re going to make that more traditionally with just the image datasets we found online. We also worked as a team on the design review sides as our presentation is coming up this week. We’re polishing up our presentation flow and the main parts we wanted to address from the project proposal presentation that we got feedback on. I am on schedule with my work.
Team Status Report for 10/2
This week, the team as a whole mostly worked on reflecting on the meeting where we reviewed our Proposal and developing our Design Review slides. Individually, we continued to do our own research into picking specific tools we would use to work on our assigned big sections of the project. Most of our work is independent and mostly done to test out different software or models for when we start developing the actual product. A new risk that was brought up was the amount of time it might take to train our model for gesture recognition as well as the amount of overhead needed to get this connected to our cursor API. This risk may impact our requirement for latency in the long term, but as we have not started connecting components we cannot make preliminary measurements yet. The system design is still the same but we are refining it in our Design Review. The schedule is also fine as it is, but we foresee that after the Design Review we may need to make some changes as we actually start getting into the real production work.
Alan Song’s Status Report for 10/2
This week, I spent most of time playing around with different libraries and APIs for controlling the mouse cursor, as well as working on our Design Review slides. I tested out pywin32, pyautogui, and mouse modules for Python. I found that while pywin32 and pyautogui also provided the necessary functionality for moving and clicking the mouse, the mouse module for Python was the easiest to work with and produced code that was easy to understand with its great attribute naming. Once I settled on using the mouse library, I made some test programs for simple actions such as moving the mouse to absolute and relative positions based on inputted number data (which will eventually be position data received from the hand detection algorithm). The clicking and scrolling functionality was much simpler to get work with and will require input from the gesture detection algorithm to be transformed into simple numerical inputs to trigger the different mouse actions. Additionally, I worked with the team on the Design Review slides, mostly focusing on fleshing out my own section about the mouse module but also participating in discussions with the team about the design overall. I am still on schedule with my own work but after the meeting with the professor and TA this week I think we can modify our schedule a bit to extend harder tasks and shrink the easier tasks instead of just assigning tasks to a 1 week length. In the next week, I will likely be assisting Brian or Andrew with getting one of their components started with development since I am a bit ahead of my own schedule.
Brian Lane’s Status Report for 10/2
I spent this week looking into potential model architectures and machine learning paradigms for use in our project. In this research I found that one of the best pre-built models included in pytorch is Microsoft’s ResNet, a deep residual convolutional neural network. I’ve decided to train this model as an intermediary to be used in the development of our project while our custom model is trained, tested, and tweaked.
I looked into transfer learning as suggested by Professor Tamal and have found the concept interesting enough to warrant further investigation as to how it would aid in the training of our model.
Further, time I spent some time creating slides for the design presentation and practicing my presentation.