Category: Andrew’s Status Reports

The week of 12/4 I prepped the final presentation as well as the finishing touches on the system in general for the techspark expo next week. I worked with Alan on the cursor smoothness metrics and we talked about changing the system at a working level by changing which gestures correspond to what. Mainly, we changed click and drag to be a fist, and single click to be an ok sign. This coming week, we’re going to work on planning out the final report and video as well as the techspark demo. I also spent a good amount of time refactoring the code by placing it in classes to make it much neater, since we originally had all the gesture recognition and flow chart logic in the main function of the demo file, so I put that all in a separate class and just called the class functions in the main. I added comments on all the major functions as well, so if we want to reuse this code and work on it in the future, it will be easy to modify. Most of the brunt work for the project is done, and all that’s left is getting our metrics set for the final report including the user testing and questions that we will include in our final report. Overall, I am on track with my work, and we are set to be good for the presentation portions next week.

This week, I looked into smoothing out the cursor motion again and started a basic implementation of the finite state machine for when the system is in cursor movement mode and in precision mode i.e. when a user is trying to be price and hover over a specific region on screen. While the team didn’t meet this week, we will be working closely this coming week for getting the system to behave a bit more smoothly incorporating the feedback from the interim demo report. I’m thinking of changing the mouse movement function a little bit to make it more drastic like scaling up the movement by some constant so the distance moved is greater so the user doesn’t have to strain as much. Then, since we’re doing some form of FSM, we can change the state to detect when the user is trying to be precise or when he wants to move. We’re placing an emphasis on our project on quality of life and user likability, and that’s what’s prompting this decision. As I said earlier, I’ll be following up with the team to test this on the webcam as they have the one we’re using right now. As per the interim demo as well, we have a functional project, we’re just working out the little kinks. As such, I’m on schedule with my work.

This week, I along with the other people in the group spent a lot of time preparing for the interim demo. We met a couple times the past weekend and to prep for it and made decent amount of headway with the integration. We now have a functioning system where we can move the cursor to an onscreen position and close our hand getting it to recognize a clicking gesture, and this recognition doesn’t produce noticeable latency within our system. My next steps are working on making the cursor movement run smoother and that will be initially attempted with some form of rolling average. Since the mouse movement gets a bit jittery when you try to click on specific locations, having some sort of rolling average will smooth out the cursor motion and make it so the cursor becomes a lot more precise at specific tasks that require precision. Another thing I’ll be working on in tandem with Brian is a hand landmark frontalization algorithm performed with some three dimensional rotation matrix performed likely with the hands that will hopefully get our gesture recognition more accurate. Overall, like the previous week, I’m on schedule with my tasks.

This week, I worked more on polishing up the code for the pose estimation. We began integration this week and met several times to discuss the demo as well as piece together the system. I’m working on some sort of smoothing algorithm for the on screen position estimation as right now when we’re updating mouse location with the pose data we’re getting slightly noisy mouse movement when we’re trying to be precise. While we’re still able to click on a unit of smallest pixel frame with relative ease, a big part of our project as we’ve stressed is user experience, and thus, getting smoother cursor movement should be somewhat of a priority. I’m currently thinking about doing an average of pixel motion. Similar to how periodic averaging smooths a signal and acts as a high pass filter, I’m going to test if the same applies here. Since our camera refresh rate is pretty high, it’s safe to assume we have some noise in our hand detection in a stationary position, so we’ll see if this averaging smooths that mild sporacity out. To stress again, this module isn’t a top priority right now as we’re able to perform within our initial specs, but it would be nice to have. Right now, integration of the system takes top priority. I’m on schedule with everything else I have outlined in the gantt chart.

This week, I worked on and started finishing up the code for the hand detector class file. I’m almost done making the classes for the cursor update object that will continuously store the cursor location and will keep it stationary in the event the camera does not detect hand on some individual frames. We also started working on integrating my code with Alan’s and we’re able to get some form of cursor movement with the video camera. Once I finish up the class for the cursor class, we’ll have a more streamlined movement of the cursor on our object.

As a personal update, I did not have too much time to work on the project this week due to catching up on a lot of other work and matters due to my injury. I am almost fully recovered, so I am seeking to get back on track this week. I’ve run into one minor issue with using the webcam (which I met up with Tao over the week to try to fix) as I have some sort of permission issue on my laptop that is not letting me access the webcam, but I am more or less on schedule although lagging behind a bit. The webcam functionality is not too big a deal since it works on other computers and I can always just use my laptop webcam to test.

Last week, we had mid-semester break so the team did not submit a status report. Our primary focus that week was working on the design review. This past week, we each completed the ethics assignment and discussed our work in class with both the guest speaker as well as other teams in 18-500 as well as 18-555. As for our actual project, the webcam we ordered arrived this week, so I had to dedicate a bit to setting it up on my computer. I’m still finishing writing up the code for pose estimation, but that is nearly complete. I have started writing the calibration process for our range of motion algorithm. I should have that finished within the next couple of days. The main difficulty now is going to be porting the individual code I have written onto an interface that connects with the OS Interface library, but this is taken into account in our integration phase. As a personal update, I have been injured the past two days and have communicated this with the professor. As such, I’ll be a bit stalled these upcoming next few days, so I’ll be a bit more communicative on my end with regards to that with the team and the teaching staff.

This week, we came off the design review report presentation and god some good feedback from TAs. For me personally, I am ahead of schedule on MediaPipe handtracking research and am in the process of building visualization tools using Matplotlib so I can more easily work on the pose estimation algorithm. I should be finishing this up by the end of this week as well as finishing up my part in the design report.

I spent most of this week researching more into the mediapipe library. I tested more with my web cam, and have a better understanding on how the actual hand landmark is parsed and implemented within the cv pipeline. I explored more on how you could use the hand landmark information as well, detecting distance between fingers, distance between hands if multiple are detected on screen, etc. While I may not use this as a feature enhancement dataset for our gesture detection algorithm as we’re going to make that more traditionally with just the image datasets we found online. We also worked as a team on the design review sides as our presentation is coming up this week. We’re polishing up our presentation flow and the main parts we wanted to address from the project proposal presentation that we got feedback on. I am on schedule with my work.

For this week, we worked on our proposal presentation and reviewed the feedback we got from the other team members as well as the professors. There is not too much to update on, we each started our assigned tasks as outlined in our proposal schedule. I personally an on track for my tasks. I reviewed the mediapipe image processing library and have a working demo of the hand tracking software. What remains next is to familiarize myself more with the library to a point where I can fully interact with and port the code I have written with to another application. Nothing else significant to report.

This week, we as a team worked on streamlining our design ideas for our project by consulting with Professor Mukherjee and TA Tao Jin. We were able to narrow down our usage environments and had a long discussion on the type of external hardware/libraries that would be most useful for our design direction. Our consensus is to have a mainly CV focused approach, so I myself started looking into CV hand detection implementations and datasets online, and the plan is to get a baseline hand tracking demo with the built in computer monitor by some time next week. We also plan on working on and completing the proposal presentation and are currently on schedule.