Anushka’s Status Report for 4/30
This week, I mainly focused on different hologram designs. We have three options right now:
- The angles in the pyramid are 53-53-127-127, and the dimensions are for a hologram that fits an iPad.
- The angles in the pyramid are 53-53-127-127, and the dimensions are for a hologram that fits a monitor.
- The angles in the pyramid are 60-60-120-120, and the dimensions are for a hologram that fits an iPad.
So far, we have tried all three, and the last one works the best but only in total darkness. Next week, we’re going to try to test new angles before the demo day, then probably pick the one with the greatest visual effect. I’ve also been working on the final sewing of the device onto the wrist band. We adjusted the height of the sensors since our hand was interfering with the input. I have also been working on the poster for our final demo.
This is the final official status report of the semester. We have all the parts completed sufficiently, but we know there are always areas of improvements. We’re going to try to make minor changes throughout the week before demo day, but otherwise, we have a completed product! I am most concerned about how it will go overall. We have tested our product amongst ourselves, but we are aiming to test it on several participants before demo day and final paper. We are excited to show everyone our progress and what we’ve learned!
Joanne’s Status Report for 4/30/2022
This week was final presentation week so we as a group worked a lot on polishing the slides and doing testing on our prototype. We conducted a lot of testing on latency, performance, accuracy and that was our focus for the latter half of the week. The early parts of the week, Edward and I took time to debug our unity/gesture code to provide better model translations. Before there was a bug of irregular spikes in the model’s movement when given gestures. However we were able to go through our whole code, and figure out what caused the issue. We were not resetting the appropriate values after the completion of one gesture, thus causing issues with data calculation. Our integrated model works fairly well in response to gestures now! After that we have been focusing on testing + hologram portion. I added new features to the web app which include changing models (now there are three models that the user can switch between), and incorporating the previous sensor data visualization we had with our current web app. I will continue this week to finish up the UI for the web app and help with finalizing other portions of the project such as display/hologram creation as well as the final report/ poster.
Team Status Report for 4/30/2022
This week, we finally finished a working demo of the wearable and Webapp! Joanne and Edward spent several hours debugging and bug squashing in order to properly filter out the incoming finger location data from the wearable. Eventually, they converged on a working system. See video here.
The current issues remaining are that WiFi is fairly slow and negatively affect latency. It isn’t too bad, but it makes our system less real-time than what we wanted. If we were to do this again, we would use BLE or perhaps some other wireless protocol.
Anushka worked on refining the hologram design. She design several version of the hologram and cut them out to see which one gave the best effect. So far, the pyramid with angles that are above 60 degrees and on a big screen work the best. She has also started working on the encasing for the hologram. At this point, we’re not really sure how to improve the visualization without changing it altogether, so we might try bigger angles on smaller screens throughout the week to see what gives the best effect.
We are on schedule to have something for the demo. This week we will work on polishing everything up and making sure things are in order to not blow up at demo day.
Edward’s Status Report for April 30
After I submitted last week’s status update, me and Joanne worked on fixing the bugs in the Webapp. We spent several hours cleaning up the code and trying to see what was causing the issues. We have been seeing that the model freezes up or moves in the opposite direction when the wearable sends is finger locations. After debugging, we finally found the issue and fixed it up. This video was made right after we fixed all the bugs!
This week, I prepared for and worked on the final presentation. Anushka was feeling better, so I gave her the wearable for her to properly sew on. So, without any hardware, I spent most of this week working on the final presentation.
I am on schedule and pretty much done with the individual tasks I needed to get done this semester! Next week, I will help on polishing everything for the final demo. We still need to build the hologram pyramid, so I will be helping Anushka on that. Super excited to demo our system in the final demo!
Anushka’s Status Report for 4/23
This week was a tough week as I got COVID and had to participate in everything remotely. Since I was incapacitated, I mainly worked on the final presentation.
The main part of the presentation I’m working on is the tradeoffs. We made a lot of algorithmic tradeoffs, so I had to revisit all the old models: the pre-ML model, the SVM model with old arm data, and with the time series classification models. Other things I wanted to mention is the usage of Jetson and different angles of holograms. I designed new hologram sides with slots and holes so that we don’t have to glue the sides together and the sides are more secure.
Since we have to finish the presentation tomorrow, I am still on track to finish this week’s goals. In the next week, I hope to be able to go back in-person and work on the hologram display and reassemble Tony to the exercise band.
Joanne’s Status Report for 4/23/2022
This week was a busy week since we are wrapping up on our project and we had some covid case situations that delayed our work schedule.
Edward and I acquired black acrylic sheets to encase our hologram pyramid and display over. We tried gluing the cut out plexiglass (for the hologram), however, the glue we used from Tech Spark did not apply as nicely as we hoped (nor did it stick). We then thought of a new design to laser cut the hologram layout so it would have a puzzle like attachment feature at the edges, so it can connect to the other sides of the hologram pyramid without all of the glue and tape. We drafted the new version to cut up. There was some covid related issues in our group, so we are in the midst of working out a schedule so we can go in to Tech Spark and cut the new version.
We are still working on our model gestures. Since our gesture algorithm did not work out, and we are relying on our finger detection algorithm for our gestures, we moved noise filtering and data queueing to the Unity side of things. The model works well in the test data for “good” swipes and zoom gestures. However during the live performance, due to noise, there are a lot of erratic model movement that we are trying to resolve. The swipe gestures look ok live, the pinches have some unknown behavior. That has been my primary goal this week. To filter our noise, I have tried taking the average of points coming in for a specific gesture and comparing that to identify a factor for gesture movement. It seemed to work well in reducing noise in the swipes, but not in the pinches for some reason.
I have been working on finalizing the web app UI and included visual components that would help with user interface (like including a finger visualization of where ur finger is on the trackpad, and battery info). As a group, we talked about collecting data for testing and have been working on our final presentation.
Team Status Report for 4/23/2022
This week, we had much to do. Redesigns, bugs, and COVID have pushed us behind schedule. We are on our last slack week next week.
Edward and Joanne have been working on fine-tuning the finger detection and model rotation. The system does not work well on live “real” data and seems to have erratic and random behaviors at times. Hopefully, by the end of this week, we can have a really good version working. We will spend the majority of next week working on this.
Edward and Joanne have also bought black acrylic for the hologram. They attempted the glue the acrylic together, but the glue was not as sticky as we wanted. So, Anushka drafted a new version of the hologram using slots to fit the walls together like a puzzle. Hopefully, we can get this cut and built next week.
We’ve also been working on the final presentation. It’s a good thing we have some time before demo day, but we’re excited to present our findings and what we’ve done so far in the semester.
In good news, the hardware has not bugged out on us yet. It seems fairly reliable so far. We still need to properly sew the board unto the wristband. This should be done by early next week.
Next week will be an extremely busy week for us. Wish us luck!
Edward’s Status Report for April 23
This week, I worked on the final project presentation as well as further improving the gesture to 3d model action algorithm. Anushka has been sick this week, so Joanne and I have been trying to finish up everything before the demo. I have been working on collecting more training data to retrain our SVM whenever an issue comes up. I think the current model is able to distinguish between one and two fingers pretty well. Some of the tricks I played to get the robustness up are: removing the noise label in the model (since it can get swipes and noise confused, and noise is easy to detect), and training on data where I just place my fingers and not move them or perform gestures.
I started collecting data for the testing section of our presentation. We discussed as a group how to determine our system’s accuracy, gesture precision, and latency.
I also worked with Joanne to cut the acrylic for our hologram, but the glue that techspark had looked really bad and tape would show up when under light, so we redesigned our hologram to use slots instead of adhesives. I am not the best at cutting, so I hope Anushka will get better so she can cut and help construct the hologram.
The gesture algorithm has proven to be much harder than we though, especially with just the two of us this week. Hopefully, next week we can have something demo-able. We are using up our slack weeks.
Anushka’s Status Report for 4/16
This week was a grind week for the team. We tried many different algorithms for determining what gesture is occurring. There are two things I tried this week:
- We queue 5 data points of what fingers are currently being detected. From there, we determine what gesture is occurring. If there are at least 4 occurrences of the number of fingers, then the gesture will be classified as such (0 for noise/rest, 1 for swipes, and 2 for pinches). If the number of fingers goes from 1 to 2, then the gesture is a pinch out, and if the vice versa occurs, then the gesture is pinch in. I set up a few cases that took care of the noise, but ultimately, it still caused a lot of errors. The frequency of data collection is too low; a gesture can range from 2-10 gestures. However, the number of fingers detected before or after the gesture can affect the window of 5 and yield an inaccurate result.
2. I tried using time series classification library called sktime: https://sktime.org/. This library is really cool because I can give the model a set of inputs over time and across different classifications, and the model can predict what class an time series data belongs too. Splitting the new arm data into training and testing, I was able to create a model with 92% accuracy, and this model is able to distinguish between pinch in, pinch out, rest, swipe left, and swipe right. However, this model would need 24 data points in advance, and as discussed before, there aren’t that many data points associated with a gesture, and with this model, we would have to predict the gesture a considerable amount of time before performing the gesture.
As a team, we’ve decided to work on different implementations until Sunday, then make a final decision on what algorithm we should select. It comes down to a tradeoff between accuracy and latency, both important metrics for our device.
I also helped Joanne with building a bigger hologram pyramid this weekend. We decided to hold off on casing until we determine the final height of the pyramid, which we’ve also decided to do a bit more exploring with.
Currently, I am a bit behind on schedule, but whatever tasks that are remaining in development and testing has to be accomplished this week as our final presentation is due next week. Everything essentially has to be done, so I’m in a bit of panic, but I have faith in Tony, Holly, and our team.