Posted on

Team Status Report for 4/2

This week was a better week for the team. We were able to tackle many problems regarding our capstone.

Our biggest concern was the Jetson. After reading online tutorials on using Edimax, we were able to finally connect the Jetson to WiFi. Next week, we’ll be focusing on connecting the sensors to the Jetson and the Jetson to the Unity web application so that we can accomplish our MVP and start formally testing it.

Since we have time now, we went back to our gesture algorithm and sought to improve the accuracy. We explored different machine learning models and decided to see if Support Vector Machines (SVM) would be effective. On the data that we already collected, the model seems to be performing well (95% accuracy on cross-validation). However, we know that the data is not comprehensive of different locations and speeds of the gestures. We collected more data and plan on training our model on that over the next week. It takes a lot of computational power, so we will try training on either a Jetson or a Shark machine. We were also suggested AWS, which we might also look into.

Biggest risk now is latency. We want to make sure that everything is fast with the addition of the Jetson. We have a Raspberry Pi as a backup, but we’re hoping that this works. Therefore, we haven’t made any changes to our implementation. We also went ahead and ordered the band to mount the sensors on. We plan on collecting more data and training on that next week after we find an effective way to train the model.

We should have a working(ish) demo for the interim demo on Monday. We also seem to be on track, but we will need to further improve our detection algorithms and thinking to have a final, usable product.

Posted on

Anushka’s Status Report for 4/2

This week, we worked more on the gesture recognition algorithm. We figured it would be best to go back to the basics and figure out a more concrete finger detection algorithm, then develop a gesture algorithm on top of that.

Currently, we abandoned the finger detection algorithm in favor of tracking either mins or maxes in the signal, then determining the relationship between them and give a metric to Unity to perform a translation. However, this metric is highly inaccurate. Edward suggested using SVMs for finger detection. The difference between pinches and swipes are the number of fingers present, so we can use existing data to train the model so that it can tell us which sensor represents one, two, or no fingers.

I added some new data that is more comprehensive to the existing data set. I also added some noise so that the model would also know what to classify if there is too many distractions.

Afterwards, I trained the data using different subsets of the new data combined with the old data. The reason behind this was because training the new data took a lot of time. It took over 4 hours to train 200 lines of new data and a lot of power.

Next week, I’m going to train the model on the Jetson with all the data. Jetsons are known for high machine learning capabilities, so maybe using it will make our computation go faster. We managed to add wifi to the Jetson this week, so it’ll be easy to download our existing code from Github. I am concerned about training the new model with the new data. With only the old data, we have a 95% accuracy, but hopefully with the new data, we’ll be prepared for more wild circumstances.

I think I’m personally back on schedule. I want to revisit the hologram soon so that we can complete integration next week. I’m excited to hear feedback from interim demo day since we have time for improvement and will likely add or improve parts of our project after Monday.

Posted on

Joanne’s Status Report for 3.26.2022

This week I continued to work on refining the Unity- Web Application portion of the project. Last week, I got up to testing fake data on just the Rotate function via MQTT protocol. In the beginning of the week, I talked with my group members and they decided for now, that the rotation gestures will be limited to rotation about the x-axis (Up). Thus, I modified the code to account for this change. The model rotated well about the x-axis during the tests with my dummy data.

I wanted to see how the Rotate function would react to real life data, so we decided to integrate the live sensor reading with the basic gesture recognition algorithm, to my Unity web application. Right now, I had a python script that I ran from my local terminal that would publish to the public MQTT broker. Then a javascript script on the DJango web application would read those published data values, and send that data over to Unity where it would be parsed for use in order to Rotate the model. I decided to create another parsing function within the javascript script so that I could call the appropriate function (i.e. Rotate or Zoom In/Out) and also send the data in the correct format for the Unity parsing function I wrote.

After testing it with live data, the model rotated well about the xaxis. Since the gesture algorithm is not complete yet, we tested by ignoring all gesture data that was not classified as a swipe. You can see a demo of the working prototype on this google drive link below, and it will also be on our main team status report. There is also another problem that we are in the midst of figuring out about the model slightly dipping away from the correct direction, then moving in the right direction immediately afterwards, during the transition between two swipe gestures. We are currently testing to see if its a problem with the data being sent over, or if it is a problem with how I am calculating Rotation values. Also we will work on making the rotation more smooth this week.

https://drive.google.com/file/d/1h3q5Os-ycagcWNNDkVVVS57qLM31H1Ls/view?usp=sharing

Lastly, I finished up a basic zoom function call. So now the model can zoom in and zoom out on calls to the Unity function from the Django web application. I am still figuring out how to zoom in based on how fast the user is pinching in and out. Right now it is just zooming in and out by a constant factor every time the function is called. Refining these two functions will be something I will be working on this week. Since we still are trying to figure out zoom out gesture detection, we have not been able to test the zoom functionality with live data, but that will also be another goal for this week.

 

Posted on

Team Status Report for 3.26.2022

We attended the ethics discussion on Wednesday. It was nice hearing what other teams are doing outside of section D and hearing their feedback on our project. The discussions did push us to think about the ethical implications of our project, which we didn’t think was a lot before. However, to be fair to our users, we will be more clear on what data we are collecting and how we are using it.

This week, we worked on a swipe detection demo and improving our gesture recognition algorithm. We are having issues with our algorithm. Since we are trying to avoid using machine learning, we are testing out different curve fitting equations and changing the parameters of how we’re collecting our data. We are limited to how fast our sensors can collect data, which is not good when a user performs a gesture quickly. We are currently collecting the accuracy when tuning hyperparameters and will map out these over the next week to determine which parameters offer the highest accuracy.

Most significant risk is the algorithm at the moment. If we are able to get the Jetson working next Wednesday, we might be able to afford better parameters for a higher accuracy, even if it takes longer to compute since we have a faster, external computer. The Jetson might change to a Raspberry Pi if we aren’t able to get the Jetson working, but we aren’t going to make that decision right now. The schedule has changed, since we weren’t expecting to be having this much trouble with the Jetson. However, this is why we allocated a lot of time for it.

We have started integration between the basic gesture detection from live sensor readings, and the web application portion of the project. We streamed data via MQTT protocol to the Django app, and confirmed we could get data into Unity and that it can rotate the model. A demo of the rotation on live data is shown below in the video on the google drive link: https://drive.google.com/file/d/1h3q5Os-ycagcWNNDkVVVS57qLM31H1Ls/view?usp=sharing.

We are continuing to work on smoothing out the rotation based on sensor data since it doesnt rotate as smoothly as it should (although it does rotate in the correct direction) and we are planning to test out the zoom in/out based on live data soon this week as well.

Posted on

Edward’s Status Report for March 26

This week, I worked on gesture recognition algorithms with Anushka and helped integrate everything with Joanne.

Our algorithm fits a curve to the 10 sensor data points and finds the relative min and max values of that curve. If there exists a max, we say the data represents a pinch. However, I am not too happy with the accuracy of this. I think we need to do something where we look at data over time, but I was having trouble figuring out what to do. Next week, I will work on this more and possibly explore other techniques.

I also wrote a script to test what we have now with Joanne. In real-time, we are able to collect data from the wearable, process it, and send it to our webapp. The webapp shows a 3D model which can be rotated as we swipe. This demo works fairly well, but there are some kinks to work out. Pinches are ignored in this demo, and swipe detection works pretty well, honestly.

I am on track this week, but I am worried we will not get a good algorithm in place and will fall behind one day. Further integration will take some time too. Next week, I will work on our deliverable, which will probably be a refined version of our swiping-to-rotate demo. The major issues for us looking forward are differentiating between a swipe and a pinch.

Posted on

Anushka’s Status Report for 3/26

This week was a cool week. I learned a lot from the ethics lesson on Wednesday. After the discussion with the other teams, I learned that the biggest concern for most engineering projects is data collection and data usage, which is something that may be lacking in our project explanation. I will keep this in mind for our final presentation.

I spent a lot of time improving the gesture recognition algorithm. With one single data collection, we are most likely not able to identify what gesture is being done. I improved it by looking at the frequency of the gestures guessed over n data collections. The accuracy improved for every gesture except zoom out, which makes sense because the beginning of the gesture looks like a swipe.

We collected more data so that we can see if the algorithm fits different variations of the the gesture. We noticed that there is high variability in the accuracy of our algorithm based on the speed and location in which we move our fingers. I decided to look into varying the n data collections and the polynomial that the data is currently being fitted into to accommodate our two discoveries. I am working with the data in Excel and am planning on looking at statistics to determine which combination yields the highest result.

Screenshot of pinching in data with n data against the order of the polynomial

I think that although this is a slow week, I’m going to be working hard to improve the algorithm before the interim demo day. Although our project is in the process of being integrated, this is a critical part of achieving our design metrics. I’m planning on meeting with my team outside of class on Monday and Tuesday so that we can work together to improve the algorithms.

Apart from algorithm updates, I need to talk to a professor about the Jetson. I’ve started playing with the Raspberry Pi, which is easier to work with since I have prior experience. I will spend Wednesday working on this part. Next week will surely be busy, but these deliverables are critical to the success of the interim demo.

Posted on

Joanne’s Status Report for 3.19.2022

So far I worked more on the Unity-Web application side of the project. I wanted to test calling Unity functions would work with streaming data. This is to mimic the data that would be streamed over from the Jetson Nano to the Web Application. I decided to use the MQTT protocol to stream over data from a Python script. I used the MQTT Eclipse Mosquitto broker. The data will be sent from the Python script to the Django Web application in the format of a string. For now, I made it so that it sends over fake dummy data that contains the gesture name, and the x and y coordinates for testing.

I wrote a script on the Django application to call a Unity function I wrote called “Rotate” every time a new data string was published onto the broker. When Rotate is called, it will parse the data that was just sent in. When I tested with dummy data, at first I ran into some issues with the model getting stuck after a certain degree change. However I was able to fix that by getting rid of a rotation limit that I incorrectly wrote initially. As of now, the model is able to rotate in the right direction (up-left, down-right, up-right, up-left) based on the data. I am still working on the algorithm to translate the data that is streamed in, into rotating at more precise angles.

At the same time, I am also looking into how to perform zoom operations with another set of dummy data. I have gotten zooming for a single view of the game object done through script, now I need to see how to access all four cameras (which is needed to get all four perspectives of the 3d model), and zoom in on all four cameras. I also am planning this week to see how to correlate our data that would be streamed in to how fast the zoom will occur.

This week, I hope to finish up a basic working of the zoom and rotate functionalities based on dummy data that is being streamed in. This way we can start trying to integrate in the later part of the week.

Posted on

Anushka’s Status Report for 3/19

This week was a productive week for our capstone project. Over Spring Break, I began working on building the gesture recognition algorithms. That week, we generated some sample data from a few gestures, so I observed the general behavior of the data points. I noticed that on an x-y plane, if the y-axis is where the sensors are lined up and if x-axis is the distance that the sensors are measuring, then a finger creates a sideways U-shape as seen in the picture below. If there are two fingers present, then the U-shape intersects to create a sideways w-shape.

Figure 1: Example of Swiping Data

I began looking at what properties of this graph stayed consistent. With both shapes, I wanted to focus on the local minima of the curves as a method of finger identification. However, the local minima would sometimes be inaccurately calculated or the values were arranged in an increasing or decreasing manner such that no local minima was detected. This was especially the case when two fingers were present. However, Edward pointed out that even though local minima was hard to detect with two fingers present, a local maxima was always present as the two Us intersected. An example is shown in the image below.

Figure 2: Example of Pinching Data

After observations from a few examples of pinching data, I reasoned that the velocity of that maxima could also serve as an indicator what direction the user is pinching and how fast their fingers are moving. If there are no local maximas present, then we can guess that a gesture is a swipe, calculate local minima’s velocity, and determine what direction their finger is moving and how fast they are swiping. We have yet to code the specifics of each gesture, but we are confident about this method of identification and will most likely move forward with this.

I also spoke with Professor Savvides about machine learning methods that would serve as viable solutions. He suggested using Markov methods, but that was something I was unfamiliar with. We were also recommended by Professor Sullivan to use curve fitting, which is why you see a few polynomials fitted to the graphs above. We are going to look into that method over the next week, but at least we have a starting point as opposed to before.

Because of the work put into gesture recognition, I would say we are back on track. I put the Jetson Nano aside for a week because I couldn’t get it to work on a monitor again, so that will definitely be my first task.I might reach out to a professor if I’m still struggling with the Jetson by Monday because that will be my main deliverable next week. I see us bringing all the parts together very soon, and it’s something I’m definitely looking forward to.

Posted on

Team Status Report for 3.19.2022

The hardware for the wearable is completed and a testbed for the physical sensor configuration and MQTT sensor data streaming have been fully set up! The device reads from all 10 sensors in a round-robin fashion (to prevent IR interference), applies a EWMA filter to the sensor readings, and sends the data to MQTT to be received by a Python program or a website. The device can also send power information like battery percentage and voltage. After this was finished, we moved on to thinking about the webapp and gesture recognition while Edward works on polishing up the hardware.

For gesture recognition, we made a lot of progress since receiving the sensors. We began by just observing the general behavior of the sensors and brainstorming ideas on how we can map them to different gestures. We noticed a few mathematical characteristics of the zooming gestures that were different from the characteristics of a swipe, so we will most likely use those as our gesture identification. We have a few theories as to how we can distinguish those further, and those will be our next step.

For the Unity-Web Application part of our project, we started working on communication of data. In order to prepare for integration, we made sure that data could be streamed to our web application via the MQTT protocol. For now, we just sent over dummy data from a python script via the protocol above. Then we made sure that we can actually communicate smoothly with the Unity interface that is embedded onto the Django web app. Now we are continuing to work on how to translate the gesture data into more precise gestures applied to the model.

The next steps are to build out the actual wearable (buy wrist straps, create enclosure for hardware, attach hardware to strap, etc.). We also need to flesh out gesture recognition and make it more robust to the various kinds of ways to perform specific gestures like a swiping left and right or a pinching in and out.

Our biggest concern still remains to be the Jetson. However, since we know the connection between the device and our computers work, we at least have one avenue for communication. That will most likely be our contingency plan, but we’re still hoping that the Jetson will work for faster computation purposes. Our schedule still remains the same, and we’re planning on beginning integration over the next week.

Posted on

Edward’s Status Report for March 19

One of the largest risk factors we had were the PCB’s not working. Before Spring Break, both our PCB’s (sensor board and support components) came in. I mocked up a rough prototype on a breadboard with both PCB’s but it didn’t seem to work. The I2C communication on the sensors was very wonky so the Photon wasn’t always able to start up all the sensors. So, after tinkering for a very long time, I eventually removed the support components PCB and ended up manually wiring up the sensors PCB only (see diagram below).
This actually ended up working! So, apparently I didn’t even need the support components board… But, the good new is, our hardware works now! The wearable is coming along well.

After writing come code to control and read from all 10 sensors as a test, I soldered the PCB and the resistors to a perfboard (see diagram below). I also added a battery.

(Eventually the Photon will go behind the resistors closer to where the blue wires are, making the form factor smaller. I will also get rid of the yellow breadboard and solder the device directly to the perfboard.)

I wrote code to stream the sensor data over MQTT and modified our visualization program to view the data in real time. I also recorded some data to be used to test out finger detection and gesture recognition algorithms.

By some stroke of luck, I am ahead of schedule, as I anticipated to be debugging and redoing the PCB’s. My next steps are to help Anushka on finger detection and gesture recognition on the Jetson.