Posted on

Joanne’s Status Report for 3.26.2022

This week I continued to work on refining the Unity- Web Application portion of the project. Last week, I got up to testing fake data on just the Rotate function via MQTT protocol. In the beginning of the week, I talked with my group members and they decided for now, that the rotation gestures will be limited to rotation about the x-axis (Up). Thus, I modified the code to account for this change. The model rotated well about the x-axis during the tests with my dummy data.

I wanted to see how the Rotate function would react to real life data, so we decided to integrate the live sensor reading with the basic gesture recognition algorithm, to my Unity web application. Right now, I had a python script that I ran from my local terminal that would publish to the public MQTT broker. Then a javascript script on the DJango web application would read those published data values, and send that data over to Unity where it would be parsed for use in order to Rotate the model. I decided to create another parsing function within the javascript script so that I could call the appropriate function (i.e. Rotate or Zoom In/Out) and also send the data in the correct format for the Unity parsing function I wrote.

After testing it with live data, the model rotated well about the xaxis. Since the gesture algorithm is not complete yet, we tested by ignoring all gesture data that was not classified as a swipe. You can see a demo of the working prototype on this google drive link below, and it will also be on our main team status report. There is also another problem that we are in the midst of figuring out about the model slightly dipping away from the correct direction, then moving in the right direction immediately afterwards, during the transition between two swipe gestures. We are currently testing to see if its a problem with the data being sent over, or if it is a problem with how I am calculating Rotation values. Also we will work on making the rotation more smooth this week.

https://drive.google.com/file/d/1h3q5Os-ycagcWNNDkVVVS57qLM31H1Ls/view?usp=sharing

Lastly, I finished up a basic zoom function call. So now the model can zoom in and zoom out on calls to the Unity function from the Django web application. I am still figuring out how to zoom in based on how fast the user is pinching in and out. Right now it is just zooming in and out by a constant factor every time the function is called. Refining these two functions will be something I will be working on this week. Since we still are trying to figure out zoom out gesture detection, we have not been able to test the zoom functionality with live data, but that will also be another goal for this week.

 

Posted on

Team Status Report for 3.26.2022

We attended the ethics discussion on Wednesday. It was nice hearing what other teams are doing outside of section D and hearing their feedback on our project. The discussions did push us to think about the ethical implications of our project, which we didn’t think was a lot before. However, to be fair to our users, we will be more clear on what data we are collecting and how we are using it.

This week, we worked on a swipe detection demo and improving our gesture recognition algorithm. We are having issues with our algorithm. Since we are trying to avoid using machine learning, we are testing out different curve fitting equations and changing the parameters of how we’re collecting our data. We are limited to how fast our sensors can collect data, which is not good when a user performs a gesture quickly. We are currently collecting the accuracy when tuning hyperparameters and will map out these over the next week to determine which parameters offer the highest accuracy.

Most significant risk is the algorithm at the moment. If we are able to get the Jetson working next Wednesday, we might be able to afford better parameters for a higher accuracy, even if it takes longer to compute since we have a faster, external computer. The Jetson might change to a Raspberry Pi if we aren’t able to get the Jetson working, but we aren’t going to make that decision right now. The schedule has changed, since we weren’t expecting to be having this much trouble with the Jetson. However, this is why we allocated a lot of time for it.

We have started integration between the basic gesture detection from live sensor readings, and the web application portion of the project. We streamed data via MQTT protocol to the Django app, and confirmed we could get data into Unity and that it can rotate the model. A demo of the rotation on live data is shown below in the video on the google drive link: https://drive.google.com/file/d/1h3q5Os-ycagcWNNDkVVVS57qLM31H1Ls/view?usp=sharing.

We are continuing to work on smoothing out the rotation based on sensor data since it doesnt rotate as smoothly as it should (although it does rotate in the correct direction) and we are planning to test out the zoom in/out based on live data soon this week as well.

Posted on

Edward’s Status Report for March 26

This week, I worked on gesture recognition algorithms with Anushka and helped integrate everything with Joanne.

Our algorithm fits a curve to the 10 sensor data points and finds the relative min and max values of that curve. If there exists a max, we say the data represents a pinch. However, I am not too happy with the accuracy of this. I think we need to do something where we look at data over time, but I was having trouble figuring out what to do. Next week, I will work on this more and possibly explore other techniques.

I also wrote a script to test what we have now with Joanne. In real-time, we are able to collect data from the wearable, process it, and send it to our webapp. The webapp shows a 3D model which can be rotated as we swipe. This demo works fairly well, but there are some kinks to work out. Pinches are ignored in this demo, and swipe detection works pretty well, honestly.

I am on track this week, but I am worried we will not get a good algorithm in place and will fall behind one day. Further integration will take some time too. Next week, I will work on our deliverable, which will probably be a refined version of our swiping-to-rotate demo. The major issues for us looking forward are differentiating between a swipe and a pinch.

Posted on

Anushka’s Status Report for 3/26

This week was a cool week. I learned a lot from the ethics lesson on Wednesday. After the discussion with the other teams, I learned that the biggest concern for most engineering projects is data collection and data usage, which is something that may be lacking in our project explanation. I will keep this in mind for our final presentation.

I spent a lot of time improving the gesture recognition algorithm. With one single data collection, we are most likely not able to identify what gesture is being done. I improved it by looking at the frequency of the gestures guessed over n data collections. The accuracy improved for every gesture except zoom out, which makes sense because the beginning of the gesture looks like a swipe.

We collected more data so that we can see if the algorithm fits different variations of the the gesture. We noticed that there is high variability in the accuracy of our algorithm based on the speed and location in which we move our fingers. I decided to look into varying the n data collections and the polynomial that the data is currently being fitted into to accommodate our two discoveries. I am working with the data in Excel and am planning on looking at statistics to determine which combination yields the highest result.

Screenshot of pinching in data with n data against the order of the polynomial

I think that although this is a slow week, I’m going to be working hard to improve the algorithm before the interim demo day. Although our project is in the process of being integrated, this is a critical part of achieving our design metrics. I’m planning on meeting with my team outside of class on Monday and Tuesday so that we can work together to improve the algorithms.

Apart from algorithm updates, I need to talk to a professor about the Jetson. I’ve started playing with the Raspberry Pi, which is easier to work with since I have prior experience. I will spend Wednesday working on this part. Next week will surely be busy, but these deliverables are critical to the success of the interim demo.

Posted on

Joanne’s Status Report for 3.19.2022

So far I worked more on the Unity-Web application side of the project. I wanted to test calling Unity functions would work with streaming data. This is to mimic the data that would be streamed over from the Jetson Nano to the Web Application. I decided to use the MQTT protocol to stream over data from a Python script. I used the MQTT Eclipse Mosquitto broker. The data will be sent from the Python script to the Django Web application in the format of a string. For now, I made it so that it sends over fake dummy data that contains the gesture name, and the x and y coordinates for testing.

I wrote a script on the Django application to call a Unity function I wrote called “Rotate” every time a new data string was published onto the broker. When Rotate is called, it will parse the data that was just sent in. When I tested with dummy data, at first I ran into some issues with the model getting stuck after a certain degree change. However I was able to fix that by getting rid of a rotation limit that I incorrectly wrote initially. As of now, the model is able to rotate in the right direction (up-left, down-right, up-right, up-left) based on the data. I am still working on the algorithm to translate the data that is streamed in, into rotating at more precise angles.

At the same time, I am also looking into how to perform zoom operations with another set of dummy data. I have gotten zooming for a single view of the game object done through script, now I need to see how to access all four cameras (which is needed to get all four perspectives of the 3d model), and zoom in on all four cameras. I also am planning this week to see how to correlate our data that would be streamed in to how fast the zoom will occur.

This week, I hope to finish up a basic working of the zoom and rotate functionalities based on dummy data that is being streamed in. This way we can start trying to integrate in the later part of the week.

Posted on

Anushka’s Status Report for 3/19

This week was a productive week for our capstone project. Over Spring Break, I began working on building the gesture recognition algorithms. That week, we generated some sample data from a few gestures, so I observed the general behavior of the data points. I noticed that on an x-y plane, if the y-axis is where the sensors are lined up and if x-axis is the distance that the sensors are measuring, then a finger creates a sideways U-shape as seen in the picture below. If there are two fingers present, then the U-shape intersects to create a sideways w-shape.

Figure 1: Example of Swiping Data

I began looking at what properties of this graph stayed consistent. With both shapes, I wanted to focus on the local minima of the curves as a method of finger identification. However, the local minima would sometimes be inaccurately calculated or the values were arranged in an increasing or decreasing manner such that no local minima was detected. This was especially the case when two fingers were present. However, Edward pointed out that even though local minima was hard to detect with two fingers present, a local maxima was always present as the two Us intersected. An example is shown in the image below.

Figure 2: Example of Pinching Data

After observations from a few examples of pinching data, I reasoned that the velocity of that maxima could also serve as an indicator what direction the user is pinching and how fast their fingers are moving. If there are no local maximas present, then we can guess that a gesture is a swipe, calculate local minima’s velocity, and determine what direction their finger is moving and how fast they are swiping. We have yet to code the specifics of each gesture, but we are confident about this method of identification and will most likely move forward with this.

I also spoke with Professor Savvides about machine learning methods that would serve as viable solutions. He suggested using Markov methods, but that was something I was unfamiliar with. We were also recommended by Professor Sullivan to use curve fitting, which is why you see a few polynomials fitted to the graphs above. We are going to look into that method over the next week, but at least we have a starting point as opposed to before.

Because of the work put into gesture recognition, I would say we are back on track. I put the Jetson Nano aside for a week because I couldn’t get it to work on a monitor again, so that will definitely be my first task.I might reach out to a professor if I’m still struggling with the Jetson by Monday because that will be my main deliverable next week. I see us bringing all the parts together very soon, and it’s something I’m definitely looking forward to.

Posted on

Team Status Report for 3.19.2022

The hardware for the wearable is completed and a testbed for the physical sensor configuration and MQTT sensor data streaming have been fully set up! The device reads from all 10 sensors in a round-robin fashion (to prevent IR interference), applies a EWMA filter to the sensor readings, and sends the data to MQTT to be received by a Python program or a website. The device can also send power information like battery percentage and voltage. After this was finished, we moved on to thinking about the webapp and gesture recognition while Edward works on polishing up the hardware.

For gesture recognition, we made a lot of progress since receiving the sensors. We began by just observing the general behavior of the sensors and brainstorming ideas on how we can map them to different gestures. We noticed a few mathematical characteristics of the zooming gestures that were different from the characteristics of a swipe, so we will most likely use those as our gesture identification. We have a few theories as to how we can distinguish those further, and those will be our next step.

For the Unity-Web Application part of our project, we started working on communication of data. In order to prepare for integration, we made sure that data could be streamed to our web application via the MQTT protocol. For now, we just sent over dummy data from a python script via the protocol above. Then we made sure that we can actually communicate smoothly with the Unity interface that is embedded onto the Django web app. Now we are continuing to work on how to translate the gesture data into more precise gestures applied to the model.

The next steps are to build out the actual wearable (buy wrist straps, create enclosure for hardware, attach hardware to strap, etc.). We also need to flesh out gesture recognition and make it more robust to the various kinds of ways to perform specific gestures like a swiping left and right or a pinching in and out.

Our biggest concern still remains to be the Jetson. However, since we know the connection between the device and our computers work, we at least have one avenue for communication. That will most likely be our contingency plan, but we’re still hoping that the Jetson will work for faster computation purposes. Our schedule still remains the same, and we’re planning on beginning integration over the next week.

Posted on

Edward’s Status Report for March 19

One of the largest risk factors we had were the PCB’s not working. Before Spring Break, both our PCB’s (sensor board and support components) came in. I mocked up a rough prototype on a breadboard with both PCB’s but it didn’t seem to work. The I2C communication on the sensors was very wonky so the Photon wasn’t always able to start up all the sensors. So, after tinkering for a very long time, I eventually removed the support components PCB and ended up manually wiring up the sensors PCB only (see diagram below).
This actually ended up working! So, apparently I didn’t even need the support components board… But, the good new is, our hardware works now! The wearable is coming along well.

After writing come code to control and read from all 10 sensors as a test, I soldered the PCB and the resistors to a perfboard (see diagram below). I also added a battery.

(Eventually the Photon will go behind the resistors closer to where the blue wires are, making the form factor smaller. I will also get rid of the yellow breadboard and solder the device directly to the perfboard.)

I wrote code to stream the sensor data over MQTT and modified our visualization program to view the data in real time. I also recorded some data to be used to test out finger detection and gesture recognition algorithms.

By some stroke of luck, I am ahead of schedule, as I anticipated to be debugging and redoing the PCB’s. My next steps are to help Anushka on finger detection and gesture recognition on the Jetson.

Posted on

Anushka’s Status Report for 3/5

This week was definitely a busy week. Our goal was to make as much progress before Spring Break, as we aren’t able to work on the hardware side of our project over break. I began this week by going over what was needed for the design review document. I made notes on what I wanted to cover in the presentation, which we ultimately incorporated into the final paper.

I began working on the Jetson Nano again. I had a feeling that something was wrong with how I wrote the SD card, so I requested one from the professors on Wednesday to try. Once I rewrote, we tried connecting the Nano to the monitor but with no success. Edward suggested I try using minicom as an interface to see the Nano, and we were both able to successfully install and run the package, thus finally giving as an SSH-like way of accessing the Nano.

I added details in the design report, which included the gesture recognition algorithm and the hologram pyramid design choices. I know there were more things that I wanted to do with the report, especially in the gesture recognition side, and after we get our feedback, I plan on incorporating them into the paper along with my changes. This is more so for us so that we understand the reasoning behind our choices so that if they don’t work, we can eliminate options.

I feel like this is the first week I feel behind. With the sensors coming in on Monday, most of us leaving on Friday, and paper also due on Friday, I felt like there was not a lot of time to test the sensors and gauge whether our current algorithm works. The team talked to the professor about ideas on gesture recognition, and we were suggested machine learning to a) help identify zooming actions, and b) help with user calibration in the beginning. I’m not too familiar with models that can test given a stream of data, so I plan on talking to Professor Savvides about potential options. I would say the gesture algorithm is the biggest risk as this point as if it doesn’t work, we have to determine a machine learning model and train on a huge dataset that we would have to produce ourselves. I think this will be my main focus over the next two weeks so that we can test as soon as we come back from Spring Break.