This week was a productive week for our capstone project. Over Spring Break, I began working on building the gesture recognition algorithms. That week, we generated some sample data from a few gestures, so I observed the general behavior of the data points. I noticed that on an x-y plane, if the y-axis is where the sensors are lined up and if x-axis is the distance that the sensors are measuring, then a finger creates a sideways U-shape as seen in the picture below. If there are two fingers present, then the U-shape intersects to create a sideways w-shape.
Figure 1: Example of Swiping Data
I began looking at what properties of this graph stayed consistent. With both shapes, I wanted to focus on the local minima of the curves as a method of finger identification. However, the local minima would sometimes be inaccurately calculated or the values were arranged in an increasing or decreasing manner such that no local minima was detected. This was especially the case when two fingers were present. However, Edward pointed out that even though local minima was hard to detect with two fingers present, a local maxima was always present as the two Us intersected. An example is shown in the image below.
Figure 2: Example of Pinching Data
After observations from a few examples of pinching data, I reasoned that the velocity of that maxima could also serve as an indicator what direction the user is pinching and how fast their fingers are moving. If there are no local maximas present, then we can guess that a gesture is a swipe, calculate local minima’s velocity, and determine what direction their finger is moving and how fast they are swiping. We have yet to code the specifics of each gesture, but we are confident about this method of identification and will most likely move forward with this.
I also spoke with Professor Savvides about machine learning methods that would serve as viable solutions. He suggested using Markov methods, but that was something I was unfamiliar with. We were also recommended by Professor Sullivan to use curve fitting, which is why you see a few polynomials fitted to the graphs above. We are going to look into that method over the next week, but at least we have a starting point as opposed to before.
Because of the work put into gesture recognition, I would say we are back on track. I put the Jetson Nano aside for a week because I couldn’t get it to work on a monitor again, so that will definitely be my first task.I might reach out to a professor if I’m still struggling with the Jetson by Monday because that will be my main deliverable next week. I see us bringing all the parts together very soon, and it’s something I’m definitely looking forward to.