Month: February 2020

Progress

I got a simple thresholding algorithm working to classify claps. I looked at OpenNI, which is a potential library we can use to interface with the Kinect on MacOSX (the default library is only for Windows, and other libraries might require us to buy an Azure Kinect).

Deliverables next week

By next week I want to have the Kinect working.

Schedule

The Kinect turned out to be a nontrivial part of our project, so I’m going to allocate a week to figure out how to use it. In terms of algorithm development, I am a little bit behind since I have to study LSTMs in more detail.

Progress

This week I finalized the design for the glove component of our project.  Below is the system level diagram that I created.

It took longer than expected to finish the design phase of my portion because I made sure to be extra careful to ensure that the system would work.  Not using a resistor to connect the buttons to the GPIO pins of the pi means that these buttons will essentially be active low, since they are connected to ground.  In the code that I will write to register the presses I will just have to account for that fact.

Schedule/Deliverables

I have made some adjustments to my overall schedule.  Now that my design is done, I will be compiling a formal bill of materials and placing an order this week.  I have already looked ahead at lead times for some of the more important components and things should not take longer than one week to arrive.

 

Chris/Tony

• Created simple classifier for stomps and claps
• Working on refining and removing false negatives, and improving the timing

For next week: create an action classifier using our research to do start and stop recognition

Most promising so far: https://github.com/felixchenfy/Realtime-Action-Recognition

Mark

Finalized glove design this week.  Created system diagram and a bill of materials to send for order.  Went through a couple iterations during design phase which is why the parts order has not been placed yet.

All:

Currently on schedule, however, we may have to use more of our slack to do more in the action classifier

This week I implemented a way to detect stomps through our OpenPose processed video of just a stomp.

Currently, however, I also processed a video with Tony just flailing his arms (should be no detection) and it still did detect it as a stomp if his legs moved at all, even when it was clearly not a stomp. Working to fix by adding more parts of the body into the classification of a stomp.

For this week, we were given AWS credit that we wanted to use to process video on so we could process longer video and test on that without having to process on our own machines

I got it partially working, but there were some issues with compatibility with our EC2 Instance so I am continuing to work on that

CV studio attempts to create a user friendly music development platform for gesture based music creation.  Using an x-box Kinect camera system along with a buttoned glove with additional features, this project intends to allow users to generate instrumental sounds with basic movements such as a clap or a stomp.  Our project will be connected to an existing music production software that allows users to save short segments of music and tune their work to their liking.  One central goal of the project is to be able to capture five different gestures with accuracy of 90%.

Progress

I read some action recognition papers (https://paperswithcode.com/search?q_meta=&q=action+recognition), many of which involve temporal in addition to spatial approaches in their models. After discussing this with Marios and Emily, Chris and I have opted instead to start with a spatial approach and then include temporal approaches if we cannot achieve the necessary accuracy. To start off, we will model claps and other gestures in terms of a change in the coordinates of the relevant joints.

Deliverables next week

By next week, I want to have a basic spatial algorithm that will recognize claps and stomps (taking videos of more actions as necessary).

Schedule

I feel slightly behind schedule, since I should have started coding algorithms by this point. I spent time on researching very complex algorithms when I should have been thinking of simpler approaches first.

Progress

I was able to get OpenPose running on the computer and I started to work on the gesture recognition using pose data from a video that was processed using OpenPose. I’m devising a way to use the pose of the hands to recognize a large clap.

I’m also looking into different action recognition “libraries” (other github projects) to do action recognition on a play pause wave.

I’ve also been able to get the Kinect working with OpenPose using libfreenect2.

Deliverables next week

Hopefully by the end of this week I’ll be able to get some basic gesture recognition using distance and frame numbers. I want to be able to recognize claps and stomps. This will still be on processed video, and when we get the Xavier Board we will be able to see if these work in real time, even with high latency.

Schedule

I think that I’m still on schedule, although I think that using machine learning classification to recognize claps and stomps will be a little more challenging than I initially thought.

Project Abstract

Planned Project Schedule

Proposal Presentation