Posted on

Cooper Weaver Status Report 8

Last week I trained and setup the endpoint for the workout classifier. This was the second batch of training that I had done, and it surpassed the metric that we had set for our classifier accuracy based on the training set of 200 images that I setup. It completed with ~95% accuracy on the dataset. This metric seems about right, as the same algorithm has been used on more complicated datasets with accuracy in the range of 92-97%. That being said there is still a decent potential that the classifier has been overtrained because I have a slightly smaller, less robust dataset so we will be tracking the accuracy of the classifier moving forward to look out for that. I have additional ideas to increase the classifier accuracy and limit overtraining that I’m prepared to implement but as we passed our metric point I’ve put those ideas on the back-burner and moved on.

Next I had to set up to endpoint to make individual predictions which has been done but currently has a bug in it that is messing up the input data and therefore not able to return a response. This is my primary task for this week, I don’t expect it to be too difficult to fix, and then our predictor will be running and integrated. At that point I will move on to helping Nakul with the backend form algorithms and Scott with the frontend. First I am going to help Scott get a testing platform up and running which will allow us to use OpenPose’s visualization tools right next to the output of our classifier as well our backend algorithms which will provide us with both key demonstration tools for how out project works as well as be hugely helpful with beta-testing and debugging.

Posted on

Cooper Status Update 7

This past week I completed a couple of scripts and wrote out a simple step-by step process for turning existing image data into sagemaker-ready data files, as well as generating new, correctly labeled image data. This allowed me to transfer the raw image data I had already collected into a form ready for training the KNN. This will also allow me to hand off data-collection in parts to the other members of my team. This is essential because I don’t want to overtrain the classifier to my body. I’ve been coercing some of my roommates into modeling workouts for me but the majority of the pictures have still been of me. I also migrated all the storage and data collection from my local machine to an s3 bucket which allows me to access my teammates pictures as well as train the algorithm without having access to the pi.

I’m now moving smoothly through data collection and training, so while I continue to accrue the huge number of data points that I need I am moving on to the data pre-processing necessary for form correction. I want to move to this job now before I finalize the classifier because I don’t want to do double-processing on the data unless it’s absolutely necessary. This means that I have to work through exactly what I need available for form correction such that I can compare to the classifier algorithm. I am hopeful that this should be a quick project that mostly involves moving the pre-processing out of my classifier and into an earlier section right after the OpenPose is completed. If that’s the case I next want to write a little python script that will allow me to test different parameters for form correction. To do this I’m going to define a number of angles between joints and set some global parameters of “correct” form as well as leniency attributes that can easily be modified. This will allow me to take OpenPose-marked images and the JSON output and visually compare the form in the picture to the actual joint angles and my predicted joint angles and leniency attributes. This will allow us to solidify the exact joints we should be defining our form correction around, as well as pinpointing the range and sensitivity we should be targeting. It will also be perfectly usable with the data collected for the classifier trainer which will provide us with ample examples of good and bad form to test with. I hope to have this done by next Monday, which will set us up nicely to push through all three types of form correction next week.

Posted on

Cooper Status Report #5

Last week I was working primarily with our hardware, setting it up both for Scott to be able to connect with Kinesis, and also writing a script to take a large number of images and label them. This is going to be hugely helpful for collecting a training set for our workout classifier.  The script takes three pictures a second for sixty seconds (180 images), and stores them locally. For the purposes of training our workout classifier I plan to store the images on my laptop, with each member of the team doing each workout once or twice for a minute in front of the camera I’ll have about 1,000 images of each workout to start training with.

I was also working on the physical box that we’re going to be putting our Pi and camera in. I went through the process of spec-ing and designing an entire case and stand before realizing buying a case online was going to be cheaper than buying the parts to build my own case.

In the coming week I’ll be working on Greek Sing a lot so I expect it to be a slow week, but I plan to help Scott get the Pi connected to Kinesis so we can use the OpenPose instance he has set up on AWS. I’ll also continue working with the hardware to get it connected to the internet and running how we want it to be (as well as exploring the number of options we have for camera settings). Once the Pi is properly connected to the internet I’ll work with Nakul to get it communicating with our backend.

Posted on

Team Report #3

Team C4 – Cooper Weaver, Nakul Garg, Scott Hamal

Status Report 3

 

Progress

  • Data preprocessing for categorization algorithm
  • Camera and RPi setup
  • Endpoints named defined and spec-ed
  • Backend class architecture defined

 

Significant Risks

  • Rotation preprocessing – Possibly very difficult. We have a plan in place to do it but we have already identified specific data-points that will break our method and require additional rotation-correction processes.
  • Incorrect classification results – If a workout is incorrectly classified the wrong backend WorkoutAnalyzer will be used this breaks the form correction, rep counting, and set counting for the rest of that workout.

 

Design Changes & Decisions

  • Each workout will have its own WorkoutAnalyzer class which will each have separate subclasses for form correction, rep counting, and set counting. Though these algorithms will be similar in many respects they will have certain stark differences and possibly require drastically different data pre-processing.
  • Http Endpoints named and http protocol for endpoint communications defined: here

 

Posted on

Cooper Status Report #3

This week we stepped back a bit to refocus our overall structure (spurred by the design review). I reconsidered the I/O of the classification algorithm making sure it fit with our refined design. I also focused on designing the data pre-processing for the classification algorithm. The data preprocessing is going to be an ongoing task for me as I expect it to be by far the most complicated part of the classification algorithm. The design as it stands now is to zero (center) the hip point, and normalize all other points to that to eliminate horizontal and vertical movement of the camera and the user. This is a relatively simple process and eliminates the most common differences between images. Following that the points will be flipped (or not) to make sure the users head is always to the left for horizontal postures and that the user is facing to the left for standing postures. Finally, the trickiest part of the data preprocessing is going to be rotation to avoid a skewed camera or a non-level floor from breaking our algorithm. For the purposes of rotation I decided to only consider a +/- of 15 degrees as anything larger than that would be frankly shocking as well as extremely difficult to recover from. In order to execute rotation correction the plan is to draw essentially a “best-fit” approximation line for the foot, hip, and head (or shoulder) points and rotate all points to bring that line to the nearest horizontal or vertical (within +/- 15 degrees). If there is no horizontal or vertical mark within that range those data-points probably should not be used and the camera likely needs to be adjusted. This solution is not foolproof, specifically there is a clear problem in creating a best-fit line during the “up” portion of a sit-up. A solution I’m considering now for that problem is to throw out the head/shoulder point if it is too far off the line of the foot-hip points and simply use the foot-hip line to rotate in that case.

This past week we also received our hardware so I began working to get the camera hooked up to the the Pi and take a few pictures before I got the flu which shut me down for the rest of the week.

 

Posted on

Cooper Status Report #2 (2/24)

I didn’t get much work done this week as I had a lot of work in other courses and we were waiting for our hardware to arrive to start collating image and OpenPose data for the classifier.

In the interim I read the AWS specs as well as a few articles on dockerizing the prediction algorithm, the correct form of the data input, and the form of the data output of a fully trained and dockerized prediction model to be able to spec out the data preprocessing that will be necessary for the classifier and clarify the communication protocols necessary for the classifier input/output.

In the coming week I will be finishing the KNN workout classification algorithm and collecting as many data points as I can hopefully to be able to train and test a first iteration by next weekend.

Posted on

Cooper Status Report #1 (2/16)

This week we were working on the overall design of our project, trying to get into the specifics about how the different portions of our project were going to meld together and communicate with each other. My primary project this week was deciding on and ordering the hardware we needed for a successful project. Specifically we had to decide on a camera to use and a conduit to communicate camera data to the cloud. We wanted our conduit to have wifi connectivity such that it could communicate with the cloud without a middleman or an ethernet cord. We also wanted it to be bluetooth capable to allow for flexibility in our design to potentially have the conduit communicate directly with the user phone if we decided we wanted to (potentially for offline storage/usage when disconnected from wifi). With this in mind my team and I chose the Raspberry Pi 3 B+ as it had all of the features we wanted as well as our team having a good amount of experience and comfort working with Raspberry Pi’s. Next I had to decide on a camera. First I had to make a decision about whether or not we needed a depth camera, I was looking at the Kinect and the Intel RealSense D415 for depth cameras. Ultimately I determined that a depth camera was necessary as the video processing done by OpenPose is not helped by depth cameras. Looking through other options for cameras I narrowed it down to the Logitech C290 and the Raspberry Pi Camera V2. The specs of the camera were relatively similar and it was unclear that we were going to be able to get any kind of performance increase from one or the other so I chose to go with the Raspberry camera as it was slightly cheaper and there would be more available documentation if we ran into any problems.

For the coming week we will be (hopefully) receiving our Pi and Camera. I plan to start work on our classification algorithm. Using the output data from OpenPose I would like to use a KNN map algorithm to identify workouts. In the first week working on this I hope to get a KNN algorithm working and running on an existing dataset that likely will not be particularly useful for us. This should give me a better sense of the functionality of the KNN algorithm and confirm it is the one we should be using, while also being able to quickly switch it to another dataset once we have accumulated an appropriate number of datapoints. At the same time, I will use the camera and possibly publicly available videos of workouts to begin compiling an extensive dataset for our particular problem. Hopefully by the following week I can begin to attempt training the algorithm on our new algorithm, or possibly move on to a different algorithm if I determine that KNN is not appropriate for this problem.