Month: February 2019

This week was a great iteration on last week’s design focussed work. Previously, what packages and what components are involved were starting to come together, but this week they were decided. Furthermore, detailed discussions over how these players speak to each other both in protocol and general work flow was ironed out. These were big next steps in creating the clarity needed to create a performant and scalable system.

A big stress of this week was communication protocols, and in our current design (diagram below) we plan to use HTTP Requests and Responses to accomplish most of the intra-system messaging. This well established and flexible protocol seems ideal for keeping our communication straightforward. All arrows in blue are those that we believe to be HTTP communications and all in black we believe can be handled through internal calls.

Throughout this detailed design process we also managed to get an idea as for what servers we need to be live as well as what those servers must do. The Raspberry Pi server must be able to initiate a connection with the backend as well as stream content to Kinesis. The Frame / Data handler must be able to take a Kinesis video stream, utilize the OpenPose Library, and ferry that data to the Core Backend Server. The Core Backend Server has a host of responsibilities including but not limited to: communicating with the frontend, managing a user session, prompting the classifier, and correcting form. The Classifier Server must be able to use the dockerized classification algorithm to classify OpenPose data and send that to the backend.

For next week, I hope to dig into the duties of the Core Backend Server. This means defining rigidly all we expect the backend to do. Then the next step is to create high level documentation defining all the endpoints for each subsystem. This list will be incredibly useful in terms of work delegation and scheduling.

I didn’t get much work done this week as I had a lot of work in other courses and we were waiting for our hardware to arrive to start collating image and OpenPose data for the classifier.

In the interim I read the AWS specs as well as a few articles on dockerizing the prediction algorithm, the correct form of the data input, and the form of the data output of a fully trained and dockerized prediction model to be able to spec out the data preprocessing that will be necessary for the classifier and clarify the communication protocols necessary for the classifier input/output.

In the coming week I will be finishing the KNN workout classification algorithm and collecting as many data points as I can hopefully to be able to train and test a first iteration by next weekend.

This week we were working on the overall design of our project, trying to get into the specifics about how the different portions of our project were going to meld together and communicate with each other. My primary project this week was deciding on and ordering the hardware we needed for a successful project. Specifically we had to decide on a camera to use and a conduit to communicate camera data to the cloud. We wanted our conduit to have wifi connectivity such that it could communicate with the cloud without a middleman or an ethernet cord. We also wanted it to be bluetooth capable to allow for flexibility in our design to potentially have the conduit communicate directly with the user phone if we decided we wanted to (potentially for offline storage/usage when disconnected from wifi). With this in mind my team and I chose the Raspberry Pi 3 B+ as it had all of the features we wanted as well as our team having a good amount of experience and comfort working with Raspberry Pi’s. Next I had to decide on a camera. First I had to make a decision about whether or not we needed a depth camera, I was looking at the Kinect and the Intel RealSense D415 for depth cameras. Ultimately I determined that a depth camera was necessary as the video processing done by OpenPose is not helped by depth cameras. Looking through other options for cameras I narrowed it down to the Logitech C290 and the Raspberry Pi Camera V2. The specs of the camera were relatively similar and it was unclear that we were going to be able to get any kind of performance increase from one or the other so I chose to go with the Raspberry camera as it was slightly cheaper and there would be more available documentation if we ran into any problems.

For the coming week we will be (hopefully) receiving our Pi and Camera. I plan to start work on our classification algorithm. Using the output data from OpenPose I would like to use a KNN map algorithm to identify workouts. In the first week working on this I hope to get a KNN algorithm working and running on an existing dataset that likely will not be particularly useful for us. This should give me a better sense of the functionality of the KNN algorithm and confirm it is the one we should be using, while also being able to quickly switch it to another dataset once we have accumulated an appropriate number of datapoints. At the same time, I will use the camera and possibly publicly available videos of workouts to begin compiling an extensive dataset for our particular problem. Hopefully by the following week I can begin to attempt training the algorithm on our new algorithm, or possibly move on to a different algorithm if I determine that KNN is not appropriate for this problem.

Significant Risks

• Speed of OpenPose and network communications must be fast enough to provide real-time updates and form correction.
  • Security is of minimal concern here which should help optimize our communication.
  • Wifi-enabled conduit eliminates middleman
  • OpenPose on AWS machines should be extremely fast
  • Potential Workaround: Provide feedback at the end of the specific workout
    • Gives us a significant buffer (~30 seconds to a minute) before the user needs the feedback
• Cloud Security
  • Our connections between the cloud and devices need to be secured. Our implementation will take security measures to ensure that our cloud endpoints are only being used for their intended purposes

Design Changes & Decisions

• Raspberry Pi 3 B+ as Conduit
• Raspberry Pi Camera V2 for Camera
• Classification algorithm will be run on OpenPose Data not directly on video/image files
  • Pro: Should make algorithm itself much simpler and potentially more accurate
  • Con: Need to compile large dataset
    • Run OpenPose on pre-existing (online) workout videos to help compile more data than we could by creating our own.
• We are no longer using a RGB-D camera and rather just RGB
  • We believe that the added depth will not improve openpose’s performance and is not needed
• Classification will have to occur on images rather than videos in order to be able to classify an exercise without missing the first rep

Nakul Garg

Team C4 — EZ-Fit

Individual Status Report 1

This week being the first week with feedback from the presentations focussed heavily on design for the whole team. I specifically was looking into the high level architecture of our system. This includes what components we need, how they interact, and how they communicate. We had a rough idea of the involved components coming into this week as can be seen in left figure, yet as a team we wanted to take this high-level model and start deciding what will actually fill the role of each part. I looked into how to link the Raspberry Pi to AWS Kinesis thus fulfilling the role of the conduit, and it turns out AWS provides a tutorial on how to do so. The Pi can act as a Kinesis Producer and the AWS Server as the Kinesis Consumer thus streaming our video straight into the processing. Then I made a sketch of the flow of how the components can interact for an average use case. The Pi will need to initially link with the backend and establish its connection to AWS Kinesis. Then the user will initiate a workout through the frontend which will prompt the backend to tell the Pi to initiate a stream. That stream can be sent through Kinesis running OpenPose and the data can be sent both to AWS SageMaker which we plan to use to run our classifier and the backend for user feedback. This week I spent focussing on ironing out these details with the flow to make sure that the technology we want to use can work together and that the architecture is clear.

For the next week, I want to iterate on this initial architecture and keep drilling deeper until we have a very clear design. This involves explicitly stating the responsibilities of each component in the above diagram. From these internal requirements, I should hopefully be able to craft a class architecture for the backend revolving around the services it is expected to provide. I also hope to specify what communication protocol the Frontend specifically uses to talk to the Backend as well as how the Pi will initially connect. We seem to be on schedule.