5/4/19 — Week 14:

Olly:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Arman:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Raunak:

What did you personally accomplish this week on the project?

This week I worked on the background subtraction task. This turned out to be harder than I thought it would be due to the background changing constantly. However, I think the background subtraction should work well with our table since we will have 3-4 sections to the dashboard which remain relatively static with solid colors that are easy to distinguish. I also started working on getting to integrate the Raspberry Pis.

Is your progress on schedule or behind?

Right now I am a little behind schedule for the demo. I need to improve the background subtraction a little bit, and finish integrating the Raspberry Pi to run OpenCV.

What deliverables do you hope to complete in the next week?

On Sunday, me and my team will be working on the final integrations and deploying our code on the hardware. After the demo on Monday, the main deliverable I need to work on will be the final report.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Were any changes made to the existing design of the system (requirements, block diagram, system spec, etc)? Why was this change necessary, what costs does the change incur, and how will these costs be mitigated going forward?

There are no design changes since last week.

Provide an updated schedule if changes have occurred.

No schedule changes.

4/27/2019 — Week 13:

Olly:

What did you personally accomplish this week on the project?

I have successfully worked to preprocess my character images so that they’re properly classified by my MNIST model about 95% of the time. From there, I move on to incorporate the EMNIST dataset into my model, which involved changing my framework slightly. I also wrote my own code to load the MNIST and EMNIST datasets to make the loading process more efficient and completely for my own purposes. I changed my preprocessing slightly, because the EMNIST and MNIST images are slightly different. My model now properly classifies about 30% of all characters.

Is your progress on schedule or behind?

This progress is roughly on schedule. My model needs some improvements to be fully successful, but there is still time for that.

What deliverables do you hope to complete in the next week?

I hope to add more layers to my model, which is a single layer neural net as of right now. I also hope to play around with other methods of OCR performance improvement.

Arman:

What did you personally accomplish this week on the project?

This week I 3D-printed an enclosure for our tablet and designed our table hardware. Additionally, I redesigned the schema of the database to save images more efficiently. Lastly, I wrote the start-up script for the raspberry pi to start our chromium browser on startup.

Is your progress on schedule or behind?

on schedule

What deliverables do you hope to complete in the next week?

I plan to finish creating the tablet and the table.

Raunak:

What did you personally accomplish this week on the project?

I worked on making the hand gesture detection much more robust. I am now able to detect a clench and 2 finger tap pretty well in most cases. I also started working on background subtraction.

Is your progress on schedule or behind?

Progress in on schedule.

What deliverables do you hope to complete in the next week?

Next week I hope to achieve 2 things. Firstly, I want to make our hand detection much more robust, and I will test this with our actual table. Second, I want to run our code on the Raspberry Pi. This will be a little challenging as getting OpenCV to deploy on a Raspberry Pi will require building it from source.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

There are no design changes since last week.

Provide an updated schedule if changes have occurred.

No schedule changes.

4/20/2019 — Week 12:

Olly:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Arman:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Raunak:

What did you personally accomplish this week on the project?

This week I worked on detecting convexity defects on the hand hull. This helped improve the performance of the clenching and swiping detection. This also helps me implement the gestures involving counting the number of fingers one has raised. I am now working on integrating the hand gesture detection with the Google Cloud API.

Is your progress on schedule or behind?

My progress is a little behind schedule right now, but I am confident I can get the rest done over the next couple weeks as the only major things remaining our integrating with the Google Cloud API and actual hardware.

What deliverables do you hope to complete in the next week?

In the next week, I will first work on finishing up the code for the in class demo. For this I will aim to have my hand gesture detection completely integrated with the Google Cloud API. After that I plan on working on transferring my code onto the Raspberry Pi, and making final changes such as background subtraction and full integration with the Google Cloud API.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

There are two main risks right now. The first is regarding integration. Currently, all the team members are working hard on integration, and this is going to be a challenge to get working well for the demo. We are managing this risk by ensuring that all the Google Cloud functionality is implemented as soon as possible, so that we can run full integration tests before the demos. The second risk is regarding hardware. We are now moving into the stage of testing on actual hardware, rather than our laptops. To mitigate the risks involved we are ensuring to write all the necessary auxiliary scripts as soon as possible.

No major changes have been made to the design.

Provide an updated schedule if changes have occurred.

None

4/6/2019 — Week 10:

Olly:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Arman:

What did you personally accomplish this week on the project?

This week, I worked more on storing information. Previously, all of the OCR results were displayed on our web UI, but nothing was stored in persistent memory. Thus, there was a loss of info between sessions. Now, that is no longer the case.

Is your progress on schedule or behind?

I’m on schedule.

What deliverables do you hope to complete in the next week?

Next week, it’s Carnival, so I don’t think I’ll accomplish much to be honest. I’ll try to clean up the UI, debug, and test.

Raunak:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

At this point, we’ve figured out a lot of the integrations, but have not begun running our code on hardware. At this point, the biggest risk will be ensuring that our code all works on the hardware that we’ve bought and hope to put together. In order to manage these risks, we did tons of research to figure out if and how our code will run on the hardware, what OS’s we should use, how these systems will communicate with each-other, etc. As a worst-case contingency plan, though, we could always run these things on a web UI.

There were no changes made to the system design since our last check-in.

Provide an updated schedule if changes have occurred.
No schedule changes.

3/30/2019 — Week 10:

Olly:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Arman:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Raunak:

What did you personally accomplish this week on the project?

This week I primarily worked on implementing the clenching detection. I ran into a little more trouble than I expected as the fine tuning of parameters was much harder than I thought it would be. I also started working on integrating my work so far with the rest of my team for the interim demo.

Is your progress on schedule or behind?

My progress is a little behind schedule right now as the clench detection took longer than I expected. However, I do not foresee this being a huge problem – I have much more time next week to catch up.

What deliverables do you hope to complete in the next week?

My goal is to a get a good start on the swipe detection. Another big goal is finishing up our integration for the interim demo. I anticipate these 2 deliverables taking up most of my time this week.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

Provide an updated schedule if changes have occurred.

3/23/2019 — Week 9:

Olly:

What did you personally accomplish this week on the project?

I spent a lot of this week working on the ethics report, so I didn’t have a lot of time to put into the project itself. I spent more time looking into TensorFlow models, but didn’t make a lot of progress with my code itself.

Is your progress on schedule or behind?

At this point, my progress is slightly behind schedule. I haven’t made as much progress on the OCR as I planned to at this point. I will put more hours into this project next week to get back on track.

What deliverables do you hope to complete in the next week?

I hope to complete my model and move onto training my model and beginning to refine it next week.

Arman:

What did you personally accomplish this week on the project?

This week, I implemented much of the cloud architecture that I wanted to. I set up the cloud endpoint and wrote all of the python code for cloud functions. I also instantiated the cloud storage bucket, the Cloud Firestore database, and used client libraries to communicate with these instances via my cloud functions endpoint. I have not tested these, though.

Is your progress on schedule or behind?

I believe that I’m mostly on schedule.

What deliverables do you hope to complete in the next week?

Next week, I plan to test my cloud infrastructure to ensure that it works as I planned. Additionally, I’d like to create a very basic dashboard that shows screenshots at the very least.

Raunak:

What did you personally accomplish this week on the project?

This week I worked on further refining the hand detection algorithm. To do this, I researched a lot about convex hull algorithms to isolate the finger tips. I continued to experiment with different parameters settings to best isolate the hands. I think I have a good working algorithm, and I have now moved onto implementing the actual hand gestures we need. I am starting to first work on the clench, which I will implement by detecting a change in the position and number of convex hull vertices. I aim to get this done by this week.

Is your progress on schedule or behind?

My progress is mainly on schedule. I did not get as much time as I would have liked to over spring break to work on the gesture detection algorithm, but now that I have the basic detection algorithm working, I am fairly confident I can implement the gestures required quickly.

What deliverables do you hope to complete in the next week?

This week I hope to complete the detection go the clenching hand gesture, and I hope to start working on the swiping motion, which I will detect by checking (x,y) coordinates of the hand over multiple frames.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

I think that currently the biggest risk we face is the different moving components that we all have going. Individually we are all making good progress on all these parts such as the OCR, cloud implementation and hand gesture recognition, but integrating all these components is going to be very challenging. We have discussed this as a team, and recognize this as a risk. Therefore, we are going to try to have a preliminary integration check sometime this week, so that we can design the necessary APIs in the best possible way – so that when we come around to integrating our components fully our task is much more straightforward.

There are no design changes since last week.

Provide an updated schedule if changes have occurred.

No schedule changes.

3/9/2019 — Week 8:

Olly:

What did you personally accomplish this week on the project?

This week, I mainly worked on building my text classification model using TensorFlow. A lot of this time still went into ramping up on TensorFlow before moving onto the OCR architecture I settled on. I have moved onto writing the barebones of my code.

Is your progress on schedule or behind?

My progress is on schedule. I have two weeks allocated to building my initial model, before testing and refinement. I’ll make sure to continue making progress.

What deliverables do you hope to complete in the next week?

I hope to complete my model and move onto training my model and beginning to refine it next week.

Arman:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Raunak:

What did you personally accomplish this week on the project?

Is your progress on schedule or behind?

What deliverables do you hope to complete in the next week?

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

As of right now, the most significant risk to our project is the number of distinct parts that we need to complete in order to reach MVP. Additionally, we have to connect each of these parts and have a successfully complete overall system. One of the ways we can mitigate the risk of having so many moving parts is not fully implementing each of the parts before connecting them. For example, we can start with having API calls or hard-coding in place of our handwritten code. Then, as we complete our tasks, we can replace each part one by one to convert our project to a handwritten system.

There are no design changes since last week.

Provide an updated schedule if changes have occurred.

No schedule changes.

3/2/2019 — Week 7:

Olly:

What did you personally accomplish this week on the project?

This week, I finished my OCR research and moved on to building my model based on my results. I also began looking into TensorFlow tutorials and examples to be able to properly implement my planned architecture for this model.

Is your progress on schedule or behind?

This progress is on schedule. This week and the next week are dedicated to building my model, before I move on to training and testing.

What deliverables do you hope to complete in the next week?

Next week, I hope to finish building my model and move on to training and testing with the EMNIST data set.

Arman:

What did you personally accomplish this week on the project?

My plan was to fully design the database and API and I mostly did that. I say mostly because I have the database fully designed and the API designed with as many of the endpoints as I can foresee. For example, I know that in Cloud Storage, we’ll need two “folders” — one for the screenshot images and another for the OCR images that Olly will be creating. In the database, I’ve designed the schema through which we’ll store the G Suite data and the screenshot metadata. I’ve also designed all of the endpoints that have to do with conducting the OCR, the storage, and everything else we’ve discussed, but I foresee things having to change or new things coming up.

Is your progress on schedule or behind?

I believe that I’m mostly on schedule.

What deliverables do you hope to complete in the next week?

Next week, I plan to implement the cloud functions to create the endpoints, instantiate the database and cloud storage instances and connect the storage instances to the endpoints.

Raunak:

What did you personally accomplish this week on the project?

This week I continued working on hand gesture recognition. I further worked with the Kinect to perform hand detection. However, while working with the Kinect I ran into a problem – the Kinect framework only allows for hand joints to be detected if the entire human body is tracked. Since we plan on having the Kinect just detect hand gestures on the table this means we cannot just use the skeleton data provided by the Kinect when the body is tracked. I have since then switched over the approach to detecting color frames as provided by the Kinect instead. I have switched over to a more rudimentary CV approach to accomplish the hand gesture detection. I wrote some basic code to find contours to detect the outline of the hand and started working on the convex hull algorithm to isolate the finger tips.

Is your progress on schedule or behind?

Its a little behind since I had to switch the computer vision approach I was using. (contours to detect hand outline rather than joint data provided by the Kinect) However, I am now confident that the current approach will work well, so I should be on track pretty soon.

What deliverables do you hope to complete in the next week?

I aim to finish implementing the hand gesture detection for basic gestures such as palm closed and palm open this week. This will involve finishing the convex hull algorithm to isolate finger tips and then using some geometry to figure out when the hand is closed vs when it’s open.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

The main risk that we face is connecting all of these parts. How is our tablet going to talk to our table? How will each of them communicate with our back-end? To mitigate these risks, we’re doing a few things.

First of all, we’re using HTTP to communicate between all of the devices. This will keep all of our communications uniform, allowing us to work with one single protocol for every part of the project.

Second, we’re using our REST API for every back-end communication. All of our backend components (Cloud Storage, Cloud ML Engine, and Firestore), have their own web client libraries, meaning that we technically need no REST API. However, having one single API to communicate with will make everything more simple and uniform.

One of the things we were most concerned about was the gesture recognition through hand-tracking. We initially started with a leapmotion sensor, but had a contingency plan in-case the sensor’s range wasn’t long enough. When we found that our concerns about the range were in-fact true, we fell back on our contingency plan of using Microsoft’s Kinect Sensor. This week, though, we found out that the Kinect sensor doesn’t work too well, because it only returns joint data if a whole body is in the frame.
Back to the drawing board, we created two options moving forward: 1) use a simple RGB camera with OCR techniques to track joints in the hand and recognize gestures and 2) use an rgb camera with OCR techniques to track physical tools besides the hands to create gestures (imagine a colored rod that we can use to move things around on the screen. Raunak will be trying these two techniques in order to decide how we move forward with the gesture recognition.

Provide an updated schedule if changes have occurred.

No schedule changes.

2/23/2019 — Week 6:

Olly:

What did you personally accomplish this week on the project?

This week, I put more research into OCR. However, due to timing issues, I wasn’t able to make a lot of progress on this. I primarily developed training and testing data in the format we would receive it from the Raspberry Pi touchscreen.

Is your progress on schedule or behind?

I’m a little behind our schedule. I wanted to be done with the primary phase of my OCR research so that I could move on to building and refining my model. However, this was a busy week for me, so I focused more on the accomplishable task of data simulation. I have about two weeks allocated for building my model, so I’m planning to re-allocate some of that time to completing my research and then working on my model more quickly. I also have about two weeks of buffer time, so I can also use this.

What deliverables do you hope to complete in the next week?

I want to completely finish my research and begin building my model.

Arman:

What did you personally accomplish this week on the project?

I created a cloud project in Google Cloud Platform Console. I also did tons of research into different backends and determined the back-end stack we’d use.

I decided we’re going to store all of our data in a Cloud Firestore database. This is good because it has Client Libraries for several languages and frameworks and additionally, it’s fully scaled and has a flexible schema. For file storage (mostly screenshots, but also other static files), I’ll store them in Firebase Storage.

To create a REST API, I’ll use the Python runtime for Cloud Functions. This API will serve our static content, allow for all GET and POST requests in communicating with our database, and will also serve as an endpoint for Olly’s OCR model.

I’ve already began designing the API and Database. Once I’m done designing it, implementing it should be fairly quick.

Is your progress on schedule or behind?

Designing the stack has taken a little longer than I’d thought, but it worked out because the technologies I settled on have very quick development times. I should be back on track next week.

What deliverables do you hope to complete in the next week?

Next week, I plan to have the API and database fully designed. I want to also implement the entire thing, but I’ll put priority on creating the dashboard.

Raunak:

What did you personally accomplish this week on the project?

This week I primarily worked on getting started with hand gesture recognition using the Kinect platform. I was able to set up the Kinect and get some basic sample code running, and I was able to toy around with it to get a better idea of the framework. However, this took longer than I expected since the product is now deprecated by Microsoft – and hence I had to use a lot of online tutorials and sample API code to understand the Kinect API framework. Now that I have a better understanding of this, I am working on a hand detection algorithm that will use the body joint data provided by the Kinect to trace out hands present in the view.

Is your progress on schedule or behind?

My progress is slightly behind schedule right now. I think it took a little longer than expected to understand how the Kinect API works. I am still flushing out my understanding of this. But after spending sometime this week working on this, I am confident that I can catch up on my progress this week. I have started working on the hand detection algorithm, which I plan to finish implementing this week. Following this, I will then use a convex hull algorithm to isolate the fingers – after which I can start looking into the implementation of preliminary hand gestures.

What deliverables do you hope to complete in the next week?

By next week I am to have the hand detection algorithm implemented and tested. I am also aiming to get the majority of the convex hull algorithm working to isolate finger tips. This will allow me to form a solid foundation for implementing hand gestures.

Team:

What are the most significant risks that could jeopardize the success of the project? How are these risks being managed? What contingency plans are ready?

There are a few risks that could jeopardize the success of our project – however we are taking the necessary steps to make sure that this risk is appropriately managed. Firstly, as mentioned in our previous report, our project has a lot of distinct moving parts – the most significant being word segmentation, OCR, hand gesture detection and the cloud enabled dashboard. We have already switched our word segmentation over to using a touch screen, so we can more easily detect the boundaries over each letter. This will allow Olly to primarily focus on the OCR, and allow us to have much better accuracy as the letters will be segmented in a more clear and efficient manner.

Another risk that we think we might face is the implementation of complicated hand gestures. After experimenting with the Kinect, we have realized that there is a few preliminary algorithms we will have to implement (such a finger tip tracking), before we can implement more complicated gestures. Since not all gestures will be suited to just using fingertip and palm data, we might have to restrict ourselves to first implementing some basic gestures such a hand swipes before moving onto more complicated ones. This is fine however, as even a handful of different gesture should be enough to operate the table. We can also always switch over more of the hand gesture features to the touch screen if absolutely necessary.

There were no changes made to the existing design. We will be sticking to the design approach we outlined last week.

Provide an updated schedule if changes have occurred.

No schedule changes. We will all be working harder collectively as a team this week to get on track, as we have fallen a little behind our expected progress by now.